Over the years, I've lost two mirrors to damage while riding single track on dual sport events. The first was an event in North Carolina (Suzuki DR650) and the second was at the Perry Mountain Tower Run in Stanton, Alabama (Kawasaki KLX250). When rolling on the tight single track in many of these locations, it's common to catch a mirror on trees and branches. Most often it survives, sometimes not.
Since many of these tracks include both pavement and offroad trail, mirrors are necessary and, quite frankly, there are times I like to have mirrors out on the trails to keep an eye on what's going on behind. However, for all other times the answer is simply folding mirrors
There are a number of folding dual sport mirror options, some very expensive. I like to keep my hobby on the cheap.
Most recently, I purchased as set of dual sport mirrors off Amazon for under $30 and added some improvement for another $5.