There have been two occasions where I turned around and headed back to paved road:
Once in Idaho I was following my pre-planned route and found myself on an ATV trail that showed as a road on the map. My FJR was fully loaded for several weeks of riding and camping and the trail was just too rough for me. It took me half of a mile to find a place wide enough to turn around.
Another time when I took a gravel road shortcut. I think I was in Maryland, but I'm not sure because it was near the west border. The road went straight across a stream and it looked to be at least 12 inches deep and 30 yards across. I defiantly peed in the stream, then humbly turned around and went back the way I came.
Other than that I have no problem with gravel roads.