I was riding my own motorcycle at age-3. I can't remember exactly when I had my first ride on the street as a passenger, but I'm sure it was somewhere around age-6 or 7. Probably could not reach the pegs since I was NOT an unusually "large" child. (Put all those thoughts aside and let's get on with the discussion...)
I've heard the "when they can reach the pegs" rule many times. I think it's a good one for all-around guidance. On a bike like the FJR I would say it's a must. On something like a GW with the armrests for support, IMO it has more to do with attention span than anything else.
How long will they sit and watch TV without getting up or moving around in the chair. That's about how long a ride you can expect on the bike.
Intercom system makes a huge difference for me. I started riding my youngest daughter at a much earlier age on the FJR, simply because I could talk to her, make sure she wasn't falling asleep and hear from her. You sure can't see them behind you when you are old and fat like me.