You can use a GPS to to see who's calling and then either press the ignore button, or just let it go to voice mail. Then pull over and make the call safely if you need to. Even if you did decide to take a call while rolling, you don't have to do anything except talk. No hands, no eyes, nothing involved.
It's really not any more distracting than listening to a good song or talking on the intercom to your passenger. Are those also "too dangerous?"
First off, a passenger will shut up or indicate a problem, while someone on the other end of a cell phone will just keep blabbering until they hear screeching and crunching sounds on the other end.
Secondly, it isn't holding the phone or whatever, it's the mental workload of carrying your side of the conversation that's the problem. It's like Keith Code's "attention dollar" in "Twist of the wrist" - there's only so much attention you can spend before it's all gone.
I won't recount the experiences with friends bending metal because they were occupied with a phone or a GPS, because it's boring and no-one will listen anyway, or feel it applies to them.
I feel everyone has the right to go to hell in their own way and make their own decisions, but I thought it was a bit hypocritical to diss folks for being on cell phones ("she nearly ran over me!") and then put one on their bike.