I did a solo ride through there in mid-June of 2005. I stayed in Baker City for a couple of days and rode the "Hells Canyon Scenic Byway" (www.byways.org), the Hells Canyon Dam road, and a bunch of others recommended by the local Yamaha dealer. Spectacular - no traffic, beautiful scenery, enough twisties to satisfy even the most daring. However, sharing those roads with several hundred HD riders would take all the fun out of it. My recommendation - go to the rally for the parties, but stay for a couple of days after it's over and do the rides solo.
I just realized that the picture in my avatar is from that trip; it's in a campground about half way around the byway loop.