When I got my concealed carry permit the trooper teaching the class said "carry" when in/on a vehicle is defined as "within your immediate grasp".
That may well be true in Montana. In Missouri I can leave a loaded gun on the car seat next to me without a permit. In Illinois it has to be unloaded and "in a case" which can be pretty much anything that closes. Both states define "carry" as on my person and require a permit for that.
In other words, its a complex subject with 51 sets of rules counting D.C. and you need to know the rules for the state in which you choose to be armed. The chances of getting caught in violation are slim unless you are really unlucky or stupid but there can be serious consequences should the dice roll against you. Make your own decision.
To the OP: I'd be more concerned about my ability to move it back and forth from bike to body discretely than the actual stowage. Its pretty rare to have a pannier broken into unless the thief has reason to believe it contains something worth the effort (like a firearm he saw you put in it).