Well, bug, it's not quite that simple. The flexible rubber also has a protrusion on the back side to hit the electrical switch mounted on the circuit board behind it. Once you completely lose the "button" back filling the panel will not do you much good.
One of mine (now on my son's bike) had a lot of use and was broken free, but I managed to grab and save it before it went off into the nether-lands.. I then was able to stick it back in place using some black RTV (silicone sealant to most folks) applied from the inside. I remember that (in my case) it wasn't the power button that shat itself, it was one of the four on the other side. Probably the volume button as I seem to hit that one oftener. BTW, that GPS is still running fine.
I have heard rumor of people finding hollow "demo units" and using the front panel of those to restore their zumo 550's. It's going to get harder and harder for us to keep these old work horse GPSes running as time goes on, but honestly, I still think it is worth the effort. I've still not seen anything that I like better.