Tire. No question. Deal with it if you don't want to replace the tire.
On a new tire, the rolling center is pretty much perfect. As the tire wears, that centerline wears a little bit to one side or the other, and when you let go, it has the effect of steering the tire, one way and then the other, as the centerline on the road moves from side to side of the bike's actual center. For most of us, the perfect storm of wobble is between 40 and 50 mph, probably a harmonic thing with the mass of the forks and bars.
That's my theory, and I'm sticking to it until somebody with important-sounding letters after their name says otherwise! :lol: