Your groups move around as the barrel warms up. This is understandable to a degree in a hunting rifle as the barrel needs to cool to be predictable and shoot to a consistent point of aim. That said, if a sighting system is faultless, the shooter faultless, then often times the action is moving in the stock due to recoil. This is where pillar/glass bedding really helps. Pressure from the stock on the barrel is the first place to look, you've found that. Floating it may have a great affect, but may not if the action is loose in the stock, (not that you can move it around by hand), but loose under the snap of recoil. If clearancing that barrel doesn't help much, I'd look to the action bedding, loads, and sighting system.
-Steve
