It's the same with the recoil of the big cal's don't feel it when you have game in the sights.
But the issue is that a rifle is shot more often at targets than at game animals (or should be). If you develop a flinch from sighting in your rifle from a bench, you'll most likely still shoot with a flinch when shooting at game.

Nothing better than listening to someone swear up and down that they don't flinch, then loading the "dummy round" in their rifle for them. Their face usually looks like this after they're done jerking themselves around....

I've found my rifle caliber that I'll probably stick with for a very long time. It's the 6.5 x 55, and I'd hunt anything and everything from coyotes to moose with it. Very little recoil, and great ballistics. It's not a "popular" cartridge, but with a little planning you'll never run out of ammo.