The 20 is smaller, lighter, easier to holster, less accurate, and more recoil using the same rounds.
I dont have a 40, so my experience is mostly comparing on paper. If you are looking for a defense sidearm to carry while walking around the woods, weight and bulk matter a lot more than accuracy. If you are looking for a handgun for hunting, the opposite would be true.