Depends on what you are using the PFDs for. If I was out in deep water on a coastal diver hunt I would keep the PFD on. But we use the boat for transport generally and then park it and hunt out of layout blinds. Or if I hunt out of the boat blind (now only 2-3X a year) we are in water shallow enough for our 48" max decoy lines to hold. Usually the boat is in no more than 3' of water. Because of this, we don't have to keep the PFDs on while we hunt. I buy Cabelas 2XL or 3XL vests and they will fit over our layered upper body outerware system on the ride out and back. No need to spend big bucks.
Another point is that if you are not needing camo for wearing a PFD while hunting, bright colors are best. You want to be seen in the water, not blend with it, should you be in the unfortunate circumstance of needing rescue.