In the real world, it doesn't matter which buck you take. There is still gonna be one less buck in the world. And there is no guarantee either one of them will survive another year no matter which one you take. They could end up shot by another hunter, taken by a predator, starve in a hard winter, get run over by a car, succumb to injuries from fighting during the rut, etc etc. Wild animals don't usually die of old age.
And depending on why deer numbers are down in that GMU, whatever good you think you're doing may not matter. If the habitat is degraded, the does will have less twins. If the area is full of coyotes and other predators, the fawns may not survive. A bad winter or a series of bad winters might do in most fawns that make it to the fall. There are a lot of factors controlling herd populations. If you think that taking any deer out of the population is going to be the deciding factor of population growth, then I'd agree with the post earlier that suggested you leave those deer alone and go to a more populated GMU that can handle the hunting pressure.