I've shot a few elk on private in the last decade including a nice bull. The stress of other people comes with the territory. I've had people seriously mess my hunts up and I've also messed someone else's hunt up (that was an accident).
Sounds like the neighbor is selfish, but honestly I feel like there is more of that than the opposite these days. Plan for it.
The hard work of elk hunting is inescapable. Either you're dealing with people or you're dealing with the pain and suffering of the back country.