I agree confrontations suck.
In Montana, I found a half section of public land that had a ton of antelope and monster whitetails on it. A beater old truck with four young guys came up and told me I was on their Uncle's land, he had bought it from the state. So, not wanting to push it I left. This year, I happened to run across the warden and asked him about it. He confirmed that he was unaware of anything. I actually looked up the records at the county and on MT's hunting maps from the state. Both confirmed it was state land. So, I head back and this year there is a lock on the gate and a fake sign saying no hunting and hinting it is official MT policy.
I guess my whole point is that I am tired of doing everything right and getting jacked by local people who think they have more right to the land than I do. Whether to push the issue or not is a personal choice.
If it were local land that I planned on hunting more than once, I would probably look into hunter harassment laws and have a chat with the warden.