Well first of all, you have discovered the challenge, which is one of the drives for photogrpahy. I've ALWAYS thought its much harder to take a good photograph of something then it is to kill it. You have to get close, stop time, get all the settings right, have the light, know the background, on and on.
I've certianly been interupted more than once. Not quite as flippant as your encounter. Usually its not their fault and they are just out doing what I am doing. But when they are breaking the rules and they do it, or they go against the grain, its much the same effect as say hunting behind a closed gate and a wheeler comes bobbing up.