I have heard people say "You're either a deer hunter, or you're an elk hunter". The idea I get is that when deer hunters are presented with a situation like you described, they'll be more patient and wait for it to play out. "Elk hunters" tend to be more aggressive and "make something happen".
I am not sure that I agree with this, but if it is true, then I am more of a deer hunter than an elk hunter. I will usually error on the side of caution/patience when hunting.
Maybe that's why I've killed about 40 deer in my life, but only a couple of elk...

In short: I think you made a good call. Maybe aggression would have worked better in this case, but most of the time it wouldn't...
