Because (if your form is perfect), your arrows are not coming off of your bow perfectly straight. In this case, your arrows have the tip slightly to the left, and the broadhead is then "steering" them to the left before the fletching can stabilize the arrow. Hence the POI is left. If your nock was high, the arrows with BHs would hit low. Nock low, BHs would hit high. Field points don't make the arrow plane, or steer it like a BH does.
Make sense?