Well the problem definitely seemed to be the temperature of the meat. After I ground it the first time with the bigger holes, I stuck the tote of meat in the freezer for a while, defintiely seemed to help not gunk everything up with fat and I was able to grind it the second time much faster through the smaller holes.
I made one batch with about 5% beef suet, and the next batch with about 10% bacon, the bacon stuff is awesome, best elk burgers ever I am realy happy with it, holds together great and the bacon isn't at all overpowering.