a long time ago I got some advice about looking at the hole when I putt and it was like a miracle cure. I was able to magically put right into the hole so accurately it was freaky, especially on the practice green. I became very dependent on this this tactic, and while it had its moments of brilliance and glory, there were far too many cases where I would mishit the ball. Mishitting the ball can simply be not hitting the putter face in the middle or topping it, etc. Those generally cause wild misses. So looking at the hole my putts would either be scary accurate or mishit way off.
For a while I did a hybrid where for long putts I would look at the hole in order to gauge the distance better, and shorter putts I knew how to stroke it and line was way more critical then distance so I would look at the ball. And my lag putts would be so scary accurate most of the time, looking at the hole my body would do these subtle last minute speed changes to my stroke and put the ball just the right distance. However I still had the problem of occasional mishits, and often enough to effect my score.
I have since gone to a visualization system, where I practice strokes looking at the hole, I get my line, then I setup to the ball and I look at the hole (or target spot on the green), back to the ball, back to the spot, a couple of times until I have a mental image in my mind of it, in terms of distance. Then I look at that ball and focus on making a clean stroke on the correct line, VISUALLIZING in my mind simulaneously that target spot so that I make those last minute body feel adjustments to my stroke for distance. In a few weeks time I have been become pretty scary accurate that way in terms of stroke distance and now I almost never mishit the ball. My errors now are when I get distracted from that visualization and then don't put the right feel into the shot. But that is over comeable with practice.
I highly recommend you move to a visualization method instead of keeping your eyes on the cup.