Take the “hit” out of your stroke by imagining the ball is transparent to the putter such that it will go right through the ball to strike it first on the side closest to the hole (yellow dot).
Taking out the hit smooths out your stroke, making it more reliable.
At the time you would brace for that little hit, it has already occurred. (And if you think you don’t brace for the hit, think again. You do.)