With a .22? Sort of normal not to be able to see the actual hole. You should be able to see a mark on the target though. Still best t get some kind of optic. Binoculars(likely the least expensive option. Cheap compact binos start at $21.99 at Cabela's. $70ish for better quality. Bushnell 10x 42 binoculars are currently on sale at Crappy Tire. $100 binos for $69.97.), spotting scope, etc. Even an old scope sight will do. Assuming you have such a thing.
"...your grip might change from shot to shot..." Not with a properly set up spotting scope. Said scope gets mounted on your shooting box so you just move your head. More expensive though.