He was hitting and defending as hard as he could physically.
I interpret the Kamehameha he used against Fat Boo the same way I do the Kamehameha he used against Pure Boo in the beginning of their fight. It was a FP KHH, but not a gathered ki one (which is what is needed to kill any given Boo and Goku knows that)