I think Yajorobe is a distant rival of Goku, like 90/100 type of gap. Later on Yajirobe called Goku a monster or something like this by him knocking down old Piccolo Daimao, hence why I think Goku was noticeable above him.
I also think both Goku and Yajirobe were warming up in the fight. Goku showed surprise at Yajirobe defeating Cymbal, calling it incredible and Yajirobe also showed surprise at Goku defeating Tambourine, saying it's better not let him angry. They were holding back power in the same proportion IMO.