There's nothing implying Goku didn't go all out vs Yajirobe. Even disregarding the anime's statement (which is something... better than nothing), Yajirobe's surprise at Goku's offensive power vs Tambourine later on might have to do with how visceral Goku was with it, also employing the after image technique firstly to confuse Tambourine, following it with a full power barrage of hits. As I noted prior, Goku was also surprised by Yajirobe's feats against Cymbal, calling him incredible, so if Yajirobe's surprise at Goku's prowess vs Tambourine is to be used as proof that Goku was holding back prior, so can Goku at Yajirobe cutting down Cymbal.
I also see Yajirobe basically equals with Tenshinhan... which could place him very close to Mutaito. It's debatable whether Goku gained any power upon his recovery and since I go with the option of him only regaining the power he displayed at the tournament, I could see Yajirobe and Tenshinhan sharing a close rivals gap with Goku with Mutaito also in the same vicinity.
Kame-Sen'nin states that "even his younger self was no match for Piccolo", which is much more a comparison between his 22nd self and prime, young Roshi... It has not bearing on 22nd Goku and Tenshinhan.