You hit the nail man, Roshi is the one who makes those Chappa's statements being explained as reputation a little fishy. Roshi's reactions to Panpoot's defeat and and to King Chappa's defeat are clearly different. He is fearing Goku when Chappa is defeated, saying things might be bad for him, yet against Panpoot he treats it as something natural... he heard what Panpoot is capable of and remembered his name as belonging to an excellent fighter, yet was not surprised after Goku easily defeating him. Even Kuririn, upon seeing some of Panpoot's movements and him destroying the wall still labeled him as "nothing special", yet he didn't say how Chappa-o was below expected... it was all about how Goku was amazing.
But on the other hand we have Yamcha saying he never imagined someone could defeat Chappa this easily, while he should also be able to replicate those feats. This sounds more like he is in kind of denial about the great legendary fighter being nothing special which is cool and all, but the aforementioned stuff goes up against this interpretation.
Also don't forget that Roshi said every one of them (Goku, Kuririn and Yamcha) were strong candidates to winning the tournament, even after seeing Goku taking out Chappa:
Chapter: 115, P14.2-3
Kuririn: “You were watching?!”
Kame-sennin: “Yup! All three of you displayed your progress magnificently! Any one of you may win the top prize this time!”
Well in depth Roshi probably knew Goku had the more chances, but saying all of them may win the tournament while Goku just showcased power considerably superior to both of them seems a little off.
If you see a video of Mike Tyson in his prime doing head movement drills, you don't need a fucking scouter to see that he will fucking kill you and he's crazy fast.
I can do that on table tennins. When seeing a table tennis match I can pretty much say both players level in seconds of playing. Fight... not so much. But considering how Yamcha, Kuririn and co. are so accustomed with fighting (as am I with table tennis), it should be easy for them.