Yaji was already 200+ in the King Piccolo saga. He was probably stronger than Goku (hungry Yajirobe could give Goku a good fight, even without a sword).
Goku here is probs like 300ish. If Yaji is 650+ that should really be enough.
He was strong enough that he was expected to help against Nappa. He should be at least stronger than Goku and Piccolo was against Raditz (416, but about 900 with an around full power attack), unless the whole cast wanted him dead. I'd expect Yaji to be around 600-1000, without a sword.
Rd 2 Without a sword Yaji loses. Dude can't even fly, no way he deals with Goku and Piccolo teaming up. With a sword, Yaji probably wins more often than not.