I think Yamcha just outmaneuvered the Saibaman. It's like when Obi Won Kanobi had the higher ground and told Anakin not to test him because of the advantage. Anakin foolishly went after him anyway and lost his limbs in the process. This is how I see the situation. It's possible Yamcha was holding back, but why on Earth would he hold back? I think it's a tactical thing, and an overly cocky statement.