Baby only brags about obtaining Saiyan power when he goes SBV1 instead 2, which was a form powered by humans all over Earth, so he’s definitely talking about how some of the strongest Saiyans in history are giving their power to him. Goku just agrees with him; “it’s true!”
Fused characters are rarely taken into account because they only exist for a brief moment in time. See how Super Boo says Gohan is the first and only warrior to surpass him, for example. I don’t see how a guy that existed for an hour 15 years ago matters.