I wouldn't really say that's true. Base Baby Vegeta more or less was able to tank SSJ3 Goku's attack without taking damage. Before Goku regressed back to Base, he was shown effortlessly evading his attack. Baby didn't look anything close to serious during that fight, which is also supported by the fact that he later told Goku that he would draw out their fight to make him suffer more.
Goku even said that he would need SSJ3 just to withstand Baby Vegeta, he never showed confidence that he'd be able to beat him outright after Baby powered up. He was banking on being able to drive Baby out of Vegeta's body somehow.
Episode 29
Time: 4m35s
Context: Goku just attacked Vegeta to no avail
Goku: "Once I've defeated you, I'll beat the eggs out of everyone else!"
Vegeta: "Once the eggs have been implanted, you cannot beat them out of someone!"
Goku: "In that case, I can still drive you out of Vegeta! I should be able to drive you out!"
[ ]
Vegeta: "You sure you can do it?"
Episode 29
Time: around 8m15s
Context: Vegeta has just powered up
Goku: "He's got awesome power! If I'm going to withstand him...I'll have to do it at Super Saiyan 3!"
Goku was at a level where he could hold his own against Base Baby Vegeta, but he was already surpassed prior to his transformation into Super Baby Vegeta.