Voice_of_Reason:
Vegeta specifically mentions choosing the night of a full moon to allow them to get the job done faster, and is likely the primary method Saiyans relied on to conquer planets for Freeza. Vegeta apparently uses the form enough that he can retain conscious control over it, unlike Goku and Gohan.
He isn't stupid enough to take out the Ginyu Force since it would only be a pointless act of treason towards Freeza, but without the immortality of the Dragon Balls to fall back on. If the Saiyan trio would be off conquering planets while the Ginyu Force are out doing their own missions, the opportunity could never arise where they'd be on the same planet with a full moon anyways.
The Oozaru is not established to reduce a Saiyan's speed, and Goku even expresses the opposite. Vegeta also tells Kui on Namek that when his battle power increases, so does his speed, and does not limit that statement to his near-death experience.
Speed has actually been expressed by the characters consistently throughout the story to not correspond with the size of one's Ki, it just often does anyways. Tenshinhan believes Goku increased his power but not his speed at the 23rd Tenkaichi Budokai, Piccolo admits his inferior power but believes he's still faster than Freeza who just transformed a 2nd time, and Gohan is confident that he's faster than Majin Buu who's power had caused him to tremble in fear:
Chapter: 177, P1.2-4, P2.1
Tenshinhan: “Son, you are truly incredible. 3 years ago, your strength was all but perfect. I’m amazed that you’ve managed to go so far above even that. However, there’s one thing that you haven’t changed that much from 3 years ago. And that’s something especially important in battle…speed!”
Chapter: 302 (DBZ 108), P9.4
Piccolo: “Damn him! You may be more powerful, but I’m faster! You couldn’t catch up to me even if you spent your entire lifetime!”
Chapter: 462 (DBZ 268), P9.2, P11.3
Context: as Gohan and Kaioshin flee from Boo
Gohan: “It’s alright! I’m confident in my speed!”
*Boo quickly catches up to them*
Gohan: “…! Th-that can’t be…!”
So Vegeta's line should not be taken as some universal explicative, it would probably just apply to his own experience up till then, since he uses the possessive "my". Which makes sense because a character can raise their Ki through meditation which is hard to see how that could affect a person's speed when they're just sitting still. The Grade forms of SSJ exemplify this precisely, where the reason Vegeta and Trunks' Grade II forms may have granted an increase in speed was due to their training in the form extensively after achieving it, unlike Goku.