What Freeza really feared was a Super Saiyan rising up. Did Freeza himself ever state that he was afraid of the Saiyans being able to beat him if they unified? If I'm not mistaken we only hear that from Carbon/Dodoria. Furthermore, Jheese states that Freeza only feared Super Saiyans.
Chapter: 285 (DBZ 91), P4.1-3, P5.2
Jheese: “Su…Su…Super Saiyan…!? Th-the legendary mightiest of warriors…[ ] So h-he’s…The one thing Lord Freeza feared, the Super Saiyan…!!”
Ginyu: “You don’t want to kill me? You don’t want any pointless fights…!? The Su-Super Saiyan is supposed to be the strongest warrior in the universe, who loves blood and battle…”
Chapter: 264 (DBZ 70), P6.5
Freeza: “It seems that the Saiyans really do hide bottomless battle power…It increases greatly the more they battle…Of course, there’s no way they could be a match for me, but if I don’t think of the future and nip them in the bud now…After all, it’d be troublesome if they were able to become Super Saiyans…”