He just doesn't like them. That's all.
His fear was fueled by cowardice and bigotry, believing the Saiyans would betray them the moment they have a chance even though Vegeta and his father (whom he killed) were the only ones who would have a reason to