Zoro had a bunch of strong and skilled sword-wielding sparring partners on Mihawk's estate, which he wouldn't have with Rayleigh. Zoro also thrives off of long, physically gruelling training sessions and I'm not sure he would have access to as much of that with Rayleigh and the wild beasts.
Rayleigh could teach him about haki, but I don't know if Zoro's current foresight haki is any worse than Luffy's, whereas I'm pretty sure Rayleigh wouldn't have taught him as good armament haki, seeing as Mihawk is famous for his oversized black blade, while Rayleigh isn't.
Maybe Zoro would turn out to have conqueror's haki.
Maybe Luffy would actually die if he had to fight the monkeys. He can fight a swordsman, but it was his forte at the time, since his armament was non-existent.