It's more secondary to the sword. WB can easily be a swordsman too, and if he fought and beat Mihawk that would disprove the entire thing (using WB because he's the only one 100% stated to be stronger than him and anyone else).
So far we only have Zoro, Vista, Mihawk, the Samurai, +Wano who are shown to be primary swordsman. When someone is primarily a swordsman it's made really obvious like with Vista.