As Kyo said, he's only Light Speed when moving in a straight line or using the Yata no Kagami. It's also worth noting that Haki users probably negate his lightspeed attributes, hence why he can be countered by characters such as Rayleigh and Whitebeard.