The problem is that we haven't seen him fight anybody noteworthy, we don't know how much stamina he has, and how good his Haki is. Current Rayliegh might be a good benchmark in that respect, who was shown to be at a disadvantage against Kizaru.
Despite being sick, Whitebeard was treated like a monster at Marineford. It seems like he was portrayed above Garp, who was healthy. A bit strange, since both were on par with Roger back in the day, but perhaps the Gura Gura no Mi is just that OP.
I'm willing to give Garp the benefit of the doubt that he could perhaps beat any commander below the Yonko (besides maybe Marco) based on his legacy alone, which includes Sanji.