He's been shown to be physically really strong. He just seems weak due to his fruit absorbing damage more, which if anything makes the fact he hasn't been KOed or completely immobilised impressive. He took out Ace easily; imagine if someone took out pre-skip Sanji in a few punches. A better example is look at how many punches it took for Luffy to take out Enel, even when negating his DF for the first time in Enel's post-DF life. BB took out Ace - who is probably pretty strong even DF aside if his brother Luffy is anything to go by - with way less hits. So dude is pretty strong.