You're not considering the scene in its entirety.
You're assuming that Vegeta intended to rough up Buu. You're entitled to your opinion, but brawling would cause a commotion, which is the opposite of what Vegeta wanted. That explains why Vegeta never followed through with fighting Buu, and...