There were only two options with the 5 elders, either they held political power only, or they were some of the strongest beings in the verse.
The only issue with them being stronger than Kaidou is why they bothered letting the yonko and shichibukai exist at all.