While I wish Oda gave them better overall portrayal outside of just hearsay, it makes more sense if you scale them relative to other armies.
If the average Elbaf warrior is at least as strong as Arlong for example, that is a very high bar as far as common foot soldiers go in the OP world.