There's nothing going against his being a reincarnation of Damaio being the reason he grew so fast either. Damaio wanted revenge on Goku and the best way to do that would be to accelerate his growth. Not being fully matured after 3 years has no bearing on whether or not his growth was accelerated compared to other Namekians.
We can't really know either way. It would make sense for Damaio to accelerate his growth so he can get revenge on Goku before Goku gets much stronger. But then Damaio showed he's capable of birthing what appear to be adults (Drum etc), so it could be that fighter types are generally birthed closer to physical maturity than magic types. Or those like Tambourine and Drum could just be another type altogether.
The only other things I can imagine would give us clues would be how quickly the Nameless Namek grew up. The Nameless Namek was sent from Namek to avoid the cataclysm, which took place in age 200-something (I'm assuming one "Age" year is the same as a DBZ Earth year). It probably took around one month to reach Earth (although there's always a possibility they deliberately programmed the ship to go slowly so he could mature enough to protect himself before reaching Earth) and then it took him around two hundred years to become guardian. There's no indication of how long he took to physically mature and leave the area around his ship (I think the anime shows him as an adult when he does finally leave his ship's vicinity). The only thing there is that he doesn't remember Namek or his parents, so he must have been incredibly young on Namek. He can't have been a reincarnate like Piccolo Jr because then he would probably remember who parents were/their intentions, as Piccolo Jr had knowledge of Damaio. Did he land on Earth and immediately leave his ship? If he did, then he probably matured quite quickly within one month (enough to explore and be somewhat consciously aware of his situation), which would somewhat support warrior types (he was a combined type I think?) naturally maturing more quickly than pure magic types.