Lets say Griffith had an early growth spurt and was near his adult height at 15. He met Casca who was 12 and was much taller than her, and 2-3 years later they both met Guts who was only 15 at the time. Makes sense to me.