He only had a few streaks of white hair and beyond his illness, there's nothing to suggest Roger had severely weakened in combat capability around the time of his death. The same should apply for Garp. We're given no indication Garp and Sengoku weakened at all between the Battle of Edd War and their fight with Shiki, and both of them had no qualms with the idea of taking on either Roger or Shiki's crew around the time of the former.