I always thought they did, especially when Luffy was using his durability against blunt attacks so much and with logias and some paramecias being hard for certain characters to hurt, but with haki it might matter less. I still think it matters though, like the Red Hawk against Doffie never would...