It's funny how few times he's even refereed to as the black swordsman in the series for how butthurt beserk fans get about it.
Even Aincrad era Kirito probably wins without duel wielding. The anime does not do justice to just how strong the clearing group players are in relation to a normal human baseline. I stopped reading somewhere on the boat ride of doom, but I never saw anything that puts guts on the level of speed or strength those characters have.
And he certainly loses to Alicization characters that have the ability to warp reality in the game and use things like energy absorption and laser like blasts from their release recollections.