That's a valid point on his part, actually. Slade can whoop like 30 dudes at once, including highly trained martial artists. He scales to Batman, who can fodderize Assassins that are skilled enough to defeat 100 men easily. No one in Uncharted is as skilled as any of the playable characters in the Arkham games.
Nate has good feats with CQC as well. In Uncharted 3, he defeated
a whole gang of pirates with his fighting skill alone after waking up from his abduction and has done so consistently throughout the game with Marlowe's men, who are agents of an ancient cult. He's also defeated men twice his size and even Lazarevic, who was so physically superior to him that he could lift him up with one hand and
throw him high into the air several meters away. Lazarevic should also be comparable to the Shambhala Guardians since they both got their powers from the same source, who can
lift and throw boulders with enough force to shatter them.
Also the fight with Rafe has different factors. In Uncharted 4, Nate was three years out of his prime and was getting back into the swing of things. In addition to this, that was a swordfight and Nate had
zero experience with a sword up until the point yet he was still able to keep up with Rafe due to his reaction timing and strength alone which allowed him to gain a brief upper hand. The fact that a rusty Nate could react fast enough to dodge Rafe's sword swings and even gain an upper hand in a swordfight without prior experience is noteworthy I'd say.
Also, where was it stated that LoA members can fight 100 men? Cause I remember one of them, while they were able to put up a fight, ended up getting caught by Penguin and his goons in Arkham City iirc (which is how Bruce was able to cure himself of Joker's toxin by tailing the member after her escape and finding Ra's).
I wouldn't worry about Slade's armor too much since Nate has high cal weapons and explosives to get past it while Slade's mask for instance can be broken off, which was what Bats did in their fight in Arkham Origins.
I'd still vouch resilience for Nate cause while Slade is a meta, he doesn't exhibit that many feats of this since he showed up in only two games (and Arkham Knight didn't do him much justice), mainly just waiting for Bats to show up and then ambush him. I'm certain Slade's endurance is great considering his experience but we just see a lot more consistency with Nate whereas Slade's isn't as fleshed out. We know Slade's resilience is high but to what degree is uncertain compared to Nate's consistent punishment.
Again, this fight is extremely hard for both regardless. I mainly think the early stages of the fight along with Nate's edged out advantages and Slade's characterization make it so that Nate could take it more likely.