Just as a reminder.
No, DBZ anime is "not the same" as manga. The anime has a lot of added scenes, different scenes, outright references and connections to most of the movies made for Dragon Ball (so that's already more than dozen different events that affect the scaling and never happen in the manga), not to mention being related to and AFFECTING every Dragon Ball GT profile, as GT is a direct continuation of the anime.
I am not saying it needs this amount of stuff, but I am just hammering again that you are all massively simplifying how different the DBZ anime Goku is from the manga.
Also, no, outliers can be outliers simply because they are ridiculously above every other conceivable feat. Anti feats aren't needed, they are just reinforcing evidence. Imagine the best anyone has pulled is 50% speed of light and then someone flies to Saturn and back in 5 seconds. Nobody has powered up, but they can keep up. How that makes any sense, I don't know.