He wasn't so fatigued that he couldn't pull off Blue Kaioken x20.
Feats show that Goku got stronger. There doesn't need to be statements when there's feats.
There are numerous people in and before the ToP Goku did not hold back against.
Base Vegeta is not stronger than SSJ2 Cabba, he cheap-shotted Monna when she didn't expect it. And again, Vegeta is stronger than Goku and Caulifla is stronger than Cabba, and she's definitely stronger after her Zenkai.
My evidence is not Goku having an easier time against them, it's that he was clearly shown to be able to keep up with them in forms that were very clearly inferior to them before.
- Base Goku was able to hold his own against Post-Zenkai SSJ2 Caulifla, which immediately shows that he got stronger as nothing even implies SSJ2 Caulifla was base Goku level at the beginning of the ToP. SSJ2 Goku was then able to fight on par with Post-Zenkai SSJ2 Caulifla after she was explicitly said to have been adapting and getting stronger.
- SSG Goku was superior to LSSJ2 Kale to a degree (she was still able to slow down one of his ki blasts and swap hands with him), despite LSSJ Kale being shown and stated several times to be SSB tier.
- UIO2 and UIO3 both gave Goku's normal forms increases in power, UIO1 would have to as well via Occam's razor.
Caulifla landed several hit on Goku, traded blows with and clashed with him. Caulifla even took a knee to the stomach from Goku and didn't even flinch. They were clearly portrayed as even in their first fight, what are you talking about?
Again, Goku was suppressed against Kale, just not very much. There are numerous things within the ToP that show and state she is Blue level. Also Goku never even had the chance to use Kaioken against Kale. She walked through the Kamehameha, palmed his face, threw him away, then lost her shiz even further and basically nuked the arena, and Goku got buried in rubble afterwards. Kale also damaged him during her meltdown, so there's that. Goku was also at the strongest he'd ever been (apart from UIO2) against Aniraza, and later against Jiren, he showed again that he wasn't so fatigued that he won't use the Kaioken.
Whis' Spirit Bomb statement is also contradicted by feats.