1. Considering the massive jump in power necessary to match Jiren in the first place, I'm pretty sure his power is still significant. Especially since in the anime, the Granolah and Moro arcs didn't happen as far as we know
2. Training with Broly doesn't equate to being on his level. In the only time we've seen Goku with Broly, he was teaching him how to control his power, not attacking him.
If Goku truly was on Broly's level, then he got a Blue level increase since the Broly movie, which even in the manga we haven't seen anything of the sort
Imma be honest, Base Jiren probably gets overwhelmed pretty easily. LB Jiren will make this more fair