Uh no, the wish is not "make goku stay in that age", the wish was for goku to return to a certain age as a child, he can still grow naturally.
Ssj4 literally just ages him back to his prime, nothing there implies a resistance, because there is no constant age manipulation happening.
Goku got aged once, ssj4 temporary ages him back, he goes back to being a kid when it finishes.
That's the same as considering the ssj multipliers as permanent power ups, no, goku's base doesn't become 50x forever, it is temporary, just like ssj4's aging stuff.