First off, GER doesn't posses infinite speed. At any moment is shown that GER can walk, speak or move during a time erasure. We see it in the manga that GER only begins to talk to Diavolo after RTZ takes action before Diavolo could even punch him. Why? Simple, because everything that occurs inside the time erasure occurs in a low flow of movements, actions; then it would be obvious why the RTZ worked at the exact moment when KC would attempt to punch him. So that's already something buffed without plot.
Now let's talk why Joker is completely able to defeat GER even with the return to zero. It is said by Igor that tricksters (aka Joker's nature) were beings that could defy and change the fate and causality of things that bind the universe of Persona. And Joker already showed to be able to do such things; he change the causality and fate of those people in Chihaya's social link; change his own fate, which was his death inside the interrogatory, and he also changes it when he was fighting Yaldabaoth, who was (SPOILER) Igor during the whole game and watched every step Joker took, every strategy he had, and even every hax, ability, Persona, everything. It was obvious that Yaldabaoth would win, and even so, he lost in the end of the game, which means that Joker can change fate and resist the causality of things that binds even type 4 acasuals (aka most of the gods of Persona).
(SPOILER) Things get even better with Maruki, who was stated to be even stronger and smarter than Yaldabaoth, since he was now changing the laws of the Metaverse too. It's a weak feat, but that also proves my point when we saw Joker being unaffected by the changes in causality and fate of the new reality Maruki (Also, he changed the causality of Akechi's death and the other ones who died too) was creating, showing that Joker being an acasual type 4 or far higher with willpower is supported by this, even if it's a little. It's not biased since it's shown in the social links, main story and even stated by one of the most wise beings in the game (Igor).
Now let's correct you there. Yes, Joker resists reality warping since he defeated Yaldabaoth and Maruki, and both could warp reality and its concepts that supports it. Yaldabaoth erased Joker from reality and even so, Joker returned stronger than before and defeated him with Satanael after unlocking him. By the way, he tanked Rays of Control and not even when Yaldabaoth was with the whole people believing him he managed to put Joker down forever. Maruki also tried to change Joker's perception and fool him with the changes in reality; guess what? Nothing of this happened in the true ending. By the way, if Joker got no resistance against reality warp, he wouldn't have won against Maruki, because it's said that if you're not resistant to it, you'll get warped as well inside it. Didn't happen with Joker at any moment.
Now since it's everything settled, it's pretty obvious why Joker is completely capable of destroying GER and Giorno with Morning Star or whatever he desires. Firstly because GER isn't invincible and never showed plausible feats to have Infinite speed (but as I can see, nobody is talking about this and I don't know why) and secondly because the RTZ wouldn't work on Joker since this power is based on warping causality, and Joker already showed (and was stated) to be completely unaffected by warps in reality, fate and causality. So forget it, the return to zero won't work. And now GER only depends on his biological attacks and his strength, and let's be honest, those won't do any harm to Joker (and the death loop won't work too because GER needs to kill the target first to put it in a deathloop, and Joker repels, absorbs and resists physical attacks).