In fact, it wasn't said anywhere that Hagrid was bad at magic. He even practices it outside of school, which hints at his curiosity. I think he's about Newt's level or stronger with extra strength. Nevertheless, he successfully masters non-verbal magic (and only skilled wizards can master it), remember how he lights a fire with his umbrella and creates a pig's tail for Dudley. He also created his own spell, with which he guided boats with freshmen to a set location automatically.
here are the moments when Hagrid fought against other wizards:
1) Hagrid’s door had burst open and by the light flooding out of the cabin they saw him quite clearly, a massive figure roaring and brandishing his fists, surrounded by six people, all of whom, judging by the tiny threads of red light they were casting in his direction, seemed to be attempting to Stun him. “No!” cried Hermione. “My dear!” said Professor Tofty in a scandalized voice. “This is an examination!” But nobody was paying the slightest attention to their star charts anymore: Jets of red light were still flying beside Hagrid’s cabin, yet somehow they seemed to be bouncing off him. He was still upright and still, as far as Harry could see, fighting. Cries and yells echoed across the grounds; a man yelled, “Be reasonable, Hagrid!” and Hagrid roared, “Reasonable be damned, yeh won’ take me like this, Dawlish!”
2) And now he saw the vast outline of Hagrid, illuminated by the light of the crescent moon revealed suddenly behind clouds; the blond Death Eater was aiming curse after curse at the gamekeeper; but Hagrid’s immense strength and the toughened skin he had inherited from his giantess mother seemed to be protecting him.
3) “How do we get out?” yelled Ron over all the screaming, but before either Harry or Hermione could answer they were bowled aside: Hagrid had come thundering down the stairs, brandishing his flowery pink umbrella.“Don’t hurt ’em, don’t hurt ’em!” he yelled. “HAGRID, NO!” Harry forgot everything else: He sprinted out from under the Cloak, running bent double to avoid the curses illuminating the whole hall. “HAGRID, COME BACK!” But he was not even halfway to Hagrid when he saw it happen: Hagrid vanished amongst the spiders, and with a great scurrying, a foul swarming movement, they retreated under the onslaught of spells, Hagrid buried in their midst.