Here are the times it has been "infiltrated":
-Big Robot climbed the Red Line, then ran out of power.
-Doflamingo saw some secret and booked it before he could die.
-Fisher Tiger climbed the Red Line with his bare hands, freed slaves, and they escaped in the chaos.
-Laffitte snuck past everybody, despite their Haki, while wearing tap shoes, and offered them a new Warlord.
-Shanks just appeared before the 5 Elders.
-Jewelry Bonney and the Revos snuck in, and Sabo literally got impaled.
-Kuma jumped in, got his face melted, and jumped out.