Ok, it seems my arrow farm has (hopefully had) a flaw in the redstone. If someone were to TP away, if the timing is just right the redstone clock would get stuck in the off position. This means that the grinder doesn't grind so the mobs keep adding up.
This wouldn't normally be an issue because once you leave the spawner it will unload the chunk and the mobs should disappear. However, I believe that sometimes if a player times out the mobs will stay loaded (at least their count towards the mob limit of the server). I think that's what happened here.
I've made some changes to the farm. Now the grinder only activates when you are on the pressure pad in the collection room. This means the player won't need to flip a switch to turn off the grinder as they leave via the nether (once they exit the collection room the redstone turns off and the clock resets).
Now what I'm unsure of is if someone /tps out of the collection room (or disconnects), will the pressure plate be deactivated before the chunk is unloaded...
In case it is still an issue there is now a master kill switch that will override all the redstone clocks (but won't reset them

) and will kill everything in the spawner.
The issue is that sometimes the repeaters if they are active when the player leaves the chunk, they stay active even if not powered. Without removing and replacing a bit of redstone (dust, torch, repeater etc) on the circuit (manually) I don't know how to reset it.
To the wiki!