As above, I found a teaspoon worked.

It was still difficult and I didn't think this was going to work without damaging the speaker, but it was fine. Take it slowly and be wary of any sharp edges of the spoon digging in, you should be fine. If I were to do this again, I'd probably cut up some rubber kitchen gloves to use as protection for the speaker, just to be on the safe side.

Prolific music listener / Hi-Fi addict!