I did not have time to test it on a server with other people. But I noticed that it was telling me that listen was disabled when I joined the game. Your hooked event was being triggered just from switching between teams and on join.
I made some changes to the plugin so it does not tell you its disabled when you leave the spectator team, also will not enable it automatically on the spectator team.
Only the !hear trigger will allow you to enable it. I find this to be a little bit cleaner.