Ohhhhhhhhhhh, Valve disabled the ability to use such plugins within non-dedicated servers. Apparently people could load the plugins in a listen server ("Create Server" like you do) and then keep the plugin loaded when playing in a game elsewhere which allowed people to use sourcemod plugins on those other servers.
http://forums.alliedmods.net/showthread.php?t=137742
You must use it within a dedicated server, yes.