[Psyk0tik]

It sounds like you're only allowing lobby connections to your server.

These are what I have for allowing players to connect to my servers in multiple ways:

sv_steamgroup "XXXXXXXX" // replace this with your Steam group's ID
sv_steamgroup_exclusive "0" // 0 = allows non-members to connect/see the server, 1 = only group members can connect/see the server
sv_allow_lobby_connect_only "0" // 0 = allow players to connect in different ways, 1 = only game lobbies can connect to the server
sv_lan "0" // 0 = allows public connections, 1 = allows local connections only