[Chdata]

TF2 [/source games] come with default settings for which to kick AFK players. Without a server.cfg set, who's to say those default settings are not kicking your players?

(Since it says kicked due to inactivity, that sounds exactly like you're being kicked by TF2 itself, not a custom mod or plugin).

Try looking at these convars:

sv_timeout 300
mp_idledealmethod 0
mp_idlemaxtime 1600