Well my system really shouldnt be a problem, the server doesn't use much cpu and I can play the game while hosting a server at the same time just fine without any performance issues.
This happens with or without me being in game, anyway here are my system specs:
Intel Core 2 Quad 9550 (stock clocks 2.8ghz)
8800 GTS 512 (evga)
Asus P5QC motherboard
4GB DDR3
Windows 7 64 bit
Well perhaps my internet spikes the upload sometimes, in a month I will be upgrading to cable, I'm currently using adsl 8megabit down / 1 megabit up. the upgrade will only have slightly better upload though, 25 megabits down and 1.5 megabits up.
Thanks for trying to help out, much appreciated.
Removing the Left4DeadToolz extension seemed to lessen the issue a bit, but it still happens sometimes.