Running a Source Dedicated Server

Hi,

When I host any source dedicated server, the server itself is fine perfect bullet reg extremely low pings ect..

but when there has been no one playing on the server for a couple of hours and then someone joins the server itself is very jerky! and the only way to fix the problem is to restart the server.

has anyone experienced this problem an how do i fix this?
the OS i'm running is Windows Server 2008 and Server 2003 R2 - Does it on both OS's.