I'm caught up in a rather common mess it seems, something that shakes my beliefs and challenges my new-found conversion to the Linux Realm. The problem seems to lie in the virtualization of the Linux OS under ESX 2.x. The process of virtualization creates issues with the system clock on the Linux box causing, in our case, a slow-down. In day we can lose more than an hour or 3+ minutes on 10.
Now, there is a fix. The real fix involves recompiling the kernel to watch the hardware clock, I'm not brave enough for that on the production server, so I took the quick-fix, the syncing of the clock to our time server. That's fine. It works, somewhat. The scheduled task runs at 4:01, 8:01,... 20:01 and it grabs the time from the time server and sets the system time and hardware clock from that. You see the problem, as it was originally, was that the guest OS (Red Hat Enterprise Linux) lost time against the vitual machine's BIOS. The entire machine is virtual and so it was keeping accurate time in the VM, but not under the guest OS. Well, about a week or so back, the VM lost sync with the HOST hardware and it hasn't recovered yet. So, what had worked, syncing with the hardware clock of the VM, now doesn't work so we're running this new script to get an "accurate" time from the time server.
But this isn't running well. The time-loss is so great that we, I, am just about ready to go back to a physical box. BTW: Our windows systems are having similar problems now, but that's not supposed to happen.
Okay, I've vented. I'm worried too. This reflects on our I.T. reputation and while I'd rather be Linux-free, The time to re-create this particular application is a nightmare. I do hope our VM vendor can find a solution soon.