So, the new DST changes in the US caused a small stir among system administrators recently. We got all of our servers updated and verified they were working before the even. Or so we thought.
I noticed today that our 3PM Eastern newsletters arrived in my inbox at 3PM alright. However, I am in Central time. My immediate assumption was that we missed the server that sends that email out. Logging in I found the time correct on the server. It had received the appropriate updates thanks to portage. So, what happened? I looked at /etc/crontab and all was fine. I then looked at the system log where cron jobs are logged. Oddly, that log line said the job started at 15:00. I knew that was not correct. I started looking around at other cron jobs on other servers, especially ones that wrote files to disk. Sure enough, every server I checked was doing things an hour behind except one. It just so happens we had restarted cron on this server however last …
[Read more]