Beware "Login Items" and Command-line tools

In March 2003 I did a clean rebuild of my G4 server, moving from 10.1.x to 10.2.x. Well, the StartupItem I wrote to launch the watchdog broke. Instead of bouncing the already-live server to debug the problem, I tried something different -- I used the "Login Items" System Preferences pane to add the perl script.

It added just fine. I logged in and out, checked the process listing and there was my watchdog. Golden, or so I thought.

This weekend a client's site went down. We found out about it fairly quickly, and I was able to bring it back up remotely in less than a hour. (It turned out wotaskd went insane and wouldn't allow new instances to register themselves. Had to bounce it.) However, the watchdog never barked. I should have gotten a "RED ALERT" email within five minutes of the site going down.

The script has been largely unchanged since initial deployment, so unless bitrot got the code, it wasn't a logic issue. I checked the process listing and sure enough, the watchdog was running. But something else caught my eye while ps dumped its output -- an instance of curl was running. Now, the chances of me seeing curl should be relatively rare -- it gets fired for a split second every five minutes. So I run ps again. I see curl again. Now this is close to impossible. Then I look at its start timestamp. March 18th. Over two months ago.

Apparently my watchdog spawned curl to HTTP-ping a website, only to have curl block forever, which blocked the watchdog forever. Something within curl blocks forever when spawned from a "Login Items"-entry, even when passed the --silent switch. This is not the case with ping, however.

After creating and installing a 10.2-happy watchdog StartupItem, all seems right again. But the lesson remains: beware command line tools launched from the "Login Items" System Preferences pane.

Monday, May 19, 2003
03:52 PM