So there we were at work, wondering why part 2 of our home-brewed customer survey system[1] wasn’t sending the mails it should be sending.
Turns out that the perl script responsible read from one table in the database to find out the recipients of the emails. It then generated a form of authentication, updated the database to say that the mails had been sent, then called
system( "/usr/local/bin/email.pl" );
where email.pl was another script written by the same developer.
Note the lack of return value from that call to system.
So what happened was that in the distant mists of time (about 2
months ago) we switched source control systems. Apparently we lost our
executable bits on some scripts, because email.pl was readonly and
owned by Apache.
Had we checked the return value, we would at least have had a warning that something was wrong. But now the system apparently “worked” for a month or so, and we only realised that nothing was being sent when the responses dried up.
I’ve re-written the code to check if the call to email.pl was
succesful, and only if it was would the database be updated.
Now we have to re-send all emails older than a month, which will probably put our server on a spam blacklist. Sigh.
Lesson: code defensively.
[1] Why we haven’t got a third-party solution for this is hard to understand, unless you worked there 2 years ago, when development time was considered free and maintainance was never factored in the cost equation.