Upstart respawn with expect fork does not respawn

This blog is for our fellow system engineers who might be cursing Upstart’s undocumented behavior. :^)

Upstart’s respawn stanza will not (always) work in expect fork or expect daemon situations. An example daemon is Nginx, which in its default mode requires expect fork. Nginx executes several operations before it forks into the background, notably: binding to a port. If this fails (because a previous instance is still alive or because the kernel still occupies the socket – see ss -tan), Nginx will exit 1, but Upstart is still waiting for the fork and will not execute the respawn.

Demonstration of nginx behavior:

There are several solutions to make Upstart work with such daemons:

  1. Do not let the daemon fork. For Nginx, you can use the -g "daemon off;" option. Most other daemons have similar options.
  2. Manually clear the way for pre-forking fatal error conditions. For this particular problem, you can use fuser --kill 80/tcp to kill occupying processes.

Do you like digging into these kinds of problems? We got plenty 😉