Running as a server

A common scenario for running an application such as fwd is to run it as a true server application. Here are some of the requirements for fwd when it runs in that mode:

1. When your application runs it will run with the privileges of a particular user. We will need to arrange for the application to run as a user who has enough privileges to do everything the application needs to do.
2. If your application needs to access files or directories on the system it will need to have the necessary privileges needed to access those files or directories.
3. Your application will run in the background on the server alongside many other applications.
4. If your application needs to work with a configuration file, that file will typically be stored in the /etc directory.
5. If your program needs to write to a log file, that log file will be stored in the /var/log directory.
6. As part of the installation process for your application you will need to supply a configuration file and put it in the /etc directory.
7. You will most likely want your application to start up automatically when the server starts up. To do this, you will need to make use of the server's init system.
8. When the application starts up it should put itself in the background and then also save its process id in the file /run/fwd.pid. The application will also need to tell the init system where to find this pid file.
9. The server may be using a feature called log rotation. This is a feature on servers that will periodically rotate out an application's log file for a fresh, empty file and then compress the previous version of the log file. The log rotation system will inform your application that the log file has changed by sending it the SIGHUP signal. Your application will have to install a signal handler for this signal that closes and reopens the log file.
10. The init system will also make it possible for administrators to restart your application. For example, an administrator may want to add a new row to the fwd.conf file and then restart the application. To restart the application the init system will send your application the SIGTERM signal, wait for it to shut down, and then restart the application.

What is an init system?

On a Unix system an init application is an application that manages the startup process. On many modern Linux distributions the init system in common use is the systemd application. You may encounter other init systems such as upstart or the even older init application.

systemd expects developers of server applications to write a special configuration file, called a unit file, to tell systemd how to manage the startup process for their application. Here are the contents of the unit file you would use for fwd.

[Unit]
Description=Folder watcher application

[Service]
Type=forking
PIDFile=/run/fwd.pid
ExecStart=/opt/fwd/fwd

[Install]
WantedBy=multi-user.target

The unit file is named fwd.service. You will place this unit file in the /lib/systemd/system/ directory, which is one of the standard directories that systemd looks in to find unit files.

A unit file consists of several sections. Here is a description of some of the options I have set in these sections.

The [Unit] section covers basic details of the service. The Description option gives a brief description of the service. If you ask systemd to show you details about running services on a machine it will print that description as part of the information it shows you.

The [Service] section is the most important section for configuring a service. The options in this section tell systemd how to actually start the service and what to expect when the service starts. The most imporant option in this section is the ExecStart option, which specifies a path to the application we want to start. The Type option tells systemd what to expect when it starts the application. In this case, since the application will fork to make a child process that runs in the background while the application itself exits, we need to tell systemd to not worry about the fact that the application appears to exit as soon as it launches. Forking applications are expected to generate a pid file that stores the process id of the background application. The PIDFile option tells systemd where it can find that file. systemd will use that process id to shut down the server application when the system shuts down or when the user asks systemd to restart the service.

The [Install] section is required for services that want to start as part of the boot process. To actually register a service for startup during the boot process, the unit has to be enabled, and must also have an [Install] section in its unit file. The WantedBy option links this service to one of the targets that systemd attempts to bring up during the boot process. The multi-user target is one of the later targets that systemd brings up at boot time.

To activate a service we construct a unit file and arrange for it to be placed in one of the standard directories where systemd looks for unit files. Once we have created a unit file and placed it in the proper location, we can ask systemd to start it by issuing the command

sudo systemctl start fwd

systemctl is the terminal application that provides a command interface to systemd.

If we want our application to start automatically when the computer boots, we have to tell systemd to enable its unit. We do this via the command

sudo systemctl enable fwd

More information about the systemctl application and the things you can do with it are available online in this article. You can read more about the options available in unit files in this article.

Becoming a daemon

Most server applications run on a system as daemons, which are processes designed for a long time in the background. The book The Linux Programming Interface by Michael Kerrisk gives an overview of the process that most daemons use to set themselves up properly. I summarize these steps below.

To become a daemon, a program typically performs the following steps at startup:

1. Perform a fork(), after which the parent exits and the child continues. (As a consequence, the daemon becomes a child of the init process.) This step is done for two reasons:
1. Assuming the daemon was started from the command line, the parent’s termination is noticed by the shell, which then displays another shell prompt and leaves the child to continue in the background.
2. The child process is guaranteed not to be a process group leader, since it inherited its process group ID from its parent and obtained its own unique process ID, which differs from the inher ited process group ID. This is required in order to be able to successfully perform the next step.
2. The child process calls setsid() to start a new session and free itself of any association with a controlling terminal. If the daemon never opens any terminal devices thereafter, then we don’t need to worry about the daemon reacquiring a controlling terminal. If the daemon might later open a terminal device, then we must take steps to ensure that the device does not become the controlling terminal. To do this, the applications performs a second fork() after the setsid() call, and again has the parent exit and the (grand)child continue. This ensures that the child is not the session leader, and thus, according to the System V conventions for the acquisition of a controlling terminal (which Linux follows), the process can never reacquire a controlling terminal.
3. After performing the final fork, the child process will save the process id of the grandchild process in a file in a known location. This makes it easier to shut down the grandchild process.
4. Clear the process umask by calling umask() to ensure that when the daemon creates files and directories they have the requested permissions.
5. Change the process’s current working directory by calling chdir(), typically to the root directory (/). This is necessary because a daemon usually runs until system shutdown; if the daemon’s current working directory is on a file system other than the one containing /, then that file system can’t be unmounted. Alternatively, the daemon can change its working directory to a location where it does its job or a location defined in its configuration file, as long as we know that the file system containing this directory never needs to be unmounted. For example, cron places itself in /var/spool/cron.
6. Close all open file descriptors that the daemon has inherited from its parent. (A daemon may need to keep certain inherited file descriptors open, so this step is optional, or open to variation.) This is done for a variety of reasons. Since the daemon has lost its controlling terminal and is running in the background, it makes no sense for the daemon to keep file descriptors 0, 1, and 2 open if these refer to the terminal. Furthermore, we can’t unmount any file systems on which the long-lived daemon holds files open. And, as usual, we should close unused open file descriptors because file descriptors are a finite resource.
7. After having closed file descriptors 0, 1, and 2, a daemon normally opens /dev/null and uses dup2() (or similar) to make all those descriptors refer to this device. This is done for two reasons:
1. It ensures that if the daemon calls library functions that perform I/O on these descriptors, those functions won’t unexpectedly fail.
2. It prevents the possibility that the daemon later opens a file using descriptor 1 or 2, which is then written to - and thus corrupted - by a library function that expects to treat these descriptors as standard output and standard error.