Service management with systemd

This is tutorial on how to manage a program or script inside Linux operating system using systemd service. This is an alternative for developer or operational team who still use third party process manager (like monit) or terminal multiplexer (screen/tmux) to run the program in the background on any Linux server that use systemd.

What is systemd?

In short, systemd is the init program that will be executed by kernel, it will have process ID (pid) 1, the parent of all processes. See the output of pstree -p for an example.

Any programs or scripts that need to be managed by systemd can register them self using systemd.service(5), which is a file that have extension ".service", ".timer", or ".path". The location of the service files for global (system) is in /etc/systemd/system/.

Use case 1: starting program at boot

Lets say that we have a Go program that run indefinitely and we want to run it when the system started or rebooted,

package main

import (
    "fmt"
    "log"
    "time"
)

func main() {
    version := "v0.1.0"
    x := 0
    for {
        fmt.Printf("stdout: %s: %d\n", version, x)
        log.Printf("stderr: %s: %d\n", version, x)
        x++
        time.Sleep(3 * time.Second)
    }
}

We can build this program and run it,

$ go build myservice.go
$ ./myservice
stdout: v0.1.0: 0
2020/11/11 17:22:05 stderr: v0.1.0: 0
stdout: v0.1.0: 1
2020/11/11 17:22:08 stderr: v0.1.0: 1
stdout: v0.1.0: 2
2020/11/11 17:22:11 stderr: v0.1.0: 2
^C

To make the program run in any linux system with systemd, we need to copy it to directory that are writeable by user. Of course, we can put it in /usr/bin or /usr/local/bin but I prefer if we put it somewhere that is not default for system packages.

For example, we create directory /data/app/bin/ and copy the myservice into it. Then, we create a service file inside /etc/systemd/system/, lets name the file myservice.service,

[Unit]
Description=My service
After=syslog.target network.target

[Service]
ExecStart=/data/app/bin/myservice
#User=<your username>

[Install]
WantedBy=multi-user.target

That is the common and basic systemd service file for common programs.

A brief explanation,

After option means the service will run after system logs and network is up.
ExecStart is the absolute path to the program that will be executed
User is the user that will run the program, by default it will run as root.

To make the program start at boot run,

$ sudo systemctl enable myservice.service
Created symlink /etc/systemd/system/multi-user.target.wants/myservice.service → /etc/systemd/system/myservice.service.

Note that the ".service" suffix is optional, you can just write myservice instead.

To make the program start now execute,

$ sudo systemctl start myservice

We can check if the program running or not by using status command,

$ sudo systemctl status myservice

or by viewing the service logs using journalctl,

$ journalctl --follow --unit myservice

Use case 2: auto restart the service when program crash

We know that non of our program is free from bugs, right? So, in case our program crash and terminate we want it to start again from beginning. With systemd service we can do this by setting "Restart=" and "RestartSec=" options to "[Service]". Here is the example of service that restart the program 5 seconds after killed or crashed,

[Unit]
Description=My service
After=syslog.target network.target

[Service]
ExecStart=/data/app/bin/myservice
Restart=on-failure
RestartSec=5s

[Install]
WantedBy=multi-user.target

We can try this by killing our myservice,

$ kill -9 `pidof myservice`
$ journalctl -f -u myservice.service
myservice[102962]: 2020/11/25 16:54:00 stderr: v0.2.0: 1
myservice[102962]: stdout: v0.2.0: 2
myservice[102962]: 2020/11/25 16:54:03 stderr: v0.2.0: 2
myservice[102962]: stdout: v0.2.0: 3
myservice[102962]: 2020/11/25 16:54:06 stderr: v0.2.0: 3
myservice[102967]: stdout: v0.2.0: 0
myservice[102967]: 2020/11/25 16:54:11 stderr: v0.2.0: 0
myservice[102967]: stdout: v0.2.0: 1
myservice[102967]: 2020/11/25 16:54:14 stderr: v0.2.0: 1

As you can see, the program restarted automatically after being killed.

There are many options for "Restart=" options, please consult your systemd.service(5) man page for more information.

Use case 3: auto restart the service when program changes

Lets say we have updated our program to add new features or fixes some bug, or let just changes the version variable from previous Go code example from v0.1.0 to v0.2.0 and rebuild the binary.

package main

import (
    "fmt"
    "log"
    "time"
)

func main() {
    version := "v0.2.0"
    x := 0
    for {
        fmt.Printf("stdout: %s: %d\n", version, x)
        log.Printf("stderr: %s: %d\n", version, x)
        x++
        time.Sleep(3 * time.Second)
    }
}

To deploy the new binary we copy them to remote server, SSH into it, and restart the service. What if we can just copy the binary and let the systemd restart it automatically? Yes, systemd can do that.

First, we create a middle service that will restart any service by using parameter, we name it systemctl-restart@.service and put it also in /etc/systemd/system/,

[Unit]
Description=systemctl-restart@%i

[Service]
ExecStart=/bin/systemctl restart %i

The %i is any string between systemctl-restart@ and .service.

Second, we create a systemd unit systemd.path(5) that will watch the program file and systemctl-restart@.service when its changed,

[Unit]
Description=Watch /data/app/bin/myservice

[Path]
PathChanged=/data/app/bin/myservice
Unit=systemctl-restart@%p.service

[Install]
WantedBy=multi-user.target

The unit file name MUST have the same name with our previous service file but with ".path" suffix. Put it in /etc/systemd/system/myservice.path. The %p parameter is the current unit file name, in this case myservice. So, when systemd detect a file changes on /data/app/bin/myservice it will activate systemctl-restart@myservice.service.

Enable and start the unit path,

$ sudo systemctl enable myservice.path
Created symlink /etc/systemd/system/multi-user.target.wants/myservice.path → /etc/systemd/system/myservice.path.
$ sudo systemctl start myservice.path

Lets try!

Oh, by the way, we can’t use scp to copy the program, we must use rsync, because scp replace the file directly, while rsync create a temporary file first and then move it to the destination.

Lets start our service first and watch the log,

$ sudo systemctl restart myservice
$ journalctl -f -u myservice
-- Logs begin at Thu 2020-10-22 05:15:25 UTC. --
Nov 11 11:51:52 myserver myservice[57614]: stdout: v0.1.0: 0
Nov 11 11:51:52 myserver myservice[57614]: 2020/11/11 11:51:52 stderr: v0.1.0: 0
Nov 11 11:51:55 myserver myservice[57614]: stdout: v0.1.0: 1
Nov 11 11:51:55 myserver myservice[57614]: 2020/11/11 11:51:55 stderr: v0.1.0: 1
Nov 11 11:51:58 myserver myservice[57614]: stdout: v0.1.0: 2
Nov 11 11:51:58 myserver myservice[57614]: 2020/11/11 11:51:58 stderr: v0.1.0: 2

Update the version in the code, rebuild it, and sync it to server,

$ GOOS=linux GOARCH=amd64 go build myservice.go
$ rsync myservice myserver:/data/app/bin/

Switch back to terminal that tailing our service logs you will see that the counter is started back to zero,

Nov 11 11:53:50 myserver myservice[57614]: 2020/11/11 11:53:50 stderr: v0.1.0: 39
Nov 11 11:53:53 myserver myservice[57614]: stdout: v0.1.0: 40
Nov 11 11:53:53 myserver myservice[57614]: 2020/11/11 11:53:53 stderr: v0.1.0: 40
Nov 11 11:53:54 myserver myservice[57667]: stdout: v0.2.0: 0
Nov 11 11:53:54 myserver myservice[57667]: 2020/11/11 11:53:54 stderr: v0.2.0: 0
Nov 11 11:53:57 myserver myservice[57667]: stdout: v0.2.0: 1
Nov 11 11:53:57 myserver myservice[57667]: 2020/11/11 11:53:57 stderr: v0.2.0: 1
Nov 11 11:54:00 myserver myservice[57667]: stdout: v0.2.0: 2
Nov 11 11:54:00 myserver myservice[57667]: 2020/11/11 11:54:00 stderr: v0.2.0: 2
Nov 11 11:54:03 myserver myservice[57667]: stdout: v0.2.0: 3

Use case 4: storing program logs into file

By default, all output of program is collected by systemd-journald(8) and you can be view it using journalctl command.

There are two methods to store the program output or error to file.

The first method is by wrapping the program with shell script. The reason for this is systemd does not support shell redirection and pipe in ExecStart option.

We can not do this,

ExecStart = /data/app/bin/myservice 2> file

but we can do this,

...
ExecStart = /data/app/bin/myservice.sh
...

where /data/app/bin/myservice.sh is a normal shell script,

#/bin/sh

/data/app/bin/myservice \
    2>> /data/app/logs/myservice-error.log \
    1>> /data/app/logs/myservice.log

The second method is by upgrading systemd to version 236 or latest with Unit options StandardOutput and StandardError set to file location, for example,

[Unit]
Description=My service
After=syslog.target network.target

[Service]
ExecStart=/data/app/bin/myservice
StandardOutput=append:/data/app/logs/myservice.log
StandardError=append:/data/app/logs/myservice-error.log

[Install]
WantedBy=multi-user.target

The append: prefix will append the log to file if its already exist.

Use case 5: start program at specific time or interval

You may think that, "well, I can do this with cronjobs". Yes, you can but systemd unit timer is more powerful than that.

Imagine that we have 30 or more cron jobs. If one of the job is failed we can not know which one, unless we pipe the job command to file, like most good sysadmin will do. We also can not inspect the current jobs status, except using another cron frontend. We also can not stop specific jobs without re-setting the whole jobs.

With systemd timer, not only we did not need to pipe it to file (since the service can have "StandardOutput=" and/or "StandardError="), we can inspect the schedule of all jobs, and stop and start specific job.

To give you an example we will create simple shell script and systemd timer that will execute it every three seconds,

#!/bin/sh

echo "Hello world!"

Put the script in file /data/app/bin/myservice.sh, and create systemd timer myservice.timer in /etc/systemd/system/,

[Unit]
Description=Run myservice every 3 seconds

[Timer]
OnCalendar=*:*:0/3
AccuracySec=1us

[Install]
WantedBy=timers.target

Note	By default `AccuracySec=` option is set to 1 minute, which means on normal scenario you did not need to set the `AccuracySec=1us` option.

The systemd know which service to be run based on the file name. So, in our case since the unit timer name is "myservice.timer" then systemd will trigger the "myservice.service". We did not need to enable or start the myservice.service, only the myservice.timer.

First, we need to stop and disable our previous myservice.service,

$ sudo systemctl stop myservice.service
$ sudo systemctl disable myservice.service

and change the ExecStart= to point to /data/app/bin/myservice.sh, and then enable and start the timer,

$ sudo systemctl enable myservice.timer
$ sudo systemctl start myservice.timer

Lets inspect the output of myservice.sh immediately,

$ journalctl -f -u myservice.service
Nov 25 17:40:18 local myservice.sh[103782]: Hello world!
Nov 25 17:40:21 local myservice.sh[103784]: Hello world!
Nov 25 17:40:24 local myservice.sh[103785]: Hello world!
^C

Now, lets see the status of all timers in our systems,

$ sudo systemctl list-timers
NEXT                        LEFT          LAST                        PASSED        UNIT                         ACTIVATES
Wed 2020-11-25 17:46:51 UTC 1s left       Wed 2020-11-25 17:46:48 UTC 1s ago        myservice.timer              myservice.service
...

We can see that our timer will run again at Wed 2020-11-25 17:46:51 UTC or 1 second from now, and has been successfully running at 17:46:48 or 1 second ago.

Summary

Systemd as the system and service manager have powerful features that user can use, especially for deploying application, either binary or script. With systemd one can auto start the program after boot, auto restart when the program crash or when the file changes, and storing the program standard output and/or error into files.