Manage Kestrel process with systemd

Kestrel is a lightweight standalone process. In order to host it on Linux, the recommended approach is to install it as a service. Systemd is a group of tools providing functionalities to manage processes on Ubuntu.
Today we will see how we can manage an ASP NET Core application together with its Kestrel process using systemd tools. This post will be composed of three parts:

1. Introduction
2. Managing process with systemctl
3. Debugging using journalctl

1. Introduction

ASP NET Core application runs on top of Kestrel which is a lightweight standalone webserver.
To be able to start to interact with our Ubuntu server we need to first establish an ssh connection. If you aren’t familiar with ssh, you can refer to my previous blog post where I provide explanations on how to setup ssh.

Once we are on the server, we can get our libraries on the server and run:

/usr/share/dotnet/dotnet myapp.dll

This will run the app from the current session. When we exit the session, the app is killed.
Hence what we need is a way to manage the Kestrel process in the background.

In Ubuntu 16.04, this is achieved using systemd functionalities.

Systemd is a set of functionalities allowing us to manage a process by providing start, stop, restart and logs functionalities.
There is much more to systemd but for this post we will only be talking about systemctl and journalctl.

2. Managing process with systemctl

systemctl allows us to control the lifecycle of the process.

A process of work in systemd term is called a unit. To configure a service unit, we need to create a myapp.service file under /etc/systemd/system.

[Unit]
Description=My application description

[Service]
WorkingDirectory=/usr/share/myapp
ExecStart=/usr/bin/dotnet /usr/share/myapp/myapp.dll
SyslogIdentifier=myapp
User=www-data

[Install]
WantedBy=multi-user.target

ExecStart specifies the command to execute at the start of the process.
ExecStartPre and ExecStartPost can be used to execute commands after or before the execution of the main command. Multiple occurances can be setup to run multiple commands.

Once we have created the unit file, we can enable it to be automatically started on boot.

systemctl enable myapp

This will create symlinks from what was specified in [Install] section under WantedBy.
In our unit we specified multi-user.target which is where most daemon run. Targets are logical group of units which can then be ordered for boot.

If we need to modify the unit once it has already been enabled, we will need to reload the daemon using the following command:

systemctl daemon-reload

This will create symlinks that notify systemd to start the service on boot. If we need the service to run immediately we can also start it:

systemctl start myapp

If we prefix our apps with a common prefix, we can check their status with list-units

systemctl list-units myapp-* --all

We can also check the state enabled/disabled:

systemctl list-unit-files myapp-*

Or simply check the status with the latest logs:

systemctl status myapp-*

Each of this commands can be targeted to a single unit too.

Removing a service

To remove a service we can reproduce the steps in reverse.

Stop and disable the service:

systemd stop myapp
systemd disable myapp

Disabling will remove the symlink created when we enabled it.

Then remove your service unit file and associated configurations if any.

rm /etc/systemd/system/myapp.service

Reload systemd and reset the fail status to remove trace of your unwanted unit:

systemctl daemon-reload
systemctl reset-failed

3. Debugging using journalctl

Once the application is running, the standard output logs are collected in journald.

To see a particular unit log, we can use journalctl:

journalctl -fxeu myapp

-f is used to follow the changes of the file.
-e brings to the last page of logs.
-u specifies the unit to look at.
-x is used to specify extra log messages when available.
--since and --until can be used to view specific date and time range. When dates aren’t specified it is assumed to be today and when time is not specified it is assumed to be 00:00. For example --since 14:00 --until 15:00 will show today’s logs from 2pm till 3pm.

The default time being UTC, it can be helpful to switch to local time.

To find your current timezone, we can use:

timedatectl status

To find your desired timezone, we can use:

timedatectl list-timezones

Then update the default timezone:

timedatectl set-timezone Asia/Singapore

From now the date and time of journald logs will be in SGT.

Lastly because journald logs are saved in a binary format, it is possible to display them under different formats. For example, we can display them in a json format:

journalctl -x -e -u myapp -o json | json_pp

This will display the logs in json prettified format.

Conclusion

Today we saw how to manage processes with Systemd taking as example Kestrel process which is the webserver used to host ASP.NET Core application. We also saw how we could access logs and how to filter them for debugging. Hope you liked this post, see you next time!