||<>|| = Introduction = This document compares [[http://upstart.ubuntu.com/|Upstart]] and [[http://freedesktop.org/wiki/Software/systemd/|systemd]] with a view to aiding in the transition to the latter. = Support status = First, it is important to note that systemd is only fully supported in Ubuntu 15.04 and later releases. While systemd is available in prior releases through the Ubuntu repositories, there is a deemphasis of support for these releases as noted [[https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1438967/comments/2|here]]. Hence, it is advised to use the default upstart on prior releases. = System Init Daemon = This has changed as part of the Ubuntu 15.04 devel cycle. Ubuntu 15.04 (using Systemd by default): * Systemd runs with PID 1 as `/sbin/init`. * Upstart runs with PID 1 as `/sbin/upstart`. Prior versions (using Upstart by default): * Upstart runs with PID 1 as `/sbin/init`. * Systemd runs with PID 1 as `/lib/systemd/systemd`. = Switching init systems = If you are running Ubuntu vivid (15.04), you can easily switch between upstart and systemd at will since both packages are installed at present. As of March 9 2015, vivid was changed to use systemd by default, before that upstart was the default. == Switch to upstart for a single boot == In grub, select "Advanced options for Ubuntu", where you will find an "Ubuntu, with Linux ... (upstart)" entry. This will boot with `init=/sbin/upstart`. If you have `upstart-sysv` installed and thus boot with upstart by default, there will be an "Ubuntu, with Linux ... (systemd)" entry, which will boot with `init=/lib/systemd/systemd`. == Permanent switch back to upstart == Install the `upstart-sysv` package, which will remove `ubuntu-standard` and `systemd-sysv` (but should not remove anything else -- if it does, yell!), and run `sudo update-initramfs -u`. After that, grub's "Advanced options" menu will have a corresponding "Ubuntu, with Linux ... (systemd)" entry where you can do an one-time boot with systemd. If you want to switch back to systemd, install the `systemd-sysv` and `ubuntu-standard` packages. = High-level startup concept = Upstart's model for starting processes ('''jobs''') is "greedy event-based", i. e. all available jobs whose startup events happen are started as early as possible. During boot, upstart synthesizes some initial events like `startup` or `rcS` as the "tree root", the early services start on those, and later services start when the former are running. A new job merely needs to install its configuration file into `/etc/init/` to become active. systemd's model for starting processes ('''units''') is "lazy dependency-based", i. e. a unit will only start if and when some other starting unit depends on it. During boot, systemd starts a "root unit" (`default.target`, can be overridden in grub), which then transitively expands and starts its dependencies. A new unit needs to add itself as a dependency of a unit of the boot sequence (commonly `multi-user.target`) in order to become active. = Job vs. unit keywords = This maps the keywords that can occur in an upstart job to the corresponding ones in a systemd unit. Keywords which don't have a direct equivalent are marked with "-". ||<#F1F1ED>'''Upstart stanza'''||<#F1F1ED>'''systemd unit file directive'''||<#F1F1ED>'''systemd unit file section'''||<#F1F1ED>'''Notes'''|| || `apparmor load` || `AppArmorProfile` || || Available in systemd version 210 and later || || `apparmor switch` || - || || || || `author` || - || || || || `chdir` || `WorkingDirectory` || Service || || || `chroot` || `RootDirectory` || || || || `console output` || `StandardOutput=tty`, `StandardError=tty` || || || || `console owner` || `StandardOutput=tty`, `StandardError=tty` || || No real equivalent? || || `console none` || `StandardOutput=null`, `StandardError=null` || || || || `description` || `Description` || Unit || || || `env` || `Environment`, `EnvironmentFile` || Service || || || `exec` || `ExecStart` || Service || || || `expect fork` || `Type=forking` || Unit || || || `expect daemon` || `Type=forking` || Unit || || || `expect stop` || `Type=notify` || Unit || Similar, not equivalent. Requires daemon to link to `libsystemd-daemon` and call [[http://www.freedesktop.org/software/systemd/man/sd_notify.html|sd_notify()]]. || || `instance` || Use "`%I`" and "`%i`" in `ExecStart`, ''etc'' to specify an instance || || See `/lib/systemd/system/getty@.service` for an example || || `kill signal` || `KillSignal` || || || || `kill timeout` || `TimeoutStopSec` || || || || `limit as` || `LimitAS` || || || || `limit core` || `LimitCORE` || || || || `limit cpu` || `LimitCPU` || || || || `limit data` || `LimitDATA` || || || || `limit fsize` || `LimitFSIZE` || || || || `limit memlock` || `LimitMEMLOCK` || || || || `limit msgqueue` || `LimitMSGQUEUE` || || || || `limit nice` || `LimitNICE` || || || || `limit nofile` || `LimitNOFILE` || || || || `limit nproc` || `LimitNPROC` || || || || `limit rss` || `LimitRSS` || || || || `limit rtprio` || `LimitRTPRIO` || || || || `limit sigpending` || `LimitSIGPENDING` || || || || `limit stack` || `LimitSTACK` || || || || `manual` || No directive(?) - use `systemctl disable foo.service` || || || || `nice` || `Nice` || Unit || || || `normal exit` || `SuccessExitStatus` || || || || `oom score` || `OOMScoreAdjust` || || || || `post-start exec/script` || `ExecStartPost` || Service || || || `post-stop exec/script` || `ExecStopPost` || Service || || || `pre-start exec/script` || `ExecStartPre` || Service || || || `pre-stop` || - || || || || `reload signal` || `ExecReload=/bin/kill -SIGFOO $MAINPID` || Service || || || `respawn` || `Restart=on-failure` || Service || || || `respawn limit` || `RestartSec` || || || || `script`/`end script` || See ''shell scripts'' below || || || || `setgid` || `Group` || Service || || || `setuid` || `User` || Service || || || `start on` || `Wants`, `Requires`, `Before`, `After` || Unit || || || `stop on` || `Conflicts`, `BindsTo` (but not commonly used) || Unit || || || `task` || `Type=oneshot` || Unit || || || `umask` || `UMask` || Unit || || || `usage` || `Documentation` || Unit || no direct equivalent || || `version` || - || || || === Shell scripts === systemd does not provide special support for shell scripts (by design). For short shell commands you can use something like {{{ ExecStart=/bin/sh -ec 'echo hello' }}} Longer scripts are usually program logic and should not be directly in a conffile and duplicated between upstart and systemd; factor it out in a proper script in e. g. `/usr/share/myapp/` and call it from both the upstart job and the systemd unit. === Automatic starting === As described above, services which want to start during boot (i. e. are not activated through sockets, D-BUS, or similar) need to become a dependency of an existing boot target. Those need an `[Install]` section with a `WantedBy=` that specifies the unit which that new service wants to become a dependency of (see `man systemd.unit`). Very commonly this is `multi-user.target`, which is roughly equivalent to `start on runlevel [2345]` in upstart; see `man systemd.special` for other common targets. == Invalid configurations == * upstart Refuses to start a job if it contains invalid syntax. * systemd Ignores invalid directives and starts the service. Note that you must test your configuration carefully since a typo will not be detected! Use systemd-analyze verify file to get warnings on typos and badly formatted options. = Override Files = * upstart Upstart allows part or all of a Job Configuration ("`/etc/init/$job.conf`") file to be overridden. To create an override file: * Create file "`/etc/init/$job.override`" containing one or more stanzas which take priority over their counterparts from the original "`$job.conf`" file. * systemd Systemd allows a similar facility via "drop-ins": a drop-in allows a directive to be modified without changing the original unit file. To create a drop-in: 1. Create a subdirectory below either "`/etc/systemd/system/`" or "`/lib/systemd/system/`" called "`${unit}.d/`". 1. Create files called `.conf` in the "`${unit}.d/`" directory containing the directives that you wish to override. = Behavioral differences (aka Gotchas) = * systemd will refuse to run a binary (via `ExecStart`, `ExecStartPre`, etc) unless the full path to the binary is specified: '''BAD''' (systemd will '''not''' search for `sleep` in `$PATH`): {{{ ExecStart=sleep 20 }}} '''GOOD''' (absolute path specified): {{{ ExecStart=/bin/sleep 20 }}} = Commands = Note that these are commands for '''interactive human''' usage. Package maintainer scripts, `ifupdown` hooks and similar must always use the init system agnostic abstractions like `invoke-rc.d`. ||<#F1F1ED>'''Operation'''||<#F1F1ED>'''Upstart Command'''||<#F1F1ED>'''Systemd equivalent''' ||<#F1F1ED>'''Notes''' || || Start service || `start $job` || `systemctl start $unit` || || || Stop service || `stop $job` || `systemctl stop $unit` || || || Restart service || `restart $job` || `systemctl restart $unit` || || || See status of services || `initctl list` || `systemctl status` || || || Check configuration is valid || `init-checkconf /tmp/foo.conf` || systemd-analyze verify || || || Show job environment || `initctl list-env` || `systemctl show-environment` || || || Set job environment variable || `initctl set-env foo=bar` || `systemctl set-environment foo=bar` || || || Remove job environment variable || `initctl unset-env foo` || `systemctl unset-environment foo` || || || View job log || `cat /var/log/upstart/$job.log` || `sudo journalctl -u $unit` || || || `tail -f` job log || `tail -f /var/log/upstart/$job.log` || `sudo journalctl -u $unit -f` || || || Show relationship between services || `initctl2dot` || `systemctl list-dependencies --all` || Shows `pstree`-style output. || = Example Services = == Example Upstart Service == `/etc/init/foo.conf`: {{{#!html 
description "Job that runs the foo daemon"

# start in normal runlevels when disks are mounted and networking is available
start on runlevel [2345]

# stop on shutdown/halt, single-user mode and reboot
stop on runlevel [016]

env statedir=/var/cache/foo

# create a directory needed by the daemon
pre-start exec mkdir -p "$statedir"

exec /usr/bin/foo-daemon --arg1 "hello world" --statedir "$statedir"
}}} == Example Systemd service == `/lib/systemd/system/foo.service`: {{{#!html 
[Unit]
Description=Job that runs the foo daemon
Documentation=man:foo(1)

[Service]
Type=forking
Environment=statedir=/var/cache/foo
ExecStartPre=/usr/bin/mkdir -p ${statedir}
ExecStart=/usr/bin/foo-daemon --arg1 "hello world" --statedir ${statedir}

[Install]
WantedBy=multi-user.target
}}} = Outstanding Work = If you'd like to help out with the migration, take a look at ... * Outstanding packages to convert: * http://people.canonical.com/~jhunt/systemd/packages-to-convert/ Note that the priority are the packages in "{{{main}}}". * blueprint: https://blueprints.launchpad.net/ubuntu/+spec/core-1411-systemd-migration ... and then come and chat to us on {{{#ubuntu-devel}}}. = Common Idioms = == Service that specifies shell meta-characters == * upstart Upstart automatically detects if a job requires a shell to expand meta characters. For example, the following job will automatically be run via "`/bin/sh -e`": {{{#!html 
   exec mydaemon --date $(date)
}}} * systemd systemd does no such detection so the equivalent unit file would need to be: {{{#!html 
   [Service]
   ExecStart=/bin/sh -ec "exec /usr/bin/mydaemon --date $(date)"
}}} == Service that does not fork (runs in foreground) == * upstart job {{{#!html 
   description "blah, blah, blah"
   exec sleep 999
}}} * systemd unit {{{#!html 
   [Unit]
   Description=blah, blah, blah
   
   [Service]
   Type=simple
   # (NOTE: "Type=simple" is the default)
   ExecStart=/usr/bin/sleep 999
}}} == Do not run service if no daemon configuration file created == * upstart job {{{#!html 
   # defines PORT variable
   env config=/etc/default/foo

   pre-start script
   [ -e "$config" ] || { stop; exit 0; }
   end script

   exec mydaemon --port="$PORT"
}}} * systemd unit {{{#!html 
   [Service]
   Type=forking
   EnvironmentFile=/etc/default/foo
   ExecStart=/usr/bin/mydaemon --port=$PORT
}}} Note: EnvironmentFile=/etc/default/foo is making /etc/default/foo mandatory. If the environment file (and so variables) are optional, you can use: EnvironmentFile=-/etc/default/foo (notice the - in front of the path). == /etc/default files which enable/disable jobs == `enable=1|0` type settings in `/etc/default` files should generally be avoided. The canonical way to enable/disable a service in an init system agnostic way is `update-rc.d enable|disable`, which will translate to init system specific actions such as adding/removing symlinks (SysV and systemd) or creating/removing job override files (upstart). For systemd in particular, admins also often call `systemctl enable|disable ` directly. Thus these settings are redundant in `/etc/default`. There is no clean way to evaluate these in a systemd unit. You can check them in `ExecStartPre=`, but that would mean that the unit will be in "failed" state if the service gets disabled that way, and so, is not desirable. For these reasons (confusing/duplication/cannot be modelled in systemd), these settings should be removed. This was done in [[https://launchpad.net/ubuntu/+source/whoopsie/0.2.42|whoopsie 0.2.42]], you can check its [[http://launchpadlibrarian.net/189472727/whoopsie_0.2.41_0.2.42.diff.gz|diff]] for a transition which respects the old default setting and removes it on upgrade. == Start a service when a file is created == * upstart job {{{#!html 
   start on file FILE=/var/crash/*.crash EVENT=create
   exec crash-handler-daemon
}}} * systemd Two files are required: * `/lib/systemd/system/foo.service`: {{{#!html 
   [Service]
   Type=forking
   ExecStart=/usr/bin/crash-handler-daemon
}}} * `/lib/systemd/system/foo.path`: {{{#!html 
   [Path]
   PathExistsGlob=/var/crash/*.crash
   Unit=foo.service
   # (NOTE: the default is "Unit=foo.service" if this is called "foo.path")
}}} == Start a service when a D-Bus name is acquired == * upstart job {{{#!html 
   start on dbus SIGNAL=NameAcquired INTERFACE=... OBJPATH=... SENDER=... DESTINATION=...
   exec mydaemon
}}} * systemd * `/lib/systemd/system/foo.service`: {{{#!html 
   [Unit]
   Description=Service that acquires a D-Bus name
   
   [Service]
   Type=dbus
   # (NOTE: systemd will consider this service ready when it claims the 'foo.bar.baz' name on dbus)
   BusName=foo.bar.baz
   ExecStart=/usr/bin/mydaemon
}}} * `/lib/systemd/system/bar.service`: {{{#!html 
   [Unit]
   Description=Service that needs the foo service
   Requires=foo.service
   After=foo.service
   
   [Service]
   ExecStart=/usr/bin/mydaemon2
}}} = Well-known Sequence Points = * Upstart provides a set of "well-known" events that jobs can make use of (such as "`runlevel`" and "`started`"). These are summarised in the `upstart-events(7)` manual page. * Systemd has a similar concept in the form of "targets". See the `systemd.special(7)` manual page for details and `bootup(7)` for the boot sequence. * For the particular case of upstart's `static-network-up`: Our ifupdown [[https://launchpad.net/ubuntu/+source/ifupdown/0.7.48.1ubuntu6|integrates that into network-online.target]] (see man systemd.special(7)). Thus, translate `start on static-network-up` to `Requires/After=network-online.target`. = Environment Differences = Both Upstart and systemd provide a very limited environment to the services they run. The environment can be modified by specifying additional stanzas (for Upstart) or directives (for systemd). However, there are some subtle differences between the ''default'' service environment provided by both init systems. The table below shows the major differences: ||<#F1F1ED>'''Description'''||<#F1F1ED>'''Upstart'''||<#F1F1ED>'''systemd'''|| || Environment variables set || `PATH`, `TERM` || `PATH`, all from `/etc/default/locale` || || Standard out and Standard error || terminal (pty pseudo-terminal) || pipes (to journal) || Note that this information is subject to change; to determine a precise difference on your system: 1. Install the [[http://packages.ubuntu.com/search?keywords=procenv|procenv]] tool: {{{#!html 
 $ sudo apt-get -y install procenv
}}} 1. Create an Upstart job to run `procenv`: {{{#!html 
 $ cat <<EOT | sudo tee /etc/init/procenv.conf
 description "Display Upstart environment"
 exec procenv --file=/tmp/procenv-upstart.log
 EOT
}}} 1. Run `procenv` under upstart: {{{#!html 
 $ sudo start procenv
}}} 1. Create a systemd unit file to run `procenv`: {{{#!html 
 $ cat <<EOT | sudo tee /lib/systemd/system/procenv.service
 [Unit]
 Description=Display systemd environment

 [Service]
 Type=oneshot
 ExecStart=/usr/bin/procenv --file=/tmp/procenv-systemd.log
 EOT
}}} 1. Run `procenv` under systemd: {{{#!html 
 $ sudo systemctl start procenv
}}} 1. Compare the environments: {{{#!html 
 $ diff /tmp/procenv-upstart.log /tmp/procenv-systemd.log
}}} = Common Problems = == How to identify which init system you are currently booting with == At the time of writing, it is possible to boot an Ubuntu system with ''either'' Upstart or systemd since both are installed. To determine which init daemon you are currently booting with, run: {{{ $ ps -p1 | grep systemd && echo systemd || echo upstart }}} == Why don't some commands (like grep) work in /lib/systemd/system ? == If you hit this problem... {{{ $ cd /lib/systemd/system $ grep foo * grep: invalid option -- '.' Usage: grep [OPTION]... PATTERN [FILE]... Try 'grep --help' for more information. }}} ... you need to add "{{{--}}}" to the {{{grep(1)}}} call: {{{ $ cd /lib/systemd/system $ grep foo -- * }}} This is required since systemd provides a file called "{{{-.slice}}}". After the shell has expanded the asterisk ("{{{*}}}"), that file gets passed to the command in question ({{{grep}}}, {{{cat}}}, ''etc'') and that command will then attempt to interpret "{{{-.slice}}}" as a command-line option rather than a filename. Alternative work-around: {{{ $ cd /lib/systemd/system $ POSIXLY_CORRECT=1 grep foo * }}} To view the pesky file: {{{ $ cd /lib/systemd/system $ cat -- -.slice }}} Note that this issue only occurs when your current working directory is the directory containing "{{{-.slice}}}". The simplest work-around is to make the {{{grep}}} or {{{cat}}} call from a different directory: {{{ $ grep foo /lib/systemd/system/* $ cat /lib/systemd/system/-.slice }}} = Debugging = Since this implies the service works under Upstart but is problematic under systemd, details of both systems are provided to allow for some comparison. == Boot Time == === Common Setup === Remove the following from the kernel command-line via the grub menu: * "`quiet`" * "`splash`" === Upstart === * Add "`--debug` to the kernel command-line via the grub menu. * Optionally add "`console=ttyS0` to the kernel command-line via the grub menu if you have a serial console. === systemd === * Add "`systemd.log_level=debug ` to the kernel command-line via the grub menu. * Optionally add one of the following too: * "`systemd.log_target=kmsg`" * "`systemd.log_target=console`" ==== Starting a rescue shell ==== * Run: {{{ $ sudo systemctl enable debug-shell.service }}} * Reboot. * If the system fails to boot, you can now switch to tty9 (CTRL+ALT+F9) for a getty console login. == From a running system == === Upstart === To switch to debug mode for the system init (PID 1): {{{ $ sudo initctl log-priority debug }}} To switch to debug mode for a session init (init PID != 1): {{{ $ initctl log-priority debug $ tail -f ~/.xsession-errors }}} = Debian Packaging = Packages installing systemd services should build-depend on `dh-systemd` and either call `dh --with systemd` (if they use `dh`) or call `dh_systemd_enable` and `dh_systemd_start` before/after `dh_installinit` respectively. Files under debian called *.service will be installed analogously to *.upstart files. See dh_systemd_enable(1p) for how to customize the installation. = Further Information = * `man systemd` -- "CONCEPTS" has an useful link collection to the per-topic pages * Ubuntu wiki [[systemd]] page (not relevant for porting) * [[http://freedesktop.org/wiki/Software/systemd/|Upstream systemd]] page with lots of documentation pointers * `man 8 init`, `man 5 init` (on an Upstart system) * systemd manpages for service writers: * http://www.freedesktop.org/software/systemd/man/systemd.unit.html * http://www.freedesktop.org/software/systemd/man/systemd.service.html * http://www.freedesktop.org/software/systemd/man/systemd.exec.html * http://www.freedesktop.org/software/systemd/man/systemd.special.html * [[Upstart]] * [[http://upstart.ubuntu.com/cookbook/|Upstart Cookbook]]