= Automated Ubuntu Server Installations on s390x with preseed =

One of the Debian installers (d-i) advanced options is the automated install mode via preseed - a combination of special boot parameters and a preseed file.

The (d-i) installer on s390x runs in netboot mode and has two stages - the initial stage (before networking has been started in the installer, usually run at the console) and the second stage that is usually run via ssh.

Automating the installation can be done in multiple ways:
 * Pass preseed keys or aliases via parmfile: e.g. debian-installer/locale=en_US domain=myhostname
 * Append complete preseed.cfg to the initrd.ubuntu
 * Preseed networking information via parmfile, and use url= parameter to fetch the rest of the config over the network

Depending on your setup, you may want to preseed some or all keys one one of the above methods, or combination of methods.

For details information on d-i preseeding please see the installation guide at [[https://help.ubuntu.com/lts/installation-guide/index.html]]

Some hints and examples are available below.

In the paramfile the format of the keys is simple space separated `key=value` pairs, e.g.:
```
priority=critical locale=en_US preseed/url=http://example.com/preseed.cfg
...
```

In the preseed files as fetched by `preseed/url`, `preseed/file`, or appended to the initrd.ubuntu the formatting is slightly more verbose - standard debconf:
```
d-i debconf/priority string critical
d-i debian-installer/locale string en_US
...
```

== Aliases ==
Because the boot prompt can become quite long, there are short forms available, called aliases, at least for some parameters, like 'locale' instead of 'debian-installer/locale' or 'url' instead of 'preseed/url'.
The boot parameters as well as their aliases form key/value pairs with an equal sign '=' as link.

||'''Alias'''||'''Normal'''||
||priority||debconf/priority||
||language||debian-installer/language||
||country||debian-installer/country||
||locale||debian-installer/locale||
||theme||debian-installer/theme||
||auto||auto-install/enable||
||classes||auto-install/classes||
||file||preseed/file||
||url||preseed/url||
||domain||netcfg/get_domain||
||hostname||netcfg/get_hostname||
||interface||netcfg/choose_interface||
||protocol||mirror/protocol||
||suite||mirror/suite||
||modules||anna/choose_modules||
||recommends||base-installer/install-recommends||
||tasks||tasksel:tasksel/first||
||desktop||tasksel:tasksel/desktop||
||dmraid||disk-detect/dmraid/enable||
||keymap||keyboard-configuration/xkb-keymap||
||preseed-md5||preseed/file/checksum||
Please find a table with all aliases [[https://help.ubuntu.com/16.04/installation-guide/s390x/apbs02.html#preseed-aliases|here]].
<<BR>>
All parameters need to be in a single line for an LPAR installation.
For an z/VM guest installation they need to be placed in the PARMFILE - more on that later.

== z Systems specific boot parameter ==
Especially for the initial stage, which is quite system dependent, there are several mainframe-specific parameters available. <<BR>>
These have the prefix: "s390-..." (don't care about the missing 'x', it also works on s390x, means 64-bit) <<BR>>
With the help of these you can overcome/answer even the initial questions.

Some example 's390-' boot parameters are: <<BR>>

 1. Network device type <<BR>>
 s390-netdevice/choose_networktype=qeth <<BR>>
 1. OSA-Express QDIO / Hipersockets device <<BR>>
 s390-netdevice/qeth/choose=0.0.0600-0.0.0601-0.0.0602 <<BR>>
 1. Layer2 or Layer3 mode <<BR>>
 s390-netdevice/qeth/layer2=true <<BR>>
 1. Relative port <<BR>>
 s390-netdevice/qeth/port=0 <<BR>>

There is not much documentation about these 's390-' parameters, the best way to look them up is therefore the full list of debconf templates available here:
 * [[https://anonscm.debian.org/cgit/d-i/|d-i Name / Description]]

Especially the following templates:
 * [[https://salsa.debian.org/installer-team/s390-netdevice/blob/master/debian/s390-netdevice.templates|QETH network devices]] <<BR>>
 * [[https://salsa.debian.org/installer-team/s390-dasd/blob/master/debian/s390-dasd.templates|DASD disk devices]] and  <<BR>>
 * [[https://salsa.debian.org/installer-team/s390-zfcp/blob/master/debian/s390-zfcp.templates|zFCP disk devices]]  <<BR>>

The following tables show the most relevant s390-specific boot parameters:

''Table: s390-netdevice parameters''
||'''Key'''||'''Value'''||'''Description'''||
||s390-netdevice/choose_networktype||selection; choices: ctc, qeth, iucv, virtio||Network device type||
||s390-netdevice/ctc/protocol||selection, choices: S/390 (0), Linux (1), OS/390 (3); default: S/390 (0)||Protocol for this connection||
||s390-netdevice/qeth/choose||selection, choices: {choices}||OSA-Express QDIO or Hipersockets device||
||s390-netdevice/qeth/port||string, default: 0||Relative Port||
||s390-netdevice/qeth/layer2||boolean, default: false||Device in layer2 mode||
||s390-netdevice/iucv/peer||string||Name of VM peer||
For further details see: [[https://anonscm.debian.org/cgit/d-i/s390-netdevice.git/plain/debian/s390-netdevice.templates|Template: s390-netdevice]]

''Table: s390-dasd''
||'''Key'''||'''Value'''||'''Description'''||
||s390-dasd/dasd||string||one or more DASD devices using comma ',' to separate multiple entries||
|| ||  ||complete dasd device number, incl. leading zeros, separated by dots '.'||
||s390-dasd/auto_format||boolean||format DASD device automatically in case it is not low-level formatted||
||s390-dasd/force_format||boolean||force DASD device format even in case it was already low-level formatted||
/* ||s390-dasd/choose_select||s390-dasd/choose_select||Available devices, select "Finish" when done|| */
/* ||s390-dasd/choose||string||Choose complete device number, including leading zeros|| */
/* ||s390-dasd/format||boolean, default: false||Format the device, in case device is already low-level formatted|| */
/* ||s390-dasd/format_required||boolean, default: true||Format the device, in case device is not low-level formatted|| */
For further details see: [[https://anonscm.debian.org/cgit/d-i/s390-dasd.git/plain/debian/s390-dasd.templates|Template: s390-dasd]]

''Table: s390-zfcp''
||'''Key'''||'''Value'''||'''Description'''||
||s390-zfcp/zfcp||string||one or more SCSI devices using comma ',' to separate multiple entries||
|| || || each entry consists of a zfcp host adapter, optionally followed by the WWPN and LUN, like:||
|| || || 0.0.1234,0.0.5678:0x2005000e11159c32:0x1234567800000000||
For further details see: [[https://anonscm.debian.org/cgit/d-i/s390-zfcp.git/plain/debian/s390-zfcp.templates|Template: s390-zfcp]]

The preseed commands of the second stage can either specified as boot parameters (they don't have the 's390-'), at least partly, or in a preseed config file, like "url=http://<your_preseed_server>/preseed.cfg". The preseed configuration file can be located on a ftp server, a http server or it can be included in the root directory of the initrd.
If there was already a ftp server prepared for an ftp-based installation, the same system can be used as preseed server, which would ensure that the preseed file can be found during boot and install time at the same system.
In case the Ubuntu servers have no direct connection to the internet, the public update server can't be directly used, hence the install and preseed system may even act a local update server.
In case of a full automated installation with dynamic network some additional resources are obviously needed, like DHCP and DNS.

Here is an example of a boot parameter set:
<<BR>>
ro locale=en_US s390-netdevice/choose_networktype=qeth s390-netdevice/qeth/choose=0.0.C000-0.0.C001-0.0.C002 s390-netdevice/qeth/layer2=true s390-netdevice/qeth/port=0 netcfg/get_ipaddress=10.245.236.12 netcfg/get_netmask=255.255.255.0 netcfg/get_gateway=10.245.236.1 netcfg/get_nameservers=10.245.236.1 netdevice/qeth/layer2=true netcfg/confirm_static=true netcfg/disable_dhcp=true netcfg/vlan_id=1234 netcfg/use_autoconfig=1 netcfg/get_hostname=ZLIN12 netcfg/get_domain=domain.com network-console/password=0123456789 network-console/password-again=0123456789 url=http://install-server/ubuntu1604/s390x/preseed.cfg

It's worth to consider using the cio_ignore parameter to hide some of the devices from Linux, which may speedup the boot and install process significantly - especially in LPAR. <<BR>>
Here is a brief overview: http://linuxmain.blogspot.de/2011/10/using-cioignore-under-zvm.html <<BR>>
Keep in mind that the boot parameters are case sensitive (like 'locale=en_US')!
A common locale, especially for the early install stage is btw. 'locale=C'. <<BR>>
Consider to put 'auto=true priority=critical' in front of 'url=...’ parameter as well:
https://help.ubuntu.com/16.04/installation-guide/s390x/apbs02.html#preseed-auto <<BR>>
IPv6 addresses of servers hosting preseed configuration files are possible.

/* == Open Issues == */
/* done: */
/* Please notice that fully automated DASD installations are currently not thoroughly possible. */
/* There is already a hint about that in the release notes: */
/* https://wiki.ubuntu.com/XenialXerus/ReleaseNotes#Fully_automated_preseed_install */
/* and as stated there there are two tickets open right now. There is work going on to get that fixed. */
/* done: */
/* In between you may temporary format the DASDs prior to the automated installation from a manual installed system. Btw. in case you have (Hyper)PAV in place you may be able to now speed up the tedious dasdfmt performance via the ‘requestsize’ argument quite a bit: */
/* http://linuxmain.blogspot.de/2016/05/dasdfmt-performance.html */

== z/VM guest preseed installation ==
In case of a z/VM guest installation and now with the required boot parameters for preseed in mind the standard PARMFILE can now either be modified, or the existing PARMFILE copied into a new one with doing some modifications afterwards.
In the latter case the EXEC file needs to be copied and modified as well: the 'PUNCH' statement needs to point to the new name of the PARMFILE.

Have a look at the relevant files located in the xenial-server-s390x.iso file's boot folder:
kernel.ubuntu and initrd.ubuntu but especially parmfile.ubuntu and ubuntu.exec .

If the PARMFILE.ubuntu and ubuntu.exec files are going to be modified on a workstation (and not - as sown below - on z/VM CMS), simple LF must be used instead of CR/LF and the files need to be transferred in ASCII mode, rather than BINARY mode in combination with 'site fix 80', which is required for the kernel.ubuntu and initrd.ubuntu files. The format of the parm file can either be in fixed record format or variable.
In this case it's recommended to open the files after the transfer to CMS with xedit to verify the format (EBCDIC) and line-ending (LF). The following steps show a sample procedure.

The file name slightly change if transferred from the workstation to z/VM CMS:
||'''Workstation'''||'''Mainframe'''||
||initrd.ubuntu||INITRD   UBUNTU   A||
||kernel.ubuntu||KERNEL   UBUNTU   A||
||parmfile.ubuntu||PARMFILE UBUNTU   A||
||ubuntu.exec||UBUNTU   EXEC     A||
(The drive letter 'A' may be different on other machines.)

Upload these files as usual to the ftp server - if not already done.
Using ASCII mode for parmfile.ubuntu and ubuntu.exec and BINARY mode with 'site fix 80' for kernel.ubuntu and initrd.ubuntu. <<BR>>
Now copy the parameter and exec file: <<BR>>
`COPYfile PARMFILE UBUNTU A PARMFILE UBUAUTO A` <<BR>>
`COPYfile UBUNTU EXEC A UBUAUTO EXEC A` <<BR>>
edit file 'UBUAUTO EXEC A': <<BR>>
`xedit UBUAUTO EXEC A` <<BR>>
and modify the line: <<BR>>
`'PUNCH PARMFILE  UBUNTU   * (NOHEADER'` <<BR>>
to: <<BR>>
`'PUNCH PARMFILE  UBUAUTO  * (NOHEADER'` <<BR>>

Now the content of the 'PARMFILE UBUAUTO A' file, which is btw empty by default, needs to be expanded with the boot parameters, taking the z/VM reader's 80 byte limit into account. <<BR>>
`xedit PARMFILE UBUAUTO A` <<BR>>
An example [[https://wiki.ubuntu.com/S390X/InstallationGuide/AutomatedInstallsWithPreseed?action=AttachFile&do=get&target=parmfile.ubuauto|PARMFILE UBUAUTO]] may look like this: <<BR>>

`ro locale=en_US auto=true priority=critical cio_ignore=01ff-0fff                ` <<BR>>
`url=ftp://server/preseed/preseed.cfg s390-netdevice/choose_networktype=qeth     ` <<BR>>
`s390-netdevice/qeth/layer2=true s390-netdevice/qeth/port=0                      ` <<BR>>
`s390-netdevice/qeth/choose=0.0.C000-0.0.C001-0.0.C002                           ` <<BR>>
`netcfg/get_ipaddress=10.245.236.12 netcfg/get_netmask=255.255.255.0             ` <<BR>>
`netcfg/get_gateway=10.245.236.1 netcfg/get_nameservers=10.245.236.1             ` <<BR>>
`netdevice/qeth/layer2=true netcfg/confirm_static=true                           ` <<BR>>
`netcfg/use_autoconfig=1 netcfg/disable_dhcp=true netcfg/vlan_id=1234            ` <<BR>>
`hostname=ZLIN42 domain=domain.com network-console/start=note                    ` <<BR>>
`network-console/password=password network-console/password-again=password       ` <<BR>>

The parmfile is limited to 11 lines only, each 80 characters long.<<BR>>
(The precise limit is 11 lines with each * 80 characters plus additional 10 characters.)<<BR>>
Hence it's recommended to try specifying more than just one parameter per line, separate parameters by single spaces and use as much (shortened) aliases as possible (see above). But do ''not'' specify a single parameter (incl. it's value) across multiple lines!<<BR>>
The case of the parameter as well as the sequence do not matter.<<BR>>
(With this specification the above parmfile can be further shortened ... and if there is really a need for more characters, one can even completely omit the specification of the second install password: 'network-console/password-again=password'.)

A general [[https://wiki.ubuntu.com/S390X/InstallationGuide/AutomatedInstallsWithPreseed?action=AttachFile&do=get&target=preseed.cfg|sample preseed file]] can be found [[https://wiki.ubuntu.com/S390X/InstallationGuide/AutomatedInstallsWithPreseed?action=AttachFile&do=get&target=preseed.cfg|here]].
For a lean pressed file for KVM guest installation please have a look at [[https://wiki.ubuntu.com/S390X/InstallationGuide/AutomatedInstallsWithPreseed?action=AttachFile&do=view&target=preseed_kvm.cfg|this]].

The output of the command 'mkpasswd -m sha-512' can be used as value for the preseed key 'passwd/user-password-crypted password'.

And finally this [[https://wiki.ubuntu.com/S390X/InstallationGuide/AutomatedInstallsWithPreseed?action=AttachFile&do=get&target=preseed+installation+console+log.txt|console log]] shows the console output of a fully automated preseed installation, starting with the ubuauto exec file up to a final login per console, that takes less than 5 min (incl. format, package installation and reboot).

These changes can also be done with the help of the 'FILEList' command, which is a simple file management tool.
<<BR>>
If a command is typed beginning with upper case and at some character position with lower case, like 'COPYfile', it will be sufficient to use the upper case part to execute the command 'copy', but the full name and long version of the command is still 'copyfile'.

Another interesting source of information on using preseed on IBM z Systems is the [[http://www.vm.ibm.com/education/lvc/zlinlvc.html|IBM Live Virtual Class Fully Automated Installation of Ubuntu Server 16.04 with preseed]] [[http://www.vm.ibm.com/education/lvc/LVC0803.pdf|presented]] and [[https://attendee.gotowebinar.com/recording/7408829682231666436|demonstrated]] by Thorsten Diehl and available, including the disk configuration file examples for two DASDs with LVM ([[http://www.vm.ibm.com/education/lvc/2dasdlvm.conf|2dasds-lvm]]), two FCP SCSI LUNs with LVM ([[http://www.vm.ibm.com/education/lvc/2zfcplvm.conf|2zfcp-lvm]]), a single DASD ([[http://www.vm.ibm.com/education/lvc/1dasd.conf|1dasd]]), a qcow2 image file ([[http://www.vm.ibm.com/education/lvc/qcow2.conf|qcow2]]) and a single FCP SCSI disk ([[http://www.vm.ibm.com/education/lvc/1zfcp.conf|1zfcp]]).