ReleaseCycle
How the Security Team interacts with the regular Ubuntu Release Cycle.
Security Updates
pre-feature-freeze
- request syncs for issues fixed in stable releases that are not fixed in the development release, but are fixed in upstream versions.
post-feature-freeze
- request feature-freeze exceptions for upstream security fixes, or push security patches, depending on extent of upstream changes.
post-beta
- focus security updates on any updates still missing from devel release that are present in stable updates (see 'Show all out-of-sync CVEs for the devel release' from $UCT/README for assistance)
add the following to the announcements section of the ubuntu-security weekly meeting: "Ubuntu <version> is scheduled to release in a few weeks. If you are interested, it would be great if people helped make sure that the Ubuntu CVE Tracker is up to date for the devel release (particularly for universe/multiverse packages."
Regression Testing
pre-Alpha 3
- run regression tests when any major changes are noticed
Alpha 3
- run kernel hardening regression tests on multiple architecture/kernel combinations (i386 w/o PAE, i386 w/ PAE, amd64) (test-kernel*.py)
- run gcc hardening regression tests on amd64 and i386 (test-gcc-security.py)
- run glibc hardening regression tests on amd64 and i386 (test-glibc-security.py)
- run other user-space hardening tests (eg PIE) on amd64 and i386 (test-built-binaries.py)
Beta 2
- run kernel hardening regression tests on multiple architecture/kernel combinations (i386 w/o PAE, i386 w/ PAE, amd64) (test-kernel*.py)
- run gcc hardening regression tests on amd64 and i386 (test-gcc-security.py)
- run glibc hardening regression tests on amd64 and i386 (test-glibc-security.py)
- run other user-space hardening tests (eg PIE) on amd64 and i386 (test-built-binaries.py)
- run remaining regression tests from qa-regression-testing, updating the tests or reporting bugs against the package
run test-rng.py --all --report from qa-regression-testing (takes many days, so may want to spread across multiple machines) and put the results in qa-regression-testing/results/test-rng.py/<release number>. This is likely a good use of EC2 (but keep in mind as of 2009/04/27 it costs ~ $100).
run qa-regression-testing/install/get_file_info.sh and review for problems
- verify guest session is confined
post-rc1
run install/get_file_info.sh and record results in qa-regression-testing/install/get_file_info_results (see qa-regression-testing/install/README for details). This also needs to be done on the final images
Review supported kernels with the kernel team
EOL Checklist
Infrastructure (EOL)
- from ubuntu-cve-tracker's README section "End of Life":
mark any active CVEs for that release with an EOL comment: ./scripts/sync-from-eol.py -WUu -r $RELEASE
- add release to 'eol_releases' in scripts/cve_lib.py
run a CVE retirement cycle:
$ ./scripts/cve_need_retire -u
remove release from boilerplates: sed -i -e '/^<release>_\(.*\): /d' -e '/^#<release>_PKG:/d' ./active/00boilerplate*
- remove any backport kernels for this release from 00boilerplate.linux
- remove any backport kernels for this release from 'kernel_srcs' in scripts/cve_lib.py
update supported_releases and ignored_releases in scripts/generate-oval
- update ubuntu-cve-tracker/scripts/packages-mirror to lose old release
update architectures to include "EOL" footnote in SecurityTeam/FAQ
regenerate Security/Features from $UCT/scripts/dump-features --editmoin
review and optionally closed bug tasks for EOL releases with $UCT/scripts/report-bugs-for-eol
- update $QRT/README.testing to remove EOL release
update people:~ubuntu-security/.ubuntu-security-tools.conf settings for (used by kernel-abi-check):
- "release_list" loses EOL release name
remove release from $UQT/security-tools/kernel-abi-check and update $UQT/security-tools/kernel-abi-check to remove any checks for EOL kernels (ie, pocket_abis_match('<release>'...). Note this also require a manual bzr pull on people
update security-tools/build-tools/security-fake-sync to ensure LTS releases point to the right place (next_release())
Individuals (EOL)
update ~/.ubuntu-security-tools.conf settings for:
- "release_list" loses EOL release name
No longer required with uvt: update ~/.uqt-vm-tools.conf settings for:
- "vm_release_list" loses EOL release name
- update debmirror/apt-cacher lists to drop old release
- delete VMs and schroots from old release
adjust sources.list to remove deb and deb-src for old release
- adjust daily build systems
remove release from build list for daily ppas (e.g. apparmor-daily recipe, ecryptfs-utils-daily recipe) and jenkins builds
remove packages built for the old release from daily ppas (e.g. apparmor-daily-trunk ppa, ecryptfs-utils-daily ppa, ufw-daily-trunk) and jenkins builds
Release Checklist
Infrastructure (release)
update $UCT/scripts/cve_lib.py:
add the new release in all_releases
devel_release is emptied
Add release entry and timestamp to release_stamps. (The time stamp can be generated based of the date stamp of the release file rounded up to the next hour; e.g. echo "(($(date -r /PATH/TO/UBUNTU/ARCHIVE/dists/RELEASE/Release.gpg +%s) / 3600 ) + 1) * 3600" | /usr/bin/bc
update release_expectations for the new release, and add a commented out section for the new devel release
- move all active CVEs and boilerplates from "devel" to release state:
cd $UCT ; ./scripts/release-cycle-released $RELEASE
Update 'releases' in $UCT/scripts/report-todo-recommend to contain the most recent LTS and most recent non-LTS in check for if not opt.all_releases:
update supported_releases and ignored_releases in $UCT/scripts/generate-oval
Update $UCT/scripts/cve.vim to include the new release.
add release's kernels to $UCT/security-tools/kernel-abi-check (based on the kernel team's ABI page) for both this release and LTS any kernels
- update usn-tool for publication
update templates/html-drupal.html (add to release_list div, add a new stanza in the 'for each' part of that div, and add a show_binaries entry). This is no longer used by the security team, but might be in landscape (TODO: check).
update templates/email-maria.txt (add a new stanza to the 'affected releases' section, add a show_packages entry, and add a show_release_urls entry). This is no longer used by the security team, but might be in landscape (TODO: check).
update templates/email-new-format.txt (add a new stanza to the 'affected releases' section, add a show_packages entry, add a show_package_description entry, add a show_source_link entry)
pull these changes down on people (this happens automatically, but it will need to happen before a USN publication):
$ cd /home/ubuntu-security/bzr-pulls/usn-tool/ $ sudo -H -u ubuntu-security bzr pull
update architectures in SecurityTeam/FAQ
update security-tools/build-tools/security-fake-sync for new stable release (ubuntu_version() and next_release())
update people:~ubuntu-security/.ubuntu-security-tools.conf settings for:
release_devel is emptied
ask webops in #webops to update the usn-website to have the version for the new release (eg, it should have "Ubuntu <version>"). Eg LP: #773627. This can be done via the '/usn/admin' interface. There shouldn't be an entry for the development release.
Update AppArmor
Update UncomplicatedFirewall
Verify versions for AppArmor in $UCT/scripts/dump-features and regenerate Security/Features from $UCT/scripts/dump-features --editmoin
- Update list of syscalls for the supported kernel for the new release:
grabbing linux-libc-dev for all architectures from the linux package for new release and putting them in a folder (eg, '4.9')
generate the syscall list (eg, 4.9.syscalls) by following http://people.canonical.com/~ubuntu-security/syscalls/README
Individuals (release)
update ~/.ubuntu-security-tools.conf settings for:
release_devel is emptied
add security-RELEASE and security-proposed-RELEASE sections to ~/.dput.cf (not normally needed if using recommended dput setup)
- pull infrastructure changes (from above):
- usn-tool
- ubuntu-cve-tracker
- ubuntu-security-tools
- ubuntu-qa-tools
Devel Opens
Infrastructure (devel)
fill in all_releases, flavor_releases, devel_release and release_names in ubuntu-cve-tracker/scripts/cve_lib.py
- Move all active CVEs and boilerplates from latest release to devel state:
./scripts/release-cycle-devel-opens $LATEST_STABLE_RELEASE
add release to non-ports and ports section, as well as the compressed_ext() function, of ubuntu-cve-tracker/scripts/packages-mirror
- It is helpful to run packages-mirror in the background now as the results are used in several future steps
$UCT/scripts/packages-mirror -p -r <new devel> ; $UCT/scripts/packages-mirror -r <new_devel>
update security-tools/build-tools/security-fake-sync for new release (next_release())
update graphing scripts on people in ~ubuntu-security/graphing (see ~ubuntu-security/graphing/README for details)
supported.data.template has the binary package counts. To update:
determine the binary package counts ($packages_mirror is set in ~/.ubuntu-cve-tracker.conf):
$ cd $packages_mirror ; zcat ./dists/$LATEST_STABLE_RELEASE/main/binary-*/Packages.gz ./dists/$LATEST_STABLE_RELEASE/restricted/binary-*/Packages.gz | grep '^Package:' | sort -u | wc -l
- in the .template file, take the last line of the file and duplicate it at the end of the file
- in the .template file, uncomment the second to last line (formerly the last line) and add the package counts for the latest stable release
- in the .template file, update the last line's commented out release name to be the new devel release and adjust counts for EOL releases
- in the .template file, add the new devel release column header name
- in the .template file, add a 0 column to all lines (except the last uncommented line), making sure everything is sensible for package counts
supported-src.data.template:
determine the source package counts:
$ cd $packages_mirror ; zcat ./dists/$LATEST_STABLE_RELEASE/main/source/Sources.gz ./dists/$LATEST_STABLE_RELEASE/restricted/source/Sources.gz | grep '^Package:' | sort -u | wc -l
- in the .template file, take the last line of the file and duplicate it at the end of the file
- in the .template file, uncomment the second to last line (formerly the last line) and add the package counts for the latest stable release
- in the .template file, update the last line's commented out release name to be the new devel release and adjust counts for EOL releases
- in the .template file, add the new devel release column header name
- in the .template file, add a 0 column to all lines (except the last uncommented line), making sure everything is sensible for package counts
update.sh REL updated, INDEX incremented by 1 (to match column in the .template files excluding the newly added devel release, since we don't know its package counts yet))
add to .bzrignore:
$NEW_RELEASE-src.data $NEW_RELEASE.data
- commit the changes
update people:~ubuntu-security/.ubuntu-security-tools.conf settings for:
release_list gains new release name (and any overlay releases, eg 'vivid/stable-phone-overlay', 'xenial/stable-phone-overlay', etc)
release_devel gains new release name (without overlay releases)
regenerate Security/Features from $UCT/scripts/dump-features
regenerate Security/Features/Historical from $UCT/scripts/dump-features --historical
- create new release in apparmor-profiles and copy profiles over from previous release
- Make sure lp:ubuntu-qa-tools/responses/update-bug includes the new release
- update $QRT/README.testing for new development release
- update ubuntu-security crontab on people to use the new development release for:
- create-dashboard.sh
- The argument should be the first letter of the new release name (lowercase)
- create-dashboard.sh
use cd $UCT && ./scripts/sync-from-versions.py -s not-affected. If satisfied with the result, use cd $UCT && ./scripts/sync-from-versions.py -s not-affected -u
use cd $UCT && ./scripts/sync-from-versions.py -r $RELEASE -s not-affected. If satisfied with the result, use cd $UCT && ./scripts/sync-from-versions.py -r $RELEASE -s not-affected -u
Individuals (devel)
- Update the following bzr trees:
- ubuntu-cve-tracker
- ubuntu-security-tools
- ubuntu-qa-tools
- qa-regression-testing
- update debmirror/apt-cacher lists to include new release
update ~/.ubuntu-security-tools.conf settings for:
release_list gains new release name
release_devel gains new release name
- build new VMs and schroots for new release (remembering to update 'vm_release_list' in '~/.uqt-vm-tools.conf' (not needed if using uvt))
- pull infrastructure changes (from above):
- ubuntu-cve-tracker
update your packages mirror with $UCT/scripts/packages-mirror -r $NEW_RELEASE
update your partner packages mirror with $UCT/scripts/packages-mirror -p
update your sources.list to have the appropriate deb-src lines for the new release (eg use $UST/build-tools/build-sources-list output in /etc/apt/sources.list.d/ubuntu-security.list)
[optional] subscribe to $NEW_RELEASE-changes at https://lists.ubuntu.com/mailman/listinfo/$NEW_RELEASE-changes
QA (devel)
For projects where the security team is the upstream maintainer (eg, AppArmor, ufw, ecryptfs):
- Update the daily build PPA to build for the development release
- As needed, update the daily build PPA packaging (eg, merge in any changes since the last release)
Click/AppArmor (devel)
- apparmor-easyprof-ubuntu, ubuntu-core-security and ubuntu-personal-security
run sh ./debian/make-new-version.sh <old> <new> which will create symlinks for the new policy to the old previous version
- increment the package version such that the major and minor versions match that of the new policy directories, and start the micro version at '1'. The policy version should match the release version. Eg, if the apparmor-easyprof-ubuntu policy is currently 15.10.17 and the new version added in the previous step is '16.04', then the new apparmor-easyprof-ubuntu version should be '16.04.1'. See README.source for details.
- adjust autopkgtests in debian/tests for the new policy
- add any new tests for the new policy to tests/* as needed
- adjust QRT/tests/image/unprivileged/apparmor-easyprof-ubuntu as necessary (not usually needed)
- click-apparmor
- update 'framework_transforms' in apparmor/click.py for the new release, using the new policy version that was added to apparmor-easyprof-ubuntu
- add a new 'ubuntu_NNNN_transform()' for the new release (must match what is in 'framework_transforms'
- add any new autopkgtests for the new 'framework_transforms' and 'ubuntu_NNNN_transform()'
- add any new tests for the new 'framework_transforms' and 'ubuntu_NNNN_transform()' to test-clicktool.py
- adjust QRT/tests/image/unprivileged/click-apparmor as necessary (not usually required)
- click-reviewers-tools
- add new frameworks/policy to ./data/apparmor-easyprof-ubuntu.json
LTS loses Desktop support
generate $UCT/$RELEASE-supported.txt with a list of all the source packages that will get 5 years of support.
update $UCT/scripts/cve_lib.py's lts_partial_supported_releases variable to include th LTS.
run $UCT/scripts/check-syntax to update all the CVEs to the "ignored" state for the EOL packages.
Debian Release/EOL Checklist
Infrastructure (Release/EOL)
- Update lp:ubuntu-security-tools for:
- README needs the proper release_extras list
- build-tools/build-sources-list has a hard-coded "supported releases" list
Update SecurityTeam/BuildEnvironment to mk-sbuild the current releases and document the proper release_extras list
Update d2u.py in lp:~ubuntu-security/debian2ubuntu/trunk with new debian release. Will need to pull the bzr tree on people.c.c as well.
IMPORTANT: only update the list for new Debian releases since d2u may show pending merges against EOL Debian releases
Individuals (Release/EOL)
- pull infrastructure changes (from above):
- ubuntu-security-tools
update ~/.ubuntu-security-tools.conf settings for:
- "release_extras" list gains/drops the release name
- [optional] create/delete VMs and schroots from the release
update your sources.list to add/remove the deb-src lines for the release (eg use $UST/build-tools/build-sources-list output in /etc/apt/sources.list.d/ubuntu-security.list)