KernelNattyStableProcessReview

Differences between revisions 2 and 3
⇤ ← Revision 2 as of 2010-10-18 16:48:09 →
Size: 1973
Editor: 192
Comment:
← Revision 3 as of 2010-10-28 20:48:41 → ⇥
Size: 5534
Editor: host194
Comment:
Deletions are marked like this. Additions are marked like this.
Line 18: Line 18:
Over time we changed our preferences on how or what we take for stable releases. But it seems the presentation on this is still distributed over too many places to give a simple and easy start. Also there are likely a few places where we could improve the work-flow of the whole process. The stable update process for released kernels has been very non-deterministic, and critical security updates during proposed kernel testing periods have occurred repeatedly, pushing back releases and resulting in released with a very high number of changes. There has been difficulty in getting verification testing completed for bugs, becuase we lack the needed hardware, and because the bug reporters have not tested the proposed kernels in a timely manner.
Line 22: Line 22:
 * Upstream stable should get into the master branch as quickly as possible.  * Upstream stable updates should get into the master branch as quickly as possible.
 * CVEs should be fixed as soon as possible
 * Certification testing needs to take place in proposed.
Line 24: Line 26:
 * While security may delay the actual uploads to proposed but the repo (and by that pre-proposed) should always be as advanced as possible.  * proposed uploads should take place often enough to prevent having a huge number of changes in any one release.
 * proposed kernel production should be scheduled as predictably as possible, and the affect of having a critical security release take place should cause minimal delay in releasing.
Line 28: Line 31:
tbd The stable kernel team will change to a regular two-week release cadence. The cycle is two weeks long, and repeats every two weeks, with the exception of holidays, UDS weeks, and other considerations. An attempt will be made to sync the schedule with the needs of the Ubuntu platform schedule and Linaro release schedule.
Line 32: Line 35:
This section should describe a plan of action (the "how") to implement the changes discussed. Could include subsections like:  * The SRU policy will be amended to indicate that changes will only be considered if we have the hardware and resources to test the fix, or a very high confidence that the bug reporter will test the proposed release.

 * Kernel stable updates will follow a two week cycle

 * At the beginning of the cycle, bug fixes, upstream stable patches, and non-embargoed security fixes will be applied, and the kernel uploaded to proposed.

 * At the end of the first week, any bug fixes which have not been verified as fixed by the reporter or by internal testing will have the patches reverted, and the associated bugs closed as "wontfix".

 * During the second week, certification and regression testing will be performed to insure that the kernel is functional. Minimum testing will insure that the kernels boots, networking operates, X starts (for workstations), and updates can be performed. Certification tests will not include the entire set of manually executed tests, but will include an automated set of tests. testing will be a joint effort by the QA and certification teams.

 * Cert and QA will begin testing the kernels currently in proposed next week (1 Nov, 2010) and report on the amount of effort and resources required to perform testing. These results will be used to evaluate the resources required to test all expected variants of released kernels.

 * Proposed releases will be tracked with a tracking bug opened by the stable kernel team, and linked via an entry in the changelog. This bug will be assigned to the QA team (or a launchpad team - TBD), and tracked on the kernel bug dashboard. The bug will be set to verified when all QA and Cert testing has passed.

 * We need to decide which systems will be tested. Should it be most common, or some subset? It will include at least one virtual environment.

 * Security kernels which are critical or embargoed will also receive the same tests required for proposed kernels.


== Action Items ==

 * [bjf] Take the whiteboard text and add it to the wiki specification
 * [sconklin] boiler plate text for bug closure on failure to verify
 * [canonical-kernel-team] look at arsenal scripts to nag the users
 * [skaet][sconklin] workout the sru schedule with respect linaro and platform
 * [sconklin] provide a document that describes the process, bug tracking, and which tests are being performed, etc
 * [jfo] make sure that the bugs that we are sending for this are on the dashboard
 * [pitti] ensure verification bug update indicates that this is going to be pulled
 * [sconklin] process to add the bug to the changelog manually during release process.
 * [marjo] QA team - define a mechanism to assign the proposed upload tracking bug to them and track results
 * [victor] QA team - provide a description of what testing they provide, and an idea of the number
of hours required to run those
 * [leann] provide info from HWDB http://reports.qa.ubuntu.com/reports/ogasawara/hwdb/system-stats-popular-20101019.html
 * [action] Victor to investigate which systems are "common" for cert testing
 * [cert-team] blueprint for cert-team
Line 36: Line 73:
This should highlight any issues that should be addressed in further specifications, and not problems with the specification itself; since any specification with problems cannot be approved. Actual start date for the new cycles (depends on examination of schedule and resources)

Do we have the capacity to test the required number of kernels?
Line 40: Line 79:
Use this section to take notes during the BoF; if you keep it in the approved spec, use it for summarising what was discussed and note any options that were rejected.

Summary

The purpose of this spec is to go over the current stable process (including the SRU policy) to see how we are doing and to see which parts can be improved or documented better.

Release Note

The impact on users would be to have a place which well documents what changes can be made to stable releases, how to ask for inclusion and why decision will be made the way they are done. This is less likely to go into the general release notes but should be linked from a prominent place on the kernel wiki page.

Rationale

The stable update process for released kernels has been very non-deterministic, and critical security updates during proposed kernel testing periods have occurred repeatedly, pushing back releases and resulting in released with a very high number of changes. There has been difficulty in getting verification testing completed for bugs, becuase we lack the needed hardware, and because the bug reporters have not tested the proposed kernels in a timely manner.

Assumptions

  • Upstream stable updates should get into the master branch as quickly as possible.
  • CVEs should be fixed as soon as possible
  • Certification testing needs to take place in proposed.
  • Regression testing needs to be done in proposed.
  • proposed uploads should take place often enough to prevent having a huge number of changes in any one release.
  • proposed kernel production should be scheduled as predictably as possible, and the affect of having a critical security release take place should cause minimal delay in releasing.

Design

The stable kernel team will change to a regular two-week release cadence. The cycle is two weeks long, and repeats every two weeks, with the exception of holidays, UDS weeks, and other considerations. An attempt will be made to sync the schedule with the needs of the Ubuntu platform schedule and Linaro release schedule.

Implementation

  • The SRU policy will be amended to indicate that changes will only be considered if we have the hardware and resources to test the fix, or a very high confidence that the bug reporter will test the proposed release.
  • Kernel stable updates will follow a two week cycle
  • At the beginning of the cycle, bug fixes, upstream stable patches, and non-embargoed security fixes will be applied, and the kernel uploaded to proposed.
  • At the end of the first week, any bug fixes which have not been verified as fixed by the reporter or by internal testing will have the patches reverted, and the associated bugs closed as "wontfix".
  • During the second week, certification and regression testing will be performed to insure that the kernel is functional. Minimum testing will insure that the kernels boots, networking operates, X starts (for workstations), and updates can be performed. Certification tests will not include the entire set of manually executed tests, but will include an automated set of tests. testing will be a joint effort by the QA and certification teams.
  • Cert and QA will begin testing the kernels currently in proposed next week (1 Nov, 2010) and report on the amount of effort and resources required to perform testing. These results will be used to evaluate the resources required to test all expected variants of released kernels.
  • Proposed releases will be tracked with a tracking bug opened by the stable kernel team, and linked via an entry in the changelog. This bug will be assigned to the QA team (or a launchpad team - TBD), and tracked on the kernel bug dashboard. The bug will be set to verified when all QA and Cert testing has passed.
  • We need to decide which systems will be tested. Should it be most common, or some subset? It will include at least one virtual environment.
  • Security kernels which are critical or embargoed will also receive the same tests required for proposed kernels.

Action Items

  • [bjf] Take the whiteboard text and add it to the wiki specification
  • [sconklin] boiler plate text for bug closure on failure to verify
  • [canonical-kernel-team] look at arsenal scripts to nag the users
  • [skaet][sconklin] workout the sru schedule with respect linaro and platform
  • [sconklin] provide a document that describes the process, bug tracking, and which tests are being performed, etc
  • [jfo] make sure that the bugs that we are sending for this are on the dashboard
  • [pitti] ensure verification bug update indicates that this is going to be pulled
  • [sconklin] process to add the bug to the changelog manually during release process.
  • [marjo] QA team - define a mechanism to assign the proposed upload tracking bug to them and track results
  • [victor] QA team - provide a description of what testing they provide, and an idea of the number

of hours required to run those

Unresolved issues

Actual start date for the new cycles (depends on examination of schedule and resources)

Do we have the capacity to test the required number of kernels?

BoF agenda and discussion

CategorySpec

KernelTeam/Specs/KernelNattyStableProcessReview (last edited 2010-10-28 21:44:22 by host194)