Launchpad Entry: qa-sru-process-streamline
Created: 2008-12-15
Contributors: sbeattie, pvillavi, heno
Summary
Improve the throughput of SRU fix validations by drawing up a clear workflow, schedule and make a pragmatic selection of tools.
Rationale
Proposed updates to the stable release need to be verified to ensure we don't introduce any regression and that the fix actually works. Currently the process is not flowing as smoothly as it could. This can be iproved with a better SRU validation workflow, including adhering more closely to existing rules:
According to the StableReleaseUpdates procedure, in order to approve a package to be uploaded to proposed the bug needs to have a set of detailed descriptions on how to reproduce the bug, this line is marked as "TEST CASE", in most of the bugs this available but in some others is not, in which case is very difficult for the SRUV team to perform the verification and most of the times the bug is delayed for weeks until the package maintainer add the TEST CASE or the bug gets feedback from directs subscribers which can reproduce the behavior.
- Bugs that require an specific hardware in order to complete the verification, if the SRUV team does not have this hardware it leads to no one except for the reporter to be able to verify the new package.
- Doing the verification for large deployments is complex (nfs example).
Use Cases
Design
- Redesign of the page for tracking the SRU bugs
- Update the documentation in order to provide detailed instructions on how to perform an SRU Verification
- Create scripts for the SRU Verification team in order to make their work easier
- Start to track the bugs without a test case.
Implementation
Workflow changes
The Ubuntu SRU Team should be contacted to ensure packages don't get uploaded to -proposed without a TEST CASE, which is a requirement in the SRU Procedure.
If a package was uploaded and does not have a detailed description on how to perform the verification, the bug should be:
marked with the tag needs-testcase
- add a comment saying that a TEST CASE is needed in order for the SRU Verification team to provide feedback on the package
- contact the Ubuntu SRU Team and assign the bug to the bug fixer for provide the required TEST CASE.
For the bugs requiring specific hardware in order to perform the verification, a HARDWARE DESCRIPTION section should be added to the bug description, but the hw-specific tag should not be removed. example:
HARDWARE DESCRIPTION This report needs the following card in order to be tested: Network controller: Atheros Communications Inc. AR5008 Wireless Network Adapter (rev 01)
Documentation
Improve the documentation regarding:
How to set up the environment (KVM, VirtualBox, Chroot, etc) in order to perform the verifications.
- Add a section with explanation on how to use the new scripts.
SRU Pages
A better identification of bugs requiring specific hardware in order to perform the verification is needed, possible create a separate list for them in order to distinguish them from the whole pile of bugs. Needs investigation on the Javascript website side since it's causing a lot of cpu usage while seeing the web page.
The page should also contain another list for the bugs that were marked as needs-testcase by the SRU Team.
Migration
Only provide one website (ie: sru.ubuntu.com) which the SRU Team should visit in order to see the status of the current SRU process, the data provided from the previous websites (http://people.ubuntu.com/~ubuntu-archive/pending-sru.html , http://people.ubuntu.com/~sbeattie/sru_todo.html) it just need to be moved and implement the ideas described previously.
Tools
A set of tools should be develop in order to make the process easier for the SRU Verification Team:
- script to enable/disable proposed and run apt-get update from /etc/apt/sources.list
- script to update package(s) to -proposed version related to bug being verified
- scripts to keep VMs updated (Vbox, KVM, Chroot, etc)
Unresolved issues
The following ideas for launchpad are not covered in this spec however they need to be filed (if not already) as a bug report in the malone product.
- make fixed SRU bugs that need verification appear at the top of the search listings (for a package)
- make it clear inside a bug report where it is in the SRU process (eg: needs ack, needs a second ack, needs to be uploaded to -proposed, is in -proposed)
- this makes it helpful to users so they don't have to read the entire bug report
- also makes it easier for the SRU team to call their attention to certain bugs
- could have a progress bar slider of sorts
Actions
Bug fixes for large setups (google nfs example) are complicated, in order to help with this process we should look at those bugs, contact the groups and coordinate efforts with them, also talk to the Ubuntu SRU Team regarding which groups are trusted.
BoF agenda and discussion
Improve the throughput of SRU fix validations by drawing up a clear workflow, schedule and make a pragmatic selection of tools.
Current tracking webpages:
- unifying these so we only have one place to go to look for packages needing verification
- better identification of bugs requiring specific hardware to perform the verification
- a "hardware description" section in the bug description
- split verification-needed bug reports by identifying hw-specfic ones and needs-testcase ones
MOTU-SRU's todo list http://qa.ubuntuwire.com/sru/todo.html
Workflow documentation
- increase documentation regarding ways to perform the verification
- require TEST CASE for a package to enter -proposed
- How can complicated bug fixes (large nfs setups) be verified?
- trusting of teams that have large infrastructures where the bug fix has been tested well
- coordination with the archive admins regarding which groups are trusted
- What can block an SRU verification?
- not having hardware, or infrastructure
not having a good test case -> assigning it back to the bug fixer (tag it so it is in different part of the list) [work with archive admin team to ensure packages don't get uploaded without a testcase]
Create template and track individual SRUs on the testcases wiki?
- justification/SRU worthy?
- person pushing
- unable to test/can delegate?
- steps taken to verify
- templates would be a good idea however tracking really should happen in bug reports
Tools
- script to enable/disable proposed and run apt-get update from /etc/apt/sources.list
- script to update package(s) to -proposed version related to bug being verified
- scripts to keep VMs updated
https://help.ubuntu.com/community/SbuildLVMHowto "Keeping your chroots up to date"
http://blog.dustinkirkland.com/2008/10/mounting-kvm-disk-image.html
- List bugs needing verification, or a link to them, at qa.ubuntu.com (the portal!)
- document ways to perform verification (via kvm, chroots etc)
- should bug reports be identified as not requiring a gui(?)
Launchpad improvements
- make fixed SRU bugs that need verification appear at the top of the search listings (for a package)
- make it clear inside a bug report where it is in the SRU process (eg: needs akk, needs a second akk, needs to be uploaded to -proposed, is in -proposed)
- this makes it helpful to users so they don't have to read the entire bug report
- also makes it easier for the SRU team to call their attention to certain bugs
- could have a progress bar slider of sorts
- bug tags can be used in the interim to indicate the state of the bug in the verification process -- verficiation-needed indicates that the packge is in -proposed and needs testing