AutomatedTestingUsingFAUmachine
|
⇤ ← Revision 1 as of 2007-09-22 15:02:31
Size: 2568
Comment: copy draft
|
Size: 2332
Comment: first draft.
|
| Deletions are marked like this. | Additions are marked like this. |
| Line 6: | Line 6: |
| * '''Packages affected''': | * '''Packages affected''': None |
| Line 12: | Line 12: |
| [http://www.faumachine.org FAUmachine] is a virtual machine for i386 and amd64, with a framework which allows elaborate automatic tests (e.g. installer tests) to be conducted. It can be used to conduct many of the test-cases listed on [:Testing] with a minimal administrative overhead. |
|
| Line 14: | Line 16: |
| This section should include a paragraph describing the end-user impact of this change. It is meant to be included in the release notes of the first release in which it is implemented. (Not all of these will actually be included in the release notes, at the release manager's discretion; but writing them is a useful exercise.) It is mandatory. |
Having widespread testing ready will reduce the chances of regressions. |
| Line 20: | Line 20: |
| This should cover the _why_: why is this change being proposed, what justifies it, where we see this justified. | Checking for regressions during the development cycle by hand is a burdensome task. Hence performing these tasks automatically seems a plausible goal. |
| Line 23: | Line 23: |
Release manager Andrea wants to know which regressions have been introduced during the development cycle of Ubuntu. Thus, she looks up the web-page containing the results of the most recent automated test run, which shows that 9 of 10 tests have succeeded. She looks at the failed and sees a screenshot of what went wrong. |
|
| Line 28: | Line 30: |
| You can have subsections that better describe specific parts of the issue. |
|
| Line 32: | Line 32: |
| This section should describe a plan of action (the "how") to implement the changes discussed. Could include subsections like: | * The test cases listed at [:Testing] need to be analyzed if these can be automated. * Experiment scripts for the applicable test-cases need to be written in the first place. * Some framework needs to be written to present the results of different test runs. * Ideally, these tests are run regularly, picking up the latest alpha/beta/daily images from the development version. * Over time, some test cases need to be modified (e.g. adapted to UI changes of the installer), but this should be a fairly easy task (merely replacing a screenshot or a string in the test-script). |
| Line 34: | Line 38: |
| === UI Changes === Should cover changes required to the UI, or specific UI that is required to implement this === Code Changes === Code changes should include an overview of what needs to change, and in some cases even the specific details. === Migration === Include: * data migration, if any * redirects from old URLs to new ones, if any * how users will be pointed to the new way of doing things, if necessary. == Test/Demo Plan == It's important that we are able to test new features, and demonstrate them to users. Use this section to describe a short plan that anybody can follow that demonstrates the feature is working. This can then be used during CD testing, and to show off after release. This need not be added or completed until the specification is nearing beta. |
|
| Line 57: | Line 41: |
| This should highlight any issues that should be addressed in further specifications, and not problems with the specification itself; since any specification with problems cannot be approved. |
Please check the status of this specification in Launchpad before editing it. If it is Approved, contact the Assignee or another knowledgeable person before making changes.
Launchpad Entry: foo
Packages affected: None
Summary
This should provide an overview of the issue/functionality/change proposed here. Focus here on what will actually be DONE, summarising that so that other people don't have to read the whole spec.
[http://www.faumachine.org FAUmachine] is a virtual machine for i386 and amd64, with a framework which allows elaborate automatic tests (e.g. installer tests) to be conducted. It can be used to conduct many of the test-cases listed on [:Testing] with a minimal administrative overhead.
Release Note
Having widespread testing ready will reduce the chances of regressions.
Rationale
Checking for regressions during the development cycle by hand is a burdensome task. Hence performing these tasks automatically seems a plausible goal.
Use Cases
Release manager Andrea wants to know which regressions have been introduced during the development cycle of Ubuntu. Thus, she looks up the web-page containing the results of the most recent automated test run, which shows that 9 of 10 tests have succeeded. She looks at the failed and sees a screenshot of what went wrong.
Assumptions
Design
Implementation
- The test cases listed at [:Testing] need to be analyzed if these can be automated.
- Experiment scripts for the applicable test-cases need to be written in the first place.
- Some framework needs to be written to present the results of different test runs.
- Ideally, these tests are run regularly, picking up the latest alpha/beta/daily images from the development version.
- Over time, some test cases need to be modified (e.g. adapted to UI changes of the installer), but this should be a fairly easy task (merely replacing a screenshot or a string in the test-script).
Outstanding Issues
BoF agenda and discussion
Use this section to take notes during the BoF; if you keep it in the approved spec, use it for summarising what was discussed and note any options that were rejected.
AutomatedTestingUsingFAUmachine (last edited 2008-08-06 16:23:15 by localhost)