Ubuntu Wiki

• Launchpad Entry: https://blueprints.launchpad.net/ubuntu-arm/+spec/arm-m-validation-dashboard

• Created: PaulLarson

• Contributors: PaulLarson, ZygmuntKrynicki

• Packages affected:

Summary

As part of the automated testing efforts on ARM, we would also like to have a dashboard interface for visualizing the current state of the image. This interface should allow the user to see, at a glance, the state of functional tests as well as performance tests, and any other data that may be useful.

Release Note

No user visible changes

Rationale

We would like to easily see how things are shaping up over time on arm, and how changes are impacting performance. A dashboard interface would help us visualize, in one place, the results of testing from multiple machines. It could also be used to generate graphs of performance metrics across different image build dates to quickly see if things are improving, or getting worse, and how far we are away from any goals that may exist.

User stories

Bob is a release manager for Ubuntu on a particular arm device. Bob wants to check the overall status of the image produced yesterday before releasing Alpha 1. Bob visits the dashboard to check for test failures. Bob marked some tests as expected to fail on this device as not all components are yet in place and some things are still broken. As all other tests have good results Bob can go forward with the release.

Jane is interested in basic performance metrics of current ubuntu image. Jane can check some synthetic benchmarks for CPU, GPU and IO performance. Jane can also check some end-to-end benchmarks for user applications (browser startup time, time to full desktop, time to render snapshot of key websites, etc). Jane can setup baseline for each metric and request to be notified of all variances that exceed given threshold.

Alice is on the Desktop QA team and would like to integrate some of the tests her team has created into the dashboard. QA Engineers quickly bootstrap a local installation of the dashboard and check the bundled documentation and examples. Within hours the dashboard can display results from some of the tests that local engineering team has already adapted.

Yung is a product manager in Big Corp ltd. Yung is building a product based on Ubuntu and wants to reuse our QA infrastructure. Yung instructs his engineers to deploy a local dashboard installation and run our tests on their new secret product. Engineers can also write an adapter that will take some of the results from the dashboard and push it to the internal QA system.

Assumptions

• Easy to deploy, including, outside of Canonical. All infrastructure components must be packaged and provided as a PPA, ready to install on a Lucid server. All device components must be packaged and uploaded to Maverick (TODO: which component? are we going straight-to-ppa or do we attempt to hit the main archive?)
• Focused on one device.
  • FIXME - is this _really_ true? - if not how to avoid multiple devices and benchmarks

    There seems to be a conflict of interests - we'd like to see this for >1 kind of device but vendors will not enjoy any device benchmark information being displayed, especially alongside competing devices

• One-way connectivity. Devices participating in the test can connect to the infrastructure services but reverse connection cannot be assumed. (TODO: what about IPv6?)
• Distributed environment. Devices and infrastructure components are places in diverse geographical and administrative zones.
• Launchpad integration for bug management. There is no plan to support third party bug trackers in the first release.

Design

TODO:

• UI design for each core use case (TODO)
  • (we want a list of tasks users have to perform to get to the goal they are after (with regard to the use case lists above)
• UI design vs UI design of other Canonical technologies (TODO)
• Design web pages we need to provide (DONE)

Dashboard will feature the following pages/views:

Project Timeline

Recurring component of each page, shown at the top. Key aspects:

• Shows milestones
• Shows number of days till next milestone
• Shows number of days/weeks? till final milestone
• Allows to click at a past day to see historical records

Project timeline could also hold global image/project properties action menu:

• Edit (add/remove/modify) test suites and benchmarks

Day Overview

Main view contains daily summary of key aspects influencing upcoming release. This is also the default view for the application. The page contains the following components:

• packages
  • summary indicators, mostly numbers and links for detail pages)
    • (could be a horizontal bar like in test cases below)
    • total number and details link
    • newly added
    • modified (version change)
    • packages that failed to build
  • action links:
    • see package details
• test cases
  • total tests suites and test cases
  • progress indicator: horizontal bar with the following components
    • skipped tests
    • successful tests
    • failed tests
    • pending tests (there is no indicator of a 'running' test)
  • action links:
    • see all tests (details)
    • edit skipped tests [optional]
• benchmarks
  • selected benchmark results (value + spark line)
    • synthetic benchmarks
      • CPU
      • FPU
      • GPU (if possible)
      • IO:
        • USB thumb drive
        • USB 2.5" HDD
        • SD card
        • Network
        • NAND [optional]
    • end-user / application benchmarks
      • time to boot
      • time to website/cached
      • time to ... (etc)
  • notable changes (value, spark line, delta)
    • (things that are not included by default but have changed radically since last test)
  • action links:
    • see all benchmarks (details)
    • define selected items
• devices
  • all devices we have at our disposal
    • percentage of time devoted to:
      • running tests
      • being 'owned' by someone
      • being idle
      • being offline
• bugs [optional]
  • all bugs filed yesterday that affect this image
    • (could use specific tag, project to detect)

Image Details

This page can be reached from the 'daily overview' pages. It should contain basic information about all packages that were used to build the image. If possible each package should be a link to a launchpad page. For packages that are not tracked by launchpad (PPAs, custom packages) and to packages that are a part of a private PPA no link will be provided.

[Optional]. This view could also provide a diff against any other day. Such difference views could be easy accessible from the project timeline.

Test Suite Details

This page can be reached from test suite list and test day overview (selected suites)

Test suite details provides access to the following components:

• Test suite URL: bzr branch that contains this test suite
• Description
• List of test cases
• Summary of historical results (pass/fail/skipped)

Actions:

• Disable whole suite for specific hardware
• Disable specific tests for specific hardware

Test Case Details

This page can be reached from test suite details.

Test case details provides access to the following components:

• Historical results (pass/fail/skip)
• Preview of the relevant part of the log file that was harvested to get this result.

Benchmark Details

TODO.

Mostly similar to test suite, except for some presentation differences.

Benchmark probe (single benchmark item) Details

TODO

Mostly similar to test case, except for some presentation differences.

Implementation

Choose basic web technology: DONE (django) Choose database system: DONE (PSQL) Design data model (including data ingest requirements, data presentation requirements, data transformations and storage): TODO Design web widgets/pieces/components that we will need and determine how each fits into the data model: TODO? (is this required in the spec?)

Architecture Overview

The dashboard is a part of a larger set of blueprints/projects that together provide the QA infrastructure. The main components are:

• Automatic Test Framework
• Test Cases (dealing with providing actual tests)

• WebKit testing/benchmarking

• Dashboard:
  • frontend - web pages, interaction with launchpad, etc backend - database model, processes, interactions with other components

Database Model

Make database model image: TODO

UI Changes

Should cover changes required to the UI, or specific UI that is required to implement this

Code Changes

Code changes should include an overview of what needs to change, and in some cases even the specific details.

Migration

Currently there is no direct migration plan. Things we could consider is migrating some bits and pieces of technology already up and running either at Canonical or somewhere else in the community/other parties that is open source and integrate their tests into our framework. If that becomes true we might want to look at migration from qa-tool-foo to our new technology.

Test/Demo Plan

It's important that we are able to test new features, and demonstrate them to users. Use this section to describe a short plan that anybody can follow that demonstrates the feature is working. This can then be used during testing, and to show off after release. Please add an entry to http://testcases.qa.ubuntu.com/Coverage/NewFeatures for tracking test coverage.

This need not be added or completed until the specification is nearing beta.

Unresolved issues

* Do we roll our own technology or do we adapt existing frameworks (like Hudson)

BoF agenda and discussion

Goal: define visualization interface for QA control that shows daily/snapshot/other summary of the 'health' of the image for a given platform

Different images based on a common base:

• server
• desktop
• netbookcurrent

Stuff we want to show:

• difference from yesterday
• performance history
• performance regressions
• performance targets (as infrastructure to see if it works during the cycle)

Dashboard mockup:

• Two columns: 1)
  • - Current build status
    • FTBFS count New package count, number of packages Latest build date/time
    - Test result - Build history
  2)
  • - News - Performance Targets

Q: What about some UI for scheculing test runs: A: we're not targeting this for the first release but we want have a UI for doing that in the future

Q: How does our project relate to other ubuntu QA projects A:

Stuff to check:

• buildbot (python) hudson (java)

Action item: check hudson out? Zygmunt Krynicki Hudson instance for Bzr at canonical: http://babune.ladeuil.net:24842/view/Ubuntu/job/selftest-jaunty/buildTimeTrend

We want to store the log file of each test run just in case (for unexpected successes)

CategorySpec