Please check the status of this specification in Launchpad before editing it. If it is Approved, contact the Assignee or another knowledgeable person before making changes.

• Launchpad Entry: developer-weather-report

Summary

The goal is to provide a dashboard covering all the variables that go into determining whether Ubuntu is release ready - bug count, RC bug count, archive analysis reports, iso testing summaries, etc, linking to detailed data about each metric.

Release Note

Not applicable — this dashboard will be part of the development process, not part of Ubuntu itself. Once in a useful state, however, the page should be announced on .... and linked to from ....

Rationale

There currently exists numerous sites which display information regarding the state of Ubuntu. However, it is inconvenient having to remember multiple sites as well as navigate to multiple locations to obtain wanted information. It would be useful if this information were gathered into one central location. A dashboard will help showcase this information at a single location and provide the release managers and Distro Team a way to measure the quality of an upcoming release.

Use Cases

• Release manager Bob wants to know the current Distro release status and what showstopping bugs exist at this point in the release. He'd also like to drill down and see which of these bugs relate to the packages his team oversees. He currently has to visit multiple different locations to gather this information.
• Mary in the distro team wants to know about the iso testing status and the current number of iso-testing related bugs. This will help her determine which areas still need help testing as well as what areas are proving to be problematic. She also currently has to visit multiple locations to find this information.
• Jack noticed one showstopper in the previous build, but a new one is now available which fixes it. He wants to see the test results from both the previous build and the current build to get a sense of our confidence level for the new build.
• In the current build we are testing there is already one showstopper known. Jill knows we're going to have to respin (and may be able to see that there's an upload queued to fix it), but she wants to continue testing the current build to catch any other issues before we reset the test cycle.
• The Ubuntu desktop ISO is considered final, validated and ready to release, but the server ISO needs to be rebuilt to fix a bug. A new build has just gone up and hasn't been tested yet. Frank wants to see the test results from the previous build, but make sure it's clear that they're old.

Assumptions

• The system should scale gracefully, i.e. it should be trivial to add a new data source and incorporate it into the dashboard display
• Visual cues (e.g. color) should be used to show at a glance whether anything is outstanding
• Dashboard should provide a link to the specific metric thus enabling the capability to drill down to the information
• A distinction could be made between critical indicators (blocking) and other metrics (e.g. bug counts)

Design

Phase I (Release readiness / critical path)

The items in this table will be the initial target.

Implementation

Example Prototype (mdz)

Information Display

• The use of color will help convey the quality or severity of a metric
  • red will indicate a failure/regression
  • green will indicate success/stability
  • grey will display non-volatile information
• additional text information will be appended in ()'s
  • this can help convey versions, timestamps, etc.
  • if this becomes too cluttered we can break it out into another column
• there should also be an indication as to when the stat was last updated as not all items will be updated at the same frequency
  • rather than clutter the table, I'd prefer to have a link to a page detailing the cronjob
• some metrics will have to incorporate multiple rows (for ex. up to date iso's)

Other Logistics

• package specific data (ex. build failures) will probably want a separate report for main / restricted / universe / multiverse

  • https://launchpad.net/ubuntu/hardy/+builds?build_text=&build_state=failed

  • http://people.ubuntu.com/~ubuntu-archive/testing/hardy_outdate.txt has FTBFS+NBS+NEEDSBUILD for main

  • http://people.ubuntu.com/~ubuntu-archive/NBS/ has NBS for all components

• severity will be package dependent
• how often do we want the reports generated?
  • hourly for many components; sections regarding CDs should be more frequent
• how far back do we want these reports archived?

  • I imagine we want to be able to look back indefinitely (MartinPitt: having the current data would be enough for my sake; SteveLangasek: I can see it being useful to be able to extract trends from historical data; for being able to get full reports, I think we would never need to go more than a week back)

• will the site poll for data or is data pushed?
  • initially poll for the data as not every data source will have the capability to push data

• dashboard displayed at http://qa.ubuntuwire.com/weatherreport/ - later on this can be moved to the new qa website, qa.ubuntu.com

UI Changes

• There is a risk the interface could get very "cluttered", esp. if we wish to show historical info (graphs, etc.)

  • One suggestion was to use "sparklines" http://en.wikipedia.org/wiki/Sparkline . This displays historical trending in a concise amount of space

Test/Demo Plan

Outstanding Issues

The dashboard will evolve over several cycles. Items listed here should be developed as separate specifications.

• all data should be tracked historically, but not in the first version
  • maybe keep all output files (data files?) in bzr
  • low-priority for implementation
• XML-RPC interface? or scrape HTML
  • Scraping HTML will be very brittle
  • One suggestion was to have each source output javascript object notation (JSON)

    • Each data source can then be "syndicated" at given locations (e.g. http://.../.../datasource1.js, and update at whatever frequency makes most sense for that particular data source (hourly, daily, etc.).

    • Can still use scraping for areas that do not output this format, but have those tools each output JSON for inclusion into the interface
    • The weather report page can then re-load these objects asynchronously, so page will update itself without needing manual reload
• ideally viewers should be able to filter the stats they want to view and save(?) preferences - most likely won't happen in the initial version
• It may also help to have collapsed sections that can be expanded to get the next level of detail. For instance, one line may show total number of new, open, etc. bugs, but when expanded it shows a further breakdown (per milestone, per area, over time, or etc.) - again, most likely wont' happen in the initial version
• As quality/stability increases the color will intensify

  • Success

    =>

    Success

    =>

    Success

  • for example, if there are multiple successful builds in a row, the green will intensify in color
• Likewise as severity escalates the color will intensify

  • Failure

    =>

    Failure

    =>

    Failure

  • for example, if the number of showstopper bugs increases, the red will intensify
• set the focus of the report based on a given build

Phase II (further metrics)

The items listed here will be taken into consideration after the initial implementation (Phase I) is satisfactory with the target audience.

• livefs up-to-dateness (cdimage) - need more info: specific source of the data and what info to parse (MartinPitt: this should definitively go into Phase 1 since it is critical, and determining this is exactly the same parsing process as for alternates)

• quantity of duplicate bugs reported (Launchpad / python-launchpad-bugs)
• determine bugs of interest based on subscriber or duplicate count (Launchpad / python-launchpad-bugs)
• open bugs with patches attached (Launchpad / python-launchpad-bugs)
• percentage change of iso testing passes/fails

• merge status (http://merges.ubuntu.com/stats.txt)

  • Number of merges needed for different areas (e.g. Desktop, Xorg)
• sync status (during freeze periods, including link to diff) to see which Debian RC bugs could be fixed by a sync

• http://popcon.ubuntu.com/ - more for bug statistics

• upstream version currency (some code in http://dehs.alioth.debian.org/ and svn://svn.debian.org/qa/trunk/)

• outstanding archive requests: syncs, removals, ... (Launchpad / python-launchpad-bugs)

  • https://launchpad.net/~ubuntu-archive/+subscribedbugs?field.searchtext=sync

  • https://launchpad.net/~ubuntu-archive/+subscribedbugs?field.searchtext=remov

• Language pack status (update package versions should be identical to -base ones; IOW: update packages are empty)

• GnomeAppInstallDesktopDatabaseUpdate: is http://people.ubuntu.com/~mvo/gnome-app-install/ up to date?

• ReadaheadListUpdate: has the /etc/readahead/boot file in the readahead-list package been updated in the last week?

  • a count of lintian errors of carefully selected error classes (not everything is worth worrying about)
    • liw will be running lintian on Ubuntu, at lesat main, and will make such a selection list anyway

• automatic ubiquity test (ubiquity-vmware-automation) results (qa tracker?)

• https://launchpad.net/ubuntu/hardy/+queue?queue_state=0 - new packages to consider in release queue

• https://launchpad.net/ubuntu/hardy/+queue?queue_state=1 - release queue packages which need to go through the 7 step SRU process

• Metrics from http://qa.debian.org/developer.php?login=debian-gcc%40lists.debian.org&comaint=yes (and DEHS)

  • upstream version
  • debian version
  • bug counts
• buildd status (including age of failed build)

  • https://launchpad.net/ubuntu/hardy/+builds et al (can select by state) - some already covered in table above

• Once we have test automation infrastructure running, we'll probably want to display this info as well.

BoF agenda and discussion

Discussions were held at UDS Boston 2007 and the comments have been incorporated into the above spec.

---

CategorySpec