• Launchpad Entry: foundations-lucid-robust-python-packaging

• Created: MichaelVogt

• Contributors:

• Packages affected: python-central

Summary

We do a lot of magic currently in postinst/preinst scripts for Python packaging. This has caused problems before. It is also hard to debug: unpack/configure ordering makes it hard to reproduce. Users have a hard time diagnosing, too.

Release Note

The python packaging system is now more robust, easier to inspect and more user-friendly.

Rationale

Design guidelines for the proposed changes:

• do building/compiling on the buildds
• catch errors on the buildds (and not on the user machines)
• let dpkg do the shuffling of files
• let dpkg do (file) conflicts checking
• as little complexity in the maintainer scripts as possible
• don't force people learn new tools if something is wrong (dpkg -L python-foo should be enough)
• as little work for the archive-admins as possible
• python application keep working during unpack/configure

User stories

Jane perform a upgrade from 8.10 to 9.04 and wants to play a game during the (boring) upgrade. This does not work because right now and she is disappointed. With the 9.04 to 9.10 upgrade it does work and she is much happier.

Design

There are currently two python packaging system implementations. For pycentral we need to tweak the way its used. For python-support we need to start a dialog with the maintainer and see if we can agree on a good way forward.

In detail:

• python-central: use DH_PYCENTRAL=include-links by default (downside: pkg needs a rebuild on a new python version)
• python-support: talk to python-support upstream about a patch that has a similar
  • as semantic DH_PYCENTRAL=include-links (and provide that patch if needed)
• tackle the problem of relative imports (and ask doko to provide more details)
• tackle the problem that pysupport adds a additional distro site-packages dir (and find out the details why that is problematic)
• clarify what supported, old-versions and unsupported in /usr/share/python/debian_defaults means

Implementation

For python-central the default is switched to "include-links". No further code changes are required. We need to also add (regression)tests for "pycentral pkginstall foo" and "pycentral pkgremove foo".

For python-support we need to start a dialog with the upstream maintainer and suggest to have "include-links" like semantics by default. With the new karmic version of python-support packages will keep working between the --unpack and --configure phase.

Migration

The migration will happen automatically as python packages get rebuild. For each new python major version (e.g. 2.7) a no-changes rebuild is required with the include-links schema.

Test/Demo Plan

Testing this change should be part of the regular upgrade testing process. We need to check for upgrade failures (that may be caused by this) and do some functional testing after the upgrade.

BoF agenda and discussion

mvo

NOTES

• keep in mind to be in sync with debian
• debian uses python-support

history why we have the current policy

• rebuild packages from source when python version changed
• plus the packaging had to be changed in debian/control
  • (python2.5-foo) on new python versions
• between versions python byte code changes
• C ABI changes with new versions of python
• we used to have "python-foo" source package, that created
  • python2.4-foo, python2.5-foo
• rebuilds required when new python is released or dropped
• archive-admins had to process those packages in the NEW queue
• (debian-release-managers had problems with the testing distro)
• GOAL: get rid of version information in the package source
  • (i.e. apps just depend on python-foo instead of python2.4-foo)
• simplify the job of the resolver (because there is just python-foo

  • not python2.3-foo -> python2.4-foo)

• some of the original requirements (problems with NEW processing,
  • testing distro, ...) have disappreared

problem space

• keep packages working during the upgrade
  • - symlinks got created during preinst so that they were available during
    • unpack (around ~interpid)
    - problematic for: update-manager, apport, hplip, python cron jobs - no longer used
• in intrepid we have DH_PYCOMPAT=nomove (multipiles the size of the
  • package by the number of python versions)
• in jaunty we have DH_PYCOMAPT=include-links (just include the
  • symlinks, not the .pyc files)

problems

• two packaging helper tools (python-support, python-central)
• python-gtk (extension) depends on python-gobject (extension)
  • - now if python-gtk builds against 2.4,2.5 but python-gobject
    • just builds against 2.4 then python-gtk breaks, but the dependencies still correct
    - python-gobject uses: Provides: python2.5-gobject, python2.6-gobject - and python-gtk depends on python2.6-gobject
    • one solution is to do the uploads in reverse order, but apparently this is not understood and no common practice.
• when adding a new version: upload new python, rebuild packages with
  • dependency chain order bottom up
• when removing a version: rebuild packages from top down without the
  • version and then remove the version
• going back python2.X-foo might be problematic for the ftp-master
  • team (NEW queue processing overhead, soyuz overwrite handling, debian/control needs to be touched) - all of this could be automated
• RELATIVE IMPORTS

possible solutions/improvements

• include DH_PYCENTRAL=include-links by default
  • - requires rebuild when a new python version comes out
    • (or one runs out of support) - just changelog entry change is enough
    - does not work with python-support so far (needs discussion
    • with python-support upstream)
    - *.pyc building is still done in maintainer scripts

action items

• use DH_PYCENTRAL=include-links by default
  • (downside: pkg needs a rebuild on a new python version; slangasek objected to this, probably should be only done for packages in main/on the CD)
• talk to python-support about a patch that has a similar
  • as semantic DH_PYCENTRAL=include-links
• tackle the problem of relative imports (and ask doko to provide
  • more details) - pysupport adds a additional distro site-packages dir that needs to be dropped relative imports break, if a python package (in the python sense) is packaged with different packaging helpers, resulting in files splitted on two different locations on sys.path. To avoid this situation all public modules must be installed in one location.

liw

We do a lot of magic currently in postinst/preinst scripts for Python packaging. This has caused problems before. It is also hard to debug: unpack/configure ordering makes it hard to reproduce. Users have a hard time diagnosing, too.

mvo would like to simplify Python packaging.

Recap of why the current code is in place (to avoid flip-flopping between problems):

- in intrepid: tried to make time when packages are non-working as short

• as possible: create symlinks in preinst script, so they're there when dpkg unpacks package (preinst is run almost immediately before unpacking) giving python modules and extensions being immediately available; this is useful so that packages like update-manager and other packages used during an upgrade are usable - hplip, apport can also break during an upgrade

- originally (before Ubuntu!) Debian had to rebuild python modules when

• a new python version was added to the archive - also required changing the source package (at least debian/control)

- python is ugly when it comes to (system) upgrades

• - upstream changes bytecode format and other things between releases
  • - abi for extensions
  - thus, have to rebuild with new python release - also, when dropping a python release from a release

- we packaged a module as python2.2-foo - sabdfl always listens! - 2006: the goal was to get remove all hardcoded python versions in the

• source package - also removing around some processing in Debian and Ubuntu (NEW queue) - also simplified apt's computation for upgrades

- Debian now no longer has problem with packages moving into testing,

• or NEW queue, but other problems

- there is no real formal Python policy that applies to this time/space

• continuum

- building for old and new versions of python eases transitions to Debian

• testing but also for users dist-upgrading
  • - impractical to have newest (default?) python version from previous
    • (LTS) release in new release

- creating symlinks in preinst is rejected by Debian and is pretty fragile - in debconf6 @ mexico some design was whiteboarded which used

• depends+provides - is this current policy? - slangasek: an app that has to depend on non-default version of python
  • should depend on the modules it requires (pythonX.Y-foo), but only apps, not libs/modules/extensions - rejected by Manoj @ Debian - some controversy about this, in the room and in Debian - I'm quite confused by this discussion

- few actual improvements to the situation, we ran out of time - using the same package names for real and virtual packages causes problems

• - but splitting name spaces causes problems too

Python-team in Debian has (practically) decided to standardize on python-support, and to avoid python-central.

proposal for just one packaging tool; emerged from discussion at Debconf9: http://lists.debian.org/debian-python/2009/08/msg00003.html

---

CategorySpec