== Dev Week -- Python packaging with dh7 and dh_python{2,3} -- barry -- Mon, Jul 11th, 2011 == {{{#!irc [20:00] we'll start in just a minute or two [20:01] Logs for this session will be available at http://irclogs.ubuntu.com/2011/07/11/%23ubuntu-classroom.html following the conclusion of the session. [20:06] hello everyone and welcome to my session. today we're going to talk about packaging python libraries using dh_python2 and dh_python3 [20:06] please feel free to ask questions at any time over in #ubuntu-classroom-chat [20:06] so first some background: [20:06] there are two common legacy python helpers you'll find in various packages, python-central and python-support [20:07] python-central has been deprecated for a while, and python-support was just recently deprecated. [20:08] of course, many packages have still not been converted, and we had a jam session a week or so back where folks from the community helped convert packages on the ubuntu cds. if we have time and interest, i can talk more about those transitions [20:08] today we have the new goodness for packaging python2 stuff: dh_python2 [20:09] the only helper for packaging python3 stuff is: dh_python3 [20:10] using dh_python2 can make most of your packaging work almost too trivial. many many packages can have no more than a 3 line rules file [20:10] provided you already have a good setup.py [20:10] mhall119 asked: can you use dh_python2 for packaging on Lucid yet? [20:11] mhall119: we are working on a back port of the full toolchain for lucid. many people need this, so stay tuned! [20:12] let's look at a simple package to see what this would look like. does everybody know how to use bzr to grab branches from launchpad? [20:13] okay, here's the url to the basic, un-debian-packaged version: lp:~barry/+junk/stupid [20:13] bzr branch lp:~barry/+junk/stupid [20:14] now, this is a very simple python package, but it does have one interesting thing: it has an extension module [20:14] if you look in src, you'll see the C file containing the extension module [20:14] if you look in stupid, you'll find the python code that wraps that [20:14] notice the unit tests :) [20:15] you can take a look at the setup.py to see it's a fairly typical thing [20:15] it's got an ext_modules defined for the extension, a few other bits of metadata, and it identifies the header file [20:15] note too, the test_suite key which names the unit tests [20:16] you could install this into a virtualenv using either python2.6, 2.7, or 3.2 using the following commands: [20:16] (this should work on natty) [20:16] virtualenv -p python2.7 [20:17] oops [20:17] virtualenv -p python2.7 /tmp/27 [20:17] source /tmp/27/bin/activate [20:17] python setup.py install [20:18] python -c 'import stupid; stupid.yes()' [20:18] that should print 'yes' [20:18] then run `deactivate` to get out of the virtualenv [20:19] you could substitute the following for the -p option python2.6 or python3.2 and that would give you a virtualenv with the appropriate python [20:19] NMinker asked: What is virtualenv? Is that a Virtual Environment? [20:20] ah. virtualenv is a python tool for creating isolated development environments. with a virtualenv, you can install stuff locally for testing without affecting your system python [20:20] it's a *very* handy tool if you're working on python code [20:20] sudo apt-get install python-virtualenv [20:21] okay, so "stupid" is a simple python package with a good setup.py, and which is compatible with python 2.6, 2.7 and 3.2 [20:21] how do we turn that into a debian package? [20:22] first we have to create the debian source package from the python package, then we can upload that source package to a ppa, or build it locally with pbuilder or sbuild [20:22] i am kind of assuming folks know basic packaging stuff, like how to use pbuilder, debuild, and such.. [20:23] okay, so let's create the source package [20:23] first, i'll introduce you to a very nice, new tool which can almost always get you started quickly. [20:24] https://launchpad.net/pkgme [20:24] pkgme is actually a packaging framework. it's not tied specifically to python, although it is written in python, and supports packaging python things [20:25] it knows about other languages and such, but for our purposes, it does a great job of creating the initial debian/ directory layout based on your setup.py [20:25] i highly recommend grabbing the ppa, using these commands: [20:25] sudo add-apt-repository ppa:pkgme-committers/dev [20:25] sudo apt-get update [20:25] sudo apt-get install pkgme [20:26] once it's installed, you just run this commands from the directory containing your setup.py [20:26] pkgme [20:26] :) [20:26] that's it [20:26] now, i've done this for you, so if you don't want to install the ppa [20:26] you can just do this: [20:26] bzr branch lp:~barry/+junk/stupid.pkgme [20:27] why don't you run pkgme locally, or grab the branch. i'll give you a minute or so and then we'll look at the details [20:27] pkgme knows about dh_python2 so it does the right thing [20:28] notice that you've now got a minimal debian/ directory. yay! you have a source branch [20:28] or "packaging branch" [20:29] take a look at the debian/control file. if you have any packaging experience, you'll see this one is bare minimum, but adequate to start with [20:30] the important things to note here are that it has a proper Build-Depends: line, and it's grabbed a few meta bits from the setup.py. it's missing a description (that is because the setup.py doesn't have one, not because pkgme missed it), so you'd want to fill that out [20:30] NMinker asked: What's the difference using pkgme and dh_make to create the debian folder? [20:31] pkgme is a framework where rules can be added for more specific knowledge of particular languages, classes of packages, etc. you could use either tool, but i like where pkgme is going, and it has very good python rules [20:31] now, bring up debian/rules in your editor, because this is where the fun stuff happens [20:32] you can see, this is just a 3 line rules file essentially, and i'll step through what is happening [20:32] the first line isn't interesting, it's just standard debian packaging [20:33] ah, slight detour [20:33] jykae: noticed that could not be found, and here's why [20:34] pkgme actually didn't quite do the right thing with the Build-Depends line (yes, i will file a bug :) [20:34] it added a dependency on python-all, but because stupid has an extension module, it needs to be compiled by the c compiler. thus it needs the python-all-dev package, which includes python's own header files and such [20:35] so you would need to change the Build-Depends line to be python-all-dev [20:35] anyway, back to the rules file [20:35] the %: line is fairly standard stuff, and introduces the make target [20:35] it basically matches anything [20:35] the really fun stuff is in the next line [20:36] dh is the magical debhelper sequence, and it almost always does the right thing for python packages [20:36] (the one exception is for python3 stuff, which we'll get to later. you have to do some manual overrides for python3, but we're working on that) [20:36] the really important thing is the `--with-python2` option [20:37] that is what tells dh to use dh_python2 to build your package [20:37] in our case, it's really the only thing you need to add [20:37] what is `--buildsystem=python_distutils` then? [20:37] well, in this specific case, it's not required, but you will often want to add it [20:37] by default dh will ignore the setup.py if there is a Makefile there [20:38] stupid doesn't have a Makefile but many packages do, e.g. to add `make build` or `make test` targets for convenience [20:38] the --buildsystem=python_distutils tells dh to use the setup.py for various steps and ignore the Makefile [20:39] anyway, that's really all you need! you'll notice that pkgme adds other standard debian/ files such as changelog, compat, and copyright. that's more packaging-fu than python-packaging-fu so i'll skip over that. i.e. none of that pertains to python packaging specifically [20:40] okay, so you should be able to take that pkgme branch, debuild -S and run pbuilder to give you a nice binary package for stupid [20:40] i'll pause for a moment for questions [20:41] okay then, moving on [20:41] remember that stupid is compatible with python2 and python3, so how would we need to modify the debian/ directory so that both versions are installed? [20:41] the first thing to understand is that in debian and ubuntu, we have completely separate stacks for python2 and python3 [20:42] this means if you want a python3 version of a package, you need to install python3-foo [20:42] a good example is python-apt and python3-apt [20:42] john_g asked: So what part of the work does setup.py do and what part do the dh_ things do? [20:43] setup.py does most of the work. my recommendation is to use virtualenv and make sure your package builds, installs, and tests exactly as you want it in a python-only world (i.e. w/o debian/ubuntu getting involved) [20:43] get a solid setup.py first, using the normal python development tools. once you have that, your debian packaging job will be *much* easier [20:44] dh_python2 does the bits to lay the package out properly within the debian file system, and to ensure that byte-compilation triggers are properly invoked when the package is installed on your system [20:44] (the .pyc files are not included in the package) [20:44] so, python3 [20:45] bzr branch lp:~barry/+junk/stupid.py3 [20:45] let's first look at the debian/control file [20:45] you'll noticed i fixed the Build-Depends :) [20:45] but also notice that it b-d's on both python-all-dev (for python 2) and python3-all-dev (for python3) [20:46] notice too that i've added an X-Python-Version line and an X-Python3-Version line. this is how you control which python versions out of all that might be installed on your system, are compatible with your package [20:46] e.g. i've said that stupid is only compatible with python 3.2 and above, and python 2.6 and above [20:47] notice that i've also created two binary package stanzas, one for python-stupid and one for python3-stupid, as per the separate stack requirements [20:47] if you pull up debian/rules you'll see the additions there [20:47] we're running short on time, so i'll run through this quickly ;) [20:48] DH_VERBOSE=1 just tells dh to spew more detailed info on what it's doing. this line is not required [20:48] the PYTHON2 and PYTHON3 lines use shell helper functions to determine which versions of python are actually installed. we'll use these in the rules below [20:49] notice line 10, where all we've added was --with=python2,python3 [20:49] that invokes dh_python3 during that part of the build process [20:49] now look at lines 13-17 [20:49] i wanted to make sure that my package's tests are run during the build process, and the build should fail if the tests fail [20:50] however, i need to make sure the tests are run for every version of python we're building for [20:50] dh does not know how to do this (yet ;), so we have to add some manual rules to make this work [20:50] the test-python% lines just invoke the package's unittests with increased verbosity [20:51] the override_dh_auto_test bit is the really key for making this work, because here we're overriding dh's standard dh_auto_test call with our own. because it depends on the test-python% target, all the tests will get invoked the way we want them to [20:51] now look at lines 20-eof [20:52] one problem we have is that dh does not yet know how to properly install the python3 built parts, so we have to do this manually [20:52] thus the override_dh_auto_install [20:52] we can just call dh_auto_install to do the right thing for python2 [20:53] but then we have to manually cycle through all python3 versions and do setup.py install with some magic arguments, in order to get the python3 parts properly installed [20:53] finally, if you look in debian/ directory, you'll see two .install files. this is how you tell the packaging toolchain which files to install for which of the multiple binary packages are getting built [20:53] look at the contents of each, and you'll see how we separate the python2 and python3 stacks [20:54] okay, i'm sorry but we've nearly run out of time. i wish i could have covered more, but hopefully this was helpful [20:54] does anybody have any questions? [20:55] please feel free to use these three branch for cargo culting :) i'll leave them alone and update them as the tools improve. stupid.py3 should build fine on natty [20:55] NMinker asked: how do I convert to dh_python2? Or has that been covered? [20:55] http://wiki.debian.org/Python/TransitionToDHPython2 [20:56] NMinker: i've done many conversions with these instructions. please join us on #ubuntu-pyjam for any questions after this session ends [20:56] questions or help [20:57] also, if you want to contribute to ubuntu, i can provide some packages that still need converting. we want to remove python-central and python-support from the oneiric cds, so this is a good way to gain some packaging cred [20:58] micahg points out also this for the larger transition effort: http://people.canonical.com/~ubuntu-archive/transitions/dh-python2.html [20:59] Cuzzie asked: If we want to package up the python application we wrote, we need to write all the rules, compat, control files ourselves? [20:59] Cuzzie: read the scrollback for the pkgme tool, or look into dh_make [21:00] pkgme is an excellent tool that will get you started [21:00] and with that, i think my session is done. i will hang around and answer more questions in #ubuntu-classroom-chat }}}