Ubuntu Wiki

Ubuntu Open Week - Packaging 101 - Daniel Holbach - Tue, Nov 4th, 2008

(11:01:17 AM) dholbach: HELLO MY FRIENDS! :-)
(11:01:26 AM) ***sebner hugs dholbach =)
(11:01:31 AM) dholbach: Who's here for the "Packaging 101" session? :-)
(11:01:35 AM) alexharrington: me
(11:01:39 AM) lvzimmer: me
(11:01:43 AM) ***lobo-ptr puts his fiiiiiiiinger up
(11:01:44 AM) radix: hi :-)
(11:01:45 AM) kippy: +1
(11:01:51 AM) charlie-tca: me
(11:01:52 AM) Webspot: yup
(11:02:01 AM) Net: Clap, clap, clap!
(11:02:28 AM) dholbach: and I guess the rest of the 237 people in here are just too shy to say "me me me!" :-)
(11:02:32 AM) chillitom: Cheers LaserJock.. Good luck!
(11:02:47 AM) dholbach: alright... let's get cracking
(11:02:48 AM) ***stefanlsd hugs dholbach!
(11:02:52 AM) niemeyer: dholbach: Thay all agree
(11:02:58 AM) ***dholbach hugs stefanlsd back :)
(11:03:03 AM) RainCT: heh
(11:03:36 AM) Net left the room.
(11:03:53 AM) ***sebner feels ignored by dholbach :P
(11:04:04 AM) dholbach: my name is Daniel Holbach, I've been member of the MOTU team for quite a while, have been involved in all kinds of teams but what I always got back to was: listening to new contributors, trying to find out what could make their lives easier and hear lots of enthusiasm in our ubuntu development community every day
(11:04:09 AM) ***dholbach hugs sebner back :)
(11:04:17 AM) sebner: :)
(11:04:25 AM) dholbach: as you can see (first lesson maybe): we hug a lot in the Ubuntu community :-)
(11:04:47 AM) dholbach: so what are we going to do in the session?
(11:05:05 AM) dholbach: we'll take a look at a simple package, identify all the key parts and I'll try to answer all the questions that come up
(11:05:20 AM) dholbach: these key parts you will find in any package you might touch later on
(11:05:40 AM) dholbach: there's one tool we're going to need, so please run
(11:05:42 AM) dholbach:   sudo apt-get install devscripts
(11:05:54 AM) dholbach: and afterwards:
(11:05:58 AM) dholbach:   dget -xu http://daniel.holba.ch/motu/hello-debhelper_2.2-2.dsc
(11:06:17 AM) dholbach: please let me know in #ubuntu-classroom-chat if you have any questions or run into trouble
(11:06:45 AM) ciprian_topala: m2a
(11:08:19 AM) dholbach: what this does is: get the source package of hello-debhelper (eqivalent of 'apt-get source hello-debhelper' - I just wanted to make sure we all have the same source package we're looking at)
(11:08:19 AM) dholbach: (you might need to run    dget -x <url>   if you're on hardy) :)
(11:08:37 AM) dholbach: if you look at your current working directory, you'll see three files and a directory
(11:08:45 AM) dholbach: ... at least ... :)
(11:09:01 AM) dholbach: hello-debhelper_2.2.orig.tar.gz is the unchanged upstream tarball that was released on the GNU FTP page
(11:09:43 AM) dholbach: hello-debhelper_2.2-2.diff.gz is the compressed patch (set of changes) we need to apply to make the source package build in the debian/ubuntu world
(11:10:08 AM) dholbach: hello-debhelper_2.2-2.dsc contains some metadata like md5sums, etc
(11:10:33 AM) dholbach: the   hello-debhelper-2.2   directory is the extracted tarball plus the applied packaging changes
(11:10:37 AM) dholbach: <demas_> QUESTION: When I install devscripts, apt-get install Exim.  Why?
(11:11:05 AM) dholbach: demas_: it's pulled in via some recommends - you can safely remove it (or install without recommends - can somebody help demas_?)
(11:11:25 AM) demas_: thanks
(11:11:35 AM) dholbach:   cd hello-debhelper-2.2
(11:11:59 AM) stefanlsd: demas_: I like to install ssmtp first instead. exim wont be pulled in then.
(11:12:00 AM) RainCT: demas_: try with --no-install-recommends
(11:12:11 AM) dholbach: if you check the contents of the directory you will see that it looks and feels like a regular source tarball you downloaded
(11:12:20 AM) dholbach: with the exception of the    debian/   directory
(11:12:23 AM) dholbach:   cd debian/
(11:13:05 AM) dholbach: this directory contains all the files that are needed to make the package build "our way"
(11:13:09 AM) dholbach: what does that mean?
(11:13:42 AM) dholbach: hello-debhelper is a simple project written in C, if you'd build it yourself, you'd probably run something like: ./configure --make-it-extra-fancy; make; sudo make install
(11:14:13 AM) dholbach: if it was a python project, you'd probably run:   python setup.py build;   sudo python setup.py install  or some such
(11:14:17 AM) dholbach: etc. etc.
(11:14:47 AM) dholbach: a lot of projects require different methods to make them build or install stuff to some directory
(11:15:06 AM) dholbach: in the Debian / Ubuntu packaging world we just apply one build process, for all kinds of packages
(11:15:31 AM) dholbach: ok, let's check out the contents one by one
(11:15:55 AM) dholbach: changelog is what the name says: it describes all the changes that were made
(11:16:11 AM) dholbach: but only the changes necessary for the packaging - it's not an upstream changelog
(11:16:29 AM) dholbach: if you look at the top line you can see
(11:17:03 AM) dholbach: <source package name> (<upstreamversion>-<debian/ubuntu revision>) <distro-release>; urgency=<something>
(11:17:26 AM) dholbach: upstream version is 2.2
(11:17:31 AM) dholbach: debian revision is -2
(11:17:44 AM) dholbach: that means 2 revisions of the upstream 2.2 have been uploaded to debian
(11:17:49 AM) dholbach: it was not modified in Ubuntu at all
(11:18:01 AM) dholbach: (you can ignore the urgency setting, we don't use it in Ubuntu)
(11:18:17 AM) catonano: sorry in the changelog, the first rows are like this, to me:
(11:18:17 AM) catonano: 2006-11-23  Karl Berry  <karl@gnu.org>
(11:18:17 AM) catonano:         * Version 2.2.
(11:18:17 AM) catonano:         
(11:18:17 AM) catonano:         * Makefile.am (po-check): add utility target (from coreutils).
(11:18:32 AM) dholbach: catonano: we're in the  debian/  directory
(11:18:42 AM) dholbach: catonano: take a look at debian/changelog
(11:18:43 AM) catonano: oh yes, sorry
(11:19:05 AM) dholbach: but you're right... ChangeLog is the upstream changelog (containing information from the software authors)
(11:19:33 AM) dholbach: every changelog entry should contain useful information about what was changed
(11:19:43 AM) dholbach: this is very important, especially in Ubuntu
(11:20:17 AM) dholbach: we maintain all packages as a team - even if you don't mind figuring out what the hell you changed half a year ago, it's very nice if you colleagues and friends don't have to guess what
(11:20:30 AM) dholbach:   * did some changes to make it work again
(11:20:33 AM) dholbach: means :)
(11:20:52 AM) dholbach: also it contains the timestamp of the change and who changed it
(11:21:15 AM) dholbach: we have a fancy tool in devscripts called  dch  that automatically adds templates for you, so you don't have to figure it out yourself :)
(11:21:50 AM) dholbach: we can ignore the  compat  file, it's not so interesting (compatibility setting for debhelper, that was used for the packaging)
(11:22:01 AM) catonano: ahem...I'm sorry I have to ask a stupid question again :-(
(11:22:18 AM) dholbach: catonano: no problem - would you mind asking in #ubuntu-classroom-chat the next time? :)
(11:22:22 AM) dholbach: catonano: shoot
(11:22:40 AM) catonano: I can't find the debian folder. This is what I see:
(11:22:40 AM) catonano: adriano@adriano-laptop:~/debhelper/hello-2.2$ ls
(11:22:40 AM) catonano: ABOUT-NLS   ChangeLog    configure.ac  INSTALL      NEWS    tests
(11:22:40 AM) catonano: aclocal.m4  ChangeLog.O  contrib       Makefile.am  po      THANKS
(11:22:40 AM) catonano: AUTHORS     config.in    COPYING       Makefile.in  README  TODO
(11:22:41 AM) catonano: build-aux   configure    gnulib        man          src
(11:22:43 AM) catonano: adriano@adriano-laptop:~/debhelper/hello-2.2$
(11:23:21 AM) dholbach: catonano: go back to the directory with the .dsc .diff.gz and .orig.tar.gz files
(11:23:31 AM) catonano: I'm there
(11:23:34 AM) dholbach: and run  dpkg-source -x hello-debhelper_2.2-2.dsc
(11:23:54 AM) dholbach: it's the magic command that   dget -x(u)   runs for you
(11:24:07 AM) dholbach: alright... let's crack on
(11:24:23 AM) dholbach: the next file is important and maybe a bit less obvious
(11:24:25 AM) catonano: thanks, I saw that
(11:24:42 AM) dholbach:   control   contains information about the source package and the binary packages
(11:24:46 AM) dholbach: so what's the difference?
(11:25:08 AM) dholbach: the source package comprises of the .diff.gz .dsc and .orig.tar.gz files and is what we need to hack on packages, it contains source
(11:25:26 AM) dholbach: the binary packages are the .deb files that my mother does not know about, but installs and updates regularly :)
(11:25:42 AM) dholbach: <demas_> QUESTION: Where can I find more information about dch? Do I need to launch this script or it is a part another script?
(11:25:59 AM) dholbach: demas_: just run   man dch   - I'll give you a couple of links for more infos and tutorials later on
(11:26:09 AM) demas_: thanks
(11:26:17 AM) dholbach: alright... control file
(11:26:30 AM) dholbach: if you take a closer look at it, you will see that it's split into two stanzas
(11:26:37 AM) dholbach: the first one is always about the source package
(11:26:57 AM) dholbach: and the following ones (in our case luckily just one) are about the resulting binary packages
(11:27:56 AM) dholbach: the source package definition contains: the name of the source (usually just what the upstream developers decided to call it), some section, some priority and a maintainer field
(11:28:17 AM) dholbach: these should be pretty obvious (the debian policy explains which values are allowed for the priority and the section)
(11:28:38 AM) dholbach: Standards-Version describes which version of the debian-policy the package complys with
(11:28:45 AM) dholbach: in this case it's 3.7.2
(11:29:05 AM) dholbach: the next line is very interesting: it tells us which packages are required to build the package
(11:29:27 AM) dholbach: this is nothing my mother has to install to get the all the goodness that hello-debhelper is, it's what we need to build the package
(11:29:41 AM) dholbach: and also what the build machines in the data center need to build the package
(11:30:05 AM) dholbach: so what happens when I upload a source package to the build machines?
(11:30:29 AM) dholbach: first it will check the GPG signature to see if it was really me who uploaded it, it will check if it knows about me at all
(11:30:53 AM) dholbach: then unpack the source package, then enter a minimal environment (that contains nothing but just a few build tools), then install the build-depends
(11:31:06 AM) dholbach: this is to make sure that it builds in a clean environment
(11:31:17 AM) dholbach: you will explicitly have to point out what's required :)
(11:31:28 AM) dholbach: <cyphermox> QUESTION: what is the Standards-Version used for? Won't lintian complain in different contexts, about the Standards-Version you use?
(11:31:39 AM) syslogd_ is now known as syslogd
(11:32:07 AM) dholbach: cyphermox: if you're the package maintainer and you learn that the new debian policy requires something, you will make the necessary fixes and update the Standards-Version
(11:32:36 AM) dholbach: for those of you who don't know: lintian is a tool that checks for all kinds of mistakes in packaging, it's seriously good stuff
(11:33:04 AM) dholbach: cyphermox: lintian will always want you to use the newest debian-policy it knows about :)
(11:33:16 AM) cyphermox: alright!
(11:33:25 AM) dholbach: but bumping the standards-version just for the sake of bumping it is unnecessary and you shouldn't do an upload of a package just for that :)
(11:33:39 AM) dholbach: <wolfger> QUESTION: What would Depends look like if we had some real dependencies?
(11:33:44 AM) dholbach: wolfger: I'll get to the depends in just a sec
(11:33:56 AM) dholbach: let's get to the binary package section now
(11:34:13 AM) dholbach: the name is pretty obvious: it's the name of the package that my mother might install
(11:34:44 AM) dholbach: Architecture is interesting: in our case it's "Any" and will tell the build machines to build this on every architecture that's available in the data center
(11:34:54 AM) dholbach: so i386, amd64, powerpc, hppa, lpia, sparc, etc
(11:35:13 AM) dholbach: this is necessary if you have architecture dependent code
(11:35:24 AM) dholbach: compiling C code on i386 and on amd64 will give you different results
(11:35:54 AM) dholbach: if your package just needs a few python scripts or just ships some HTML files (something-doc package), you will set  Architecture: all
(11:37:05 AM) dholbach: which means: all architectures will use one and the same package (and the package gets only built on one build machine)
(11:37:09 AM) dholbach: now we get to "Depends:"
(11:37:24 AM) dholbach: this is the packages you need to install to make the package work
(11:37:26 AM) dholbach: in our cases it's pretty funny: ${shlibs:Depends}
(11:37:29 AM) dholbach: this doesn't look like a package name, right you are :)
(11:37:39 AM) dholbach: it's a substitution variable
(11:38:01 AM) dholbach: ${shlibs:Depends} will at the end of the build be replaced with all the names of packages that contain libraries that the binary files in hello-debhelper are linked against
(11:38:18 AM) dholbach: I hope that was not too cryptic :)
(11:39:12 AM) dholbach: so if /usr/bin/hello in the resulting hello-debhelper package is linked against /lib/libc.so.6 the package will automagically depends on libc6
(11:39:28 AM) dholbach: <bhk_f> dholbach: QUESTION: "Architecture: all" Vs. "Architecture: any" difference ?
(11:40:37 AM) dholbach: bhk_f: I explained it above, in short: "any" => build on every release architecture because of architecture dependant resulting code (C, C++, etc, etc), "all": all architectures use the same package, architecture indepedent
(11:40:39 AM) dholbach: <poef> QUESTION: If I modify someone elses package and build new sources (and binaries), should I use Original-Maintainer or XSBC-Original-Maintainer? What's XSBC?
(11:41:27 AM) dholbach: poef: that's an Ubuntu speciality. our friends at Debian asked us to change the maintainer fields whenever we do changes so they don't get mails from our users about bugs, etc
(11:41:51 AM) dholbach: https://wiki.ubuntu.com/DebianMaintainerField has more info about that and the  update-maintainer  tool (ubuntu-dev-tools package) is useful for that
(11:42:00 AM) dholbach: XSBC stand for:
(11:42:07 AM) dholbach: X = custom header entry
(11:42:24 AM) dholbach: S = add it to the source package
(11:42:29 AM) dholbach: B = add it to the binary package
(11:42:35 AM) dholbach: C = add it to the .changes file
(11:42:45 AM) dholbach: <demas_> QUESTION: Can I rely on ${shlibs:Depends} variable or sometimes I need to fill this field manually
(11:43:06 AM) dholbach: demas_: yes, it's really good stuff and you should never have to explicitly list library package names
(11:43:31 AM) dholbach: because sometimes library package change their names (ABI breakage, etc), in those cases you can just rebuild the package and it will have the new library package name as a resulting depends
(11:43:37 AM) dholbach: it's REALLY good stuff
(11:43:46 AM) dholbach: <demas_> QUESTION: How system can know really packages to replace Depends variable?  Where can it find this information?
(11:44:52 AM) dholbach: I can't go into too much detail here about it, but if you check the output of  ldd <some binary>  you will see the libraries it's linked against
(11:47:22 AM) dholbach: the library packages contain SHLIBS information, which will show which library with which soname relates to which library package
(11:47:22 AM) dholbach: this is automatically generated as well
(11:47:22 AM) dholbach: <demas_> QUESTION:  For example, I create python program which uses sql-alchemy.  How Depends variable can understand that my package depends on the python-sqlalchemy package ?
(11:47:22 AM) dholbach: demas_: unforunately there is no automagic for python
(11:47:24 AM) dholbach: demas_: ${python:Depends} will just give you the current python versions
(11:47:25 AM) dholbach: alright... let's move on :)
(11:47:33 AM) dholbach: Conflicts: hello
(11:47:33 AM) dholbach: Provides: hello
(11:47:33 AM) dholbach: Replaces: hello
(11:48:01 AM) dholbach: these entries basically make sure that hello and hello-debhelper are not installed at the same time
(11:49:24 AM) dholbach: they are both the same examples, just one time built with debhelper (packaging tool) and one time without it
(11:49:24 AM) dholbach: the rest of the   control   file is just a short and along description which you will see in aptitude/synaptic/adept/etc.
(11:50:10 AM) dholbach: I picked an easy example for this session, but there are packages the build like literally 100 binary packages from one source
(11:50:11 AM) dholbach: just for kicks, try out
(11:50:16 AM) dholbach:   $ apt-cache showsrc mono
(11:50:26 AM) dholbach: take a look at the "Binary:" section
(11:50:30 AM) dholbach: :-)
(11:50:38 AM) dholbach: <bhk_f> QUESTION: last time i packaged a python app, it needed a setup.py before i could debianize it, so thats twice the effort, even ubuntu wiki suggested this way, is this the only recommended approach ?
(11:51:26 AM) dholbach: bhk_f: unfortunately there are lots and lots of different ways to package something - I like using distutils (in setup.py) because it makes writing debian/rules trivial, but there are other ways for sure
(11:51:26 AM) dholbach: let's move on, we're running late :)
(11:51:31 AM) dholbach: the  copyright  file explains who the authors of the package are, what licenses each and every part are under and who the copyright holders are
(11:52:05 AM) dholbach: getting all this information is time-consuming and something we shouldn't take lightly
(11:53:02 AM) RainCT: bhk_f: if you use cdbs but there's no setup.py you can just ommit the python-distutils.mk include and call dh_pycentral/dh_pysupport directly in the binary-install/<pkgname>:: target (and more or less the same is true if you don't use cdbs)
(11:53:02 AM) dholbach: it's important to get this right before you upload the package, because the archive admins look at every bit and make sure that we can ship it at all and that you got all the licensing bits right
(11:53:25 AM) dholbach: saying "but COPYRIGHT said it was GPLv3" is not enough
(11:53:25 AM) dholbach: you need to check all the source ane make sure you listed all the bits that were "borrowed from somewhere else" as well
(11:53:25 AM) dholbach: <cyphermox> dholbach: QUESTION: in the mono example, what does dfsg stand for in the Version line: Version: 1.9.1+dfsg-4ubuntu2
(11:53:25 AM) dholbach: cyphermox: DFSG stands for debian free software guidelines - this is something we follow as well
(11:53:44 AM) dholbach: sometimes we have to remove certain bits from the upstream code we're shipping to comply with the DFSG
(11:53:55 AM) lvzimmer: ll
(11:54:10 AM) dholbach: in this case   1.9.1   is not the "upstream version number", but "1.9.1+dfsg"
(11:54:29 AM) dholbach: sometimes just having a chat with the upstream developers about it is enough to get the issue resolved
(11:55:41 AM) cyphermox: ok.
(11:55:41 AM) dholbach: in the particular case of mono, I don't know what the issue was
(11:55:41 AM) dholbach: ok, let's come to the last piece of the puzzle: debian/rules
(11:55:41 AM) dholbach: this is the heart of the packaging process and it would take very long to go through it very carefully
(11:55:41 AM) dholbach: basically it's a Makefile with specific targets
(11:55:41 AM) dholbach: the clean target is similar to what "make clean" would normally do
(11:55:49 AM) dholbach: the install target takes care of installing the resulting files into the right places
(11:56:08 AM) dholbach: you can see that we run ./configure as well, etc.
(11:56:19 AM) dholbach: note all the   dh_*  calls towards the bottom of the file
(11:57:05 AM) dholbach: these are all debhelper commands that make packaging very simple, common packaging tasks: like compress the changefiles, update the database of menu entries, etc are done almost automatically for us
(11:57:15 AM) dholbach: there are toolkits like CDBS that simplify this Makefile even more
(11:57:38 AM) dholbach: if you're interested in packaging and getting involved, please check out https://wiki.ubuntu.com/MOTU/GettingStarted
(11:57:45 AM) dholbach: it links to the packaging guide (including tutorials)
(11:59:17 AM) dholbach: it links to tutorial videos on youtube
(11:59:19 AM) dholbach: it explains how to get in touch with developers and lists simple tasks you can get started with
(11:59:20 AM) dholbach: <dsestero> QUESTION: I am considering packaging Java J2EE applications. Is that a viable route? Are there any special steps needed to package this kind of applications that run in a container like JBoss AS? Are there any tutorial you would suggest?
(11:59:20 AM) dholbach: dsestero: definitely, watch out for the next Ubuntu Java meeting - there are people who can explain much better what's required for that
(11:59:21 AM) dholbach: <hophet> QUESTION: I packed a binary tool from ibm, but i havent the source code. Can i put it in the tree? (Sorry by my poor english)
(11:59:30 AM) dholbach: hophet: if the resulting binary is redistributable at all, it might get into multiverse, I'm not sure how likely that is - it really depends on the licensing terms
(11:59:38 AM) dholbach: getting it to build from source would be MUCH MUCH better
(12:00:06 PM) dholbach: OK my friends... thanks a lot for this great session - I had a lot of fun and hope so did you
(12:00:12 PM) dholbach: I hope to see you all in #ubuntu-motu soon
(12:00:24 PM) dholbach: jaunty is   Y O U R   development cycle - make me proud :-)
(12:01:22 PM) Nuc134rB0t: thanks dholbach for this session
(12:01:34 PM) dholbach: thanks Nuc134rB0t - I'm glad you're happy :)
(12:01:42 PM) ***RainCT hugs dholbach :)
(12:01:46 PM) ***dholbach hugs y'all!