Changes between Initial Version and Version 1 of TalkPackaging

03/13/10 09:37:49 (11 years ago)
David Fraser

Added notes for talk this afternoon


  • TalkPackaging

    v1 v1  
     1= Talk on Packaging 2010-03-13 =
     3These are the notes from a talk at [wiki:Meeting20100313] by David Fraser on Python packaging for Fedora, RHEL, Windows, etc
     5== General Packaging ==
     7 * We have source code for Python libraries, applications and tools
     8 * These have dependencies on other Python libraries, as well as non-Python libraries (database drivers etc)
     9 * We need users to run our applications etc on  various platforms
     11=== Terms of Endearment ===
     13 * ''Distribution'' here refers to a built Python thing
     14 * ''Distro'' refers to a Linux distribution
     15 * ''Package'' refers to a Python package (a module with submodules)
     16 * ''rpm'' refers to a package file in the RPM format (to distinguish from Python packages)
     18=== Distutils ===
     20 * [ distutils] is the standard Python packaging system
     21 * Various extensions exist that augment its capabilities, and because you use Python, you can do that to
     22 * Input: you write a `` script (and optionally a `setup.cfg` configuration file) that specifies:
     23   * Package metadata (name, version, authorship, licensing, dependencies, etc)
     24   * Package contents (pure python modules/packages, extensions, executable Python scripts, ''data'' - non-code that needs to be in the package, scripts that aid installation etc)
     25 * Output: running `` supports various commands that produce different outputs:
     26   * `sdist` produces a source distribution - it helps if you can regenerate the distribution from a clean source distribution
     27     * `` template `->` `MANIFEST` list of files to include - these are not regenerated automatically by default (`--force-manifest`)
     28   * `bdist` contains a generic ''built'' installation (but I've hardly ever seen it used as it's not a useful package format)
     29   * `bdist_rpm` generates RPM and SRPM package files
     30   * `bdist_wininst` produces a Windows executable installer that will install the distribution into an existing Python installation
     31   * The output generally gets put in a `dist` subdirectory
     32 * Commands include the above output commands, as well as intermediate commands that are run on the way to producing the above
     33   * Each command can have command-specific options (as well as the generic options passed to ``)
     34   * Options can be specified per-command either in `` or in `setup.cfg`, or on the command-line
     35   * `build` is an intermediate command which runs:
     36     * `build_py` for pure modules (simple copy to the build directory)
     37     * `build_ext` for compiling C/C++ extensions, links them to build directory
     38     * `build_clib` for building C/C++ libraries
     39     * `build_scripts` which copies them and alters the `#!` line
     40   * `install` then installs everything from the build directory to the target (separate steps like `build`)
     41   * `clean` cleans up the build directory
     42   * You can also `register` the command with the Python package index, and `upload` it
     43 * Scripts can include things to run (which can be installed into the path), as well as a post-installation/pre-removal script
     44 * Things to consider:
     45   * If you've got a bunch of related files, should they be in a package? Otherwise they can clutter the standard Python `site-packages` directory
     46   * Can you cleanly regenerate your source distribution from itself
     47   * Package dependencies: a distribution can provide, require or obsolete packages
     48   * Consider creating distributions on the same platform as you're targetting, or from other platforms, or both
     49   * How are you going to build your distribution for different platforms?
     50   * How are you going to deliver your distribution to people on different platforms, including dependencies?
     51   * How will your distribution interact with the native packaging on the target platforms (if any)
     53=== The world of eggs ===
     55[ setuptools] is a set of extensions to `distutils` that try and bring it into the modern age:
     56 * Adds proper dependency support to Python packages
     57 * Lots of surrounding tools - `easy_install`, `pkg_resources` etc - very simple ways of installing stuff from the standard Python repositorya
     58 * Lots more tools being built around this format
     59 * Simply import `setuptools` instead of `distutils`
     60 * Does not integrate directly with distros' packaging systems
     61 * Supports parallel versions of the same library reasonably well (you can `require` a specific version and use it even if a different one is present - however conflicts can arise)
     62 * Not that good at uninstallation etc
     64=== The world of RPM ===
     66 * [ RPM] is a packaging format that originated with [ RedHat], and is now used in [ RHEL]/[ CentOS]/[ Fedora], [ SUSE Enterprise]/[ openSUSE], [ Mandriva] as well as being part of the [ Linux Standard Base]
     67 * Input: A `.spec` file defines a rpm's metadata, how to build both a source and a binary rpm, what sources to use, patches to apply, etc, etc
     68   * This is a more data-driven format than ``
     69   * The `.spec` file contains a header section for general information as well as
     70     * script sections for `%prep`aring the build, `%setup` (unpacking the source and packaging), `%build`ing the source, `%install`ing (into a `BUILD_ROOT` directory`), `%check`ing the results, and `%clean`ing up
     71     * `%pre` and `%post` installation scripts, and `%preun` and `%postun` uninstallation scripts
     72     * A list of the `%files` included in the rpm (including categories like `%doc`umentation, and file attributes)
     73     * A `%changelog`
     74   * Support for macros, including shell execution to define macros
     75   * Support for subpackages - multiple rpms can be built from the same `.spec` file
     76 * From `distutils`, `bdist_rpm` generates a source distribution, creates a spec file, generates a source rpm from that, and then a binary rpm
     77 * What's really useful about `rpm` is dependency based on repositories
     78   * Different distros have different tools for this, and they vary in capability between releases - `RHEL/CentOS/Fedora` use [ yum]
     79   * There isn't that much functional difference between the `.deb` format/the `.rpm` format, and `apt-get`/`yum` etc - for the user. There is for the packager...
     80   * You can specify dependencies as options to `bdist_rpm` (in ``, `setup.cfg` or on the commandline) - they don't seem to get included from the normal package metadata
     81 * Distros/repositories have different standards and requirements for inclusion of rpms and `.spec` files
     82   * `bdist_rpm`'s automatically generated `.spec` files will generally not meet these requirements
     83   * The general feeling is that for inclusion into a distro, `.spec` files should be hand-generated and maintained
     84   * Are you targeting your rpms for inclusion in a Linux distro? Read and follow the rules and follow the procedures...
     85   * Otherwise you may be happy with the standard `distutils` stuff
     86   * Generally packages using `setuptools` can install their `.egg` information alongside the source code and include that in the rpm
     87 * Targetting older distros can be tricky if you are using lots of modern Python stuff
     88   * Distros have their own version of Python - you may require a newer one. Typically this can be installed alongside as something like `python25.rpm` - often these rpms exist, sometimes you have to rebuild them
     89   * In that case you will need to package all the dependencies for the new Python - usually called `python25-babel` etc
     90   * Doing this on multiple distros can be tiring. Since you're not targetting inclusion in the old distros, try and get away with murder (or at least, functional packages rather than beautifully crafted ones)
     91 * We have a tool called [ centuryegg] for targetting older distros
     92   * This is currently reasonably specific to our set of requirements
     93   * Order of priority:
     94     * Use existing `rpm`s from the target distro
     95     * Backport `rpm`s from newer releases of the target distro
     96     * Spin our own `rpm`s from the eggs in PyPI
     97   * Target is to be able to automatically source and download all the requirements from a simple list, generate the rpms, and upload them into a repository
     98   * We should make it more generic - is anyone interested?
     99 * Generating your own repository
     100   * You will generally need different repositories for different distros and versions (even apparently equivalent ones like `RHEL4`/`CentOS 4`)
     101   * Usually this just involves compiling (strongly recommend doing this ''on the target distro'' - we had strange crashes due to minor library versioning differences etc)
     102   * You then just need to run something like `createrepo` and put the files in a web space
     104=== The world of Windows ===
     106 * `bdist_win32` is fine for lots of purposes - especially if distributing packages to other developers
     107 * `bdist_msi` has now also been added, that produces packages in Windows installer's MSI format (this is also used to produce the python msi itself)
     108 * For distributing applications, most Windows users expect a single install, and may be confused by having the Python runtime environment set up on their machine with lots of libraries
     109 * [ py2exe] is the most popular of a variety of tools for producing a ''frozen'' Python distribution on Windows:
     110   * An extension to `distutils`
     111   * Packages up the Python runtime, and a set of Python packages, modules, extensions, scripts and data, into a target directory
     112   * Scripts are converted to stub `win32` executables that load the Python dll and execute some code
     113   * Automatic search for Python library dependencies (by scanning your source code for `import` statements), as well as manual specification of requirements
     114   * All the libraries you depend on need to be included - sometimes having things installed as `eggs` on the build environment can produce problems
     115   * Running in frozen mode often requires some changes to the underlying code for compatibility - location of files etc - lots of tips on the wiki site
     116 * It's fairly common to use [ InnoSetup] to produce an installer containing all the required files, Start menu items, etc
     118=== Putting it all together ===
     120 * It makes life easier if you can do all the above, and whatever other targets you require, from the same `` script
     121 * Separate out options for the different commands as much as possible
     122 * We found we had to hack `distutils` a lot with derived code to make it all work