Connexer Ltd. | Automatic Debian Package Repository HOWTO

Automatic Debian Package Repository HOWTO

Revision History

Revision	Date	Revised by	Comments
0.31	20131217	RCS	Added CC license
0.3	20070928	RCS	Deprecated this HOWTO in favor of reprepro
0.2	20050606	RCS	Updated to reflect that Sarge is the new stable distribution
0.1	20050112	RCS	Added information about alternatives and fixed numerous typos
0.0	20050109	RCS	Initial draft

Copyright

Licensing

This article is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.

WARNING

I now consider this HOWTO deprecated. At the time that I wrote it, the available tools were seriously lacking. Nowadays, the tools are much better and much easier to setup. I encourage you to use the setup described in Debian Administration's Setting up your own APT repository with upload support article. I now use a setup that is nearly identical to the one described in the article. However, I will leave this HOWTO here indefinitely as it may be of interest to some.

Introduction
Alternatives
Getting Started
Setting Up
Populating the Repository
Preparing the Config Files
Updating the Repository
Using the Repository
Provide Feedback

Introduction

This HOWTO was inspired by several things:

A desire to more effectively manage some packages I have backported and created locally (kernel images and JDK packages).
A desire to help others who are interested in creating a local repository.
A desire to help extend some of the available documentation, which does not go into sufficient detail.
An interest in creating an automatic repository (which is not so well documented), as opposed to a trivial repository (which limits flexibility).

There are also many good reasons as to why you may want to create your own package repository. Say you are trying to install something that depends on the package j2sdk1.4. This is not possible without outside sources, since Debian does not distribute JREs or JDKs. Naturally, you could just add the Blackdown deb sources to your sources.list and be on your way. But what if you prefer the Sun JRE? You would have to download it from Sun and manually install it. If you do this with enough packages you will lose track or begin forgetting where you have placed things.

In my case, I used the excellent package java-package to create a .deb file from the Sun JRE .bin file. Now I can place that .deb file into my repository, and it only gets installed if I have packages that depend on it. If I remove all of the depending packages, the JRE is also removed. I can apt-get install the JRE onto all of my machines without need for a manual installation each time. If an update is released, I can create the requisite .deb package and place it in my repository so that all of my machines update automatically.

Additionally, what if you want to customize an existing Debian package and have it available to all your Debian machines? I also have a detailed walkthrough on customizing Debian packages. Once you have customized more than a couple of packages, it will be much easier to keep track of them if they are all in a local repository accessible by APT. Clearly, there are many reasons why you may want to create your own repository.

An additional motivation for me to write this was because some people out there are unhappy with all of the trivial repositories floating around out there. Apparently, trivial repositories are not compatible with apt-pinning and make life difficult for those people who make use of unofficial package repositories and like to pin their packages.

The main source of information during my discovery process was Aaron Isotton's Debian Repository HOWTO. This document gives an excellent overview of the process, but is lacking in some areas. Specifically, the finer details of generating all the files necessary to make your repository work.

This document assumes that you have a good level of understanding of the Debian distribution and feel comfortable using the command line.

Alternatives

It would not be fair of me to let you think that this is the only possible way to setup a Debian package repository. Naturally, you could just install your packages locally using dpkg -i, or simply find packages in someone else's repository, like, for example, those listed at apt-get.org.

Two possible alternatives are mirrorer and dak. The mirrorer software is currently only available via CVS, but provides support for more advanced repositories (i.e., a pooled structure without a database server and the ability to check and generate signatures). The dak package is used directly in the maintenance of the official Debian repository and is now available officially in the Debian archive.

Another alternative is an automated tool by Ola Lundqvist called debarchiver. This tool will automatically create a repository for you in /var/lib/debarchiver. It also creates an incoming directory where you place your packages along with a .changes file and sets up a cron job to monitor the directory and automatically migrate the packages for you. This is more in line with how the real Debian repository operates. I was not really aware of all the functionality of this tool when I embarked on my quest.

Since I change my repository infrequently, I have chosen to not use debarchiver. Naturally, you can choose to make use of derbarchiver or to take my more manual approach. If you still want to do it this way or if you just want to see better how a Debian package repository is structured, then read on.

Getting Started

Necessary packages:

apt-utils

Other necessities:

A place to store your packages (local directory, or remote FTP or web space).

I have made the assumption that you will be creating your package repository locally and then using it locally or uploading it to a remote machine and accessing it from there. If the machine you will be using to create the repository has an HTTP or FTP server running, you can access your packages locally and remotely from other machines. If you have a remote server that you can access via SSH, or some other shell, then you can simply create everything on the remote machine and access it from anywhere you like.

Setting Up

You first need a directory structure to house your packages. Here are some example choices for a root:

~/packages/
/packages/
/var/www/debian/
/usr/local/packages/

Your choice depends on whether you intend to serve your packages to remote machines or use them only on the local machine. You should create the desired directory and cd into it.

Now you need to create the remainder of your directory tree. You can visit the Debian Repository HOWTO for a detailed look at exactly how the real Debian repository is structured. I have personally chosen to model mine after that structure, but your needs will dictate your choice. Regardless, you need to start by creating a directory below your repository root called dists/.

In my case, I serve my repository to all the machines on my LAN via Apache, so my repository is located at /var/www/debian/, with the directory dists/ one level below that.

Under dists/, you need to create one or more directories for the distributions you will serve packages to. In my case, I have one server and two PCs running Sarge, so I have the directory dists/sarge/.

Now you need to make yet another choice. What sections will you have? The sections are those which follow your distribution line the sources.list. For example, the Debian package source:

deb http://ftp.es.debian.org/debian/ sarge main non-free contrib

contains the distribution sarge, and the sections main, contrib and non-free.

To see an example of a repository with a slightly different structure, visit backports.org. They have structured their sections so that they are named after individual packages. That way you can make certain that you get only the backported packages you want. Personally, I take the former approach, since it is my personal repository and I don't introduce any packages I don't want.

Below each section you will need one or more binary directories and an optional source directory. In my case, all of my machines are i386 architecture, so I have a binary-i386/ and a source/, since I choose to also maintain source packages whenever possible.

When you are done creating all the directories, you should have a structure similar to this:

 $ find . -type d
 .
 /dists
 /dists/sarge
 /dists/sarge/contrib
 /dists/sarge/contrib/binary-i386
 /dists/sarge/contrib/source
 /dists/sarge/main
 /dists/sarge/main/binary-i386
 /dists/sarge/main/source
 /dists/sarge/non-free
 /dists/sarge/non-free/binary-i386
 /dists/sarge/non-free/source

Populating the Repository

Now you must place your package files in their corresponding directories. How you do this is a matter of choice, though taking the time to organize yourself properly in the beginning pays off as your repository grows larger. Typically, binary .debs (those which you can install directly with an apt-get install command, would go in their corresponding binary-i386/ directory. If you support a different processor architecture, then you would use that instead.

The only thing I have not managed to figure out yet is a pooled repository, like the Debian repository. This is where every package is kept in a common directory and the respective binary-$(ARCH)/ directories only contain a reference to the appropriate files. This comes in handy when you have to support more than one architecture, since certain package (e.g., documentation, interpreted scripts, etc.) are the same and do not require architecture-dependent compilation. However, since I only have i386 machines, I have not bothered to learn how to do this. Perhaps in the future.

Each binary-i386/ directory and source/ directory requires a file called Release, which indicates the target architecture and some additional information. For example, the Release file for the for the Sarge main binary files is located at /dists/sarge/main/binary-i386/Release and has these contents:

 Archive: stable
 Component: main
 Origin: <Your name or organization>
 Label: <A descriptive label>
 Architecture: <Your processor arch or "source">

Once you are finished creating the necessary files, you should have something like this:

 $ find . -name Release
 /dists/sarge/contrib/binary-i386/Release
 /dists/sarge/contrib/source/Release
 /dists/sarge/main/binary-i386/Release
 /dists/sarge/main/source/Release
 /dists/sarge/non-free/binary-i386/Release
 /dists/sarge/non-free/source/Release

Don't forget to update the archive, component and architecture fields for each file.

Preparing the Config Files

I have two configuration files:

apt-sarge-release.conf: This file is used to generate the Sarge "Release" file
apt-ftparchive.conf: This file is used to tell apt-ftparchive what distributions and sections need to be processed.

All of these files reside in the repository root. The contents of each are listed below. If you are really interested, you can read the apt-ftparchive(1) documentation. It is a bit hard to follow, but still informative.

Contents of apt-sarge-release.conf:

 APT::FTPArchive::Release::Origin "Your name or organization";
 APT::FTPArchive::Release::Label "Descriptive label";
 APT::FTPArchive::Release::Suite "stable";
 APT::FTPArchive::Release::Codename "sarge";
 APT::FTPArchive::Release::Architectures "i386 source";
 APT::FTPArchive::Release::Components "main contrib non-free";
 APT::FTPArchive::Release::Description "More detailed description";

Below are the contents of the apt-ftparchive.conf file. I know that there are more efficient ways to accomplish the same thing. However, I was unable to get them to work. The documentation makes is seem as though it is possible to specify a generic directory structure and let apt-ftparchive find all the specific directories. But nothing I could do would make it find a directory that I had not explicitly stated. So, here is the brute force approach:

 Dir {
   ArchiveDir ".";
   CacheDir ".";
 };
 
 Default {
   Packages::Compress ". gzip bzip2";
   Sources::Compress "gzip bzip2";
   Contents::Compress "gzip bzip2";
 };
 
 BinDirectory "dists/sarge/main/binary-i386" {
   Packages "dists/sarge/main/binary-i386/Packages";
   Contents "dists/sarge/Contents-i386";
   SrcPackages "dists/sarge/main/source/Sources";
 };
 
 BinDirectory "dists/sarge/contrib/binary-i386" {
   Packages "dists/sarge/contrib/binary-i386/Packages";
   Contents "dists/sarge/Contents-i386";
   SrcPackages "dists/sarge/contrib/source/Sources";
 };
 
 BinDirectory "dists/sarge/non-free/binary-i386" {
   Packages "dists/sarge/non-free/binary-i386/Packages";
   Contents "dists/sarge/Contents-i386";
   SrcPackages "dists/sarge/non-free/source/Sources";
 };
 
 Tree "dists/sarge" {
   Sections "main contrib non-free";
   Architectures "i386 source";
 };
 
 Default {
   Packages {
     Extensions ".deb";
   };
 };

Once you have the configuration files tweaked to your liking, you are ready to actually make your repository "visible" to apt. Personally, I found it easier and more efficient to create a short shell script to do this for me. I called it update-archive.sh and put in the repository root along with the configuration files. The contents are listed below:

 #!/bin/sh
 apt-ftparchive generate apt-ftparchive.conf
 apt-ftparchive -c apt-sarge-release.conf release dists/sarge/ >dists/sarge/Release

Updating the Repository

Once all the files and packages are in place, you can proceed to update the repository databases. You can do this by running the shell script listed above or manually executing the commands. The output should resemble this:

 $ ./update-archive.sh
  dists/sarge/main/binary-i386: 21 files 42.7MB 2s
  dists/sarge/contrib/binary-i386: 0 files 0B 0s
  dists/sarge/non-free/binary-i386: 4 files 114MB 5s
  dists/sarge/main/binary-i386/: 21 files 42.7MB 0s
  dists/sarge/contrib/binary-i386/: 0 files 0B 0s
  dists/sarge/non-free/binary-i386/: 4 files 114MB 0s
  dists/sarge/main/source/: 2 pkgs in 0s
  dists/sarge/contrib/source/: 0 pkgs in 0s
  dists/sarge/non-free/source/: 0 pkgs in 0s
 Done Packages, Starting contents.
 Done. 314MB in 50 archives. Took 7s

You will need to update your repository periodically. You can either set a cron job to run your script, or run it manually whenever you add or remove packages.

Using the Repository

You can either use the repository locally or remotely.

For local access:

 deb file:///path/to/repository/ sarge main non-free contrib

For remote access (like an official apt repository):

 deb http://your.server/repository/ sarge main non-free contrib

If you managed to read this whole thing, please let me know what you thought, whether it was useful, or if you have any suggestions for improvement.