PkgForge - Submitting Packages

The entire purpose of the Package Forge system is to make it easy for a user to submit a set of software source packages and have them built on multiple platforms with the products being copied into the central repositories. Parts of this process are complex but from the user point of view it can be considered fairly simplistically. After a job has been submitted it is transformed into a set of tasks, one per target platform, and they placed into queues for the package builders. The builders select the jobs in the order of submission, attempt the build and then run through any checks (such as running rpmlint). If the build was successful and the checks have all passed then the generated packages are submitted into the central package repository. At the end of the process any requested reports are generated. Full details of each job are always viewable through the web interface.

The source packages are submitted using the pkgforge command-line tool by invoking the submit command. There are a number of options which can be used, the only obligatory option is the bucket (explained below). For example:

pkgforge submit --bucket lcfg foo-1-2.src.rpm

It is possible to submit more than one source package as part of a job. The builds will be attempted in the order in which they are specified by the user. The RPM builder can use the generated packages from building one source package as the build-dependencies for subsequent source packages within the same build job. Further to this, if a package fails to build then it will be put onto the back of the queue in the hope that the error was due to missing build-dependencies which can be satisfied within the current build job. The builder will continue attempting to build source packages within a job until all are successful or no more can be built. This makes it possible to build large sets of source packages without knowing the correct ordering required.

Validity checks will be carried out on each file before the build job is submitted. This includes checking that the SRPM is valid and that it contains a specfile with the .spec suffix.

Getting Help

If you do not know the the name of the command you require, then the pkgforge tool can be invoked without any command and it will print to the screen a list of all available commands with short details on each. For example:

% pkgforge
Available commands:

  commands: list the application's commands
      help: display a command's help screen

    submit: Submit a set of source packages for building

The help command can be used to get a full list of options for any particular command. For example:

% pkgforge help submit
Required option missing: bucket
pkgforge submit [-?aBcpr] [long options...]
	-c --configfile     The configuration file for this application
	--target            Location into which jobs should be submitted
	--id                The unique identifier for this job
	-? --usage --help   Prints this usage information.
	-p --platforms      The platforms on which packages should be built
	-a --archs          The architectures for which packages should be
	                    built
	-B --bucket         The bucket to which the packages will be
	                    submitted.
	-r --report         Email addresses to which build reports will be
	                    sent
	--verbose           Verbose output

Most options have a single-character short-form prefixed with a single-dash or alternatively can be reduced to the shortest unique string prefixed with a double-dash. For example, --plat is acceptable.

If there is an option for which you want to specify the same value every time the command is invoked (e.g. the email address for reports) then it is better added into the configuration file (see below for details).

The options which are most likely to be regularly useful are --platforms, --archs, --bucket and --report and each is explained in greater detail below. One further option that can occasionally be useful is --id. If the job identifier is not specified then a UUID is generated but it can be useful to use an ID string which is more meaningful to the individual (i.e. contains the user name or a package name). Note that the ID MUST not have been used previously, MUST be no more than 50 characters long and MUST only contain characters in the A-Za-z0-9_.- set. Also beware that there is nothing stopping other users also generating jobs using your chosen naming scheme...

Selecting the Bucket

The bucket is only option which must be specified for each build job. It is LCFG terminology for the repository into which a package is stored. Typically there are different buckets for packages with different license restrictions. Taking the School of Informatics as an example we have:

lcfg
Packages which globally distributable and part of the LCFG project.
world
Other packages which are globally distributable.
uoe
Packages which can only be distributed within the University of Edinburgh.
inf
Packages which can only be distributed with the School of Informatics.
devel
Development code. Nothing in this bucket is distributed and it is regularly swept to remove old packages.

It is important that a package submitted into a particular bucket does not have dependencies which can only be satisfied by packages in a different bucket with a more restrictive license. For example, a package in the world bucket cannot depend on packages in the inf bucket. To achieve this goal the set of buckets available for satisfying build-dependencies is different depending on the target bucket. In the School of Informatics the sets of buckets used in build configurations are:

Target Available Buckets
lcfg base, updates, epel, lcfg
world base, updates, epel, lcfg, world
uoe base, updates, epel, lcfg, world, uoe
inf base, updates, epel, lcfg, world, uoe, inf
devel base, updates, epel, lcfg, world, uoe, devel

Selecting the Platform

If no platform is specified then tasks will be registered for the default set of platforms. The PkgForge system has two different pre-defined groups of platforms which can be requested, these are auto and all. The default set of platforms is the auto group. Tasks can be scheduled for any platform which is marked in the registry as active but it may well be that not all active platforms are marked as auto. As suggested by the name, the all group selects all active platforms.

A job can be submitted for all active platforms like this:

% pkgforge submit -B lcfg --platforms all foo-1-2.src.rpm

If a set of source packages will only build on particular platforms then it is worthwhile restricting the target set. Any number of platform names can be specified in a comma-separate list. For example:

% pkgforge submit -B lcfg --platforms 'f13,sl6' foo-1-2.src.rpm

Note that it is not a problem for the PkgForge system if a package is submitted for a platform and it fails to build. It is down to the individual user to decide on whether or not a build failure is important.

If a platform name is requested which is not active or not supported then no tasks will be registered for that platform. Any other platforms will be registered as normal, the overrall job submission will not fail. The PkgForge web interface will flag up build jobs which reference inactive or unsupported platforms.

Selecting the Architecture

Further to the possibility of restricting the target platforms it is possible to restrict to particular architectures. Any architecture restrictions are applied after the platform list has been calculated. This means that it is not possible, in a single build job, to target different architectures on different platforms (e.g. just f13/i386 and sl6/x86_64 is not possible).

By default all architectures are selected, this normally means i386 and x86_64. Selecting a specific architecture can be done like this:

% pkgforge submit -B lcfg --arch i386 foo-1-2.src.rpm

Feedback

Once a job has been submitted full feedback on the progress can be obtained from the PkgForge web interface. It is also possible to get email reports.

Email Reports

Email reports can be selected by passing in an email address to the --report option or by setting it permanently in the configuration file (see below).

The email reports will be sent by the individual builders after each task has completed (either success or failure). This means that if the job is submitted for 3 platforms and 2 architectures for each platform the user will receive 6 email reports.

As well as the subject and details in the body of the message, the following email headers will be added to each report email to make it easy to do filtering.

Configuration

Typically the pkgforge submit command will already have been configured on a system-wide basis in the /etc/pkgforge/pkgforge.yml and /etc/pkgforge/submit.yml files. If the LCFG pkgforge component is being used then this can be done using the lcfg/options/pkgforge.h header. Typically the only option which must be set is the target directory, which is the location of the directory into which new jobs should be submitted. If a networked filesystem is being used then this will usually be the same as the setting for the pkgforge.incoming resource.

The primary pkgforge configuration file (/etc/pkgforge/pkgforge.yml) should look something like:

--- 
"accepted": '/afs/example.org/pkgs/pkgforge/accepted/'
"incoming": '/afs/example.org/pkgs/pkgforge/incoming/'
"results": '/afs/example.org/pkgs/pkgforge/results/'
"website": 'https://pkgforge.example.org/'

and the submit-specific configuration file should look something like:

--- 
"target": '/afs/example.org/pkgs/pkgforge/incoming/'

Further to this the user can override any settings using a configuration file named submit.yml which is stored in the .pkgforge directory within the user's home directory. In particular this is useful for permanently setting the report option, like this:

report: user1@example.org

It can also be useful for permanently setting defaults for the bucket, platform and architecture options. All settings can still be overridden at any time from the command-line.