Package Forge - Build Jobs

Central to the design of the Package Forge system is the concept of a build job which is what the user submits. In essence a build job has a unique identifier and consists of a set of, one or more, source packages along with some instructions which tell the target build systems how and where the packages should be built and submitted.

The Job Class

A build job is represented by the PkgForge::Job class. This has the following important attributes:

The Identifier

This can be any string the user chooses which has not been previously seen by the system. The only limitation on the contents is that placed by the column definition in the job table of the registry database, this requires it to be a maximum of 50 characters long. If the user does not select their own tracking identifier then a Universally Unique Identifiers (UUID) is generated using the Data::UUID module.

Platforms

This controls the set of platforms for which the build job should be submitted. By default, if the user does not express an opinion, all supported, active platforms will be selected. As well as the straightforward, positive inclusive approach of just listing the suitable target platforms (e.g. "sl5, f13") it is possible to exclude particular unsuitable platforms from the whole set (e.g. "!f13").

In PkgForge terminology a platform name is based only on the distribution name and version (e.g. "f13" or "sl5"). The platform name does not contain any reference to the architecture. Note that this is different from LCFG which appends an "_64" for the x86_64 architecture. The target architectures are handled separately, see below for details.

It should be noted that there are no limits on the strings which can be placed in the list of platforms. Anything which refers to a platform which does not exist will be ignored, no errors are reported. However, a helpful warning is given on the web interface when it spots that a submitted job refers to unsupported or inactive platforms.

Architectures

The architecture list is used in a very similar way to the platform list. This controls the set of architectures for which the build job should be submitted. By default, if the user does not express an opinion, all architectures for the selected platforms will be selected. Again, as well as the straightforward, positive inclusive approach of just listing the suitable target architectures (e.g. "i386, x86_64") it is possible to exclude particular unsuitable architectures from the whole set (e.g. "!x86_64").

It should be noted that there are no limits on the strings which can be placed in the list of architectures. Anything which refers to a platform which does not exist will be ignored, no errors are reported. However, a helpful warning is given on the web interface when it spots that a submitted job refers to unsupported or inactive architectures.

Further to this, it is worth noting that there are some limits on what can be achieved with this method of specifying the platform and architecture lists. If a job should be built for different architectures on different platforms (e.g. i386-only on sl5 but all on f13) then it will need to be submitted as multiple separate jobs. It is not expected that this will cause much difficulty.

The Source Packages

Clearly the most important part of a build job is the set of source packages which should be built. A job may contain one or more source packages of multiple file types. Any file type which has an associated Perl class which implements the PkgForge::Source Moose role can be added to a job. Individual build daemons will filter the set of source packages and only attempt to build from those which are appropriate (e.g. an RPM builder might only choose SRPMs). This feature makes it very easy for a user to submit a job for all platforms without needing prior knowledge as to which will support a particular file type.

When a job is submitted the source packages are validated and the SHA1 sums are calculated. These are then used throughout the system to ensure that the source package files remain consistent. The validation varies from type to type but includes basics such as checks that the file exists and is of a non-zero size. For example, the SRPM source package type also checks that the package is a valid source RPM.

The builds of the source packages will be attempted in the order that they are specified by the user. When building a set of packages some platforms will stop immediately when an error is encountered, on others the build will move onto the next package and the failed package will be retried later. See the documentation for the relevant build daemon for full details.

The Package Bucket

The package bucket is LCFG/DICE terminology for a directory in which built packages are stored. In the Package Forge build system the bucket is typically used for two related issues. The first is the selection of the build system configuration to be used and the second is the selection of the location into which built packages will be stored.

There are no restrictions on what the user can set as the bucket for a submitted job. The knowledge of which buckets are supported is only available to the build daemons (and varies on each platform) so the incoming queue processor cannot check for validity. Depending on the build system being used an incorrect bucket name could trigger an error and cause a complete failure to build the job.

The bucket name must be specified by the user. Where it is not applicable a build daemon may choose to completely ignore the bucket though, in which case it can be set to anything the user chooses. For example, an external site using their own pkgforge system may choose to put all packages into a single location or may not use the pkgsubmit tool.

The RPM builder uses mock to make packages. On these platforms the mock configuration is closely related to the directory to be used for storing built packages. Packages in certain directories should only have build-dependencies on packages in other particular directories. For example, on F13 the LCFG packages in the lcfg bucket should only depend on software which is in F13 or already in the bucket. There should not be any dependencies on packages in buckets with license restrictions (e.g. inf or uoe). The list of buckets is site-dependent, see local documentation for details.

Reports

Currently Package Forge only supports sending reports via email. The report attribute is used to specify the (comma-separated) list of email addresses to which a report should be sent after each build job is completed. If nothing is specified then no reports will be sent.

Due to the way in which a job is split into multiple tasks, one for each target platform, a separate report is sent for each task. So, for example, one submitted job may generate four reports (sl5/i386, sl5/x86_64, f13/i386, f13/x86_64). There is no way to easily aggregate the messages as the data is stored separately for each task.

Submitting a new job

A build job is submitted as a directory which contains all the source packages and a meta-data manifest file. This directory is placed into a standard location in the file-system where the user has write access. In the School of Informatics the file-system used is AFS so that a build job may be submitted from anywhere.

Build jobs are submitted using the submit command of the pkgforge command-line tool. Here is the current short help-text for the command:

pkgforge submit [-aBcpr] [long options...]
	--verbose           Verbose output
	-c --configfile     The configuration file for this application
	-a --archs          The architectures for which packages should be built
	--target            Location into which jobs should be submitted
	--id                The unique identifier for this job
	-p --platforms      The platforms on which packages should be built
	-B --bucket         The bucket to which the packages will be submitted.
	-r --report         Email addresses to which build reports will be sent

Most of these options (--archs, --platforms, --id, --bucket and --report) have already been discussed above. It is unlikely that it will ever be necessary to set the target option, this controls where the new build job should be stored and the default should be correct. It is entirely possible to use the submission tool with multiple separate pkgforge systems though in which the target option is essential. The only other option which might come in handy is that for changing the configfile. Again, if multiple pkgforge systems are being used it might be necessary to have different configurations for each. The standard configuration file handling is discussed below.

Job Submission Examples

Submitting a job for only F13 (both i386 and x86_64 architectures):

$ pkgforge submit --bucket lcfg --platform f13 foobar-1-2.src.rpm

Submitting a job for all platforms except SL5 (again, for all architectures):

$ pkgforge submit --bucket world --platform '!sl5' foobar-1-2.src.rpm

Email reports are needed:

$ pkgforge submit --bucket devel --report somebody@example.org foobar-1-2.src.rpm

Job Submission Configuration

For the job submission tool the following configuration files are examined and, if they exist, the contents are used to set the default values for the submission client options. Values for options set in files which are later in the list override any values from files which are earlier in the list.

  1. /etc/pkgforge/pkgforge.yml
  2. /etc/pkgforge/submit.yml
  3. ~/.pkgforge/pkgforge.yml
  4. ~/.pkgforge/submit.yml

Default values for any submit option can be set, the configuration file format is YAML. The easiest approach is to look at the system-wide configuration files and copy the applicable settings into the local directory. For example, this makes it possible to avoid having to specify the email addresses for reports every time a job is submitted.