The Package Forge Registry

The Package Forge registry is designed to facilitate the scheduling and tracking of build jobs once they have been submitted. It does not necessarily contain everything which defines a job (e.g. the list of packages or instructions for the specific build daemons).

Once a newly submitted job has been processed and accepted a set of associated tasks are registered, one per target platform and architecture. As well as the registry of jobs and tasks, the registry stores information on the supported platforms and the build daemons for those platforms. When a build daemon needs to find a new task or update the status of a task it talks directly to the database. For simplicity and safety, the database layer handles all the necessary locking, transactions and consistency checks, there is no middle layer of extra code.

The Schema

The full details of the database schema are available or you can view a relationship diagram generated using postgresql_autodoc and GraphViz.

Platforms

In the Package Forge Registry a platform is a build target which is uniquely identified by the combination of the name and the architecture, for example f13/i386 or sl5/x86_64.

For new tasks to be registered for a platform it must be marked as active. To maintain referential integrity, a platform should never be removed once tasks have been registered. If a platform is no longer supported it should be marked as inactive (i.e. active) is false.

As well as the concept of a platform being active there is a secondary attribute (named auto) which controls whether or not tasks are registered for this platform by default. If a platform is in the default set then tasks will be registered unless a user has specifically requested that it should be ignored. If the auto attribute is set to false (which is the default) then a user must either specifically request that platform name or must use the all string to show that tasks should be registered for every active platform.

Builders

In the Package Forge Registry a uniquely named builder entry should exist for each build daemon. Typically, it is expected that there will be one builder per platform but there are no limits. It may well be that there is a period during which there are no builders and new tasks will just be queued as long as the platform is active. It is also possible to register extra builders for a platform which can be used during busy times. Note that having a builder registered does not necessarily mean it will take new jobs. The builder has to have the necessary permissions to be able to talk to the registry database and request new jobs, that must be managed separately (normally via the LCFG postgresql component).

When a builder takes on a new task the ID is stored into the current field and an entry is added into the build log. Note that in some circumstances a builder is allowed to give up on a job halfway through without considering this to be a failure. This means that a task can appear in the build log multiple times for the same or different build daemons (as long as they all support the same platform).

Build Jobs

When a new job is accepted from the incoming queue it is added to the Package Forge Registry along with a subset of the parameters specified in the submission. As it goes through the job life-cycle the status changes to reflect the stage reached. Whenever any aspect of the job changes the modification time is automatically updated.

The sequence of allowed job statuses is:

incoming
The job is in the incoming queue and currently being processed.
valid
The job is in the incoming queue and has been validated.
invalid
The job is in the incoming queue and is invalid.
accepted
The job has been validated and accepted.
registered
The job has been validated and the associated tasks have been registered.
partial fail
At least one associated task has failed to build.
fail
All associated tasks have failed.
partial success
At least one task has succeeded and none have failed.
success
All tasks have succeeded.
cancelled
All tasks were cancelled.

In normal circumstances, the first few stages - incoming, valid, accepted and registered fly by very quickly. The rest can take a while and depend on the availability of the various builders for each platform.

Tasks

When a newly submitted job has been processed and validated a set of tasks will then be registered, one for each required platform and architecture. Note that tasks are only registered for active platforms. Typically, a job will be submitted for all platforms and architectures in which case a task will be registered for every target platform in the default set. It is possible to restrict the set of build platforms by either name or architecture. Beware that if you submit a job for non-existent or inactive platforms they will just be ignored during the task scheduling stage.

Unlike the parent job a task has a much more straightforward set of statuses.

needs build
The task requires building (default on registration).
building
The task is currently building.
fail
The task has failed to build.
success
The task has been successfully built (and submitted).
cancelled
The task has been cancelled.

Whenever a task status is updated the parent job status is automatically updated within the database. All tasks failed leads to a job fail status, any task failure leads to partial fail. If all tasks succeed then the job status is success, if any succeeds and none has failed then the job status is partial success. If all tasks are cancelled then the job status is set to cancelled.