package list wrapper scripts port and rewrite

After the rewrite was merged (before full testing) a series of small follow-ups (#1328, #1332, #1333) were required to bring everything into working order. After that the code has been happily deployed for Leap 15.0.

A few items remain in order to bring the SLE wrappers into the fold.

flesh out comment tests

As part of the long-term goal of increasing test coverage a detailed set was added to cover the code responsible for interacting with OBS comments. Additionally, the ReviewBot specific comment code was also covered. The tests are a combination of unit and functional tests that utilize the local OBS instance brough up on travis-ci for a net gain of around 1.5% coverage.

adi packages with baselibs.conf staged for all archs

Packages not contained within the ring projects are placed in adi stagings which are only built for x86_64. For packages utilizing a baselibs.conf such as wine this is less than ideal since the imported packages are not built. For wine specifically this causes the package to never pass the repo-checker since the wine (x86_64) package requires the wine-32bit package which is not built.

To solve the wine case and allow baselibs.conf to be tested in stagings the adi process was enhanced to detect baselibs.conf and enable all archs from the target project (for Factory this includes i586). This handles both static baselibs.conf and those that are generated during by the spec file.

config generalization

As part of an ongoing process to standardize the execution and configuration of the various tools to reduce the management overhead, various flags have been migrated to be read out of project configuration. This allows tools to be run without flags via standardized service files while the config defaults can be managed centrally and overridden in OBS remote config.

The config system was expanded to support non-distribution projects to allow for tools to be run against devel projects like GNOME:Factory which has long utilized factory-auto. This exposes the flags that were previously available via the project config. Interestingly, the change had to be reverted and re-introduced with pattern ordering. Had been luck that the issue had not been a problem prior to this point.

NonFree workflow

A discussion of the NonFree workflow was started regarding including such requests in the normal staging workflow. As a consequence of not being staged the new repo_checker can not process the requests.

obs_clone improvements

The obs_clone tool clones data between OBS instances which is used to load a facsimile of Factory into a local OBS instance for testing. Without a large amount of specific project setup the various tools cannot function and thus cannot be tested. After a hiccup in Factory the project had to be rebuilt by bootstrapping against a staging project. This caused a circular dependency which the tool cannot currently handle. A workaround was added to resolve the issue.

As part of the debugging process a variety of improvements were made furthering the goal of being able to run all the tools against the local instance.

staging-bot improvements

The quick staging strategy was improved to handle the new Leap 15.0 workflow and future proof the implementation. In order to process requests as quickly as possible the requests that do not require human review (ex. coming from SLE were they were already reviewed) are grouped together to avoid being blocked by other requests. The strategy has already been seen functioning correctly in production.

Towards the same goal, the special strategy, which is used to isolate important packages and/or those that routinely cause problems, was modified to be disablable. This is utilize for Leap which generally sees requests from other sources which have already passed various processes and are less likely to cause issues. Thus rather than isolate such packages and thus take up an entire staging they are simply grouped normally.

last year

Some major work was completed last year at this time.

API request cache

In an effort to drastically improve the runtime of the various staging tools and bots an HTTP cache was implemented for caching specific OBS API queries and expiring them when the OBS projects were changed. The result was a significant improvement and in many cases removed over 90% of execution time once the caches were warmed up. Given the number of staging queries run in a day this was a major quality of life improvement.

ignored request pseudo-state

Another quality of life improvement was the introduction of an ignored request pseudo-state for use in the staging process. This allowed staging masters to place requests in a backlog with a message indicating the reason. Not only did this improve the workflow for communicating such things between staging masters, but it also paved the way for automated staging improvements since this could be used to exclude requests. The pseudo-state has since become an integral part of the workflow.

Ignored requests can be seen along with their reason in the list output:


A major step forward was the creation of the RequestSplitter which not only replaced the different request grouping implementations, but providing a vastly more powerful solution. Not only were the previous static grouping options available, but xpath filters and group-bys could be specified to create complex request groupings. Prior to this such grouping had to be done manually via copy-paste from list sub-command or other primitive means. Given that all ring packages must be staged manually using this process any improvement helps.

Additionally, this made the select sub-command much more powerful with a variety of new options and modes including an --interactive mode which allowed a proposal to be modified in a text editor. The combination provides a vastly superior experience and laid the groundwork for further automation via strategies (mentined above and to be covered in future post).

An example of interactive mode, which also includes the strategies introduced later, can be seen below.


A utility was created to print the list of devel projects for a given project. This tool was intended to eventually replace the expensive query with the cached version which has since been done.

Yet another quality of life improvement was to indicate if origin differs from the expected origin in the leaper review. This simple change aided in being able to more quickly scan through large numbers of such reviews.