ReviewBot overrides via comment commands

Something which has been discussed for quite a while, a method for overriding bot reviews, was added recently. The functionality ended up being to interpret comments in the form @bot <command> [args...]. Comment commands are then restricted to members of either a configured group or the staging group. For example release overrides addressed to leaper may be issued by anyone in leap-reviewers group or factory-staging (responsible for staging process).

repo-checker gained the ability to skip the cycles check via a comment command placed on a staging project (@repo-checker skip-cycle). Previously, this had to be done via the command line by manually running the bot.

The final decision towards this workflow was clear after it turned out OBS groups were insufficient for the override workflow.

leaper was also modified to leave previously declined reviews open for SLE manager override. A similar change will be made to the openSUSE portion of the bot.

Quality of life changes

Instead of printing the list of packages, requiring the package to be deleted, to build as a flat list it is formatted as a bulleted list by repo-checker.

sad-package is still a build requirement of package1, package2, package3

sad-package is still a build requirement of:

- package1
- package2
- package3

Similarly, ReviewBots were changed to accept unhandled request types without posting a comment to reduce e-mail spam.

All config options are now lazy-loaded to allow for placement in remote config.

metrics.o.o update for Grafana 5.x and TW review site

In light of the Grafana 5.x series changing the dashboard format the existing 4.x dashboards were migrated. The Tumbleweed snapshot review site data can now also be ingested and presented. Both improvements are scheduled for deployment to metrics.opensuse.org next week.

pkglistgen adventures

A rather interesting bug was causing the pkglistgen.py code to crash on Leap 15.0 after Build153.1 during drop list creation. It ended up being solved by adding a flush() call before passing the file to read by libsolv. The situation would have been easier to resolve if the libsolv components complained about in incomplete XML file, but instead the invalid data makes it through a chain of tools before being read by a tool that complains. In addition to fixing the problem the metadata hash is now validated as further assurance that the correct data is present.

The repo metadata hash for update repos is now used to avoid downloading unchanged information.

To complete the week for pkglistgen the addition of services to Leap 15.0 exposed the need to skip local service run during update.

repo-checker for SLE 15

Lastly, the repo-checker is now utilized for SLE 15.0 development which should reduce both the pre-release problems in SLE 15, but also in Leap 15.0 since it sources core packages from SLE. In combination with Factory this should elevate the quality of pre-releases and even the final releases in addition to reducing negative staging results.

last year

One of the biggest acheivements of last year was the introduction of select automation: merge and strategies which ended up taking over the majority of staging grouping work. An example of using the strategies in interactive mode can be seen in 2018w01-02. The work was based on observations and involvement with Leap 42.3 and Factory staging. Five strategies were created which are evaluated against the open requests based on a priority order. The result is rather powerful automated staging and follow-up staging.

Stagings which were initially filled using a strategy can have further requests --merged into them that match the same strategy. This combined with configurable time period since first staging and other conditions makes for a solid follow-up workflow allowing maintainers to send requests organically. An initial waiting period of an hour is used to allow for patterns to emerge.

A staging-bot user was created under which this code and more now runs as a service. One can clearly see it take over for Factory in the Staged - User* graphs on metrics.opensuse.org.

The strategies were also integrated into the OBS dashboard which allows one to quickly tell what strategy was used for each staging. Note the white section within the Project column on the left in the below images (ex. special, kernel-source).

The existing feature set was documented in repo wiki for staging masters and future plans were organized in a github project.

Towards the same goal of automating staging work several commands were given --cleanup flags to remove mundane and neglected tasks. Similarly, a rebuild command was introduced to trigger rebuilds of flaky packages rather than waiting hours for a staging master to come around.

A couple quality of life improvements were made:

frozenage defaults to all letter stagings (to avoid typing all of them)
include a --message option in unselect to allow for ignoring a request while un-staging since both are commonly done together
provide an unlock command to resolve stuck staging locks

All in the same week, a tool for interacting with devel projects was introduced. The tool provides an overview of the requests to devel projects and the open reviews for devel project (requests targeting Factory). The --remind feature is run as a service to create review reminder comments of which contributors are likely familiar.

On the OBS end, changes were made to the obs_factory plugin in order to improve development experience, harden validation, and enhance the API for use by the staging tools. Rather than make hundreds of individual API calls a single aggregate call could be made which drastically improved performance.