uWatch - Upstream version watching tool

About

uWatch purpose

uWatch has been designed to help maintainers to detect releases of new versions of the software they package. This tool, written in python, offers several functions to get upstream version, compare upstream versions to versions from GAR build descriptions, and report version change to a central database.

uWatch script is integrated with GAR build system and QA pages. This tool can be either used directly from command line, or through gmake targets. The later being the easiest way to use it for maintainers, since GAR will take in charge the argument passing.

uWatch and GAR integration

TODO: insert drawing

uWatch and QA integration

TODO: insert drawing

uWatch configuration

uWatch behaviour is controlled by three configuration variables, the following paragraphs describe each variable.

Enabling / disabling uWatch

uWatch can be enabled or disabled by the mean of the ENABLE_UWATCH variable. uWatch is activated by default. You can disable it by setting this variable to 0 in the Makefile. The scope of this variable is the whole Makefile, which means if the Makefile is used to produce several packages, uWatch will be activated or deactivated for all the packages.

By disabling uWatch, it will be disabled for both the maintainers using Makefile target, and the bot producing QA information. Even if disabled for Makefile use, uWatch will be usable from command line by calling directly the uwatch binary.

Upstream site configuration

uWatch use the content of the UPSTREAM_MASTER_SITES variable as the URL of the upstream site. The content of this URL is retrieved and parsed using a regular expression (see next paragraph). The web page (or ftp) retrieved from this URL should contain version numbers, and of course the latest. A typical URL to use should be the download page of the upstream software (or sources).

Examples :

Upstream files regular expression

UFILES_REGEX variable contains a regular expression used to parse the page retrieved from UPSTREAM_MASTER_SITES url. Even if there exists similarity and patterns in the regular expression to use. The regular expression is specific to each upstream page. It should have a capturing group matching the version number itself.

Examples of regular expressions matching a source filename :

  • commons-codec-(\d+(?:\.\d+)*).tar.gz
  • libXinerama-(\d+(?:\.\d+)*).tar.bz2

Example of regular expressions matching a simple list of version from the mozilla download page :

  • "(\d+(?:\.\d+)*)/

Regular expression guessing

In most of the case regular expressions can be guessed from the content of the DISTFILES variable used by GAR. This means that it is not mandatory to define the regular expression in the Makefile. If the UFILES_REGEX variable is not defined, guessing algorithm will be used automatically.

UFILES_REGEX should be defined only if the guessed regular expression does not match properly the page retrieved from the upstream url. This is the case for Mozilla packages, which have version number in the URL itself.

Many thanks to Maciej for contributing the regular expression guessing code.

Using uWatch from GAR

uWatch is designed to be used as a Makefile target for GAR users. There should not be use cases leading you to use it directly from command line (otherwise please let me know). The following paragraphs describe the list of makefile targets and give you a detailed view of each target.

Targets list

The folliwing targets are available :

  • get-gar-version
  • get-uwatch-configuration
  • get-upstream-latest-version
  • get-upstream-version-list
  • check-upstream
  • upgrade-to-latest-upstream
  • update-package-version-database

Using get-gar-version

This target displays the version of the software currently defined in the build description. This version is the version which will be download and used during packaging. The version is defined by the VERSION variable in the Makefile.

[wbonnet@current9s:~/gar/pkg/firefox/trunk]$ gmake get-gar-version
firefox - GAR version is 3.0.18

Using get-uwatch-configuration

This target displays the uWatch configuration stored in the GAR build description.

[wbonnet@current9s:~/gar/pkg/firefox/trunk]$ gmake get-uwatch-configuration
firefox - Upstream Watch is enabled
firefox - UFILES_REGEX is : (\d+(?:\.\d+)*)
firefox - UPSTREAM_MASTER_SITES is : http://releases.mozilla.org/pub/mozilla.org/firefox/releases/
firefox - GAR version is : 3.0.18
firefox - http_proxy is : http://proxy:3128
firefox - ftp_proxy is not set

Enabling uWatch is the default. This behavior is controlled by the UNABLE_UWATCH variable (1 = enable, 0 = disable). UFILES_REGEX variable defines the regular expression used to match the page retrieved from the url defined in UPSTREAM_MASTER_SITES.

Using get-upstream-latest-version

This target retrieve the latest version available from the upstream url. Regular expression and upstream site are defined by UFILES_REGEX and UPSTREAM_MASTER_SITES variables.

[wbonnet@current9s:~/gar/pkg/firefox/trunk]$ gmake get-upstream-latest-version
firefox - Latest upstream version is 3.6.13

Using get-upstream-version-list

This target retrieve the complete list of versions matching the UFILES_REGEX. Previous target retrieve only the latest version. This is useful to check that the regular expression is correct and retrive the full list of available versions.

[wbonnet@current9s:~/gar/pkg/firefox/trunk]$ gmake get-upstream-version-list
firefox - 2.0.0.20
firefox - 3.5.16
firefox - 3.6.13

Using check-upstream

This target retrieve the lastest version available from upstream url and compares this version to the GAR version. There are two cases, either local version is outdated

[wbonnet@current9s:~/gar/pkg/firefox/trunk]$ gmake check-upstream
firefox : A new version of upstream files is available. Package can be upgraded from version 3.0.18 to 3.6.13

or it is up to date

wbonnet@current9s :~/gar/pkg/cpan/Parse-Syslog/trunk > gmake check-upstream
Parse-Syslog : Package is up-to-date. Current version is 1.10

Using upgrade-to-latest-upstream

This target is used to automatically create an upgrade branch.

uWatch retrieves the latest upstream version and compares it to the gar version. If a newer upstream version exist, then a branch called upgrade_from_GARVERSION_to_UPSTREAMVERSION is created. The content of the trunk is copied to the newly created branch, then the version in Makefile is automatically modified to contain the value of the upstream version.

[wbonnet@current9s:~/gar/pkg/firefox/trunk]$ gmake check-upstream
firefox : A new version of upstream files is available. Package can be upgraded from version 3.0.18 to 3.6.13

[wbonnet@current9s:~/gar/pkg/firefox/trunk]$ gmake upgrade-to-latest-upstream
firefox : a new version of upstream files is available. Creating upgrade branch from version 3.0.18 to 3.6.13

[wbonnet@current9s:~/gar/pkg/firefox/trunk]$ ls ../branches/
firefox2-maintenance           firefox3.5-packaging           upgrade_from_3.0.18_to_3.6.13
firefox3-packaging             mwatters

[wbonnet@current9s:~/gar/pkg/firefox/trunk]$ ls ../branches/upgrade_from_3.0.18_to_3.6.13/
Makefile                   files                      removeBuildPathFromJar.sh
checksums                  gar

[wbonnet@current9s:~/gar/pkg/firefox/trunk]$ cd ../branches/upgrade_from_3.0.18_to_3.6.13/

[wbonnet@current9s:~/gar/pkg/firefox/branches/upgrade_from_3.0.18_to_3.6.13]$ gmake check-upstream
firefox : Package is up-to-date. Current version is 3.6.13

Package from the branch is now up to date, speaking of version. Of course, this does not mean the package can be released nor even compiled. This is the starting point of the maintainer's work. Once the package upgraded, content of the branch can be merged to the trunk.

Branch creation generates a time stamp cookie under the work directory in package trunk. This cookie is used to prevent the same branch from being created twice. By removing this cookie, the maintainer will let uWatch recreate the branch the next time this target will be executed.

Using update-package-version-database

This target is used to automatically report information about GAR and upstream version to the QA database. The following information are reported :

  • Date and time of the checks
  • Package name
  • Catalog name
  • GAR path
  • GAR version
  • UFILES_REGEX variable content
  • UPSTREAM_MASTER_SITES variable contents
  • DISTFILES variable contents
  • uWatch enable or not
  • Latest upstream version
  • uWatch execution status (successful or not)
  • uWatch output (especially error messages)

The uwatch bot, running on the build farm, is executing this target daily to update the qa database. Information are used by pages like the maintainer qa page.

Using uWatch from command line

uWatch is automatically called from the Makefile when you use one of the targets previously described. The following paragraphs describes how to use uWatch from the command line. This should not be the main use case for maintainers, nor for the QA bot running uWatch. The following paragraphs are provided mostly to describe what is behind the GAR wrappers. If you, as maintainers, have use cases that need to use uWatch from command line, it probably means something is missing from the Makefiles wrappers. So please let me know :)

List of supported commands

The following commands are supported from the command line :

* check-upstream
* get-upstream-latest-version
* get-upstream-version-list
* upgrade-to-version
* update-package-version-database

Command : check-upstream

This command retrieve latest version from upstream site and compares it to the version supplied through command line. Upstream URL and current version are mandatory arguments. The regular expression used to parsed the upstream URL can either be supplied trough command line or guessed if the user supplies the DISTFILES informations.

The command output either nothing if current (local) version is newer or the same as the latest upstream version, or the latest upstream version if it is newer than the current version supplied through command line arguments.

[wbonnet@current9s:~/gar/pkg/cpan/Parse-Syslog/trunk]$ gar/bin/uwatch check-upstream 
--upstream-url="http://search.cpan.org/~dschwei/Parse-Syslog" --gar-distfiles="Parse-Syslog-1.10.tar.gz" 
--catalog-name="pm_parsesyslog" --current-version="1.09"
1.10
[wbonnet@current9s:~/gar/pkg/cpan/Parse-Syslog/trunk]$

The previous example shows the output when current version is older than upstream, while the following example shows the output of the command when the current version is newer than upstream versions.

[wbonnet@current9s:~/gar/pkg/cpan/Parse-Syslog/trunk]$ gar/bin/uwatch check-upstream 
--upstream-url="http://search.cpan.org/~dschwei/Parse-Syslog" --gar-distfiles="Parse-Syslog-1.10.tar.gz"
--catalog-name="pm_parsesyslog" --current-version="1.19"
[wbonnet@current9s:~/gar/pkg/cpan/Parse-Syslog/trunk]$

Command : get-upstream-latest-version

This command retrieves the latest version from the upstream site. Upstream URL is a mandatory argument. The regular expression used to parsed the upstream URL can either be supplied trough command line or guessed if the user supplies the DISTFILES informations. Output and command syntax is similar to the get-upstream-version-list command. Difference stand in the fact that only the latest version is displayed.

Syntax with regular expression supplied :

[wbonnet@current9s:~/gar/pkg/firefox/trunk]$ gar/bin/uwatch get-upstream-latest-version
--upstream-url="http://releases.mozilla.org/pub/mozilla.org/firefox/releases/" --regexp="\"(\d+(?:\.\d+)*)/"
3.6.13

Syntax with regular expression guessing :

[wbonnet@current9s:~/gar/pkg/cpan/Parse-Syslog/trunk]$ gar/bin/uwatch get-upstream-latest-version
--upstream-url="http://search.cpan.org/~dschwei/Parse-Syslog" --gar-distfiles="Parse-Syslog-1.10.tar.gz"
--catalog-name="pm_parsesyslog"
1.10

Command : get-upstream-version-list

This command retrieves the list of version from the upstream site. Upstream URL is a mandatory argument. The regular expression used to parsed the upstream URL can either be supplied trough command line or guessed if the user supplies the DISTFILES informations. Output and command syntax is similar to the get-upstream-latest-version command. Difference stand in the fact that all the versions matching the regular expression are displayed.

Syntax with regular expression supplied :

[wbonnet@current9s:~/gar/pkg/firefox/trunk]$ gar/bin/uwatch get-upstream-version-list 
--upstream-url="http://releases.mozilla.org/pub/mozilla.org/firefox/releases/" --regexp="\"(\d+(?:\.\d+)*)/"
2.0.0.20
3.5.16
3.6.13

Syntax with regular expression guessing :

[wbonnet@current9s:~/gar/pkg/cpan/Parse-Syslog/trunk]$ gar/bin/uwatch get-upstream-version-list 
--upstream-url="http://search.cpan.org/~dschwei/Parse-Syslog" --gar-distfiles="Parse-Syslog-1.10.tar.gz"
--catalog-name="pm_parsesyslog"
1.10

Command : upgrade-to-version

This command is used to create an upgrade branch from the current trunk version. Files from the trunk are copied to a branch named branches/upgrade_from_A_to_B. Where :

  • A is the current version
  • B is the upstream latest version
[wbonnet@current9s:~/gar/pkg/cpan/Parse-Syslog/trunk]$ ls ../branches/
[wbonnet@current9s:~/gar/pkg/cpan/Parse-Syslog/trunk]$ gar/bin/uwatch upgrade-to-version --current-version="1.09" --target-version="1.10"
[wbonnet@current9s:~/gar/pkg/cpan/Parse-Syslog/trunk]$ ls ../branches/
upgrade_from_1.09_to_1.10
[wbonnet@current9s:~/gar/pkg/cpan/Parse-Syslog/trunk]$ ls ../branches/upgrade_from_1.09_to_1.10/
Makefile   checksums  files      gar        work

Command : update-package-version-database

This command is used to report, to the uWatch database, information about GAR and Upstream versions. The following switches are used to defines informations :

  • catalog-name
  • package-name
  • execution-date
  • gar-path
  • gar-version
  • upstream-version
  • upstream-url
  • regexp
  • gar-distfiles
  • uwatch-output
  • uwatch-error

Unless you want to run a local database storing information version, you will probably never need to use this command, especially directly from command line instead of using Makefile wrapper from GAR ( gmake update-package-version-database ). If you need more information about this command, please look to the code of the wrapper in gar.lib.mk, or ask me for adding more information in this page.

Unless otherwise stated, the content of this page is licensed under Creative Commons Attribution-ShareAlike 3.0 License