Usage

Experimental automated package upload utility is available for testing. To test, please update your gar sources and run csw-upload-pkg inside the bin/ directory.

maciej@login [login]:~ > ~/src/opencsw-git/gar/v2/bin/csw-upload-pkg --help
Usage: csw-upload-pkg [ options ] <pkg1> [ <pkg2> [ ... ] ]

Uploads a set of packages to the unstable catalog in opencsw-future.

- When an ARCH=all package is sent, it's added to both sparc and i386 catalogs
- When a SunOS5.x package is sent, it's added to catalogs SunOS5.x,
  SunOS5.(x+1), up to SunOS5.11.
- If a package update is sent, the tool uses catalogname to identify the
  package it's supposed to replace

The --remove option affects the same catalogs as the regular use, except that
it removes assignments of a given package to catalogs, instead of adding them.

For more information, see:
http://wiki.opencsw.org/automated-release-process#toc0

Options:
  -h, --help   show this help message and exit
  -d, --debug  
  --remove     Remove packages from catalogs instead of adding them

Once your packages are sent, you can verify the state of the checkpkg database by visiting the web frontend of the checkpkg database¹ and inspecting the catalogs you expect your package to be.

The opencsw-future tree² on the mirror host is updated a couple times per day. Your uploaded packages will appear there eventually.

In case of any problems with the usage, please re-run the tool in debug mode (—debug) and send full output to Maciej.

Infrastructure

• Buildfarm - a collection of Solaris zones and hardware hosting them
• Buildfarm database is a MySQL database on the buildfarm, on the mysql zone (mysql.bo.opencsw.org)
• Web interface for the buildfarm database http://buildfarm.opencsw.org/pkgdb/
• External REST interface at http://buildfarm.opencsw.org/pkgdb/rest/
• Internal REST interface at http://buildfarm.opencsw.org/releases/
• Master mirror at http://mirror.opencsw.org
• unstable catalog at http://mirror.opencsw.org/opencsw-future/unstable/
• Source code of those applications is available from SourceForge.

Uploading a package

Here's a rough description of what happens when you upload a package using csw-upload-pkg.

1. csw-upload-pkg examines the given file name set for correctness
   • It alerts in certain conditions, e.g. a present i386 file but missing sparc file, or an UNCOMMITTED tag
2. csw-upload-pkg runs 'pkgdb importpkg' to make sure that your package's metadata are imported to the buildfarm database
3. csw-upload-pkg queries the external rest interface for package metadata; it verifies that metadata have been uploaded successfully
4. csw-upload-pkg queries the internal rest interface to know whether package data (as opposed to metadata) are uploaded to the master mirror
   • if not yet there, csw-upload-pkg sends a POST request with package data
5. csw-upload-pkg queries the external rest interface for contents of catalogs
6. csw-upload-pkg calculates which packages to insert to which catalogs
   • Depending on the given file set and catalog contents, 5.9 packages may or may not be inserted into 5.10 and 5.11 catalogs
7. csw-upload-pkg sends a sequence of DELETE and PUT queries to the internal rest interface to modify catalogs

Catalog generation

web zone runs a cron job which wakes up every 3h and performs a set of tasks. It generates a new unstable catalog from the database, and pushes it to the master mirror.

• pkgdb is invoked, and it generates a catalog at the given location
  • pkgdb uses a direct MySQL connection
• catalog notifier is run, and sends e-mails to the maintainers of modified packages
• A cron job on unstable9x generates atom feeds for each catalog every hour. For the time being, they can be found in Ben's directory.

Connections

This image is generated from a graphviz file available from gar source code repository. It describes connections made by various infrastructure components.

Discussion

Pains with the current process:

• Released packages are sometimes still in testing/ (fixed by Dago)
• The whole process includes too many manual steps, which could be automated
• Mixed usage of testing/ for developer testing and user testing

Comparison of new catalog layout with Debian:

For each catalog there is a set of machines equipped with the packages from the catalog. The catalog is always an overlay over the more stable version. That means the catalog for testing contains all packages from current, where the packages from testing supersede the ones from current.

Experimental has the notion of "projects", which allows a grouping of packages for release (like for XML Toolchain, X11, Gnome, etc.)
Package releases are directly made out of GAR:

gmake submitpkg[-<project>]

TBD: This triggers the automated buildbot which puts the packages automatically in experimental after building.

The same procedure is triggered by the upstream watch:

1. Check out <pkg>
2. Copy trunk to tags/<pkg>-<version>_REV=… with the new version
3. Update Revision
4. gmake makesum
5. gmake garchive
6. Commit changes
7. (Trigger automatic buildbot build)
8. Mail maintainer after build has finished

In Oslo we talked about additional commands like csw-build and csw-release initiated by Trygve, maybe he can write something about the usage?

See also

• Releases and staging
• Release process