Experimental automated package upload utility is available for testing. To test, please update your gar sources and run csw-upload-pkg inside the bin/ directory.
maciej@login [login]:~ > ~/src/opencsw-git/gar/v2/bin/csw-upload-pkg --help Usage: csw-upload-pkg [ options ] <pkg1> [ <pkg2> [ ... ] ] Uploads a set of packages to the unstable catalog in opencsw-future. - When an ARCH=all package is sent, it's added to both sparc and i386 catalogs - When a SunOS5.x package is sent, it's added to catalogs SunOS5.x, SunOS5.(x+1), up to SunOS5.11. - If a package update is sent, the tool uses catalogname to identify the package it's supposed to replace The --remove option affects the same catalogs as the regular use, except that it removes assignments of a given package to catalogs, instead of adding them. For more information, see: http://wiki.opencsw.org/automated-release-process#toc0 Options: -h, --help show this help message and exit -d, --debug --remove Remove packages from catalogs instead of adding them
Once your packages are sent, you can verify the state of the checkpkg database by visiting the web frontend of the checkpkg database1 and inspecting the catalogs you expect your package to be.
The opencsw-future tree2 on the mirror host is updated a couple times per day. Your uploaded packages will appear there eventually.
In case of any problems with the usage, please re-run the tool in debug mode (—debug) and send full output to Maciej.
- Buildfarm - a collection of Solaris zones and hardware hosting them
- Buildfarm database is a MySQL database on the buildfarm, on the mysql zone (mysql.bo.opencsw.org)
- Web interface for the buildfarm database http://buildfarm.opencsw.org/pkgdb/
- External REST interface at http://buildfarm.opencsw.org/pkgdb/rest/
- Internal REST interface at http://buildfarm.opencsw.org/releases/
- Master mirror at http://mirror.opencsw.org
- unstable catalog at http://mirror.opencsw.org/opencsw-future/unstable/
- Source code of those applications is available from SourceForge.
Uploading a package
Here's a rough description of what happens when you upload a package using csw-upload-pkg.
- csw-upload-pkg examines the given file name set for correctness
- It alerts in certain conditions, e.g. a present i386 file but missing sparc file, or an UNCOMMITTED tag
- csw-upload-pkg runs 'pkgdb importpkg' to make sure that your package's metadata are imported to the buildfarm database
- csw-upload-pkg queries the external rest interface for package metadata; it verifies that metadata have been uploaded successfully
- csw-upload-pkg queries the internal rest interface to know whether package data (as opposed to metadata) are uploaded to the master mirror
- if not yet there, csw-upload-pkg sends a POST request with package data
- csw-upload-pkg queries the external rest interface for contents of catalogs
- csw-upload-pkg calculates which packages to insert to which catalogs
- Depending on the given file set and catalog contents, 5.9 packages may or may not be inserted into 5.10 and 5.11 catalogs
- csw-upload-pkg sends a sequence of DELETE and PUT queries to the internal rest interface to modify catalogs
web zone runs a cron job which wakes up every 3h and performs a set of tasks. It generates a new unstable catalog from the database, and pushes it to the master mirror.
- pkgdb is invoked, and it generates a catalog at the given location
- pkgdb uses a direct MySQL connection
- catalog notifier is run, and sends e-mails to the maintainers of modified packages
- A cron job on unstable9x generates atom feeds for each catalog every hour. For the time being, they can be found in Ben's directory.
This image is generated from a graphviz file available from gar source code repository. It describes connections made by various infrastructure components.
Pains with the current process:
- Released packages are sometimes still in testing/ (fixed by Dago)
- The whole process includes too many manual steps, which could be automated
- Mixed usage of testing/ for developer testing and user testing
Comparison of new catalog layout with Debian:
|Debian||OpenCSW||Who puts stuff there||an example command|
|Experimental||experimental/||by the maintainer||cp <pkg> /home/experimental/<name>|
|Unstable||unstable/||by the maintainer||csw-upload-pkg|
|Testing / Named release||testing/ | kiel/||by an automated job||not implemented|
|Stable / Named release||stable/ | dublin/||by release manager||rm stable; ln -s dublin stable|
For each catalog there is a set of machines equipped with the packages from the catalog. The catalog is always an overlay over the more stable version. That means the catalog for testing contains all packages from current, where the packages from testing supersede the ones from current.
|e8s / e8x||u8s / u8x||t8s / t8x||s8s / s8x|
|e9s / e9x||u9s / u9x||t9s / t9x||s9s / s9x|
|e10s / e10x||u10s / u10x||t10s / t10x||s10s / s10x|
|eosols / eosolx||uosols / uosolx||tosols / tosolx||sosols / sosolx|
Experimental has the notion of "projects", which allows a grouping of packages for release (like for XML Toolchain, X11, Gnome, etc.)
Package releases are directly made out of GAR:
This checks if everything has been committed (like now when CSW is put in instead of UNCOMMITTED) and makes a server-side copy in SVN to tags as tags/[<project>_]<pkg>-<version>,REV=… (implemented in gar/v2-pbuild).
TBD: This triggers the automated buildbot which puts the packages automatically in experimental after building.
The same procedure is triggered by the upstream watch:
- Check out <pkg>
- Copy trunk to tags/<pkg>-<version>_REV=… with the new version
- Update Revision
- gmake makesum
- gmake garchive
- Commit changes
- (Trigger automatic buildbot build)
- Mail maintainer after build has finished
In Oslo we talked about additional commands like csw-build and csw-release initiated by Trygve, maybe he can write something about the usage?