Proposal: shared library placement

Background

Historically, shared libraries have a standard place at OpenCSW: /opt/csw/lib, with optional subdirectories for additional architectures, if necessary. When compiling new software, typically the following linker flags are used: -L/opt/csw/lib -R/opt/csw/lib. They link binaries against libraries in /opt/csw/lib and set the RPATH to allow binaries to find their needed shared libraries at runtime.

There are certain shared objects which are not public shared libraries; for example Kerberos ships libraries against which Kerberos binaries link, but other binaries don't. Therefore, there is a need to distinguish between private and public shared libraries.

Shared libraries have the status of a common resource, and once put into place, they serve many binaries and many packages. The original software that put them in place is a client of that resource. Once any other package starts to link against a shared library, it has to stay in place until all the dependent binaries go away or stop depending on it. The original package that provided them might be even removed from the catalog, but shared libraries must stay until relieved.

Many shared libraries in the OpenCSW catalog are kept in /opt/csw/lib, but some aren't. One typical reason for some libraries to be placed elsewhere is that in order to support more than one version of certain piece of software, a custom prefix was used. This was typically done by passing --prefix=/opt/csw/foo to the configure script. Usually, the goal has been to have a separate directory for executables that would otherwise conflict with the ones in /opt/csw/bin. As a side effect, shared libraries produced by that piece of software, were put into /opt/csw/foo/lib, as opposed to /opt/csw/lib.

Practical problems arise when compiling a new piece of software, which is meant to link against libraries installed under a custom prefix. If the dependency was hard, the typical scenario was that the configure script returned an error, stating that it couldn't find a needed set of libraries. There's also a worse scenario, in which the dependent project does not display an error message, but instead simply skips support for the project under a custom prefix. This might be discovered belatedly, potentially after the release.

Unlike executables, shared libraries do have their own mechanism of supporting multiple versions of the same library. Every time a backward-incompatible change is made, the shared library receives a new SONAME, which effectively creates a new file on disk, allowing both versions of a shared library to coexist in one directory. Therefore, supporting multiple versions is not a valid technical reason for shared libraries to be kept outside the standard location, /opt/csw/lib.

Shared libraries should be viewed as a system-wide, common resource1. The fact that some installers, e.g. driven by the --prefix flag, may put shared libraries into a nonstandard directory, should not determine the final visibility of shared libraries.

This proposal is intended to be merged (when accepted) to packaging shared libraries.

Goals

  • Focused and consistent management of shared libraries
  • Non-ambiguous discrimination between public and private shared libraries
  • Reduction of anomalies in build procedures of dependent software projects
  • Decoupling the issue of placing shared libraries from the issue of supporting multiple versions of the same software on one system

Non-goals

  • Suggesting a way to support multiple versions of the same software on one system
  • Placing all executables under /opt/csw/bin
  • Codifying a method of linking to older versions of a library

Overview

Public shared libraries

Public shared libraries, the ones that other software projects can link to, should2 have presence in /opt/csw/lib. Presence is defined, in OS terminology, as availability of library data, through an open() call on the /opt/csw/lib/<soname> path. In the case of 64-bit binaries and binaries optimized for specific architectures, it can be /opt/csw/lib/<isa>/<soname>.

Private shared libraries

Private shared libraries, the ones that other software projects cannot and/or should not link to, should be put under a custom subdirectory under /opt/csw/lib, e.g. /opt/csw/lib/foo. Binaries that need private shared libraries need to be linked using additional -L and -R flags. This way, other compiled software will be safe from accidentally linking to a private library. If a library previously thought to be private is in fact needed by other software, it has to be first made available from the shared library location, and ensured to conform to the OpenCSW shared library packaging standard.

64-bit libraries should be visible under /opt/csw/lib/foo/amd64 and /opt/csw/lib/foo/sparcv9 on their respective platforms.

For more information, see private shared libraries section of the shared library policy.

*-config scripts

Development packages often provide *-config script, e.g. /opt/csw/bin/curl-config. These script should return the /opt/csw/lib path when asked for --libs or --ldflags. For example:

$ curl-config --libs
-L/opt/csw/lib -lcurl -xarch=v8 (...)

Implementation details

If a piece of software is compiled with the standard OpenCSW prefix (/opt/csw), public shared libraries are usually already at the right place. If there are any private libraries in /opt/csw/lib, they can be moved to a subdirectory after the "make install" step. It is important for these libraries to not be available from the default shared library location. A specific linker flag (e.g. -R /opt/csw/lib/foo) might be necessary to ensure that private libraries are found at runtime.

Software compiled with a nonstandard prefix defined by the --prefix flag, will most likely also accept the --libdir flag. An example of ./configure flags:

./configure --prefix=/opt/csw/foo --libdir=/opt/csw/lib

If the build system does not put shared libraries into /opt/csw/lib by default, two approaches are possible:

  1. Patching the build scripts, passing arguments or environment variables
  2. Moving libraries after the "make install" step (or its equivalent, depending on the build system)

In the second case, care needs to be taken that the *-config script (e.g. /opt/csw/lib/python-config) returns the /opt/csw/lib path.

Caveats

Linking to older versions of a library

Some pieces of software require linking to old versions of libraries. There are at least two known ways of achieving this, at least one of them is compatible with this proposal. Specific solutions to this problem are outside of the scope of this document.

C++ libraries compiled with GCC

GCC and Solaris Studio use different name mangling conventions. C++ libraries compiled with GCC cannot be used by binaries compiled with Solaris Studio. However, the SONAMEs are the same, independent of the compiler. This leads to a SONAME collision. If there is such a conflict, it can be worked around by isolating GCC-compiled C++ libraries in a specific directory, e.g. /opt/csw/lib/gcc.

The problem does not affect C libraries.

Unless otherwise stated, the content of this page is licensed under Creative Commons Attribution-ShareAlike 3.0 License