Adding Gnulib To A Project

Last Updated: November 17, 2011
Version: 1.1

History:
1.1: Updated license discussion; Added See Also section


As many projects these days are written for systems with glibc, they may not always build cleanly on Solaris. While it may be possible #ifdef around the problem to include alternate headers or use alternate but equivalent functions, this doesn't always work. In these situations, it may be simpler to add gnulib support to the project. The gnulib project is a source-level compatibility library that provides compatible headers and function implementations for many of the commonly use functions that cause problems going from glibc -> $anything_else.

Knowing When gnulib is Helpful

If you should encounter a project that won't build, you may see errors like:

tmpwatch.c:387: warning: implicit declaration of function 'stpcpy'
tmpwatch.c:643: warning: implicit declaration of function 'getopt_long'
tmpwatch.c:696: warning: implicit declaration of function 'strtoimax'

You may also get linker errors about similar function names when the linker can't find the symbols in any available library. In these cases, where the functions are standard fare on glibc systems, gnulib becomes a good candidate.

Prerequisites

While gnulib isn't that difficult to use, it is heavily dependent on the build system of the project you're integrating it into. In short, if the project isn't using Autoconf and Automake, you aren't using gnulib. It's still technically possible, of course, but the amount of work required would go up by several orders of magnitude.

Licensing is also something to take into consideration. Many of the gnulib modules are now licensed under GPL3. You should ensure that the modules you need to use are license compatible with your project. To see which license a particular module is using, see the modules/$modulename file. The Fedora Project maintains a nice license compatibility matrix. If your project isn't compatible with the current gnulib licensing, you could consider checking out commit id 5861339993f3014cfad1b94fc7fe366fc2573598, which just preceeds the bulk of the license changes.

Getting Started

Before you add gnulib support to a project, you'll need to have gnulib availabile. The project maintainers use git for version control, so you can create a local clone of the project. I keep my clone in ~/. To clone the repo, use the command:

git clone git://git.savannah.gnu.org/gnulib.git

Steps To Integrate

Integrating gnulib can be fairly simple, but it depends on the layout of the existing project. You'll need to do the following high level things to leverage this source library:

  1. Import the required gnulib modules to this project with gnulib-tool
  2. Hook gnulib into Automake
  3. Hook gnulib into Autoconf
  4. Ensure that proper include statements are used in your code.

Some Practical "How to Get Nice Patches" Stuff

To generate patches useful for GAR, you're going to want to use some sort of local VCS tool. I prefer git due to it's incredible flexibility, but feel free to use the same tool as upstream if you have access to the public view of their project. I'll assume git is the tool in use for this purpose and that your upstream source is a tarball.

Extract the upstream tarball into a working directory, preferably somewhere that won't get wiped out in a GAR clean operation. Technically, you could do this all within the $(WORKSRC) directory since GAR now has integrated Git support. I'd recommend not doing this though as, at least until you've done it a few times, this is something that will require lots of iterative fiddling. You'd be well advised to do this outside of GAR until you're familiar with the process.

Initialize the upstream source into a new git repository with the command:

git init

Add the existing source to the repository with:

git add .

And then snapshot this source tree:

git commit -m 'upstream master'

Now, do all of your work on a branch:

git checkout -b add_gnulib

At each logical step in the following discussion, you'll want to make a new commit. This is done by adding the files (or parts of files) that represent a logical chunk in the process. This would look something like:

git add lib/
git add m4/
git commit -m 'imported gnulib modules'

When you've completed a successful gnulib surgery, you can obtain a nicely formatted list of patches that can be used as PATCHFILES in your GAR recipe with the command:

git format-patch master

The token 'master' in the above command is the default primary branch in a git project. If you followed along exactly with the above, it would represent the commit with the message 'upstream master.' You'll receive a set of files, numbered in order of commit, with filenames built from the first line of your commit message.

The above assumes that you're not expecting upstream to accept the patches. You wouldn't necessarily want to create a patch including lib/ and m4/ for submission to the project, as they'd run the gnulib checkout command themselves (more below). As long as you've got logical units for your patches though, this one could be left out of your submission. It's still useful locally though, until such time as upstream has merged in the gnulib support (eg: continue to include it in your PATCHFILES as necessary).

Importing The Modules

The gnulib library is very modular, allowing you to pull in only the required functionality. For example, the tmpwatch project required portable versions of stpcpy, strtoimax and getopt.

The getopt-gnu name is one option of several getopt modules. You can use getopt-posix or simply getopt. To see a list of all available modules, use the command:

gnulib-tool --list

You're now ready to actually import the modules. You'll notice that dependencies for the required modules are added automatically and that the tail end of the output outlines the next steps to finish the integration:

$ ~/gnulib/gnulib-tool --no-vc-files --import getopt-gnu stpcpy strtoimax
Module list with included dependencies:
  arg-nonnull
  extensions
  getopt-gnu
  getopt-posix
  gettext-h
  include_next
  inttypes
  multiarch
  stddef
  stdint
  stdlib
  stpcpy
  string
  strtoimax
  strtol
  strtoll
  unistd
  verify
  warn-on-use
  wchar
File list:
  build-aux/arg-nonnull.h
  build-aux/warn-on-use.h
  lib/dummy.c
  lib/getopt.c
  lib/getopt.in.h
  lib/getopt1.c
  lib/getopt_int.h
  lib/gettext.h
  lib/inttypes.in.h
  lib/stddef.in.h
  lib/stdint.in.h
  lib/stdlib.in.h
  lib/stpcpy.c
  lib/string.in.h
  lib/strtoimax.c
  lib/strtol.c
  lib/strtoll.c
  lib/unistd.in.h
  lib/verify.h
  lib/wchar.in.h
  m4/00gnulib.m4
  m4/extensions.m4
  m4/getopt.m4
  m4/gnulib-common.m4
  m4/include_next.m4
  m4/inttypes-pri.m4
  m4/inttypes.m4
  m4/longlong.m4
  m4/multiarch.m4
  m4/stddef_h.m4
  m4/stdint.m4
  m4/stdlib_h.m4
  m4/stpcpy.m4
  m4/string_h.m4
  m4/strtoimax.m4
  m4/strtol.m4
  m4/strtoll.m4
  m4/unistd_h.m4
  m4/warn-on-use.m4
  m4/wchar.m4
  m4/wchar_t.m4
  m4/wint_t.m4
Creating directory ./lib
Creating directory ./m4
Copying file ./arg-nonnull.h
Copying file ./warn-on-use.h
Copying file lib/dummy.c
Copying file lib/getopt.c
Copying file lib/getopt.in.h
Copying file lib/getopt1.c
Copying file lib/getopt_int.h
Copying file lib/gettext.h
Copying file lib/inttypes.in.h
Copying file lib/stddef.in.h
Copying file lib/stdint.in.h
Copying file lib/stdlib.in.h
Copying file lib/stpcpy.c
Copying file lib/string.in.h
Copying file lib/strtoimax.c
Copying file lib/strtol.c
Copying file lib/strtoll.c
Copying file lib/unistd.in.h
Copying file lib/verify.h
Copying file lib/wchar.in.h
Copying file m4/00gnulib.m4
Copying file m4/extensions.m4
Copying file m4/getopt.m4
Copying file m4/gnulib-common.m4
Copying file m4/gnulib-tool.m4
Copying file m4/include_next.m4
Copying file m4/inttypes-pri.m4
Copying file m4/inttypes.m4
Copying file m4/longlong.m4
Copying file m4/multiarch.m4
Copying file m4/stddef_h.m4
Copying file m4/stdint.m4
Copying file m4/stdlib_h.m4
Copying file m4/stpcpy.m4
Copying file m4/string_h.m4
Copying file m4/strtoimax.m4
Copying file m4/strtol.m4
Copying file m4/strtoll.m4
Copying file m4/unistd_h.m4
Copying file m4/warn-on-use.m4
Copying file m4/wchar.m4
Copying file m4/wchar_t.m4
Copying file m4/wint_t.m4
Creating lib/Makefile.am
Creating m4/gnulib-cache.m4
Creating m4/gnulib-comp.m4
Finished.

You may need to add #include directives for the following .h files.
  #include <getopt.h>
  #include <inttypes.h>
  #include <string.h>

You may need to use the following Makefile variables when linking.
Use them in <program>_LDADD when linking a program, or
in <library>_a_LDFLAGS or <library>_la_LDFLAGS when linking a library.
  $(LTLIBINTL) when linking with libtool, $(LIBINTL) otherwise

Don't forget to
  - add "lib/Makefile" to AC_CONFIG_FILES in ./configure.ac,
  - mention "lib" in SUBDIRS in Makefile.am,
  - mention "-I m4" in ACLOCAL_AMFLAGS in Makefile.am,
  - mention "m4/gnulib-cache.m4" in EXTRA_DIST in Makefile.am,
  - invoke gl_EARLY in ./configure.ac, right after AC_PROG_CC,
  - invoke gl_INIT in ./configure.ac.

The —no-vc-files options may or may not be of use to you. It directs gnulib-tool not to alter .gitignore, .cvsignore or other VCS ignore files when doing the import. If you'll be submitting the patches upstream, you should likely omit this option as your patches don't need to include any of the imported files save m4/gnulib-cache.m4. If, on the other hand, you'll be carrying the patch locally within GAR, you'll want to include the lib/* and m4/* files as part of your patch series.

The m4/gnulib-cache.m4 file provides the info required for a later gnulib-tool call to be simply:

$ ~/gnulib/gnulib-tool --import

Thus, the upstream author(s) can apply your first patch that contains only the addition of this file and then import their own copies of the required modules.

Other useful options to gnulib-tool include —source-base=DIR and —m4-base=DIR. If your project already has its own files in the lib/ directory or m4 macros in the m4/ directory, you should use these to have your import place files in alternate locations. When used, the instructions at the tail end of the import output will be modified accordingly.

Hook gnulib into Automake

Now that you've imported the source files, it's time to tell Automake about them. For small projects, this is reasonably easy. For larger projects, more changes to Makefile.am (and potentially Makefile.am in sub directories) will be required. Again, we'll use tmpwatch as an example here. It is a very simply Automake setup, but provides good examples of the key ideas.

The SUBDIRS variable tells Automake which other directories in your project also contain Makefile.am files. This list either needs to be altered to include the 'lib' directory (assuming standard gnulib-tool import options) or created with 'lib' as the only value. The following line sufficed for tmpwatch:

SUBDIRS = lib

The ACLOCAL_AMFLAGS value in Makefile.am allows you to set arguments that the aclocal program will make use of. Aclocal actually parses configure.ac, but values in Makefile.am are pulled in. Modifying the value to include:

-I m4

instructs aclocal to make use of the m4 macros that gnulib-tool imported. It saves you having to call aclocal like `aclocal -I m4` by hand, which in turns keeps the tool `autoreconf` as a possibility.

Adding m4/gnulib-cache.m4 to EXTRA_DIST in Makefile.am ensures that this file will be included in tarballs that `gmake dist` creates.

Next, you'll want to ensure that headers and libraries can be found by the preprocessor and the linker. In a simple project, it should suffice to add '-Ilib' to both AM_CPPFLAGS and AM_LDFLAGS. You'll want these near or at the beginning of these variable declarations so that they're given preference where required.

Lastly, you need to tell your Makefile.am where and when to include the archive lib/gnulib.a (or .la if you have libtool in use support). This is controlled with various $target_LDADD variables. For example, the following statement in the tmpwatch Makefile.am tells Automake that the tmpwatch target needs to be linked with libgnu.a:

tmpwatch_LDADD = $(top_srcdir)/lib/libgnu.a

At this point, as far as Automake is concerned, you should be in good shape.

Hook gnulib into Autoconf

Now, you need to alter your configure.ac so that it becomes aware of gnulib. The following two statements should be added to configure.ac:

gl_EARLY
gl_INIT

gl_EARLY, as instructed, should immediately follow AC_PROG_CC. I typically add gl_INIT right after _EARLY, but it could go later in the file. Lastly, you need to add lib/Makefile to the list of files that configure should generate after running. This is done by adding it to the list in AC_CONFIG_FILES.

Ensure Proper Include Statements

Last, you should check that the .c files in your project use the appropriate include statements for the preprocessor. In most projects, this will already be done. You may encounter cases where some auto-detection of appropriate headers is already done. In this case, you might need to override this. One example might be a project that knew to use <sys/inttypes.h> on Solaris while using <stdint.h> on Linux. You'd want to ensure that <stdint.h> was used in all cases now.

Hooking gnulib Patches Into GAR

Now that you've got a set of patches representing the addition of gnulib to your project, you'll need to ensure that you make GAR use them. Adding the list of patches to your PATCHFILES statement won't quite be enough. You'll also need to recalibrate autoconf and automake prior to running the configure script. To do this, use a pre-configure-modulated hook along the lines of:

pre-configure-modulated:
   @(cd $(WORKSRC); autoreconf -i )
   @$(MAKECOOKIE)

The above command assumes a fairly standard autoconf/automake setup. It may not work in all cases, depending on how the project uses the tools (eg: is it following GNU style, foreign style, etc). The autoreconf command tells the auto* tools to recreate Makefile.in's from Makefile.am's, configure from configure.ac, etc. For more information on this, see the (soon to be written) autoconf/automake tutorial. In some projects, you may find some already included script that can be used to re-bootstrap the auto* tools. I've seen this named 'bootstrap.sh' and 'autogen.sh,' but other names are sure to be found. If autoreconf -i isn't working, poke around for such a script.

See Also

Unless otherwise stated, the content of this page is licensed under Creative Commons Attribution-ShareAlike 3.0 License