Using multiple versions of BLAS and LAPACK with Gentoo/Linux Sébastien Fabbro Markus Dittrich Joshua Saddler This guide explains the use of the different implementations of the BLAS and LAPACK libraries that are available via Portage. 1.0 2007-10-22 Introduction

The Basic Linear Algebra Subroutines (BLAS) and Linear Algebra PACKage (LAPACK) are well designed linear algebra software libraries developed by the High Performance Computing (HPC) community. BLAS implements dense matrix and vectors products, while LAPACK provides routines for solving systems of linear equations. Both are widely used in many scientific applications and it is, therefore, important to have efficient implementations available.

Originally written in FORTRAN 77, a number of additional language wrappers have been developed for languages like C, C++, FORTRAN 95, and others. The following legacy libraries are available via the reference ebuilds:

  • BLAS: FORTRAN 77 implementation of BLAS
  • CBLAS: C implementation of BLAS
  • LAPACK: FORTRAN 77 implementation of LAPACK

In addition, Gentoo provides a number of optimized BLAS and LAPACK implementations that will be described below. Different implementations are bundled together with Gentoo's eselect system and the widely used pkg-config tool.

It is important to note that if you require, e.g., a well performing BLAS implementation, simply emerging X over Y often is not enough. Rather, you will have to carefully benchmark your applications since performance may depend on many factors, such as hardware or network. If you are simply looking for a well performing and well tested implementation, the reference ebuilds will likely be your best choice.

For Users
Installing

If best possible performance is not of paramount importance for you and you simply need BLAS and/or LAPACK, just emerge the virtual packages:

# emerge blas cblas lapack

These will install the reference legacy packages from http://netlib.org. They are well tested, easy to debug implementations. They should satisfy most users; if they're all you need, you're done reading.

However, if:

  • BLAS/LAPACK are critical for the speed of your applications
  • you absolutely need to build the fastest computer
  • you want to help Gentoo sci project to improve their packages

... then read on, and be sure to file bugs both to Gentoo and upstream.

There is a number of optimized implementations of these libraries in the Portage tree:

  • ATLAS: Automatically Tuned Linear Algebra Software is an open-source package that empirically tunes the library to the machine it is being compiled on. It provides BLAS (FORTRAN 77 and C), and LAPACK implementations on various architectures.
  • GotoBLAS: Goto BLAS provide open-source, free for academic use, hand-coded machine language, processor optimized versions of the FORTRAN 77 BLAS routines. Claims to be the fastest BLAS.
  • ACML: AMD Core Math Library is a closed-source but free package containing BLAS (FORTRAN 77 only) and LAPACK for x86 and amd64 architectures, but also other math tools such as statistical libraries and FFTs.
  • MKL: Intel® Math Kernel Library is a closed-source but free package for non-commercial use containing BLAS (FORTRAN 77 and C), LAPACK optimized for Intel® based architectures: x86, amd64 and ia64.

Usually performance gain is noticeable mainly with BLAS, since LAPACK routines depend on BLAS kernels.

Compiling and linking with installed libraries

We took great care to make sure that each package provides consistent pkg-config files. Hence, compiling and linking with BLAS/LAPACK should be simple and straightforward:

# pkg-config --libs blas    (To link with FORTRAN 77 BLAS library)
# pkg-config --cflags cblas (To compile against C BLAS library)
# pkg-config --libs cblas   (To link with C BLAS library)
# pkg-config --libs lapack  (To link with FORTRAN 77 LAPACK library)

pkg-config files are available for whichever implementation you select with eselect. More information on using pkg-config can be obtained with man pkg-config.

Choosing libraries

You can easily switch BLAS, CBLAS and LAPACK implementations with eselect. At this point, you probably have eselect-blas, eselect-cblas and eselect-lapack installed. If you do not, install them now! Using eselect, you can view which implementations of CBLAS are available.

# eselect cblas list
Installed CBLAS for library directory lib64
[1]   atlas
[2]   atlas-threads
[3]   gsl
[4]   mkl-threads *
[5]   reference

The implementation marked with an asterisk (*) is the currently selected implementation. To switch implementations, run:

# eselect lapack set atlas

To learn more about the eselect tool, visit the eselect guide

When selecting blas, cblas or lapack profiles try to avoid mixing different implementations since we don't have any mechanism to enforce reasonable profiles. However, here is a list of well performing profile combinations that have been used successfully in the past:

  • Most CPUs:
    • blas,cblas: atlas (or atlas-threads with multi-processor)
    • lapack:atlas
  • Most CPUs:
    • blas: goto
    • cblas,lapack: reference
  • AMD based CPUs:
    • blas,lapack: acml-gfortran (or acml-gfortran-openmp with multi-processors)
    • cblas: reference
  • Intel based CPUs:
    • blas,cblas,lapack: mkl-threads
Choosing a compiler

Most of the libraries can compile with both the GNU compiler collection and the Intel® compilers on the x86, amd64 and ia64 architectures. By default, you are probably using gcc. You can also compile the *-reference packages with ifort and icc. To do this, you need to define the F77 environment variable and don't forget the FFLAGS.

# F77=ifort FFLAGS="-O2 -mp" emerge blas-reference

Depending on your hardware, a small performance gain can be noticed thanks to vectorization. The -mp flag maintains floating-point precision, since by default ifort is pretty aggressive on floating point arithmetic, and we are actually compiling a math package. Try man ifort to see additional flags to fit your hardware.

Some of the implementations let you specify the Intel® C compiler as well. Please beware that not all libraries compile with all combinations. You should receive an error during the emerge in case you have chosen an incompatible combination.

As usual for Gentoo, there are many combinations of USE flags and compilers with which you could compile a package. Unfortunately switching compilers between BLAS and LAPACK might not be always compatible. For example:

# F77=ifort FFLAGS="-O2 -mp" USE="openmp" emerge =acml-3.6.0-r1
# eselect blas set acml-ifort-openmp
# F77=gfortran FFLAGS="-O2" emerge lapack-reference

This will most likely break things or not even compile.

Try to be consistent in your choice. Stay with the GCC most of the time will avoid you some trouble, unless you want to use the MKL, in which case the Intel® C and FORTRAN compilers make a good combination.

Documentation

If you need BLAS or LAPACK to develop your own programs, the documentation becomes pretty handy. Setting the USE="doc" flag for the corresponding BLAS or LAPACK package will install man pages and quick reference sheets from the app-doc/blas-docs and app-doc/lapack-docs packages. They are standard and valid for all implementations. For optimized packages, the USE="doc" flags will usually install extra doc in PDF or HTML format.

For Developers
Providing new implementations

The Portage tree contains many ebuilds that depend on the BLAS/CBLAS/LAPACK libraries. As there is more than one possible implementation, the Gentoo Scientific Project reorganized all the packages to provide virtual/blas, virtual/cblas, and virtual/lapack. All ebuilds using BLAS should depend on this virtual package, unless it is explicitly known to break with different BLAS implementations.

To work with Gentoo's configuration tools app-admin/eselect-{blas,cblas,lapack}, and the virtual, every ebuild that installs a BLAS implementation must fulfill following requirements:

  1. The ebuild must install an eselect file for each profile it provides. The libraries should link to the ones in /usr/$(get_libdir) directories and the include files in /usr/include:
    • libblas.so[.0] - Shared object for FORTRAN BLAS applications
    • libblas.a - Static library for FORTRAN BLAS applications
    • libcblas.so[.0] - Shared object for C/C++ CBLAS applications
    • libcblas.a - Static library for C/C++ CBLAS applications
    • cblas.h - Include header for C/C++ applications
    • liblapack.so[.0] - Shared object for FORTRAN LAPACK applications
    • liblapack.a - Static library for FORTRAN LAPACK applications
  2. The ebuild must install a blas.pc, cblas.pc and/or lapack.pc pkg-config file and therefore RDEPEND on dev-util/pkgconfig. They should also be included in the eselect files, and link to the /usr/$(get_libdir)/pkgconfig directory:
    • blas.pc - BLAS pkg-config file
    • cblas.pc - CBLAS pkg-config file
    • lapack.pc - LAPACK pkg-config file
  3. Be included in the virtual package as a possible provider:
    • virtual/blas - BLAS virtual package
    • virtual/cblas - CBLAS virtual package
    • virtual/lapack - LAPACK virtual package

The easiest way of understanding all this is probably getting inspiration from one of the available packages. Currently the Portage tree provide the following virtuals:

sci-libs/acml**sci-libs/blas-atlas**sci-libs/blas-goto*sci-libs/blas-reference*sci-libs/cblas-reference*sci-libs/gsl*sci-libs/lapack-atlas*sci-libs/lapack-reference*sci-libs/mkl***
Package name virtual/blas virtual/cblas virtual/lapack
Packages with BLAS/LAPACK dependencies

Simply use virtual/{blas,cblas,lapack} as a [R]DEPEND. To build some packages, you might need to use the pkg-config tool. If you are lucky, the package uses autotools together with common BLAS and LAPACK M4 macros.In this case, the configuration step becomes simple. For example:

econf --with-blas="$(pkg-config --libs blas)"
Don't forget to add dev-util/pkgconfig in DEPEND.
In the near future

We plan to include more standard BLAS, LAPACK libraries: BLAS 95, LAPACK 95, Sparse BLAS, ScaLAPACK. If you feel inclined to write an ebuild for these, you are more than welcomed to file it on our Bugzilla.