Linear Algebra on Gentoo Sébastien Fabbro This guide explains the use of linear algebra libraries and focus on how to use the different implementations of BLAS and LAPACK available on Gentoo. 1.0 2010-12-22 Introduction

There are many performant numerical libraries available. The Basic Linear Algebra Subprograms (BLAS) and the Linear Algebra PACKage (LAPACK) are well designed linear algebra libraries developed by the High Performance Computing (HPC) community. BLAS is an API of dense matrix and vectors products, while LAPACK provides routines for solving systems of linear equations. Both are widely used in many scientific applications and it is, therefore, important to have efficient implementations available.

BLAS and LAPACK were originally written in FORTRAN 77. Since then, a number of additional language wrappers have been developed for languages like C, C++, FORTRAN 95, Java, Python, etc... Netlib offers exact implementations of the APIs and they are called "reference" libraries.

  • BLAS: FORTRAN 77 and C (CBLAS) implementations of BLAS
  • LAPACK: FORTRAN 77 and C (LAPACKE) implementations of LAPACK

  • BLACS: FORTRAN 77 and C implementations of BLACS
  • ScaLAPACK: FORTRAN 77 and C implementations of PBLAS and ScaLAPACK

In addition, Gentoo provides a number of optimized implementations of the above linear algebra libraries that will be described below. You can switch between implementations with the Gentoo's eselect system and the widely used pkg-config tool.

It is important to note that if you require, e.g., a well performing BLAS implementation, simply emerging X over Y often is not enough. Rather, you will have to carefully benchmark your applications since performance may depend on many factors, such as hardware or network. If you are simply looking for a well performing and well tested implementation, the reference ebuilds will likely be your best choice.

For Users
Installing

If best possible performance is not of paramount importance for you and you simply need BLAS and/or LAPACK, just emerge the virtual package:

# emerge lapack

This will install both <><> and <><> the reference packages from http://www.netlib.org/ . They are well tested, easy to debug implementations. They should satisfy most users; if they're all you need, you're done reading.

However, if:

  • linear algebra libraries are critical for the speed of your applications
  • you absolutely need to build the fastest computer
  • you want to help Gentoo sci project to improve their packages

... then read on, and be sure to file bugs both to Gentoo and upstream.

There is a number of optimized implementations of these libraries in the Portage tree:

  • ATLAS: Automatically Tuned Linear Algebra Software is an open-source package that empirically tunes the library to the machine it is being compiled on. It provides BLAS (FORTRAN 77 and C), and LAPACK implementations on various architectures.
  • GotoBLAS: Goto BLAS provides open-source, free for academic use, hand-coded machine language, processor optimized versions of the FORTRAN 77 and C BLAS routines. Still claims to be the fastest BLAS.
  • ACML: AMD Core Math Library is a closed-source but free package containing BLAS (FORTRAN 77 only) and LAPACK for x86 and x86_64 architectures, but also other math tools such as statistical libraries and FFTs.
  • MKL: Intel® Math Kernel Library is a closed-source but free package for non-commercial use on Linux systems containing implementations of all the linear algebra libraries mentioned here.

Usually performance gain is noticeable mainly with BLAS, since LAPACK routines depend on BLAS kernels.

Developping with the installed linear algebra libraries

We took great care to make sure that each package provides consistent pkg-config files generated by us. Compiling and linking becomes straightforward:

# pkg-config --libs blas    (To link with FORTRAN 77 BLAS library)
# pkg-config --cflags cblas (To compile against C BLAS library)
# pkg-config --libs cblas   (To link with C BLAS library)
# pkg-config --libs scalapack  (To link with the ScaLAPACK library)

pkg-config files are available for all implementations and various alternatives within implementations. The default names of the implementations are: blas, cblas, lapack, lapacke, blacs and scalapack, and they can be chosen with eselect. You can also always compile or link with an library not selected for the More information on using pkg-config can be obtained with man pkg-config.

Selecting libraries

You can switch BLAS, CBLAS and LAPACK implementations with eselect. you can view which implementations of CBLAS are available.

# eselect cblas list
Installed CBLAS for library directory lib64
[1]   atlas
[2]   atlas-threads
[3]   gsl
[4]   mkl-threads *
[5]   reference

The implementation marked with an asterisk (*) is the currently selected implementation. To switch implementations, run:

# eselect blas set atlas-threads

To learn more about the eselect tool, visit the eselect guide

When selecting your linear algebra profiles try to avoid mixing different implementations since we don't have any mechanism to enforce reasonable profiles. However, here is a list of well performing profile combinations that have been used successfully in the past:

  • performant on most CPUs:
    • blas, cblas: atlas (or atlas-threads with multi-processor)
    • lapack, lapacke: atlas
  • performant on most CPUs:
    • blas, cblas: goto2
    • lapack, lapacke: reference
  • performant on AMD based CPUs:
    • blas, lapack: acml-gfortran (or acml-gfortran-openmp with multi-processors)
    • cblas: reference
  • performant on Intel based CPUs:
    • blas,cblas,lapack: mkl-threads
Choosing a compiler

All the above libraries have been tested with the GNU compiler collections (gcc, gfortran). There are many available C compilers and a few FORTRAN (ifort, Open64) compilers on Gentoo and many other FORTRAN compilers outside of Gentoo ().

# F77=ifort FFLAGS="-O2 -mp1" emerge blas-reference

Depending on your hardware, a small performance gain can be noticed thanks to vectorization. The -mp flag maintains floating-point precision, since by default ifort is pretty aggressive on floating point arithmetic, and we are actually compiling a math package. Try man ifort to see additional flags to fit your hardware.

Some of the implementations let you specify the Intel® C compiler as well. Please beware that not all libraries compile with all combinations. You should receive an error during the emerge in case you have chosen an incompatible combination.

As usual for Gentoo, there are many combinations of USE flags and compilers with which you could compile a package. Unfortunately switching compilers between BLAS and LAPACK might not be always compatible. For example:

# USE=ifort emerge acml
# eselect blas set acml-ifort-openmp
# FC=gfortran FFLAGS="-O2" emerge lapack-reference

This will most likely break things or not even compile.

Try to be consistent in your choice. Stay with the GCC most of the time will avoid you some trouble, unless you want to use the MKL, in which case the Intel compilers make a good combination.

Documentation

If you need BLAS or LAPACK to develop your own programs, the documentation becomes pretty handy. Setting the USE="doc" flag for the corresponding BLAS or LAPACK package will install man pages and quick reference sheets from the app-doc/blas-docs and app-doc/lapack-docs packages. They are standard and valid for all implementations. For optimized packages, the USE="doc" flags will usually install extra doc in PDF or HTML format.

For ebuild developers
Packages with BLAS/LAPACK dependencies

You need two things: set [R]DEPEND to virtual/. To build some packages, you m need to use the pkg-config tool. If you are lucky, the package uses autotools together with the autoconf <>AX_BLAS and <>AX_LAPACK M4 macros. In this case, the configuration step becomes simple. For example:

econf --with-blas="$(pkg-config --libs blas)"
Providing new implementations

The Portage tree contains many ebuilds that depend on the BLAS/CBLAS/LAPACK/BLACS/ScaLAPACK libraries. As there is more than one possible implementation, the Gentoo Science Project reorganized all the packages to provide virtual. All ebuilds using should depend on this virtual package, unless it is explicitly known to break with a specific implementation.

To work with Gentoo's configuration tools app-admin/eselect-{blas,cblas,lapack}, and the virtual, every ebuild that installs a BLAS implementation must fulfill following requirements:

  1. The ebuild must install an eselect file for each profile it provides. The libraries should link to the ones in /usr/$(get_libdir) directories and the include files in /usr/include:
    • libblas.so[.0] - Shared object for FORTRAN BLAS applications
    • libblas.a - Static library for FORTRAN BLAS applications
    • libcblas.so[.0] - Shared object for C/C++ CBLAS applications
    • libcblas.a - Static library for C/C++ CBLAS applications
    • cblas.h - Include header for C/C++ applications
    • liblapack.so[.0] - Shared object for FORTRAN LAPACK applications
    • liblapack.a - Static library for FORTRAN LAPACK applications
  2. The ebuild must install a blas.pc, cblas.pc and/or lapack.pc pkg-config file and therefore RDEPEND on dev-util/pkgconfig. They should also be included in the eselect files, and link to the /usr/$(get_libdir)/pkgconfig directory:
    • blas.pc - BLAS pkg-config file
    • cblas.pc - CBLAS pkg-config file
    • lapack.pc - LAPACK pkg-config file
  3. Be included in the virtual package as a possible provider:
    • virtual/blas - BLAS virtual package
    • virtual/cblas - CBLAS virtual package
    • virtual/lapack - LAPACK virtual package

The easiest way of understanding all this is probably getting inspiration from one of the available packages. Currently the Portage tree provide the following virtual packages:

sci-libs/acml**sci-libs/atlas****sci-libs/gotoblas2**sci-libs/blas-reference*sci-libs/cblas-reference*sci-libs/gsl*sci-libs/lapack-reference*sci-libs/mkl******
Package name virtual/blas virtual/cblas virtual/lapack virtual/lapacke virtual/blacs virtual/scalapack
Benchmarks

If you feel inclined to write an ebuild for these, you are more than welcomed to file it on our Bugzilla.