From 62d213b65e148467ad41e3a073a2123c66dd1572 Mon Sep 17 00:00:00 2001 From: Sébastien Fabbro Date: Sat, 8 Jan 2011 19:05:57 +0000 Subject: mask root svn --- docs/proj/#linalg.xml# | 551 +++++++++++++++++++++++++++++++++++++++++++++++++ docs/proj/linalg.xml | 550 ++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 1101 insertions(+) create mode 100644 docs/proj/#linalg.xml# create mode 100644 docs/proj/linalg.xml (limited to 'docs/proj') diff --git a/docs/proj/#linalg.xml# b/docs/proj/#linalg.xml# new file mode 100644 index 000000000..3105039cc --- /dev/null +++ b/docs/proj/#linalg.xml# @@ -0,0 +1,551 @@ + + + + + +Linear Algebra on Gentoo + + + Sébastien Fabbro + + + + This guide explains the use of linear algebra libraries and focus on + how to use the different implementations of BLAS and LAPACK available on Gentoo. + + + + + + +1.0 +2010-12-22 + + +Introduction +
+ + +

+ There are many + performant numerical libraries available. + The Basic Linear Algebra Subprograms (BLAS) and the Linear Algebra PACKage (LAPACK) + are well designed linear algebra libraries developed by the + High Performance Computing (HPC) community. BLAS is an API of dense + matrix and vectors products, while LAPACK provides routines for + solving systems of linear equations. Both are widely used in + many scientific applications and it is, therefore, important to + have efficient implementations available. +

+ +

+ BLAS and LAPACK were originally written in FORTRAN 77. Since then, a + number of additional language wrappers have been developed for + languages like C, C++, FORTRAN 95, Java, Python, etc... + Netlib offers exact implementations of the APIs and they are called + "reference" libraries. There is also some parallel implementations + for +

+ +
    +
  • + BLAS: FORTRAN 77 and C + (CBLAS) implementations of BLAS +
    • +
  • + LAPACK: FORTRAN 77 and + C (LAPACKE) implementations of LAPACK +
    • +
+ +

+ +

+ +
    +
  • + BLACS: FORTRAN 77 and C + implementations of BLACS +
    • +
  • + ScaLAPACK: FORTRAN 77 and + C implementations of PBLAS and ScaLAPACK +
    • +
+ +

+ In addition, Gentoo provides a number of optimized implementations + of the above linear algebra libraries that will be described + below. You can switch between implementations with the + Gentoo's eselect system and the widely used pkg-config + tool. +

+ +

+ It is important to note that if you require, e.g., a well performing + BLAS implementation, simply emerging X over Y often is not enough. Rather, you will have + to carefully benchmark your applications since performance may depend + on many factors, + such as hardware or network. + If you are simply looking for a well performing and well tested + implementation, the reference ebuilds will likely be your best choice. +

+ + + +
+ + + +For Users +
+Installing + + +

+ If best possible performance is not of paramount importance for you + and you simply need BLAS and/or LAPACK, just emerge the virtual + package: +

+ +
+# emerge lapack
+
+ +

+ This will install both <><> and <><> the reference packages from + http://www.netlib.org/ . They are well tested, easy to debug + implementations. They should satisfy most users; if they're all you need, you're + done reading. +

+ +

+However, if: +

+ +
    +
  • linear algebra libraries are critical for the speed of your applications
    • +
  • you absolutely need to build the fastest computer
    • +
  • you want to help Gentoo sci project to improve their packages
    • +
+ +

+... then read on, and be sure to file bugs both to Gentoo and upstream. +

+ +

+ There is a number of optimized implementations of these libraries in the Portage + tree: +

+ +
    +
  • + ATLAS: Automatically + Tuned Linear Algebra Software is an open-source package that empirically + tunes the library to the machine it is being compiled on. It provides BLAS + (FORTRAN 77 and C), and LAPACK implementations on various architectures. +
    • +
  • + GotoBLAS: + Goto BLAS provides open-source, free for academic use, hand-coded + machine language, processor optimized versions of the FORTRAN 77 + and C BLAS routines. Still claims to be the fastest BLAS. +
    • +
  • + ACML: + AMD Core Math Library is a closed-source but free package containing BLAS (FORTRAN 77 + only) and LAPACK for x86 and x86_64 architectures, but also other math tools + such as statistical libraries and FFTs. +
    • +
  • + MKL: + Intel® Math Kernel Library is a closed-source but free package for + non-commercial use on Linux systems containing implementations of all the linear + algebra libraries mentioned here. +
    • +
+ +

+ Usually performance gain is noticeable mainly with BLAS, since LAPACK routines + depend on BLAS kernels. +

+ + +
+ + +
+Developping with the installed linear algebra libraries + + +

+ We took great care to make sure that each package provides + consistent pkg-config files generated by us. + Compiling and linking becomes straightforward: +

+ +
+# pkg-config --libs blas    (To link with FORTRAN 77 BLAS library)
+# pkg-config --cflags cblas (To compile against C BLAS library)
+# pkg-config --libs cblas   (To link with C BLAS library)
+# pkg-config --libs scalapack  (To link with the ScaLAPACK library)
+
+ +

+ pkg-config files are available for all implementations and + various alternatives within implementations. The default names of + the implementations are: blas, cblas, lapack, lapacke, blacs and + scalapack, and they can be chosen with eselect. You can also always compile or link + with an library not selected for the + More information on using pkg-config can be obtained with man pkg-config. +

+ + +
+
+Selecting libraries + + +

+ You can switch BLAS, CBLAS and LAPACK implementations with + eselect. you can view which implementations of CBLAS + are available. +

+ +
+# eselect cblas list
+Installed CBLAS for library directory lib64
+[1]   atlas
+[2]   atlas-threads
+[3]   gsl
+[4]   mkl-threads *
+[5]   reference
+
+ +

+ The implementation marked with an asterisk (*) is the currently + selected implementation. To switch implementations, run: +

+ +
+# eselect blas set atlas-threads
+
+ +

+ To learn more about the eselect tool, visit the + eselect guide +

+ +

+ When selecting your linear algebra profiles try to avoid mixing + different implementations since we don't have any mechanism to enforce + reasonable profiles. However, here is a list of well performing + profile combinations that have been used successfully in the past: +

+
    +
  • performant on most CPUs: +
      +
    • blas, cblas: atlas (or atlas-threads with multi-processor)
      • +
    • lapack, lapacke: atlas
      • +
    +
    • +
  • performant on most CPUs: +
      +
    • blas, cblas: goto2
      • +
    • lapack, lapacke: reference
      • +
    +
    • +
  • performant on AMD based CPUs: +
      +
    • blas, lapack: acml-gfortran (or acml-gfortran-openmp with + multi-processors)
      • +
    • cblas: reference
      • +
    +
    • +
  • performant on Intel based CPUs: +
      +
    • blas,cblas,lapack: mkl-threads
      • +
    +
    • +
+ + +
+ +
+Choosing a compiler + + +

+ All the above libraries have been tested with the GNU compiler + collections (gcc, gfortran). + There are many available C compilers and a few FORTRAN (ifort, + Open64) compilers on Gentoo and many other FORTRAN compilers outside + of Gentoo (). +

+ +
+# F77=ifort FFLAGS="-O2 -mp1" emerge blas-reference
+
+ +

+ Depending on your hardware, a small performance gain can be noticed thanks to + vectorization. The -mp flag maintains floating-point precision, since by + default ifort is pretty aggressive on floating point arithmetic, and we are + actually compiling a math package. Try man ifort to see additional flags + to fit your hardware. +

+ +

+ Some of the implementations let you specify the Intel® C compiler as + well. Please beware that not all libraries compile with all + combinations. You should receive an error during the emerge in case you have + chosen an incompatible combination. +

+ +

+ As usual for Gentoo, there are many combinations of USE flags and + compilers with which you could compile a package. Unfortunately + switching compilers between BLAS and LAPACK might not be always + compatible. For example: +

+ +
+# USE=ifort emerge acml
+# eselect blas set acml-ifort-openmp
+# FC=gfortran FFLAGS="-O2" emerge lapack-reference
+
+ +

+ This will most likely break things or not even compile. +

+ +

+ Try to be consistent in your choice. Stay with the GCC most of the time will + avoid you some trouble, unless you want to use the MKL, in which case the Intel + compilers make a good combination. +

+ + +
+
+Documentation + + +

+ If you need BLAS or LAPACK to develop your own programs, the documentation + becomes pretty handy. Setting the USE="doc" flag for the corresponding BLAS or + LAPACK package will install man pages and quick reference sheets from the + app-doc/blas-docs and app-doc/lapack-docs packages. They are + standard and valid for all implementations. For optimized packages, the + USE="doc" flags will usually install extra doc in PDF or HTML format. +

+ + +
+ + + +For ebuild developers +
+ +
+Packages with BLAS/LAPACK dependencies + + +

+ You need two things: + set [R]DEPEND to virtual/. To build some + packages, you m need to use the pkg-config tool. If you are lucky, the + package uses autotools together with the autoconf <>AX_BLAS and <>AX_LAPACK M4 + macros. In this case, the configuration step becomes simple. For example: +

+ +
+econf --with-blas="$(pkg-config --libs blas)"
+
+ + +
+ + + +Providing new implementations + + +

+ The Portage tree contains many ebuilds that depend on the + BLAS/CBLAS/LAPACK/BLACS/ScaLAPACK libraries. As there is more than + one possible implementation, the Gentoo Science Project + reorganized all the packages to provide virtual. All ebuilds using + should depend on this virtual package, unless it is explicitly + known to break with a specific implementation. +

+ +

+ To work with Gentoo's configuration tools + app-admin/eselect-{blas,cblas,lapack}, and the virtual, every ebuild that + installs a BLAS implementation must fulfill following requirements: +

+ +
    +
  1. + The ebuild must install an eselect file for each profile it provides. The + libraries should link to the ones in /usr/$(get_libdir) + directories and the include files in /usr/include: +
      +
    • + libblas.so[.0] - Shared object for FORTRAN BLAS + applications +
      • +
    • + libblas.a - Static library for FORTRAN BLAS applications +
      • +
    • + libcblas.so[.0] - Shared object for C/C++ CBLAS applications +
      • +
    • + libcblas.a - Static library for C/C++ CBLAS applications +
      • +
    • cblas.h - Include header for C/C++ applications
      • +
    • + liblapack.so[.0] - Shared object for FORTRAN LAPACK + applications +
      • +
    • + liblapack.a - Static library for FORTRAN LAPACK applications +
      • +
    +
    2. +
  2. + The ebuild must install a blas.pc, cblas.pc and/or + lapack.pc pkg-config file and therefore RDEPEND on + dev-util/pkgconfig. They should also be included in the eselect + files, and link to the /usr/$(get_libdir)/pkgconfig directory: +
      +
    • blas.pc - BLAS pkg-config file
      • +
    • cblas.pc - CBLAS pkg-config file
      • +
    • lapack.pc - LAPACK pkg-config file
      • +
    +
    3. +
  3. Be included in the virtual package as a possible provider: +
      +
    • virtual/blas - BLAS virtual package
      • +
    • virtual/cblas - CBLAS virtual package
      • +
    • virtual/lapack - LAPACK virtual package
      • +
    +
    4. +
+ +

+ The easiest way of understanding all this is probably getting inspiration from + one of the available packages. Currently the Portage tree provide the following + virtual packages: +

+ + + + + + + + + + + + + sci-libs/acml + * + + * + + + + + + sci-libs/atlas + * + * + * + * + + + + + sci-libs/gotoblas2 + * + * + + + + + + + sci-libs/blas-reference + * + + + + + + + + sci-libs/cblas-reference + + * + + + + + + + sci-libs/gsl + + * + + + + + + + sci-libs/lapack-reference + + + * + + + + + + sci-libs/mkl + * + * + * + * + * + * + +
Package namevirtual/blasvirtual/cblasvirtual/lapackvirtual/lapackevirtual/blacsvirtual/scalapack
+ + +
+ + + + +Benchmarks +
+ + +

+ If you feel inclined to write an ebuild for these, you + are more than welcomed to file it on our Bugzilla. +

+ + +
+ + + diff --git a/docs/proj/linalg.xml b/docs/proj/linalg.xml new file mode 100644 index 000000000..9e83460f7 --- /dev/null +++ b/docs/proj/linalg.xml @@ -0,0 +1,550 @@ + + + + + +Linear Algebra on Gentoo + + + Sébastien Fabbro + + + + This guide explains the use of linear algebra libraries and focus on + how to use the different implementations of BLAS and LAPACK available on Gentoo. + + + + + + +1.0 +2010-12-22 + + +Introduction +
+ + +

+ There are many + performant numerical libraries available. + The Basic Linear Algebra Subprograms (BLAS) and the Linear Algebra PACKage (LAPACK) + are well designed linear algebra libraries developed by the + High Performance Computing (HPC) community. BLAS is an API of dense + matrix and vectors products, while LAPACK provides routines for + solving systems of linear equations. Both are widely used in + many scientific applications and it is, therefore, important to + have efficient implementations available. +

+ +

+ BLAS and LAPACK were originally written in FORTRAN 77. Since then, a + number of additional language wrappers have been developed for + languages like C, C++, FORTRAN 95, Java, Python, etc... + Netlib offers exact implementations of the APIs and they are called + "reference" libraries. +

+ +
    +
  • + BLAS: FORTRAN 77 and C + (CBLAS) implementations of BLAS +
    • +
  • + LAPACK: FORTRAN 77 and + C (LAPACKE) implementations of LAPACK +
    • +
+ +

+ +

+ +
    +
  • + BLACS: FORTRAN 77 and C + implementations of BLACS +
    • +
  • + ScaLAPACK: FORTRAN 77 and + C implementations of PBLAS and ScaLAPACK +
    • +
+ +

+ In addition, Gentoo provides a number of optimized implementations + of the above linear algebra libraries that will be described + below. You can switch between implementations with the + Gentoo's eselect system and the widely used pkg-config + tool. +

+ +

+ It is important to note that if you require, e.g., a well performing + BLAS implementation, simply emerging X over Y often is not enough. Rather, you will have + to carefully benchmark your applications since performance may depend + on many factors, + such as hardware or network. + If you are simply looking for a well performing and well tested + implementation, the reference ebuilds will likely be your best choice. +

+ + + +
+ + + +For Users +
+Installing + + +

+ If best possible performance is not of paramount importance for you + and you simply need BLAS and/or LAPACK, just emerge the virtual + package: +

+ +
+# emerge lapack
+
+ +

+ This will install both <><> and <><> the reference packages from + http://www.netlib.org/ . They are well tested, easy to debug + implementations. They should satisfy most users; if they're all you need, you're + done reading. +

+ +

+However, if: +

+ +
    +
  • linear algebra libraries are critical for the speed of your applications
    • +
  • you absolutely need to build the fastest computer
    • +
  • you want to help Gentoo sci project to improve their packages
    • +
+ +

+... then read on, and be sure to file bugs both to Gentoo and upstream. +

+ +

+ There is a number of optimized implementations of these libraries in the Portage + tree: +

+ +
    +
  • + ATLAS: Automatically + Tuned Linear Algebra Software is an open-source package that empirically + tunes the library to the machine it is being compiled on. It provides BLAS + (FORTRAN 77 and C), and LAPACK implementations on various architectures. +
    • +
  • + GotoBLAS: + Goto BLAS provides open-source, free for academic use, hand-coded + machine language, processor optimized versions of the FORTRAN 77 + and C BLAS routines. Still claims to be the fastest BLAS. +
    • +
  • + ACML: + AMD Core Math Library is a closed-source but free package containing BLAS (FORTRAN 77 + only) and LAPACK for x86 and x86_64 architectures, but also other math tools + such as statistical libraries and FFTs. +
    • +
  • + MKL: + Intel® Math Kernel Library is a closed-source but free package for + non-commercial use on Linux systems containing implementations of all the linear + algebra libraries mentioned here. +
    • +
+ +

+ Usually performance gain is noticeable mainly with BLAS, since LAPACK routines + depend on BLAS kernels. +

+ + +
+ + +
+Developping with the installed linear algebra libraries + + +

+ We took great care to make sure that each package provides + consistent pkg-config files generated by us. + Compiling and linking becomes straightforward: +

+ +
+# pkg-config --libs blas    (To link with FORTRAN 77 BLAS library)
+# pkg-config --cflags cblas (To compile against C BLAS library)
+# pkg-config --libs cblas   (To link with C BLAS library)
+# pkg-config --libs scalapack  (To link with the ScaLAPACK library)
+
+ +

+ pkg-config files are available for all implementations and + various alternatives within implementations. The default names of + the implementations are: blas, cblas, lapack, lapacke, blacs and + scalapack, and they can be chosen with eselect. You can also always compile or link + with an library not selected for the + More information on using pkg-config can be obtained with man pkg-config. +

+ + +
+
+Selecting libraries + + +

+ You can switch BLAS, CBLAS and LAPACK implementations with + eselect. you can view which implementations of CBLAS + are available. +

+ +
+# eselect cblas list
+Installed CBLAS for library directory lib64
+[1]   atlas
+[2]   atlas-threads
+[3]   gsl
+[4]   mkl-threads *
+[5]   reference
+
+ +

+ The implementation marked with an asterisk (*) is the currently + selected implementation. To switch implementations, run: +

+ +
+# eselect blas set atlas-threads
+
+ +

+ To learn more about the eselect tool, visit the + eselect guide +

+ +

+ When selecting your linear algebra profiles try to avoid mixing + different implementations since we don't have any mechanism to enforce + reasonable profiles. However, here is a list of well performing + profile combinations that have been used successfully in the past: +

+
    +
  • performant on most CPUs: +
      +
    • blas, cblas: atlas (or atlas-threads with multi-processor)
      • +
    • lapack, lapacke: atlas
      • +
    +
    • +
  • performant on most CPUs: +
      +
    • blas, cblas: goto2
      • +
    • lapack, lapacke: reference
      • +
    +
    • +
  • performant on AMD based CPUs: +
      +
    • blas, lapack: acml-gfortran (or acml-gfortran-openmp with + multi-processors)
      • +
    • cblas: reference
      • +
    +
    • +
  • performant on Intel based CPUs: +
      +
    • blas,cblas,lapack: mkl-threads
      • +
    +
    • +
+ + +
+ +
+Choosing a compiler + + +

+ All the above libraries have been tested with the GNU compiler + collections (gcc, gfortran). + There are many available C compilers and a few FORTRAN (ifort, + Open64) compilers on Gentoo and many other FORTRAN compilers outside + of Gentoo (). +

+ +
+# F77=ifort FFLAGS="-O2 -mp1" emerge blas-reference
+
+ +

+ Depending on your hardware, a small performance gain can be noticed thanks to + vectorization. The -mp flag maintains floating-point precision, since by + default ifort is pretty aggressive on floating point arithmetic, and we are + actually compiling a math package. Try man ifort to see additional flags + to fit your hardware. +

+ +

+ Some of the implementations let you specify the Intel® C compiler as + well. Please beware that not all libraries compile with all + combinations. You should receive an error during the emerge in case you have + chosen an incompatible combination. +

+ +

+ As usual for Gentoo, there are many combinations of USE flags and + compilers with which you could compile a package. Unfortunately + switching compilers between BLAS and LAPACK might not be always + compatible. For example: +

+ +
+# USE=ifort emerge acml
+# eselect blas set acml-ifort-openmp
+# FC=gfortran FFLAGS="-O2" emerge lapack-reference
+
+ +

+ This will most likely break things or not even compile. +

+ +

+ Try to be consistent in your choice. Stay with the GCC most of the time will + avoid you some trouble, unless you want to use the MKL, in which case the Intel + compilers make a good combination. +

+ + +
+
+Documentation + + +

+ If you need BLAS or LAPACK to develop your own programs, the documentation + becomes pretty handy. Setting the USE="doc" flag for the corresponding BLAS or + LAPACK package will install man pages and quick reference sheets from the + app-doc/blas-docs and app-doc/lapack-docs packages. They are + standard and valid for all implementations. For optimized packages, the + USE="doc" flags will usually install extra doc in PDF or HTML format. +

+ + +
+ + + +For ebuild developers +
+ +
+Packages with BLAS/LAPACK dependencies + + +

+ You need two things: + set [R]DEPEND to virtual/. To build some + packages, you m need to use the pkg-config tool. If you are lucky, the + package uses autotools together with the autoconf <>AX_BLAS and <>AX_LAPACK M4 + macros. In this case, the configuration step becomes simple. For example: +

+ +
+econf --with-blas="$(pkg-config --libs blas)"
+
+ + +
+ + + +Providing new implementations + + +

+ The Portage tree contains many ebuilds that depend on the + BLAS/CBLAS/LAPACK/BLACS/ScaLAPACK libraries. As there is more than + one possible implementation, the Gentoo Science Project + reorganized all the packages to provide virtual. All ebuilds using + should depend on this virtual package, unless it is explicitly + known to break with a specific implementation. +

+ +

+ To work with Gentoo's configuration tools + app-admin/eselect-{blas,cblas,lapack}, and the virtual, every ebuild that + installs a BLAS implementation must fulfill following requirements: +

+ +
    +
  1. + The ebuild must install an eselect file for each profile it provides. The + libraries should link to the ones in /usr/$(get_libdir) + directories and the include files in /usr/include: +
      +
    • + libblas.so[.0] - Shared object for FORTRAN BLAS + applications +
      • +
    • + libblas.a - Static library for FORTRAN BLAS applications +
      • +
    • + libcblas.so[.0] - Shared object for C/C++ CBLAS applications +
      • +
    • + libcblas.a - Static library for C/C++ CBLAS applications +
      • +
    • cblas.h - Include header for C/C++ applications
      • +
    • + liblapack.so[.0] - Shared object for FORTRAN LAPACK + applications +
      • +
    • + liblapack.a - Static library for FORTRAN LAPACK applications +
      • +
    +
    2. +
  2. + The ebuild must install a blas.pc, cblas.pc and/or + lapack.pc pkg-config file and therefore RDEPEND on + dev-util/pkgconfig. They should also be included in the eselect + files, and link to the /usr/$(get_libdir)/pkgconfig directory: +
      +
    • blas.pc - BLAS pkg-config file
      • +
    • cblas.pc - CBLAS pkg-config file
      • +
    • lapack.pc - LAPACK pkg-config file
      • +
    +
    3. +
  3. Be included in the virtual package as a possible provider: +
      +
    • virtual/blas - BLAS virtual package
      • +
    • virtual/cblas - CBLAS virtual package
      • +
    • virtual/lapack - LAPACK virtual package
      • +
    +
    4. +
+ +

+ The easiest way of understanding all this is probably getting inspiration from + one of the available packages. Currently the Portage tree provide the following + virtual packages: +

+ + + + + + + + + + + + + sci-libs/acml + * + + * + + + + + + sci-libs/atlas + * + * + * + * + + + + + sci-libs/gotoblas2 + * + * + + + + + + + sci-libs/blas-reference + * + + + + + + + + sci-libs/cblas-reference + + * + + + + + + + sci-libs/gsl + + * + + + + + + + sci-libs/lapack-reference + + + * + + + + + + sci-libs/mkl + * + * + * + * + * + * + +
Package namevirtual/blasvirtual/cblasvirtual/lapackvirtual/lapackevirtual/blacsvirtual/scalapack
+ + +
+ + + + +Benchmarks +
+ + +

+ If you feel inclined to write an ebuild for these, you + are more than welcomed to file it on our Bugzilla. +

+ + +
+ + + -- cgit v1.2.3-18-g5258