PyCBC for Einstein@Home

This page is about building the PyCBC software suite and in particular the pycbc_inspiral binary to run on various Einstein@Home clients.

The files referred to on this page are maintained in the standard PyCBC repository at github (tools/einsteinathome).

Debian 7 (Wheezy) & 5.0 (Lenny)

(Note: this section is outdated and kept for historical reference only)

I started with the oldest Debian version that natively features Python 2.7, which is Debian 7.

To reproduce the build, set up a minimal system from a netinstall image ( debian-7.9.0-amd64-netinst.iso) on a VM. Then install the following packages (screen and emacs are only for convenience):

screen emacs openssh-server git-core ant gcc g++ gfortran automake autoconf make libtool pkg-config libfftw3-dev libgsl0-dev libpcre3-dev libfreetype6-dev libjpeg-dev libpng-dev libhdf5-dev libmysqlclient-dev libpq-dev liblapack-dev python python-pip python-virtualenv python-dev

The build script can be downloaded directly from github ( pycbc_build_lenny-wheezy.sh), should be made executable and run.

It should just run and leave you with a self-contained binary (PyInstaller "onefile" bundle) in pycbc/environment/dist/pycbc_inspiral.

The script then was extended to compile what's not available or too old on Debian 5 (Lenny), including Python 2.7, and to automatically detect this particular Debian version (by checking /etc/debian_version). libhdf5-dev is not available on Debian Lenny and should be left out from the list of packages to install above.

Notes

• the binary is ~70MB in size, every instance expands to 210MB on the client machine

• the build script has not been maintained for a long time, i.e. since this build worked once on Debian 5.0 (Lenny)

• this build is not suitable for running on Einstein@Home, as the libc the binary refers to is too new for some clients (GLIBC 2.7 on Debian 5). However it may serve as a reasonable basis for a VM application.

• this build is set up for maximum compatibility instead of maximum performance. LAPACK & BLAS are the plain vanilla reference implementations, FFTW is 3.3.3 without SIMD or thread usage (on Lenny)

• The PyCBC suite is a beast. For nothing else than building the PyCBC suite on a Debian 7 system I had to install ~150 Debian packages (libraries and tools), 50+ Python (pip) packages and manually compile half a dozen software packages (LALSuite being only one of these).

Debian 4.0 (Etch) / Ubuntu 6.06 (Dapper Drake)

This is the target system for Einstein@Home Linux clients (GLIBC 2.3.6).

Machine Setup

• set up a VM with Debian 4 "Etch" ( debian-40r0-amd64-netinst.iso, minimal system w/o network mirror, IDE HD, 2048 MB RAM, 2 cores) (or ubuntu-6.06.2-server-amd64.iso)

• download pycbc_etch64_compile-usr-local.sh to your user account, e.g. "boinc":

wget --no-check-certificate https://github.com/ligo-cbc/pycbc/raw/master/tools/einsteinathome/pycbc_etch64_compile-usr-local.sh
chmod +x pycbc_*.sh

• as root: cd /usr/local/src and run /home/boinc/pycbc_etch64_compile-usr-local.sh. This will install Python 2.7 and newer versions of some software (autotools, git etc.) to /usr/local

Build

Download and run the build script:

rm -f pycbc_build_eah.sh
wget --no-check-certificate https://github.com/ligo-cbc/pycbc/raw/master/tools/einsteinathome/pycbc_build_eah.sh
chmod +x pycbc_build_eah.sh
./pycbc_build_eah.sh

This should leave you with a zipped "onedir" PyInstaller bundle pycbc_inspiral.zip linking to GLIBC 2.3.6 (and newer) and a "scipy weave code cache" pythoncompiled.zip as well as the tools progress and fstab, all in pycbc-build/environment/dist.

Notes

• pycbc_etch64_compile-usr-local.sh compiles a gcc-4.8.5 suite, the newest that compiles without problems on that old system, and the one compatible with the other installations (OSX, Cygwin). gcc-7.1.0 compiles when configured with --disable-libsanitizer.

• There is a hidden dependency between the gcc used (and shared objects compiled with it) and libpython.so, so if gcc is updated, Python and the whole application needs to be recompiled (from scratch/sources).

• To ease maintenance, the build script has been merged with that for Cygwin and OSX, and has been extended for other Linux versions, too. It uses various OS-specific methods to auto-detect the system which it runs on.

• OpenSSL-1.0.1p is compiled manually. It turned out to be non-trivial to find a version that builds on top of the old GLIBC and is compatible with pyOpenSSL-0.13

• LAPACK and BLAS are compiled from their unoptimized reference implementation. To be used with scipy-0.16.0 it's important that the deprecated functions are built and available (see RedHat Bugzilla: scipy/linalg/_flapack.so: undefined symbol: sgegv_)

• Obsolete For quite some time I struggeled with what looked like an error in the scipy build. "ImportError: cannot import name doccer" is seriously misleading, as it's not actually the "doccer" module that fails to load. You get to see the actual linking error ("python2.7/site-packages/scipy/linalg/cython_lapack.so: undefined symbol: zlacn2_ ") when you try to run a pyinstaller directory bundle or just run "python -c 'from scipy.io.wavfile import write as write_wav'" after scipy was installed. The reason I figured is that the atlas3 liblapack.so of the system that a self-built (pip installed) scipy links to is apparently broken (or possibly doesn't feature the "deprecated" functions that this scipy version requires). The solution is to remove the atlas3 lapack/blas libs from the system and compile an own liblapack. I still don't know what stupid limitation of Python swallowed the actual linking error message, getting to see this would have saved me a week of work. The "import" line above is now part of the build script testing the scipy installation right after the build.

• Obsolete (but kept for reference): psycopg-2.6 (required by pegasus-wms) doesn't compile on Debian Etch (or Ubuntu Dapper Drake). The linker complains "/usr/bin/ld: build/temp.linux-x86_64-2.7/psycopg/psycopgmodule.o: relocation R_X86_64_PC32 against `lobjectType' can not be used when making a shared object; recompile with -fPIC". However, according to the logs, the compilation of psycopgmodule.c is done with "-fPIC", and compiling it again doesn't help. I tried different versions of libpq, gcc and binutils (ld) to no avail. So the script actually compiles psycopg-2.5.5 and fakes its version information to be 2.6.

• Obsolete due to different requirements of different python modules involved (pyinstaller,virtualenv vs. pycbc) the version of setuptools is downgraded during the build process (setuptools==18.7.1 -> setuptools==18.2). I sure hope this doesn't introduce some more hidden incompatibilities.

• The Linux build also produces "onefile" binaries of "pycbc_condition_strain" (for "autogating" a timeseries in Frame files) and "pycbc_inspiral_osg", the latter contains the weave cache bundle with the application and thus is really self-contained.

• For use with Jenkins, pycbc_etch64_compile-usr-local.sh installs JRE 1.6.45u in /opt/java

Windows 7 / Cygwin

Machine Setup

1. Setup a (virtual) machine with Win7 64 Bit (I used 2 cores, 3GB RAM, 180GB HD). The name of the user account should not contain blanks/spaces

2. Install Cygwin 64 Bit Version from Cygwin.com (https://cygwin.com/setup-x86_64.exe). I used the mirror http://ftp.inf.tu-dresden.de . Install (required):

• Archive: hdf5, unzip, zip
• Database: libmysqlclient-devel, libmysqlclient18
• Devel: autoconf (2.5+wrapper), automake (1.15+wrapper), binutils, cmake, gcc-core, gcc-g++, gcc-gfortran, git, libhdf5-devel, libhdf5_10, libhdf5cpp, libtool, make, mingw64-x86_64-gcc-core, openssl-devel, patch, pkg-config, swig
• Libs: libfreetype-devel, libfreetype6, libgsl-devel, libgsl0, libjpeg-devel, libpcre-devel, libpng-devel, libpq-devel
• Net: openssh
• Python: python
• Science: gsl
• Web: wget
• Utils: upx

Recommended:

• Admin: cygrunsrv, shutdown
• Editors: emacs
• Utils: screen

NOTE: unlike in e.g. Debian apt, -devel packages don't include / require the corresponding standard/runtime packages, so you need to select both separately (e.g gsl, hdf5)

2a. Alternatively (and probably simpler) install from an old, but pre-packaged version of Cygwin:

• Download Install_Cygwin64_PyCBC.zip and unpack to a local directory
• run Cygwin-setup-x86_64.exe
• At "Choose A Download Source" select "Install from Local Directory"
• Confirm "Select Root Directory" (C:\cygwin64)
• At "Select Local Package Directory" just confirm the local directory of the unpacked archive
• At "Select Packages" click on "Default" next to "+ All" to read "Install", then "Next"
• Confirm all following warnings & messages

3. After Cygwin installation finished, open a Cygwin shell (a "shortcut" should be on the desktop) and run a few commands:

python -m ensurepip
pip install --upgrade pip
pip install virtualenv

Build

Download and run the build script:

rm -f pycbc_build_eah.sh
wget --no-check-certificate https://github.com/ligo-cbc/pycbc/raw/master/tools/einsteinathome/pycbc_build_eah.sh
chmod +x pycbc_build_eah.sh
./pycbc_build_eah.sh

This should leave you with a zipped "onedir" PyInstaller bundle pycbc_inspiral.zip and a "scipy weave code cache" pythoncompiled.zip, as well as tools progress.exe and fstab.exe, all in pycbc-build/environment/dist/. The latter creates a fake "/etc/fstab" for the Cygwin process with a mount point referring to the project directory of the BOINC client.

Notes

• This was done with a Cygwin installation as of late 2015 (setup 2.873), with gcc-4.9.3. Newer installations may install a newer version of gcc and may behave differently, these have not been tested. To reproduce that installation, use method 2a above.

• Notes for Debian 4.0 (Etch) apply here as well (with exception of OpenSSL)

• to build DLLs with libtool, -no-undefined gets added to LDFLAGS of libraries by the script for LibFrame, METAIO and LALSuite. Although I'm not entirely sure this really applies to all libraries that are built, it seems to work without errors or problems at least for the ones that are actually used in the current application.

• The additional program "fstab.exe" is only needed on Windows/Cygwin. It creates a fake "/etc/fstab", which make the BOINC project directory accessible as e.g. "/project/albert.phys.uwm.edu/", such that the application can access boinc-resolved paths like "../../project/albert.phys.uwm.edu/job.xml". On all other platforms, a copy of (/usr)/bin/true can be used.

• The "wrapper" for Windows that is necessary to run this as a BOINC application needs to be built separately with M$VC. Use the branch eah_wrapper_improvements of git://gitlab.aei.uni-hannover.de/einsteinathome/boinc.git. As of Apr 2017, this may also be cross-built with MinGW-W64 and the Makekefile.mingw in BOINC/lib.

• It was tried (April 2017) to install and use Cygwin64 in a wine-2.6 (development), but without success. The installation runs through, but even a dash crashes, apparently due to unimplemented Windows API calls.

Hints for convenience

1. Autologin: In a VM running on your local machine with only local networking login usually just holds you up. Open a terminal ("cmd.exe") and type "control userpasswords2" to disable this.

2. If you really need case-sensitive filenames in Cygwin (so far I didn't), set registry value HKLM\SYSTEM\CurrentControlSet\Control\Session Manager\kernel\obcaseinsensitive to 0 and reboot the (virtual) machine.

3. Setup ssh server for remote command-line access:

• open a Cygwin terminal as Admin (right-click -> run as administrator)
• in that shell, run "ssh-host-config" and follow the instructions. You will probably run "cygrunsrv -S sshd" at the end
• Make sure the Windows firewall permits connection to port 22 from outside

3a. For public key authentication I had to

• open a Cygwin terminal as Admin (right-click -> run as administrator)
• mkdir /home/cyg_server && chown cyg_server /home/cyg_server && chmod 700 /home/cyg_server
• 'manage Computer' -> Local Users and Groups -> cyg_server -> change /var/empty to /home/cyg_server and /bin/false to /bin/bash

4. if you get errors from fork of the type "Resource temporarily unavailable", do

• stop "Cygwin sshd" service (serch for "services")
• close all locally open Cygwin windows/terminals
• with Windows Explorer, navigate to C:\Cygwin64\bin, right-click on "ash" and "Run as Administrator"
• type "/bin/rebaseall"
• close the shell ("exit") and start "Cygwin sshd" again

Mac OSX 10.7 (Lion)

10.7 is the minimum OSX version required by PyInstaller

Machine Setup

• Install Xcode (4.6.3), command-line tools (April 2013) and MacPorts
• run (once)

sudo port selfupdate
sudo port install python27 gcc48 git wget autoconf automake pkgconfig
( cd /opt/local/bin; sudo ln -s /usr/bin/clang )
sudo port select --set python python27
sudo python -m ensurepip
export PATH="$PATH:/opt/local/Library/Frameworks/Python.framework/Versions/2.7/bin"
hash -r
sudo pip install virtualenv

Build

Download and run the build script:

rm -f pycbc_build_eah.sh
wget --no-check-certificate https://github.com/ligo-cbc/pycbc/raw/master/tools/einsteinathome/pycbc_build_eah.sh
chmod +x pycbc_build_eah.sh
./pycbc_build_eah.sh

This should leave you with a zipped "onedir" PyInstaller bundle pycbc_inspiral.zip and a "scipy weave code cache" pythoncompiled.zip, as well as tools progress and fstab, all in pycbc-build/environment/dist/.

Notes

• Getting the wrapper to build (needs to be done on OSX 10.6) is a mess, in particular to link the static version of libstdc++

• The most work for the OSX port actually was required to overcome the limitations of the Frame-reading code in LALSupport / LALFrame / FrameLib, that chokes at various place when there are blanks / spaces in the "sanitized" "realpath" of the frame files. This never happens on standard Linux installations and is overcome differently on Windows/Cygwin (see "fstab.exe" tool), but is pretty common on OSX. The patch necessary to fix parsing of URLs with blanks in LAL Frame Cache files has been pushed upstream (9706666a), the OSX application uses ldas-tools FrameCPP instead of the FrameLib.

Einstein@Home

The PyCBC application is currently set up on Albert@Home.

Application

The application consists of

• A BOINC "wrapper" as the main executable. As the standard wrappers published by BOINC are outdated, more recent versions (built from 3765e54e) are attached here:
  • wrapper_x86_64-pc-linux-gnu
  • wrapper_windows_x86_64.exe
  • wrapper_x86_64-apple-darwin

• A program "fstab". This is built by the script, it only actually does something for Cygwin/Windows. For all other OS, a copy of /bin/true (Linux) or /usr/bin/true (OSX) may be used as well.

• (currently) a program "progress". This reads the stderr output of pycbc_inspiral looking for indicators of (filtering) progress, and writes a file "progress.txt" to be read by the wrapper, which then communicates the "fraction done" to the BOINC client. Recently this functionality has been built into pycbc_inspiral itself to avoid the additional I/O that verbose pycbc_inspiral output causes, so this program can be dropped in future configurations.

• The zip archives pycbc_inspiral.zip (application) and pythoncompiled.zip (weave code cache) as built by the script.

The wrapper needs a "job (description) file" that describes what to do. In current configuration this is shipped with the workunits, as it might be different for each such workunit. If the only differences are in the data files and command-lines to use, the job file might become part of the application in future configurations, the fake "fstab" programs may then be omitted from Linux and OSX "applications".

Workunits

Workunits (that repeat the search for GW150914) can be created using the script create_work_CBC1.sh. It references

• the standard boinc create_work utility
• the input- and output templates result_template_CBC1+stderr and input_template_CBC1
• the job file job-GW150914.xml for the wrapper
• the publicly available GW150914 data file H-H1_LOSC_4_V1-1126257414-4096.gwf and
• the template bank H1L1-PREGEN_TMPLTBANK_SPLITBANK_BANK16-1126051217-3331800.xml.gz

Validator

A standalone validator (that natively reads HDF5 files) is being worked on. For now, the following setup is used:

• The actual validator is a standard BOINC script_validator with command-line arguments --init_script CBC1H1hdf2txt.sh --compare_script validator_compare_CBC1.sh. It references and uses:
• A standalone validator_test_CBC1 build from einsteinathome/project-daemons
• A Python script CBC1H5tool5col.py that converts the HDF5 result file to an ASCII table
• A wrapper script CBC1H1hdf2txt.sh that converts all input files and vaildates each with validator_test_CBC1
• A wrapper script validator_compare_CBC1.sh that compares only the first file of each result (file name ending in "_0") using validator_test_CBC1

Assimilator, File Deleter

As the results of PyCBC are ordinary files, the usual E@H components assimilator (build from einsteinathome/project-daemons or copy binary from old search) and file deleter (build from standard BOINC or copy binary from old search) are used.

Condor-BOINC-GAHP

A HTCondor (submit) interface to a BOINC project (here: Albert@Home) is under heavy development, and currently even the future development process is subject to discussion. The test setup on Albert@Home is documented in an (AEI internal) Redmine ticket.

Resources

Aditional info that was or might become useful

Debian

• HowToForge: Using Old Debian Versions

Cygwin

• Using Cygwin
• Setting up a custom Cygwin packge server / mirror
• cygwin.mirror.constant.com - older Cygwin packages
• upx-3.07-1.tar.bz2
• command-line options for setup.exe
• installing Cygwin offline

Manage Cygwin packages from command-line (wasn't used)

• Manage Cygwin packages on command-line
• apt-cyg · GitHub
• How do I install cygwin components from the command line? - Stack Overflow

DLLs

• HOWTO Create and Deploy a Sample DLL using MinGW
• DLLs with Libtool
• Cygwin Wiki: rebaseall
• Cygwin rebase-3.0.1 README
• couldn't allocate cygwin heap, Win32 error 487 (and I've already done rebaseall)
• Advanced MinGW DLLl Topics: Setting The DLL Base Address
• blog omega prime: Everything You Never Wanted To Know About DLLs
• Looking for exported symbols in a DLL with objdump?

Useful tools for (non-Cygwin) Windows

• Sourceforge GnuWin32 unzip-5.51-1-bin.zip
• UltraVNC (download from heise.de)
• PuTTY
• BOINC Wrapper application
• Sanos source, including glob.h/c

OSX

• Homebrew — The missing package manager for OS X
• Fink: Debian tools for OSX/Darwin
• MacPorts
• Should I use Fink, MacPorts, Homebrew, or something else
• Deploying Applications With the C++ Runtime

Portable coding

• Insane Coding: Creating portable Linux binaries
• Platform Independent Coding - DLLs and SOs
• GCC 4.8.5 manual

GLIBC installation (dead end)

• Glibc 2 HOWTO
• Glibc Installation HOWTO

Python

• Instant Python
• Crash into Python
• Traps for the Unwary in Python’s Import System
• Python 2 Tutorial (German)
• Scipy Weave Tutorial
• Python traceback module
• Installing Python Modules
• Python Packaging User Guide: Installing Packages
• Easy Install

PyInstaller

• PyInstaller Bundled Software for PyCBC Analysis: v1.2.6 README: Hints on hidden imports for PyInstaller
• PyInstaller manual: giving run-time options
• PyInstaller 2.1 Manual ( PDF)

HDF5

• HDF5 Source Code
• HDF5 C++ API Class list
• HDF5 Tutorial: Command-line Tools for Viewing HDF5 Files
• HDF5 Predefined Datatypes
• Introduction to HDF5
• HDF5 Tools API Specification

BOINC

• BOINC Wrapper
• Job submission
• Job templates
• Remote submission
• Condor-BOINC
• AEI issue tracker #2282: Set up (Condor) "remote submission" on Albert@Home

LSC

• Data release for event GW150914
• Linux binaries for use on OSG