PyCBC for Einstein@Home
This page is about building the PyCBC software suite and in particular the
pycbc_inspiral
binary to run on various Einstein@Home clients.
The files referred to on this page are maintained in the
standard PyCBC repository at github (tools/einsteinathome).
Debian 7 (Wheezy) & 5.0 (Lenny)
(Note: this section is outdated and kept for historical reference only)
I started with the oldest Debian version that natively features Python 2.7, which is Debian 7.
To reproduce the build, set up a minimal system from a netinstall image (
debian-7.9.0-amd64-netinst.iso) on a VM. Then install the following packages (screen and emacs are only for convenience):
screen emacs openssh-server git-core ant gcc g++ gfortran automake autoconf make libtool pkg-config libfftw3-dev libgsl0-dev libpcre3-dev libfreetype6-dev libjpeg-dev libpng-dev libhdf5-dev libmysqlclient-dev libpq-dev liblapack-dev python python-pip python-virtualenv python-dev
The build script can be downloaded directly from github (
pycbc_build_lenny-wheezy.sh), should be made executable and run.
It should just run and leave you with a self-contained binary (PyInstaller "onefile" bundle) in
pycbc/environment/dist/pycbc_inspiral
.
The script then was extended to compile what's not available or too old on Debian 5 (Lenny), including Python 2.7, and to automatically detect this particular Debian version (by checking
/etc/debian_version
). libhdf5-dev is not available on Debian Lenny and should be left out from the list of packages to install above.
Notes
- the binary is ~70MB in size, every instance expands to 210MB on the client machine
- the build script has not been maintained for a long time, i.e. since this build worked once on Debian 5.0 (Lenny)
- this build is not suitable for running on Einstein@Home, as the libc the binary refers to is too new for some clients (GLIBC 2.7 on Debian 5). However it may serve as a reasonable basis for a VM application.
- this build is set up for maximum compatibility instead of maximum performance. LAPACK & BLAS are the plain vanilla reference implementations, FFTW is 3.3.3 without SIMD or thread usage (on Lenny)
- The PyCBC suite is a beast. For nothing else than building the PyCBC suite on a Debian 7 system I had to install ~150 Debian packages (libraries and tools), 50+ Python (pip) packages and manually compile half a dozen software packages (LALSuite being only one of these).
Debian 4.0 (Etch) / Ubuntu 6.06 (Dapper Drake)
This is the target system for Einstein@Home Linux clients (GLIBC 2.3.6).
Machine Setup
- download
pycbc_etch64_compile-usr-local.sh
to your user account, e.g. "boinc":
wget --no-check-certificate https://github.com/ligo-cbc/pycbc/raw/master/tools/einsteinathome/pycbc_etch64_compile-usr-local.sh
chmod +x pycbc_*.sh
- as root:
cd /usr/local/src
and run /home/boinc/pycbc_etch64_compile-usr-local.sh
. This will install Python 2.7 and newer versions of some software (autotools, git etc.) to /usr/local
Build
Download and run the build script:
rm -f pycbc_build_eah.sh
wget --no-check-certificate https://github.com/ligo-cbc/pycbc/raw/master/tools/einsteinathome/pycbc_build_eah.sh
chmod +x pycbc_build_eah.sh
./pycbc_build_eah.sh
This should leave you with a zipped "onedir" PyInstaller bundle
pycbc_inspiral.zip
linking to GLIBC 2.3.6 (and newer) and a "scipy weave code cache"
pythoncompiled.zip
as well as the tools
progress
and
fstab
, all in
pycbc-build/environment/dist
.
Notes
-
pycbc_etch64_compile-usr-local.sh
compiles a gcc-4.8.5 suite, the newest that compiles without problems on that old system, and the one compatible with the other installations (OSX, Cygwin). gcc-7.1.0 compiles when configured with --disable-libsanitizer
.
- There is a hidden dependency between the gcc used (and shared objects compiled with it) and libpython.so, so if gcc is updated, Python and the whole application needs to be recompiled (from scratch/sources).
- To ease maintenance, the build script has been merged with that for Cygwin and OSX, and has been extended for other Linux versions, too. It uses various OS-specific methods to auto-detect the system which it runs on.
- OpenSSL-1.0.1p is compiled manually. It turned out to be non-trivial to find a version that builds on top of the old GLIBC and is compatible with pyOpenSSL-0.13
- Obsolete For quite some time I struggeled with what looked like an error in the scipy build. "ImportError: cannot import name doccer" is seriously misleading, as it's not actually the "doccer" module that fails to load. You get to see the actual linking error ("python2.7/site-packages/scipy/linalg/cython_lapack.so: undefined symbol: zlacn2_ ") when you try to run a pyinstaller directory bundle or just run "python -c 'from scipy.io.wavfile import write as write_wav'" after scipy was installed. The reason I figured is that the atlas3 liblapack.so of the system that a self-built (pip installed) scipy links to is apparently broken (or possibly doesn't feature the "deprecated" functions that this scipy version requires). The solution is to remove the atlas3 lapack/blas libs from the system and compile an own liblapack. I still don't know what stupid limitation of Python swallowed the actual linking error message, getting to see this would have saved me a week of work. The "import" line above is now part of the build script testing the scipy installation right after the build.
- Obsolete (but kept for reference): psycopg-2.6 (required by pegasus-wms) doesn't compile on Debian Etch (or Ubuntu Dapper Drake). The linker complains "/usr/bin/ld: build/temp.linux-x86_64-2.7/psycopg/psycopgmodule.o: relocation R_X86_64_PC32 against `lobjectType' can not be used when making a shared object; recompile with -fPIC". However, according to the logs, the compilation of psycopgmodule.c is done with "-fPIC", and compiling it again doesn't help. I tried different versions of libpq, gcc and binutils (ld) to no avail. So the script actually compiles psycopg-2.5.5 and fakes its version information to be 2.6.
- Obsolete due to different requirements of different python modules involved (pyinstaller,virtualenv vs. pycbc) the version of setuptools is downgraded during the build process (setuptools==18.7.1 -> setuptools==18.2). I sure hope this doesn't introduce some more hidden incompatibilities.
- The Linux build also produces "onefile" binaries of "pycbc_condition_strain" (for "autogating" a timeseries in Frame files) and "pycbc_inspiral_osg", the latter contains the weave cache bundle with the application and thus is really self-contained.
- For use with Jenkins,
pycbc_etch64_compile-usr-local.sh
installs JRE 1.6.45u in /opt/java
Windows 7 / Cygwin
Machine Setup
1. Setup a (virtual) machine with Win7 64 Bit (I used 2 cores, 3GB RAM, 180GB HD).
The name of the user account should not contain blanks/spaces
2. Install Cygwin 64 Bit Version from Cygwin.com (
https://cygwin.com/setup-x86_64.exe).
I used the mirror
http://ftp.inf.tu-dresden.de . Install (required):
- Archive: hdf5, unzip, zip
- Database: libmysqlclient-devel, libmysqlclient18
- Devel: autoconf (2.5+wrapper), automake (1.15+wrapper), binutils, cmake, gcc-core, gcc-g++, gcc-gfortran, git, libhdf5-devel, libhdf5_10, libhdf5cpp, libtool, make, mingw64-x86_64-gcc-core, openssl-devel, patch, pkg-config, swig
- Libs: libfreetype-devel, libfreetype6, libgsl-devel, libgsl0, libjpeg-devel, libpcre-devel, libpng-devel, libpq-devel
- Net: openssh
- Python: python
- Science: gsl
- Web: wget
- Utils: upx
Recommended:
- Admin: cygrunsrv, shutdown
- Editors: emacs
- Utils: screen
NOTE: unlike in e.g. Debian apt, -devel packages don't include / require the corresponding standard/runtime packages, so you need to select both separately (e.g gsl, hdf5)
2a. Alternatively (and probably simpler) install from an old, but pre-packaged version of Cygwin:
- Download Install_Cygwin64_PyCBC.zip and unpack to a local directory
- run
Cygwin-setup-x86_64.exe
- At "Choose A Download Source" select "Install from Local Directory"
- Confirm "Select Root Directory" (
C:\cygwin64
)
- At "Select Local Package Directory" just confirm the local directory of the unpacked archive
- At "Select Packages" click on "Default" next to "+ All" to read "Install", then "Next"
- Confirm all following warnings & messages
3. After Cygwin installation finished, open a Cygwin shell (a "shortcut" should be on the desktop) and run a few commands:
python -m ensurepip
pip install --upgrade pip
pip install virtualenv
Build
Download and run the build script:
rm -f pycbc_build_eah.sh
wget --no-check-certificate https://github.com/ligo-cbc/pycbc/raw/master/tools/einsteinathome/pycbc_build_eah.sh
chmod +x pycbc_build_eah.sh
./pycbc_build_eah.sh
This should leave you with a zipped "onedir" PyInstaller bundle
pycbc_inspiral.zip
and a "scipy weave code cache"
pythoncompiled.zip
, as well as tools
progress.exe
and
fstab.exe
, all in
pycbc-build/environment/dist/
. The latter creates a fake "/etc/fstab" for the Cygwin process with a mount point referring to the project directory of the BOINC client.
Notes
- This was done with a Cygwin installation as of late 2015 (setup 2.873), with gcc-4.9.3. Newer installations may install a newer version of gcc and may behave differently, these have not been tested. To reproduce that installation, use method 2a above.
- Notes for Debian 4.0 (Etch) apply here as well (with exception of OpenSSL)
- to build DLLs with libtool,
-no-undefined
gets added to LDFLAGS
of libraries by the script for LibFrame, METAIO and LALSuite. Although I'm not entirely sure this really applies to all libraries that are built, it seems to work without errors or problems at least for the ones that are actually used in the current application.
- The additional program "fstab.exe" is only needed on Windows/Cygwin. It creates a fake "/etc/fstab", which make the BOINC project directory accessible as e.g. "/project/albert.phys.uwm.edu/", such that the application can access boinc-resolved paths like "../../project/albert.phys.uwm.edu/job.xml". On all other platforms, a copy of (/usr)/bin/true can be used.
- The "wrapper" for Windows that is necessary to run this as a BOINC application needs to be built separately with M$VC. Use the branch
eah_wrapper_improvements
of git://gitlab.aei.uni-hannover.de/einsteinathome/boinc.git
. As of Apr 2017, this may also be cross-built with MinGW-W64 and the Makekefile.mingw in BOINC/lib.
- It was tried (April 2017) to install and use Cygwin64 in a wine-2.6 (development), but without success. The installation runs through, but even a dash crashes, apparently due to unimplemented Windows API calls.
Hints for convenience
1. Autologin: In a VM running on your local machine with only local networking login usually just holds you up.
Open a terminal ("cmd.exe") and type "control userpasswords2" to disable this.
2. If you really need case-sensitive filenames in Cygwin (so far I didn't), set registry value
HKLM\SYSTEM\CurrentControlSet\Control\Session Manager\kernel\obcaseinsensitive
to 0 and reboot the (virtual) machine.
3. Setup ssh server for remote command-line access:
- open a Cygwin terminal as Admin (right-click -> run as administrator)
- in that shell, run "ssh-host-config" and follow the instructions. You will probably run "cygrunsrv -S sshd" at the end
- Make sure the Windows firewall permits connection to port 22 from outside
3a. For public key authentication I had to
- open a Cygwin terminal as Admin (right-click -> run as administrator)
-
mkdir /home/cyg_server && chown cyg_server /home/cyg_server && chmod 700 /home/cyg_server
- 'manage Computer' -> Local Users and Groups -> cyg_server -> change
/var/empty
to /home/cyg_server
and /bin/false
to /bin/bash
4. if you get errors from fork of the type "Resource temporarily unavailable", do
- stop "Cygwin sshd" service (serch for "services")
- close all locally open Cygwin windows/terminals
- with Windows Explorer, navigate to C:\Cygwin64\bin, right-click on "ash" and "Run as Administrator"
- type "/bin/rebaseall"
- close the shell ("exit") and start "Cygwin sshd" again
Mac OSX 10.7 (Lion)
10.7 is the minimum OSX version required by PyInstaller
Machine Setup
- Install Xcode (4.6.3), command-line tools (April 2013) and MacPorts
- run (once)
sudo port selfupdate
sudo port install python27 gcc48 git wget autoconf automake pkgconfig
( cd /opt/local/bin; sudo ln -s /usr/bin/clang )
sudo port select --set python python27
sudo python -m ensurepip
export PATH="$PATH:/opt/local/Library/Frameworks/Python.framework/Versions/2.7/bin"
hash -r
sudo pip install virtualenv
Build
Download and run the build script:
rm -f pycbc_build_eah.sh
wget --no-check-certificate https://github.com/ligo-cbc/pycbc/raw/master/tools/einsteinathome/pycbc_build_eah.sh
chmod +x pycbc_build_eah.sh
./pycbc_build_eah.sh
This should leave you with a zipped "onedir" PyInstaller bundle
pycbc_inspiral.zip
and a "scipy weave code cache"
pythoncompiled.zip
, as well as tools
progress
and
fstab
, all in
pycbc-build/environment/dist/
.
Notes
- Getting the wrapper to build (needs to be done on OSX 10.6) is a mess, in particular to link the static version of libstdc++
- The most work for the OSX port actually was required to overcome the limitations of the Frame-reading code in LALSupport / LALFrame / FrameLib, that chokes at various place when there are blanks / spaces in the "sanitized" "realpath" of the frame files. This never happens on standard Linux installations and is overcome differently on Windows/Cygwin (see "fstab.exe" tool), but is pretty common on OSX. The patch necessary to fix parsing of URLs with blanks in LAL Frame Cache files has been pushed upstream (9706666a), the OSX application uses ldas-tools FrameCPP instead of the FrameLib.
Einstein@Home
The PyCBC application is currently set up on
Albert@Home.
Application
The application consists of
- A BOINC "wrapper" as the main executable. As the standard wrappers published by BOINC are outdated, more recent versions (built from
3765e54e
) are attached here:
- A program "fstab". This is built by the script, it only actually does something for Cygwin/Windows. For all other OS, a copy of
/bin/true
(Linux) or /usr/bin/true
(OSX) may be used as well.
- (currently) a program "progress". This reads the stderr output of
pycbc_inspiral
looking for indicators of (filtering) progress, and writes a file "progress.txt" to be read by the wrapper, which then communicates the "fraction done" to the BOINC client. Recently this functionality has been built into pycbc_inspiral
itself to avoid the additional I/O that verbose pycbc_inspiral
output causes, so this program can be dropped in future configurations.
- The zip archives
pycbc_inspiral.zip
(application) and pythoncompiled.zip
(weave code cache) as built by the script.
The wrapper needs a "job (description) file" that describes what to do. In current configuration this is shipped with the workunits, as it might be different for each such workunit. If the only differences are in the data files and command-lines to use, the job file might become part of the application in future configurations, the fake "fstab" programs may then be omitted from Linux and OSX "applications".
Workunits
Workunits (that repeat the search for GW150914) can be created using the script
create_work_CBC1.sh. It references
- the standard boinc
create_work
utility
- the input- and output templates result_template_CBC1+stderr and input_template_CBC1
- the job file job-GW150914.xml for the wrapper
- the publicly available GW150914 data file
H-H1_LOSC_4_V1-1126257414-4096.gwf
and
- the template bank
H1L1-PREGEN_TMPLTBANK_SPLITBANK_BANK16-1126051217-3331800.xml.gz
Validator
A standalone validator (that natively reads HDF5 files) is being worked on. For now, the following setup is used:
- The actual validator is a standard BOINC script_validator with command-line arguments
--init_script CBC1H1hdf2txt.sh --compare_script validator_compare_CBC1.sh
. It references and uses:
- A standalone
validator_test_CBC1
build from einsteinathome/project-daemons
- A Python script CBC1H5tool5col.py that converts the HDF5 result file to an ASCII table
- A wrapper script CBC1H1hdf2txt.sh that converts all input files and vaildates each with
validator_test_CBC1
- A wrapper script validator_compare_CBC1.sh that compares only the first file of each result (file name ending in "_0") using
validator_test_CBC1
Assimilator, File Deleter
As the results of PyCBC are ordinary files, the usual E@H components
assimilator
(build from
einsteinathome/project-daemons or copy binary from old search) and
file deleter
(build from standard BOINC or copy binary from old search) are used.
Condor-BOINC-GAHP
A HTCondor (submit) interface to a BOINC project (here: Albert@Home) is under heavy development, and currently even the future development process is subject to discussion. The test setup on Albert@Home is documented in an (AEI internal)
Redmine ticket.
Resources
Aditional info that was or might become useful
Debian
Cygwin
Manage Cygwin packages from command-line (wasn't used)
DLLs
OSX
Portable coding
GLIBC installation (dead end)
Python
PyInstaller
HDF5
BOINC
LSC