Common guide lines for cluster usage

This document describes common pitfalls and guide lines when using a large computing cluster. Some of the details are specific to Atlas but most should apply everywhere where a large number of users use shared resources. The ultimate goal should be to make all programs as efficient as possible while creating as little impact on system resources and users as possible.

Preface

Using computing resources to solve problems is always limited. Limits encountered often are CPU speeds, available memory and disk space, shortcomings of programming languages and data formats, limited knowledge of people programming, using and maintaining complex analysis pipelines and systems.

Usually, to make pipelines go faster one needs to identify the most stringent bottle neck(s) and try to optimize those away. Getting to a healthy mix of compromises may be difficult and lengthy! And by no means does finding such a compromise guarantee that a pipeline will run efficiently (or at all) on a different cluster and/or on the same cluster a year later. Thus, unfortunately, one needs to be aware of code and system changes throughout the life-time of an analysis pipeline!

-- CarstenAulbert - 18 Jul 2014