Programming Examples

Remote command example (sequential mode)

The following example shows how to send a command on some nodes, how to get a specific buffer and how to get gathered buffers:

from ClusterShell.Task import task_self
task = task_self()"/bin/uname -r", nodes="green[36-39,133]")

print task.node_buffer("green37")

for buf, nodes in task.iter_buffers():
        print nodes, buf

if task.max_retcode() != 0:
    print "An error occurred (max rc = %s)" % task.max_retcode()


['green37', 'green38', 'green36', 'green39'] 2.6.32-431.el6.x86_64
['green133'] 3.10.0-123.20.1.el7.x86_64
Max return code is 0

Remote command example with live output (event-based mode)

The following example shows how to use the event-based programmation model by installing an EventHandler and listening for EventHandler.ev_read() (we've got a line to read) and EventHandler.ev_hup() (one command has just completed) events. The goal here is to print standard outputs of uname -a commands during their execution and also to notify the user of any erroneous return codes:

from ClusterShell.Task import task_self
from ClusterShell.Event import EventHandler

class MyHandler(EventHandler):

   def ev_read(self, worker, node, sname, msg):
       print "%s: %s" % (node, msg)

   def ev_hup(self, worker, node, rc):
       if rc != 0:
           print "%s: returned with error code %s" % (node, rc)

task = task_self()

# Submit command, install event handler for this command and run task"/bin/uname -a", nodes="fortoy[32-159]", handler=MyHandler()) example script

The following script is available as an example in the source repository and is usually packaged with ClusterShell:

# ClusterShell simple example script.
# This script runs a simple command on remote nodes and report node
# availability (basic health check) and also min/max boot dates.
# It shows an example of use of Task, NodeSet and EventHandler objects.
# Feel free to copy and modify it to fit your needs.
# Usage example: ./ -n node[1-99]

import optparse
from datetime import date, datetime
import time

from ClusterShell.Event import EventHandler
from ClusterShell.NodeSet import NodeSet
from ClusterShell.Task import task_self

class CheckNodesResult(object):
    """Our result class"""
    def __init__(self):
        """Initialize result class"""
        self.nodes_ok = NodeSet()
        self.nodes_ko = NodeSet()
        self.min_boot_date = None
        self.max_boot_date = None

    def show(self):
        """Display results"""
        if self.nodes_ok:
            print "%s: OK (boot date: min %s, max %s)" % \
                (self.nodes_ok, self.min_boot_date, self.max_boot_date)
        if self.nodes_ko:
            print "%s: FAILED" % self.nodes_ko

class CheckNodesHandler(EventHandler):
    """Our ClusterShell EventHandler"""

    def __init__(self, result):
        """Initialize our event handler with a ref to our result object."""
        self.result = result

    def ev_read(self, worker, node, sname, msg):
        """Read event from remote nodes"""
        # this is an example to demonstrate remote result parsing
        bootime = " ".join(msg.strip().split()[2:])
        date_boot = None
        for fmt in ("%Y-%m-%d %H:%M",): # formats with year
                # datetime.strptime() is Python2.5+, use old method instead
                date_boot = datetime(*(time.strptime(bootime, fmt)[0:6]))
            except ValueError:
        for fmt in ("%b %d %H:%M",):    # formats without year
                date_boot = datetime(, \
                    *(time.strptime(bootime, fmt)[1:6]))
            except ValueError:
        if date_boot:
            if not self.result.min_boot_date or \
                self.result.min_boot_date > date_boot:
                self.result.min_boot_date = date_boot
            if not self.result.max_boot_date or \
                self.result.max_boot_date < date_boot:
                self.result.max_boot_date = date_boot

    def ev_close(self, worker, timedout):
        """Worker has finished (command done on all nodes)"""
        if timedout:
            nodeset = NodeSet.fromlist(worker.iter_keys_timeout())

def main():
    """ Main script function """
    # Initialize option parser
    parser = optparse.OptionParser()
    parser.add_option("-d", "--debug", action="store_true", dest="debug",
                      default=False, help="Enable debug mode")
    parser.add_option("-n", "--nodes", action="store", dest="nodes",
                      default="@all", help="Target nodes (default @all group)")
    parser.add_option("-f", "--fanout", action="store", dest="fanout",
                      default="128", help="Fanout window size (default 128)",
    parser.add_option("-t", "--timeout", action="store", dest="timeout",
                      default="5", help="Timeout in seconds (default 5)",
    options, _ = parser.parse_args()

    # Get current task (associated to main thread)
    task = task_self()
    nodes_target = NodeSet(options.nodes)
    task.set_info("fanout", options.fanout)
    if options.debug:
        print "nodeset : %s" % nodes_target
        task.set_info("debug", True)

    # Create ClusterShell event handler
    handler = CheckNodesHandler(CheckNodesResult())

    # Schedule remote command and run task (blocking call)"who -b", nodes=nodes_target, handler=handler, \

if __name__ == '__main__':

Using NodeSet with Parallel Python Batch script using SLURM

The following example shows how to use the NodeSet class to expand $SLURM_NODELIST environment variable in a Parallel Python batch script launched by SLURM. This variable may contain folded node sets. If ClusterShell is not available system-wide on your compute cluster, you need to follow Installing ClusterShell as user using pip first.

Example of SLURM pp.sbatch to submit using sbatch pp.sbatch:


#SBATCH --ntasks-per-node 1

# run the servers
srun ~/.local/bin/ -w $SLURM_CPUS_PER_TASK -t 300 &
sleep 10

# launch the parallel processing
python -u ./

Example of a script:

#!/usr/bin/env python

import os, time
import pp
from ClusterShell.NodeSet import NodeSet

# get the nodelist form Slurm
nodeset = NodeSet(os.environ['SLURM_NODELIST'])

# start the servers (ncpus=0 will make sure that none is started locally)
# casting nodelist to tuple/list will correctly expand $SLURM_NODELIST
job_server = pp.Server(ncpus=0, ppservers=tuple(nodelist))

# make sure the servers have enough time to start

# test function to execute on the remove nodes
def test_func():
    print os.uname()

# start the jobs
job_1 = job_server.submit(test_func,(),(),("os",))
job_2 = job_server.submit(test_func,(),(),("os",))

# retrieve the results
print job_1()
print job_2()

# Cleanup