Commands with inputs¶

Some nodeset commands require input (eg. node names, node sets or node groups), and some only give output. The following table shows commands that require some input:

There are three ways to give some input to the nodeset command:

• from command line arguments,
• from standard input (enabled when no arguments are found on command line),
• from both command line and standard input, by using the dash special argument “-“ meaning you need to use stdin instead.

The following example illustrates the three ways to feed nodeset:

$ nodeset -f node1 node6 node7
node[1,6-7]

$ echo node1 node6 node7 | nodeset -f
node[1,6-7]

$ echo node1 node6 node7 | nodeset -f node0 -
node[0-1,6-7]

Furthermore, nodeset‘s standard input reader is able to process multiple lines and multiple node sets or groups per line. The following example shows a simple use case:

$ mount -t nfs | cut -d':' -f1
nfsserv1
nfsserv2
nfsserv3

$ mount -t nfs | cut -d':' -f1 | nodeset -f
nfsserv[1-3]

Other usage examples of nodeset below show how it can be useful to provide node sets from standard input (sinfo is a SLURM [1] command to view nodes and partitions information and sacct is a command to display SLURM accounting data):

$ sinfo -p cuda -o '%N' -h
node[156-159]

$ sinfo -p cuda -o '%N' -h | nodeset -e
node156 node157 node158 node159

$ for node in $(sinfo -p cuda -o '%N' -h | nodeset -e); do
        sacct -a -N $node > /tmp/cudajobs.$node;
  done

Previous rules also apply when working with node groups, for example when using nodeset -r reading from standard input (and a matching group is found):

$ nodeset -f @gpu
node[156-159]

$ sinfo -p cuda -o '%N' -h | nodeset -r
@gpu

Most commands described in this section produce output results that may be formatted using --output-format and --separator which are described in Output result formatting.

Commands with no input¶

The following table shows all other commands that are supported by nodeset. These commands don’t support any input (like node sets), but can still recognize options as specified below.

Output result formatting¶

When using the expand command (-e, --expand), a separator string is used when displaying results. The option -S, --separator allows you to modify it. The specified string is interpreted, so that you can use special characters as separator, like \n or \t. The default separator is the space character ” “. This is an example showing such separator string change:

$ nodeset -e --separator='\n' node[0-3]
node0
node1
node2
node3

The -O, --output-format option can be used to format output results of most nodeset commands. The string passed to this option is used as a base format pattern applied to each node or each result (depending on the command and other options requested). The default format string is “%s”. Formatting is performed using the Python builtin string formatting operator, so you must use one format operator of the right type (%s is guaranteed to work in all cases). Here is an output formatting example when using the expand command:

$ nodeset --output-format='%s-ipmi' -e node[1-2]x[1-2]
node1x1-ipmi node1x2-ipmi node2x1-ipmi node2x2-ipmi

Output formatting and separator combined can be useful when using the expand command, as shown here:

$ nodeset -O '%s-ipmi' -S '\n' -e node[1-2]x[1-2]
node1x1-ipmi
node1x2-ipmi
node2x1-ipmi
node2x2-ipmi

When using the output formatting option along with the folding command, the format is applied to each node but the result is still folded:

$ nodeset -O '%s-ipmi' -f mgmt1 mgmt2 login[1-4]
login[1-4]-ipmi,mgmt[1-2]-ipmi

Union operation¶

Union is the easiest arithmetic operation supported by nodeset: there is no special command line option for that, just provide several node sets and the union operation will be computed, for example:

$ nodeset -f node[1-3] node[4-7]
node[1-7]

$ nodeset -f node[1-3] node[2-7] node[5-8]
node[1-8]

Other operations¶

As an extension to the above, other arithmetic operations are available by using the following command-line options (working set is the node set currently processed on the command line – always from left to right):

If rangeset mode (-R) is turned on, all arithmetic operations are supported by replacing NODESET by any RANGESET. See Range sets for more info about nodeset‘s rangeset mode.

Arithmetic operations usage examples:

$ nodeset -f node[1-9] -x node6
node[1-5,7-9]

$ nodeset -f node[1-9] -i node[6-11]
node[6-9]

$ nodeset -f node[1-9] -X node[6-11]
node[1-5,10-11]

$ nodeset -f node[1-9] -x node6 -i node[6-12]
node[7-9]

Extended patterns support¶

nodeset does also support arithmetic operations through its “extended patterns” (inherited from NodeSet extended pattern feature, see Extended String Pattern, there is an example of use:

$ nodeset -f node[1-4],node[5-9]
node[1-9]

$ nodeset -f node[1-9]\!node6
node[1-5,7-9]

$ nodeset -f node[1-9]\&node[6-12]
node[6-9]

$ nodeset -f node[1-9]^node[6-11]
node[1-5,10-11]

Slicing¶

Slicing is a way to select elements from a node set by their index (or from a range set when using -R toggle option, see Range sets. In this case actually, and because nodeset‘s underlying NodeSet class sorts elements as observed after folding (for example), the word set may sound like a stretch of language (a set isn’t usually sorted). Indeed, NodeSet further guarantees that its iterator will traverse the set in order, so we should see it as a ordered set. The following simple example illustrates this sorting behavior:

$ nodeset -f b2 b1 b0 b c a0 a
a,a0,b,b[0-2],c

Slicing is performed through the following command-line option:

Some slicing examples are shown below:

$ nodeset -f -I 0 node[4-8]
node4

$ nodeset -f --slice=0 bnode[0-9] anode[0-9]
anode0

$ nodeset -f --slice=1,4,7,9,15 bnode[0-9] anode[0-9]
anode[1,4,7,9],bnode5

$ nodeset -f --slice=0-18/2 bnode[0-9] anode[0-9]
anode[0,2,4,6,8],bnode[0,2,4,6,8]

Splitting into n subsets¶

Splitting a node set into several parts is often useful to get separate groups of nodes, for instance when you want to check MPI comm between nodes, etc. Based on NodeSet.split() method, the nodeset command provides the following additional command-line option (since v1.4):

MAXSPLIT is an integer specifying the number of separate groups of nodes to compute. Input’s node set is divided into smaller groups, whenever possible with the same size (only the last ones may be smaller due to rounding). Obviously, if MAXSPLIT is higher than or equal to the number N of elements in the set, then the set is split to N single sets.

Some node set splitting examples:

$ nodeset -f --split=4 node[0-7]
node[0-1]
node[2-3]
node[4-5]
node[6-7]

$ nodeset -f --split=4 node[0-6]
node[0-1]
node[2-3]
node[4-5]
node6

$ nodeset -f --split=10000 node[0-4]
foo0
foo1
foo2
foo3
foo4

$ nodeset -f --autostep=3 --split=2 node[0-38/2]
node[0-18/2]
node[20-38/2]

Splitting off non-contiguous subsets¶

It can be useful to split a node set into several contiguous subsets (with same pattern name and contiguous range indexes, eg. node[1-100] or dc[1-4]node[1-100]). The --contiguous option allows you to do that. It is based on NodeSet.contiguous() method, and should be specified with standard commands (fold, expand, count, regroup). The following example shows how to split off non-contiguous subsets of a specified node set, and to display each resulting contiguous node set in a folded manner to separated lines:

$ nodeset -f --contiguous node[1-100,200-300,500]
node[1-100]
node[200-300]
node500

Similarly, the following example shows how to display each resulting contiguous node set in an expanded manner to separate lines:

$ nodeset -e --contiguous node[1-9,11-19]
node1 node2 node3 node4 node5 node6 node7 node8 node9
node11 node12 node13 node14 node15 node16 node17 node18 node19

Choosing fold axis (nD)¶

The default folding behavior for multidimensional node sets is to fold along all nD axis. However, other cluster tools barely support nD nodeset syntax, so it may be useful to fold along one (or a few) axis only. The --axis option allows you to specify indexes of dimensions to fold. Using this option, rangesets of unspecified axis there won’t be folded. Please note however that the obtained result may be suboptimal, this is because NodeSet algorithms are optimized for folding along all axis. --axis value is a set of integers from 1 to n representing selected nD axis, in the form of a number or a rangeset. A common case is to restrict folding on a single axis, like in the following simple examples:

$ nodeset --axis=1 -f node1-ib0 node2-ib0 node1-ib1 node2-ib1
node[1-2]-ib0,node[1-2]-ib1

$ nodeset --axis=2 -f node1-ib0 node2-ib0 node1-ib1 node2-ib1
node1-ib[0-1],node2-ib[0-1]

Because a single nodeset may have several different dimensions, axis indices are silently truncated to fall in the allowed range. Negative indices are useful to fold along the last axis whatever number of dimensions used:

$ nodeset --axis=-1 -f comp-[1-2]-[1-36],login-[1-2]
comp-1-[1-36],comp-2-[1-36],login-[1-2]

Picking N node(s) at random¶

Use --pick with a maximum number of nodes you wish to pick randomly from the resulting node set (or from the resulting range set with -R):

$ nodeset --pick=1 -f node11 node12 node13
node12
$ nodeset --pick=2 -f node11 node12 node13
node[11,13]

Node group expression rules¶

The general node group expression is @source:groupname. For example, @slurm:bigmem represents the group bigmem of the group source slurm. Moreover, a shortened expression is available when using the default group source (defined by configuration); for instance @compute represents the compute group of the default group source.

Valid group source names and group names can contain alphanumeric characters, hyphens and underscores (no space allowed). Indeed, same rules apply to node names.

Listing group sources¶

As already mentioned, the following nodeset command is available to list configured group sources and also display the default group source (unless -q is provided):

$ nodeset --groupsources
local (default)
genders
slurm

Listing group names¶

If the list external shell command is configured (see node groups configuration), it is possible to list available groups from the default source with the following commands:

$ nodeset -l
@mgnt
@mds
@oss
@login
@compute

Or, to list groups from a specific group source, use -l in conjunction with -s (or –groupsource):

$ nodeset -l -s slurm
@slurm:parallel
@slurm:cuda

Or, to list groups from all available group sources, use -L (or –list-all):

$ nodeset -L
@mgnt
@mds
@oss
@login
@compute
@slurm:parallel
@slurm:cuda

You can also use nodeset -ll or nodeset -LL to see each group’s associated node sets.

Using node groups in basic commands¶

The use of node groups with the nodeset command is very straightforward. Indeed, any group name, prefixed by @ as mentioned above, can be used in lieu of a node name, where it will be substituted for all nodes in that group.

A first, simple example is a group expansion (using default source) with nodeset:

$ nodeset -e @oss
node40 node41 node42 node43 node44 node45

The nodeset count command works as expected:

$ nodeset -c @oss
6

Also nodeset folding command can always resolve node groups:

$ nodeset -f @oss
node[40-45]

There are usually two ways to use a specific group source (need to be properly configured):

$ nodeset -f @slurm:parallel
node[50-81]

$ nodeset -f -s slurm @parallel
node[50-81]

Finding node groups¶

As an extension to the list command, you can search node groups that a specified node set belongs to with nodeset -l[ll] as follow:

$ nodeset -l node40
@all
@oss

$ nodeset -ll node40
@all node[1-159]
@oss node[40-45]

This feature is implemented with the help of the NodeSet.groups() method (see Finding node groups for further details).

Resolving node groups¶

If needed group configuration conditions are met (cf. node groups configuration), you can try group lookups thanks to the -r, --regroup command. This feature is implemented with the help of the NodeSet.regroup() method (see Regrouping node sets for further details). Only exact matching groups are returned (all containing nodes needed), for example:

$ nodeset -r node[40-45]
@oss

$ nodeset -r node[0,40-45]
@mgnt,@oss

When resolving node groups, nodeset always returns the largest groups first, instead of several smaller matching groups, for instance:

$ nodeset -ll
@login node[50-51]
@compute node[52-81]
@intel node[50-81]

$ nodeset -r node[50-81]
@intel

If no matching group is found, nodeset -r still returns folded result (as does -f):

$ nodeset -r node40 node42
node[40,42]

Indexed node groups¶

Node groups are themselves some kind of group sets and can be indexable. To use this feature, node groups external shell commands need to return indexed group names (automatically handled by the library as needed). For example, take a look at these indexed node groups:

$ nodeset -l
@io1
@io2
@io3

$ nodeset -f @io[1-3]
node[40-45]

Arithmetic operations on node groups¶

Arithmetic and special operations (as explained for node sets in nodeset-arithmetic and nodeset-special are also supported with node groups. Any group name can be used in lieu of a node set, where it will be substituted for all nodes in that group before processing requested operations. Some typical examples are:

$ nodeset -f @lustre -x @mds
node[40-45]

$ nodeset -r @lustre -x @mds
@oss

$ nodeset -r -a -x @lustre
@compute,@login,@mgnt

More advanced examples, with the use of node group sets, follow:

$ nodeset -r @io[1-3] -x @io2
@io[1,3]

$ nodeset -f -I0 @io[1-3]
node40

$ nodeset -f --split=3 @oss
node[40-41]
node[42-43]
node[44-45]

$ nodeset -r --split=3 @oss
@io1
@io2
@io3

Extended patterns support with node groups¶

Even for node groups, the nodeset command supports arithmetic operations through its extended pattern feature (see Extended String Pattern). A first example illustrates node groups intersection, that can be used in practice to get nodes available from two dynamic group sources at a given time:

$ nodeset -f @db:prod\&@compute

The following fictive example computes a folded node set containing nodes found in node group @gpu and @slurm:bigmem, but not in both, minus the nodes found in odd @chassis groups from 1 to 9 (computed from left to right):

$ nodeset -f @gpu^@slurm:bigmem\!@chassis[1-9/2]

Also, version 1.7 introduces a notation extension @* (or @SOURCE:*) that has been added to quickly represent all nodes (please refer to Selecting all nodes for more details).

Working with range sets¶

By default, the nodeset command works with node or group sets and its functionality match most NodeSet class methods. Similarly, nodeset will match RangeSet methods when you make use of the -R option switch. In that case, all operations are restricted to numerical ranges. For example, to expand the range “1-10”, you should use:

$ nodeset -e -R 1-10
1 2 3 4 5 6 7 8 9 10

Almost all commands and operations available for node sets are also available with range sets. The only restrictions are commands and operations related to node groups. For instance, the following command options are not available with nodeset -R:

• -r, --regroup as this feature is obviously related to node groups,
• -a / --all as the all external call is also related to node groups.

Using range sets instead of node sets doesn’t change the general command usage, like the need of one command option presence (cf. nodeset-commands), or the way to give some input (cf. nodeset-stdin), for example:

$ echo 3 2 36 0 4 1 37 | nodeset -fR
0-4,36-37

$ echo 0-8/4 | nodeset -eR -S'\n'
0
4
8

Stepping and auto-stepping are supported (cf. Stepping and auto-stepping) and also zero-padding (cf. nodeset-zpad), which are both RangeSet class features anyway.

The following examples illustrate these last points:

$ nodeset -fR 03 05 01 07 11 09
01,03,05,07,09,11

$ nodeset -fR --autostep=3 03 05 01 07 11 09
01-11/2

Arithmetic and special operations¶

All arithmetic operations, as seen for node sets (cf. nodeset-arithmetic), are available for range sets, for example:

$ nodeset -fR 1-14 -x 10-20
1-9

$ nodeset -fR 1-14 -i 10-20
10-14

$ nodeset -fR 1-14 -X 10-20
1-9,15-20

For now, there is no extended patterns syntax for range sets as for node sets (cf. nodeset-extended-patterns). However, as the union operator , is available natively by design, such expressions are still allowed:

$ nodeset -fR 4-10,1-2
1-2,4-10

Besides arithmetic operations, special operations may be very convenient for range sets also. Below is an example with -I / --slice (cf. nodeset-slice):

$ nodeset -fR -I 0 100-131
100

$ nodeset -fR -I 0-15 100-131
100-115

There is another special operation example with --split (cf. nodeset-splitting-n):

$ nodeset -fR --split=2 100-131
100-115
116-131

Finally, an example of the special operation --contiguous (cf. nodeset-splitting-contiguous):

$ nodeset -f -R --contiguous 1-9,11,13-19
1-9
11
13-19

rangeset alias¶

When using nodeset with range sets intensively (eg. for scripting), it may be convenient to create a local command alias, as shown in the following example (Bourne shell), making it sort of a super seq(1) command:

$ alias rangeset='nodeset -R'
$ rangeset -e 0-8/2
0 2 4 6 8

Command line options¶

The -w option allows you to specify remote hosts by using ClusterShell NodeSet syntax, including the node groups @group special syntax (cf. Node group expression rules) and the Extended String Patterns syntax (see Extended String Pattern) to benefits from NodeSet basic arithmetics (like @Agroup&@Bgroup). Additionally, the -x option allows you to exclude nodes from remote hosts list (the same NodeSet syntax can be used here). Nodes exclusion has priority over nodes addition.

Using node groups¶

If you have ClusterShell node groups configured on your cluster, any node group syntax may be used in place of nodes for -w as well as -x.

For example:

$ clush -w @rhel6 cat /proc/loadavg
node26: 0.02 0.01 0.00 1/202 23042

For pdsh backward compatibility, clush supports two -g and -X options to respectively select and exclude nodes group(s), but only specified by omitting any “@” group prefix (see example below). In general, though, it is advised to use the @-prefixed group syntax as the non-prefixed notation is only recognized by clush but not by other tools like nodeset.

For example:

$ clush -g rhel6 cat /proc/loadavg
node26: 0.02 0.01 0.00 1/202 23033

Selecting all nodes¶

The special option -a (without argument) can be used to select all nodes, in the sense of ClusterShell node groups (see node groups configuration for more details on special all external shell command upcall). If not properly configured, the -a option may lead to a runtime error like:

clush: External error: Not enough working external calls (all, or map +
list) defined to get all node

Picking node(s) at random¶

Use --pick with a maximum number of nodes you wish to pick randomly from the targeted node set. clush will then run only on selected node(s). The following example will run a script on a single random node picked from the @compute group:

$ clush -w @compute --pick=1 ./nonreg-single-client-fs-io.sh

Host files¶

The option --hostfile (or --machinefile) may be used to specify a path to a file containing a list of single hosts, node sets or node groups, separated by spaces and lines. It may be specified multiple times (one per file).

For example:

$ clush --hostfile ./host_file -b systemctl is-enabled httpd

This option has been added as backward compatibility with other parallel shell tools. Indeed, ClusterShell provides a preferred way to provision node sets from node group sources and flat files to all cluster tools using NodeSet (including clush). Please see node groups configuration.

Configuration¶

The system-wide library configuration file /etc/clustershell/topology.conf defines the routes of default command propagation tree. It is recommended that all connections between parent and children nodes are carefully pre-configured, for example, to avoid any SSH warnings when connecting (if using the default SSH remote worker, of course).

The content of the topology.conf file should look like this:

[routes]
rio0: rio[10-13]
rio[10-11]: rio[100-240]
rio[12-13]: rio[300-440]

This file defines the following topology graph:

rio0
|- rio[10-11]
|  `- rio[100-240]
`- rio[12-13]
   `- rio[300-440]

At runtime, ClusterShell will pick an initial propagation tree from this topology graph definition and the current root node. Multiple admin/root nodes may be defined in the file.

Enabling tree mode¶

Since version 1.7, the tree mode is enabled by default when a configuration file is present. When the configuration file /etc/clustershell/topology.conf exists, clush will use it by default for target nodes that are defined there. The topology file path can be changed using the --topology command line option.

Enabling tree mode should be as much transparent as possible to the end user. Most clush options, including options defined in clush.conf or specified using -O or -o (ssh options) are propagated to the gateways and taken into account there.

Tree mode specific options¶

The --remote=yes|no command line option controls the remote execution behavior:

• Default is yes, that will make clush establish connections up to the leaf nodes using a distant worker like ssh.

• Changing it to no will make clush establish connections up to the leaf parent nodes only, then the commands are executed locally on the gateways (like if it would be with --worker=exec on the gateways themselves). This execution mode allows users to schedule remote commands on gateways that take a node as an argument. On large clusters, this is useful to spread the load and resources used of one-shot monitoring, IPMI, or other commands on gateways. A simple example of use is:

  $ clush -w node[100-199] --remote=no /usr/sbin/ipmipower -h %h-ipmi -s
  

  This command is also valid if you don’t have any tree configured, because in that case, --remote=no is an alias of --worker=exec worker.

The --grooming command line option allows users to change the grooming delay (float, in seconds). This feature allows gateways to aggregate responses received within a certain timeframe before transmitting them back to the root node in a batch fashion. This contributes to reducing the load on the root node by delegating the first steps of this CPU intensive task to the gateways.

Fanout considerations¶

ClusterShell uses a “sliding window” or fanout of processes to avoid too many concurrent connections and to conserve resources on the initiating hosts.

The --fanout (or -f) option of clush allows the user to change the default fanout value defined in clush.conf or in the library defaults if not specified.

In tree mode, the same fanout value is used on the head node and on each gateway. That is, if the fanout is 16, each gateway will initate up to 16 connections to their target nodes at the same time.

Debugging Tree mode¶

To debug Tree mode, you can define the following environment variable before running clush (or any other applications using ClusterShell):

$ export CLUSTERSHELL_GW_LOG_LEVEL=DEBUG  (default value is INFO)
$ export CLUSTERSHELL_GW_LOG_DIR=/tmp     (default value is /tmp)

This will generate log files of the form $HOSTNAME.gw.log in CLUSTERSHELL_GW_LOG_DIR.

Output gathering options¶

If option -b or --dshbak is specified, clush waits for command completion while displaying a progress indicator and then displays gathered output results. If standard output is redirected to a file, clush detects it and disable any progress indicator.

The following is a simple example of clush command used to execute uname -r on node40, node41 and node42, wait for their completion and finally display digested output results:

$ clush -b -w node[40-42] uname -r
---------------
node[40-42]
---------------
2.6.35.6-45.fc14.x86_64

It is common to cancel such command execution because a node is hang. When using pdsh and dshbak, due to the pipe, all nodes output will be lost, even if all nodes have successfully run the command. When you hit CTRL-C with clush, the task is canceled but received output is not lost:

$ clush -b -w node[1-5] uname -r
Warning: Caught keyboard interrupt!
---------------
node[2-4] (3)
---------------
2.6.31.6-145.fc11
---------------
node5
---------------
2.6.18-164.11.1.el5
Keyboard interrupt (node1 did not complete).

Performing diff of cluster-wide outputs¶

Since version 1.6, you can use the --diff clush option to show differences between common outputs. This feature is implemented using Python unified diff. This special option implies -b (gather common stdout outputs) but you don’t need to specify it. Example:

$ clush -w node[40-42] --diff dmidecode -s bios-version
--- node[40,42] (2)
+++ node41
@@ -1,1 +1,1 @@
-1.0.5S56
+1.1c

A nodeset is automatically selected as the “reference nodeset” according to these criteria:

1. lowest command return code (to discard failed commands)
2. largest nodeset with the same output result
3. otherwise the first nodeset is taken (ordered (1) by name and (2) lowest range indexes)

Standard input bindings¶

Unless option --nostdin is specified, clush detects when its standard input is connected to a terminal (as determined by isatty(3)). If actually connected to a terminal, clush listens to standard input when commands are running, waiting for an Enter key press. Doing so will display the status of current nodes. If standard input is not connected to a terminal, and unless option --nostdin is specified, clush binds the standard input of the remote commands to its own standard input, allowing scripting methods like:

$ echo foo | clush -w node[40-42] -b cat
---------------
node[40-42]
---------------
foo

Another stdin-bound clush usage example:

$ ssh node10 'ls /etc/yum.repos.d/*.repo' | clush -w node[11-14] -b xargs ls
---------------
node[11-14] (4)
---------------
/etc/yum.repos.d/cobbler-config.repo

Progress indicator¶

In output gathering mode, clush will display a live progress indicator as a simple but convenient way to follow the completion of parallel commands. It can be disabled just by using the -q or --quiet options. The progress indicator will appear after 1 to 2 seconds and should look like this:

clush: <command_completed>/<command_total>

If writing is performed to clush standard input, like in command | clush, the live progress indicator will display the global bandwidth of data written to the target nodes.

Finally, the special option --progress can be used to force the display of the live progress indicator. Using this option may interfere with some command outputs, but it can be useful when using stdin while remote commands are silent. As an example, the following command will copy a local file to node[1-3] and display the global write bandwidth to the target nodes:

$ dd if=/path/to/local/file | clush -w node[1-3] --progress 'dd of=/path/to/remote/file'
clush: 0/3 write: 212.27 MiB/s

Single-character interactive commands¶

clush also recognizes special single-character prefixes that allows the user to see and modify the current nodeset (the nodes where the commands are executed). These single-character interactive commands are detailed below:

To leave an interactive session, type quit or Control-D. As of version 1.6, it is not possible to cancel a command while staying in clush interactive session: for instance, Control-C is not supported and will abort current clush interactive command (see ticket #166).

Example of clush interactive session:

$ clush -w node[11-14] -b
Enter 'quit' to leave this interactive mode
Working with nodes: node[11-14]
clush> uname
---------------
node[11-14] (4)
---------------
Linux
clush> !pwd
LOCAL: /root
clush> -node[11,13]
Working with nodes: node[12,14]
clush> uname
---------------
node[12,14] (2)
---------------
Linux
clush>

The interactive mode and commands described above are subject to change and improvements in future releases. Feel free to open an enhancement ticket if you use the interactive mode and have some suggestions.

Overriding clush.conf settings¶

clush default settings are found in a configuration described in clush configuration. To override any settings, use the --option command line option (or -O for the shorter version), and repeat as needed. Here is a simple example to disable the use colors in the output nodeset header:

$ clush -O color=never -w node[11-12] -b echo ok
---------------
node[11-12] (2)
---------------
ok

Worker selection¶

By default, clush is using the default library worker configuration when running commands or copying files. In most cases, this is ssh (See Changing default worker for default worker selection).

Worker selection can be performed at runtime thanks to --worker command line option (or -R for the shorter version in order to be compatible with pdsh remote command selection option):

$ clush -w node[11-12] --worker=rsh echo ok
node11: ok
node12: ok

By default, ClusterShell supports the following worker identifiers:

• exec: this local worker supports parallel command execution, doesn’t rely on any external tool and provides command line placeholders described below:

  • %h and %host are substitued with each target hostname
  • %hosts is substitued with the full target nodeset
  • %n and %rank are substitued with the remote rank (0 to n-1)

  For example, the following would request the exec worker to locally run multiple ipmitool commands across the hosts foo[0-10] and automatically aggregate output results (-b):

  $ clush -R exec -w foo[0-10] -b ipmitool -H %h-ipmi chassis power status
  ---------------
  foo[0-10] (11)
  ---------------
  Chassis Power is on
  

• rsh: remote worker based on rsh

• ssh: remote worker based on ssh (default)

• pdsh: remote worker based on pdsh that requires pdsh to be installed; doesn’t provide write support (eg. you cannot cat file | clush --worker pdsh); it is primarily an 1-to-n worker example.

Using NodeSet¶

If you are used to Python sets, NodeSet interface will be easy for you to learn. The main conceptual difference is that NodeSet iterators always provide ordered results (and also NodeSet.__getitem__() by index or slice is allowed). Furthermore, NodeSet provides specific methods like NodeSet.split(), NodeSet.contiguous() (see below), or NodeSet.groups(), NodeSet.regroup() (these last two are related to Node groups). The following code snippet shows you a basic usage of the NodeSet class:

>>> from ClusterShell.NodeSet import NodeSet
>>> nodeset = NodeSet()
>>> nodeset.add("node7")
>>> nodeset.add("node6")
>>> print nodeset
node[6-7]

NodeSet class provides several object constructors:

>>> print NodeSet("node[1-5]")
node[1-5]
>>> print NodeSet.fromlist(["node1", "node2", "node3"])
node[1-3]
>>> print NodeSet.fromlist(["node[1-5]", "node[6-10]"])
node[1-10]
>>> print NodeSet.fromlist(["clu-1-[1-4]", "clu-2-[1-4]"])
clu-[1-2]-[1-4]

All corresponding Python sets operations are available, for example:

>>> from ClusterShell.NodeSet import NodeSet
>>> ns1 = NodeSet("node[10-42]")
>>> ns2 = NodeSet("node[11-16,18-39]")
>>> print ns1.difference(ns2)
node[10,17,40-42]
>>> print ns1 - ns2
node[10,17,40-42]
>>> ns3 = NodeSet("node[1-14,40-200]")
>>> print ns3.intersection(ns1)
node[10-14,40-42]

Unlike Python sets, it is important to notice that NodeSet is somewhat not so strict about the type of element used for set operations. Thus when a string object is encountered, it is automatically converted to a NodeSet object for convenience. The following example shows an example of this (set operation is working with either a native nodeset or a string):

>>> nodeset = NodeSet("node[1-10]")
>>> nodeset2 = NodeSet("node7")
>>> nodeset.difference_update(nodeset2)
>>> print nodeset
node[1-6,8-10]
>>>
>>> nodeset.difference_update("node8")
>>> print nodeset
node[1-6,9-10]

NodeSet ordered content leads to the following being allowed:

>>> nodeset = NodeSet("node[10-49]")
>>> print nodeset[0]
node10
>>> print nodeset[-1]
node49
>>> print nodeset[10:]
node[20-49]
>>> print nodeset[:5]
node[10-14]
>>> print nodeset[::4]
node[10,14,18,22,26,30,34,38,42,46]

And it works for node names without index, for example:

>>> nodeset = NodeSet("lima,oscar,zulu,alpha,delta,foxtrot,tango,x-ray")
>>> print nodeset
alpha,delta,foxtrot,lima,oscar,tango,x-ray,zulu
>>> print nodeset[0]
alpha
>>> print nodeset[-2]
x-ray

And also for multidimensional node sets:

>>> nodeset = NodeSet("clu1-[1-10]-ib[0-1],clu2-[1-10]-ib[0-1]")
>>> print nodeset
clu[1-2]-[1-10]-ib[0-1]
>>> print nodeset[0]
clu1-1-ib0
>>> print nodeset[-1]
clu2-10-ib1
>>> print nodeset[::2]
clu[1-2]-[1-10]-ib0

To split a NodeSet object into n subsets, use the NodeSet.split() method, for example:

>>> for nodeset in NodeSet("node[10-49]").split(2):
...     print nodeset
...
node[10-29]
node[30-49]

To split a NodeSet object into contiguous subsets, use the NodeSet.contiguous() method, for example:

>>> for nodeset in NodeSet("node[10-49,51-53,60-64]").contiguous():
...     print nodeset
...
node[10-49]
node[51-53]
node[60-64]

For further details, please use the following command to see full NodeSet API documentation.

Multidimensional considerations¶

Version 1.7 introduces full support of multidimensional NodeSet (eg. da[2-5]c[1-2]p[0-1]). The NodeSet interface is the same, multidimensional patterns are automatically detected by the parser and processed internally. While expanding a multidimensional NodeSet is easily solved by performing a cartesian product of all dimensions, folding nodes is much more complex and time consuming. To reduce the performance impact of such feature, the NodeSet class still relies on RangeSet when only one dimension is varying (see RangeSet class). Otherwise, it uses a new class named RangeSetND for full multidimensional support (see RangeSetND class).

Extended String Pattern¶

NodeSet class parsing engine recognizes an extended string pattern, adding support for union (with special character ”,”), difference (with special character ”!”), intersection (with special character “&”) and symmetric difference (with special character “^”) operations. String patterns are read from left to right, by proceeding any character operators accordinately. The following example shows how you can use this feature:

>>> print NodeSet("node[10-42],node46!node10")
node[11-42,46]

Using node groups¶

Node groups are prefixed with @ character. Please see Node group expression rules for more details about node group expression/syntax rules.

Please also have a look at Node groups configuration to learn how to configure external node group bingings (sources). Once setup (please use the nodeset command to check your configuration), the NodeSet parsing engine automatically resolves node groups. For example:

>>> print NodeSet("@oss")
example[4-5]
>>> print NodeSet("@compute")
example[32-159]
>>> print NodeSet("@compute,@oss")
example[4-5,32-159]

That is, all NodeSet-based applications share the same system-wide node group configuration (unless explicitly disabled — see Disabling node group resolution).

When the all group upcall is configured (node groups configuration), you can also use the following NodeSet constructor:

>>> print NodeSet.fromall()
example[4-6,32-159]

When group upcalls are not properly configured, this constructor will raise a NodeSetExternalError exception.

Finding node groups¶

In order to find node groups a specified node set belongs to, you can use the NodeSet.groups() method. This method is used by nodeset -l <nodeset> command (see Finding node groups). It returns a Python dictionary where keys are groups found and values, provided for convenience, are tuples of the form (group_nodeset, contained_nodeset). For example:

>>> for group, (group_nodes, contained_nodes) in NodeSet("@oss").groups().iteritems():
...     print group, group_nodes, contained_nodes
...
@all example[4-6,32-159] example[4-5]
@oss example[4-5] example[4-5]

More usage examples follow:

>>> print NodeSet("example4").groups().keys()
['@all', '@oss']
>>> print NodeSet("@mds").groups().keys()
['@all', '@mds']
>>> print NodeSet("dummy0").groups().keys()
[]

Regrouping node sets¶

If needed group configuration conditions are met (cf. node groups configuration), you can use the NodeSet.regroup() method to reduce node sets using matching groups, whenever possible:

>>> print NodeSet("example[4-6]").regroup()
@mds,@oss

The nodeset command makes use of the NodeSet.regroup() method when using the -r switch (see Resolving node groups).

Overriding default groups configuration¶

It is possible to override the libary default groups configuration by changing the default NodeSet resolver object. Usually, this is done for testing or special purposes. Here is an example of how to override the resolver object using NodeSet.set_std_group_resolver() in order to use another configuration file:

>>> from ClusterShell.NodeSet import NodeSet, set_std_group_resolver
>>> from ClusterShell.NodeUtils import GroupResolverConfig
>>> set_std_group_resolver(GroupResolverConfig("/other/groups.conf"))
>>> print NodeSet("@oss")
other[10-20]

It is possible to restore NodeSet default group resolver by passing None to the NodeSet.set_std_group_resolver() module function, for example:

>>> from ClusterShell.NodeSet import set_std_group_resolver
>>> set_std_group_resolver(None)

Disabling node group resolution¶

If for any reason, you want to disable host groups resolution, you can use the special resolver value RESOLVER_NOGROUP. In that case, NodeSet parsing engine will not recognize @ group characters anymore, for instance:

>>> from ClusterShell.NodeSet import NodeSet, RESOLVER_NOGROUP
>>> print NodeSet("@oss")
example[4-5]
>>> print NodeSet("@oss", resolver=RESOLVER_NOGROUP)
@oss

Any attempts to use a group-based method (like NodeSet.groups() or NodeSet.regroups()) on such “no group” NodeSet will raise a NodeSetExternalError exception.

Getting a Task object¶

To get the Task object bound to the current thread, you use one of the following:

• Use the Task.task_self() function available at the root of the Task module
• or use task = Task(); Task objects are only instantiated when needed.

Example of getting the current task object:

>>> from ClusterShell.Task import task_self
>>> task = task_self()

So for a single-threaded application, a Task is a simple singleton (which instance is also available through Task.task_self()).

To get the Task object associated to a specific thread identified by the identifier tid, you use the following:

>>> from ClusterShell.Task import Task
>>> task = Task(thread_id=tid)

Configuring the Task object¶

Each Task provides an info dictionary that shares both internal Task-specific parameters and user-defined (key, value) parameters. Use the following Task class methods to get or set parameters:

• Task.info()
• Task.set_info()

For example, to configure the task debugging behavior:

>>> task.set_info('debug', True)
>>> task.info('debug')
True

You can also use the Task info dictionary to set your own Task-specific key, value pairs. You may use any free keys but only keys starting with USER_ are guaranteed not to be used by ClusterShell in the future.

Task info keys and their default values:

Below is an example of print_debug override. As you can see, we set the function print_csdebug(task, s) as the value. When debugging is enabled, this function will be called for any debug text line. For example, this function searchs for any known patterns and print a modified debug line to stdout when found:

def print_csdebug(task, s):
   m = re.search("(\w+): SHINE:\d:(\w+):", s)
   if m:
       print "%s<pickle>" % m.group(0)
   else:
       print s

# Install the new debug printing function
task_self().set_info("print_debug", print_csdebug)

Submitting a shell command¶

You can submit a set of commands for local or distant execution in parallel with Task.shell().

Local usage:

task.shell(command [, key=key] [, handler=handler] [, timeout=secs])

Distant usage:

task.shell(command, nodes=nodeset [, handler=handler] [, timeout=secs])

This method makes use of the default local or distant worker. ClusterShell uses a default Worker based on the Python Popen2 standard module to execute local commands, and a Worker based on ssh (Secure SHell) for distant commands.

If the Task is not running, the command is scheduled for later execution. If the Task is currently running, the command is executed as soon as possible (depending on the current fanout).

To set a per-worker (eg. per-command) timeout value, just use the timeout parameter (in seconds), for example:

task.shell("uname -r", nodes=remote_nodes, handler=ehandler, timeout=5)

This is the prefered way to specify a command timeout. EventHandler.ev_timeout() event is generated before the worker has finished to indicate that some nodes have timed out. You may then retrieve the nodes with DistantWorker.iter_keys_timeout().

Submitting a file copy action¶

Local file copy to distant nodes is supported. You can submit a copy action with Task.copy():

task.copy(source, dest, nodes=nodeset [, handler=handler] [, timeout=secs])

This method makes use of the default distant copy worker which is based on scp (Secure CoPy) which comes with OpenSSH.

If the Task is not running, the copy is scheduled for later execution. If the Task is currently running, the copy is started as soon as possible (depending on the current fanout).

Starting the Task¶

Before you run a Task, you must add at least one worker (shell command, file copy) or timer to it. If a Task does not have any worker to execute and monitor, it exits immediately when you try to run it with:

task.resume()

At this time, all previously submitted commands will start in the associated Task thread. From a library user point of view, the task thread is blocked until the end of the command executions.

Please note that the special method Task.run() does a Task.shell() and a Task.resume() in once.

To set a Task execution timeout, use the optional timeout parameter to set the timeout value in seconds. Once this time is elapsed when the Task is still running, the running Task raises TimeoutError exception, cleaning by the way all scheduled workers and timers. Using such a timeout ensures that the Task will not exceed a given time for all its scheduled works. You can also configure per-worker timeout that generates an event EventHandler.ev_timeout() but will not raise an exception, allowing the Task to continue. Indeed, using a per-worker timeout is the prefered way for most applications.

Getting Task results¶

After the task is finished (after Task.resume() or Task.run()) or after a worker is completed when you have previously defined an event handler (at EventHandler.ev_close()), you can use Task result getters:

• Task.iter_buffers()
• Task.iter_errors()
• Task.node_buffer()
• Task.node_error()
• Task.max_retcode()
• Task.num_timeout()
• Task.iter_keys_timeout()

Note: buffer refers to standard output, error to standard error.

Please see some examples in Programming Examples.

Exiting the Task¶

If a Task does not have anymore scheduled worker or timer (for example, if you run one shell command and then it closes), it exits automatically from Task.resume(). Still, except from a signal handler, you can always call the following method to abort the Task execution:

• Task.abort()

For example, it is safe to call this method from an event handler within the task itself. On abort, all scheduled workers (shell command, file copy) and timers are cleaned and Task.resume() returns, unblocking the Task thread from a library user point of view. Please note that commands being executed remotely are not necessary stopped (this is due to ssh(1) behavior).

Configuring a Timer¶

A timer is bound to a Task (and its underlying Engine) and fires at a preset time in the future. Timers can fire either only once or repeatedly at fixed time intervals. Repeating timers can also have their next firing time manually adjusted (see Task.timer()).

A timer is not a real-time mechanism; it fires when the Task’s underlying Engine to which the timer has been added is running and able to check if the timer firing time has passed.

When a timer fires, the method EventHandler.ev_timer() of the associated EventHandler is called.

To configure a timer, use the following (secs in seconds with floating point precision):

task.timer(self, fire=secs, handler=handler [, interval=secs])

Changing default worker¶

When calling Task.shell() or Task.copy() the Task object creates a worker instance for each call. When the nodes argument is defined, the worker class used for these calls is based on Task default distant_worker. Change this value to use another worker class, by example Rsh:

from ClusterShell.Task import task_self
from ClusterShell.Worker.Rsh import WorkerRsh

task_self().set_default('distant_worker', WorkerRsh)

Thread safety and Task objects¶

ClusterShell is an event-based library and one of its advantage is to avoid the use of threads (and their safety issues), so it’s mainly not thread-safe. When possible, avoid the use of threads with ClusterShell. However, it’s sometimes not so easy, first because another library you want to use in some event handler is not event-based and may block the current thread (that’s enough to break the deal). Also, in some cases, it could be useful for you to run several Tasks at the same time. Since version 1.1, ClusterShell provides support for launching a Task in another thread and some experimental support for multiple Tasks, but:

• you should ensure that a Task is configured and accessed from one thread at a time before it’s running (there is no API lock/mutex protection),
• once the Task is running, you should modify it only from the same thread that owns that Task (for example, you cannot call Task.abort() from another thread).

The library provides two thread-safe methods and a function for basic Task interactions: Task.wait(), Task.join() and Task.task_wait() (function defined at the root of the Task module). Please refer to the API documentation.