Node sets handling

NodeSet class

NodeSet is a class to represent an ordered set of node names (optionally indexed). It's a convenient way to deal with cluster nodes and ease their administration. NodeSet is implemented with the help of two other ClusterShell public classes, RangeSet and RangeSetND, which implement methods to manage a set of numeric ranges in one or multiple dimensions. NodeSet, RangeSet and RangeSetND APIs match standard Python sets. A command-line interface (nodeset) which implements most of NodeSet features, is also available.

Other classes of the ClusterShell library make use of the NodeSet class when they come to deal with distant nodes.

Using NodeSet

If you are used to Python sets, NodeSet interface will be easy for you to learn. The main conceptual difference is that NodeSet iterators always provide ordered results (and also NodeSet.__getitem__() by index or slice is allowed). Furthermore, NodeSet provides specific methods like NodeSet.split(), NodeSet.contiguous() (see below), or NodeSet.groups(), NodeSet.regroup() (these last two are related to Node groups). The following code snippet shows you a basic usage of the NodeSet class:

>>> from ClusterShell.NodeSet import NodeSet
>>> nodeset = NodeSet()
>>> nodeset.add("node7")
>>> nodeset.add("node6")
>>> print(nodeset)
node[6-7]

NodeSet class provides several object constructors:

>>> print(NodeSet("node[1-5]"))
node[1-5]
>>> print(NodeSet.fromlist(["node1", "node2", "node3"]))
node[1-3]
>>> print(NodeSet.fromlist(["node[1-5]", "node[6-10]"]))
node[1-10]
>>> print(NodeSet.fromlist(["clu-1-[1-4]", "clu-2-[1-4]"]))
clu-[1-2]-[1-4]

All corresponding Python sets operations are available, for example:

>>> from ClusterShell.NodeSet import NodeSet
>>> ns1 = NodeSet("node[10-42]")
>>> ns2 = NodeSet("node[11-16,18-39]")
>>> print(ns1.difference(ns2))
node[10,17,40-42]
>>> print(ns1 - ns2)
node[10,17,40-42]
>>> ns3 = NodeSet("node[1-14,40-200]")
>>> print(ns3.intersection(ns1))
node[10-14,40-42]

Unlike Python sets, it is important to notice that NodeSet is somewhat not so strict about the type of element used for set operations. Thus when a string object is encountered, it is automatically converted to a NodeSet object for convenience. The following example shows an example of this (set operation is working with either a native nodeset or a string):

>>> nodeset = NodeSet("node[1-10]")
>>> nodeset2 = NodeSet("node7")
>>> nodeset.difference_update(nodeset2)
>>> print(nodeset)
node[1-6,8-10]
>>>
>>> nodeset.difference_update("node8")
>>> print(nodeset)
node[1-6,9-10]

NodeSet ordered content leads to the following being allowed:

>>> nodeset = NodeSet("node[10-49]")
>>> print(nodeset[0])
node10
>>> print(nodeset[-1])
node49
>>> print(nodeset[10:])
node[20-49]
>>> print(nodeset[:5])
node[10-14]
>>> print(nodeset[::4])
node[10,14,18,22,26,30,34,38,42,46]

And it works for node names without index, for example:

>>> nodeset = NodeSet("lima,oscar,zulu,alpha,delta,foxtrot,tango,x-ray")
>>> print(nodeset)
alpha,delta,foxtrot,lima,oscar,tango,x-ray,zulu
>>> print(nodeset[0])
alpha
>>> print(nodeset[-2])
x-ray

And also for multidimensional node sets:

>>> nodeset = NodeSet("clu1-[1-10]-ib[0-1],clu2-[1-10]-ib[0-1]")
>>> print(nodeset)
clu[1-2]-[1-10]-ib[0-1]
>>> print(nodeset[0])
clu1-1-ib0
>>> print(nodeset[-1])
clu2-10-ib1
>>> print(nodeset[::2])
clu[1-2]-[1-10]-ib0

Conversely, to find the zero-based position of a node in the set, use the NodeSet.index() method (the inverse of __getitem__), for example:

>>> print(NodeSet("node[10-49]").index("node20"))
10

To split a NodeSet object into n subsets, use the NodeSet.split() method, for example:

>>> for nodeset in NodeSet("node[10-49]").split(2):
...     print(nodeset)
...
node[10-29]
node[30-49]

To split a NodeSet object into contiguous subsets, use the NodeSet.contiguous() method, for example:

>>> for nodeset in NodeSet("node[10-49,51-53,60-64]").contiguous():
...     print(nodeset)
...
node[10-49]
node[51-53]
node[60-64]

For further details, please see the full NodeSet API documentation.

Multidimensional considerations

Version 1.7 introduces full support of multidimensional NodeSet (e.g. da[2-5]c[1-2]p[0-1]). The NodeSet interface is the same, multidimensional patterns are automatically detected by the parser and processed internally. While expanding a multidimensional NodeSet is easily solved by performing a Cartesian product of all dimensions, folding nodes is much more complex and time consuming. To reduce the performance impact of such feature, the NodeSet class still relies on RangeSet when only one dimension is varying (see RangeSet class). Otherwise, it uses a new class named RangeSetND for full multidimensional support (see RangeSetND class).

Extended String Pattern

NodeSet class parsing engine recognizes an extended string pattern, adding support for union (with special character ","), difference (with special character "!"), intersection (with special character "&") and symmetric difference (with special character "^") operations. String patterns are read from left to right, by processing any character operators accordingly. The following example shows how you can use this feature:

>>> print(NodeSet("node[10-42],node46!node10"))
node[11-42,46]

Node groups

Node groups are very useful and are needed to group similar cluster nodes in terms of configuration, installed software, available resources, etc. A node can be a member of more than one node group.

Using node groups

Node groups are prefixed with @ character. Please see Node group expression rules for more details about node group expression/syntax rules.

Please also have a look at Node groups configuration to learn how to configure external node group bindings (sources). Once setup (please use the nodeset command to check your configuration), the NodeSet parsing engine automatically resolves node groups. For example:

>>> print(NodeSet("@oss"))
example[4-5]
>>> print(NodeSet("@compute"))
example[32-159]
>>> print(NodeSet("@compute,@oss"))
example[4-5,32-159]

That is, all NodeSet-based applications share the same system-wide node group configuration (unless explicitly disabled --- see Disabling node group resolution).

When the all group upcall is configured (node groups configuration), you can also use the following NodeSet constructor:

>>> print(NodeSet.fromall())
example[4-6,32-159]

When group upcalls are not properly configured, this constructor will raise a NodeSetExternalError exception.

Finding node groups

In order to find node groups a specified node set belongs to, you can use the NodeSet.groups() method. This method is used by the nodeset -l <nodeset> or cluset -l <nodeset> command (see Finding node groups). It returns a Python dictionary where keys are groups found and values, provided for convenience, are tuples of the form (group_nodeset, contained_nodeset). For example:

>>> for group, (group_nodes, contained_nodes) in NodeSet("@oss").groups().items():
...     print(group, group_nodes, contained_nodes)
...
@all example[4-6,32-159] example[4-5]
@oss example[4-5] example[4-5]

More usage examples follow:

>>> list(NodeSet("example4").groups().keys())
['@all', '@oss']
>>> list(NodeSet("@mds").groups().keys())
['@all', '@mds']
>>> list(NodeSet("dummy0").groups().keys())
[]

Regrouping node sets

If needed group configuration conditions are met (cf. node groups configuration), you can use the NodeSet.regroup() method to reduce node sets using matching groups, whenever possible:

>>> print(NodeSet("example[4-6]").regroup())
@mds,@oss

The nodeset and cluset commands make use of the NodeSet.regroup() method when using the -r switch (see Resolving node groups).

Overriding default groups configuration

It is possible to override the library default groups configuration by changing the default NodeSet resolver object. Usually, this is done for testing or special purposes. Here is an example of how to override the resolver object using NodeSet.set_std_group_resolver() in order to use another configuration file:

>>> from ClusterShell.NodeSet import NodeSet, set_std_group_resolver
>>> from ClusterShell.NodeUtils import GroupResolverConfig
>>> set_std_group_resolver(GroupResolverConfig("/other/groups.conf"))
>>> print(NodeSet("@oss"))
other[10-20]

It is possible to restore NodeSet default group resolver by passing None to the NodeSet.set_std_group_resolver() module function, for example:

>>> from ClusterShell.NodeSet import set_std_group_resolver
>>> set_std_group_resolver(None)

Disabling node group resolution

If for any reason, you want to disable host groups resolution, you can use the special resolver value RESOLVER_NOGROUP. In that case, NodeSet parsing engine will not recognize @ group characters anymore, for instance:

>>> from ClusterShell.NodeSet import NodeSet, RESOLVER_NOGROUP
>>> print(NodeSet("@oss"))
example[4-5]
>>> print(NodeSet("@oss", resolver=RESOLVER_NOGROUP))
@oss

Any attempts to use a group-based method (like NodeSet.groups() or NodeSet.regroup()) on such "no group" NodeSet will raise a NodeSetExternalError exception.

NodeSet object serialization

The NodeSet class supports object serialization through the standard pickling. Group resolution is done before pickling.