CLUSH(1)

NAME
clush - execute shell commands on a cluster

SYNOPSIS

clush -a | -g group | -w nodes [OPTIONS]

clush -a | -g group | -w nodes [OPTIONS] command

clush -a | -g group | -w nodes [OPTIONS] --copy file | dir [ file | dir ...] [ --dest path ]

clush -a | -g group | -w nodes [OPTIONS] --rcopy file | dir [ file | dir ...] [ --dest path ]

DESCRIPTION

clush is a program for executing commands in parallel on a cluster and for gathering their results. clush executes commands interactively or can be used within shell scripts and other applications. It is a partial front-end to the ClusterShell library that ensures a light, unified and robust parallel command execution framework. Thus, it allows traditional shell scripts to benefit from some of the library features. clush currently makes use of the Ssh worker of ClusterShell, by default, that only requires ssh(1) (OpenSSH SSH client).

INVOCATION

clush can be started non-interactively to run a shell command, or can be invoked as an interactive shell. To start a clush interactive session, invoke the clush command without providing command.

Non-interactive mode
    When clush is started non-interactively, the command is executed on the specified remote hosts in parallel. If option -b or --dshbak is specified, clush waits for command completion and then displays gathered output results.

    The -w option allows you to specify remote hosts by using ClusterShell NodeSet syntax, including the node groups @group special syntax and the Extended Patterns syntax to benefit from NodeSet basic arithmetic (like @Agroup\&@Bgroup). See EXTENDED PATTERNS in nodeset(1) and also groups.conf(5) for more information.

    Unless the option --nostdin (or -n) is specified, clush detects when its standard input is connected to a terminal (as determined by isatty(3)). If actually connected to a terminal, clush listens to standard input when commands are running, waiting for an Enter key press. Doing so will display the status of current nodes. If standard input is not connected to a terminal, and unless the option --nostdin is specified, clush binds the standard input of the remote commands to its own standard input, allowing scripting methods like:
    # echo foo | clush -w node[40-42] -b cat
    ---------------
    node[40-42]
    ---------------
    foo

    Please see some other great examples in the EXAMPLES section below.

Interactive session
    If a command is not specified, and its standard input is connected to a terminal, clush runs interactively. In this mode, clush uses the GNU readline library to read command lines. Readline provides commands for searching through the command history for lines containing a specified string. For instance, type Control-R to search in the history for the next entry matching the search string typed so far. clush also recognizes special single-character prefixes that allow the user to see and modify the current nodeset (the nodes where the commands are executed).

Single-character interactive commands are:

clush> ?
    show current nodeset

clush> @<NODESET>
    set current nodeset

clush> +<NODESET>
    add nodes to current nodeset

clush> -<NODESET>
    remove nodes from current nodeset

clush> !COMMAND
    execute COMMAND on the local system

clush> =
    toggle the output format (gathered or standard mode)

    To leave an interactive session, type quit or Control-D.

Local execution ( --worker=exec or -R exec )
    Instead of running provided command on remote nodes, clush can use the dedicated exec worker to launch the command locally, for each node. Some parameters could be used in the command line to make a different command for each node. %h or %host will be replaced by node name and %n or %rank by the remote rank [0-N] (to get a literal % use %%). %hosts will be replaced by the full target nodeset, in which case the command is executed only once.

File copying mode ( --copy )
    When clush is started with the -c or --copy option, it will attempt to copy specified files and/or directories to the provided cluster nodes. The --dest option can be used to specify a single path where all the file(s) should be copied to on the target nodes. In the absence of --dest, clush will attempt to copy each file or directory found in the command line to their same location on the target nodes.

Reverse file copying mode ( --rcopy )
    When clush is started with the --rcopy option, it will attempt to retrieve specified file and/or dir from provided cluster nodes. If the --dest option is specified, it must be a directory path where the files will be stored with their hostname appended. If the destination path is not specified, it will take each file or directory's parent directory as the local destination.

OPTIONS

--version
    show clush version number and exit

-s GROUPSOURCE, --groupsource=GROUPSOURCE
    optional groups.conf(5) group source to use

-n, --nostdin
    do not watch for possible input from stdin; this should be used when clush is run in the background (or in scripts).

--groupsconf=FILE
    use alternate config file for groups.conf(5)

--conf=FILE
    use alternate config file for clush.conf(5)

-O <KEY=VALUE>, --option=<KEY=VALUE>
    override any key=value clush.conf(5) options (repeat as needed)

Selecting target nodes:

-w NODES
    nodes where to run the command

-x NODES
    exclude nodes from the node list

-a, --all
    run command on all nodes

-g GROUP, --group=GROUP
    run command on a group of nodes

-X GROUP
    exclude nodes from this group

--hostfile=FILE, --machinefile=FILE
    path to a file containing a list of single hosts, node sets or node groups, separated by spaces and lines (may be specified multiple times, one per file)

--topology=FILE
    topology configuration file to use for tree mode

--pick=N
    pick N node(s) at random in nodeset

Output behavior:

-q, --quiet
    be quiet, print essential output only

-v, --verbose
    be verbose, print informative messages

-d, --debug
    output more messages for debugging purpose

-G, --groupbase
    do not display group source prefix (always @groupname)

-L  disable header block and order output by nodes; if -b/-B is not specified, clush will wait for all commands to finish and then display aggregated output of commands with same return codes, ordered by node name; alternatively, when used in conjunction with -b/-B (e.g. -bL), clush will enable a "live gathering" of results by line, so that the next line is displayed as soon as possible (e.g. when all nodes have sent the line)

-N  disable labeling of command output

-P, --progress
    show progress during command execution; if writing is performed to standard input, the live progress indicator will display the global bandwidth of data written to the target nodes

-b, --dshbak
    display gathered results in a dshbak-like way (note: it will only try to aggregate the output of commands with same return codes)

-B  like -b but including standard error

-r, --regroup
    fold nodeset using node groups

--axis=RANGESET
    for nD nodesets, fold the displayed nodeset (e.g. in gathered output headers) along provided axis only. Axes are indexed from 1 to n and can be specified here either using the rangeset syntax, e.g. '1', '1-2', '1,3', or by a single negative number meaning that the index is counted from the end. Because some nodesets may have several different dimensions, axis indices are silently truncated to fall in the allowed range. This is the per-invocation equivalent of the fold_axis library default.

-S, --maxrc
    return the largest of command return codes

--color=WHENCOLOR
    clush can use NO_COLOR, CLICOLOR and CLICOLOR_FORCE environment variables. NO_COLOR takes precedence over CLICOLOR_FORCE which takes precedence over CLICOLOR. When --color option is used these environment variables are not taken into account. --color tells whether to use ANSI colors to surround node or nodeset prefix/header with escape sequences to display them in color on the terminal. WHENCOLOR is never, always or auto (which uses color if standard output/error refer to a terminal). Colors are set to [34m (blue foreground text) for stdout and [31m (red foreground text) for stderr, and cannot be modified.

--diff
    show diff between common outputs (find the best reference output by focusing on largest nodeset and also smaller command return code)

--outdir=OUTDIR
    output directory for stdout files (OPTIONAL)

--errdir=ERRDIR
    output directory for stderr files (OPTIONAL)

File copying:

-c, --copy
    copy local file or directory to remote nodes

--rcopy
    copy file or directory from remote nodes

--dest=DEST_PATH
    destination file or directory on the nodes (optional: use the first source directory path when not specified)

-p  preserve modification times and modes

Connection options:

-f FANOUT, --fanout=FANOUT
    do not execute more than FANOUT commands at the same time, useful to limit resource usage. In tree mode, the same fanout value is used on the head node and on each gateway (the fanout value is propagated). That is, if the fanout is 16, each gateway will initiate up to 16 connections to their target nodes at the same time. Default fanout value is defined in clush.conf(5).

-l USER, --user=USER
    execute remote command as user

-o OPTIONS, --options=OPTIONS
    can be used to give ssh options, e.g. -o "-p 2022 -i ~/.ssh/myidrsa"; these options are added first to ssh and override default ones

-t CONNECT_TIMEOUT, --connect_timeout=CONNECT_TIMEOUT
    limit time to connect to a node

-u COMMAND_TIMEOUT, --command_timeout=COMMAND_TIMEOUT
    limit time for command to run on the node

-m MODE, --mode=MODE
    run mode; define MODEs in <confdir>/*.conf

-R WORKER, --worker=WORKER
    worker name to use for connection (exec, ssh, rsh, pdsh, or the name of a Python worker module), default is ssh

--remote=REMOTE
    whether to enable remote execution: in tree mode, 'yes' forces connections to the leaf nodes for execution, 'no' establishes connections up to the leaf parent nodes for execution (default is 'yes')

    For a short explanation of these options, see -h, --help.

EXIT STATUS

By default, an exit status of zero indicates success of the clush command but gives no information about the remote commands exit status. However, when the -S option is specified, the exit status of clush is the largest value of the remote commands return codes.

For failed remote commands whose exit status is non-zero, and unless the combination of options -qS is specified, clush displays messages similar to:

clush: node[40-42]: exited with exit code 1

EXAMPLES

  Remote parallel execution

# clush -w node[3-5,62] uname -r
    Run command uname -r in parallel on nodes: node3, node4, node5 and node62

  Local parallel execution

# clush -w node[1-3] --worker=exec ping -c1 %host
    Run locally, in parallel, a ping command for nodes: node1, node2 and node3. You may also use -R exec as the shorter and pdsh compatible option.

  Display features

# clush -w node[3-5,62] -b uname -r
    Run command uname -r on nodes[3-5,62] and display gathered output results (integrated dshbak-like).

# clush -w node[3-5,62] -bL uname -r
    Line mode: run command uname -r on nodes[3-5,62] and display gathered output results without default header block.

# ssh node32 find /etc/yum.repos.d -type f | clush -w node[40-42] -b xargs ls -l
    Search some files on node32 in /etc/yum.repos.d and use clush to list the matching ones on node[40-42], and use -b to display gathered results.

# clush -w node[3-5,62] --diff dmidecode -s bios-version
    Run this Linux command to get BIOS version on nodes[3-5,62] and show version differences (if any).

# clush -w dc[1-2]n[1-2] --axis=1 -b uname -r
    Gather output and fold the displayed nodeset header along axis 1 only, e.g. show dc[1-2]n1,dc[1-2]n2 rather than dc[1-2]n[1-2] in the header block. See also fold_axis in the Library Defaults configuration to change it permanently.

  All nodes

# clush -a uname -r
    Run command uname -r on all cluster nodes, see groups.conf(5) to set up all cluster nodes (all: field).

# clush -a -x node[5,7] uname -r
    Run command uname -r on all cluster nodes except on nodes node5 and node7.

# clush -a --diff cat /some/file
    Run command cat /some/file on all cluster nodes and show differences (if any), line by line, between common outputs.

  Node groups

# clush -w @oss modprobe lustre
    Run command modprobe lustre on nodes from node group named oss, see groups.conf(5) to setup node groups (map: field).

# clush -g oss modprobe lustre
    Same as previous example but using -g to avoid @ group prefix.

# clush -w @mds,@oss modprobe lustre
    You may specify several node groups by separating them with commas (please see EXTENDED PATTERNS in nodeset(1) and also groups.conf(5) for more information).

  Copy files

# clush -w node[3-5,62] --copy /etc/motd
    Copy local file /etc/motd to remote nodes node[3-5,62].

# clush -w node[3-5,62] --copy /etc/motd --dest /tmp/motd2
    Copy local file /etc/motd to remote nodes node[3-5,62] at path /tmp/motd2.

# clush -w node[3-5,62] -c /usr/share/doc/clustershell
    Recursively copy local directory /usr/share/doc/clustershell to the same path on remote nodes node[3-5,62].

# clush -w node[3-5,62] --rcopy /etc/motd --dest /tmp
    Copy /etc/motd from remote nodes node[3-5,62] to local /tmp directory, each file having their remote hostname appended, e.g. /tmp/motd.node3.

FILES

$CLUSTERSHELL_CFGDIR/clush.conf
    Global clush configuration file. If $CLUSTERSHELL_CFGDIR is not defined, /etc/clustershell/clush.conf is used instead.

$XDG_CONFIG_HOME/clustershell/clush.conf
    User configuration file for clush. If $XDG_CONFIG_HOME is not defined, $HOME/.config/clustershell/clush.conf is used instead.

{sys.prefix}/etc/clustershell/clush.conf
    Configuration file when ClusterShell is installed in a Python virtual environment.

$HOME/.local/etc/clustershell/clush.conf
    Local user configuration file for clush (default installation for pip --user)

~/.clush.conf
    Deprecated per-user clush configuration file (no longer read since version 1.9).

~/.clush_history
    File in which interactive clush command history is saved.

SEE ALSO

clubak(1), cluset(1), nodeset(1), readline(3), clush.conf(5), groups.conf(5).

https://clustershell.readthedocs.io/

BUG REPORTS

Use the following URL to submit a bug report or feedback:
    https://github.com/clustershell/clustershell/issues

AUTHOR
Stephane Thiell

COPYRIGHT
GNU Lesser General Public License version 2.1 or later (LGPLv2.1+)