helm - Easy server and cluster automation

        helm TASK [OPTIONS]
        helm help tasks
        helm help [TASK]

        # patch the same file on all of the machines in your cluster 
        helm patch --file my_fix.patch --target /opt/cool_system/

        # run a command on only the servers which run memcache
        helm run "pidof memcached" --roles memcache 

        # copy a file to every server in your cluster
        helm put --local foo.tar.gz --remote /tmp/bar.tar.gz

        # rsync a local folder to specific servers
        helm rsync_put --local lib/important --remote /tmp/important_libs --servers web1,web2,web4

    "helm" is command-line utility to make it easy to automate system tasks
    for individual machines, a cluster of machines or a subset of machines
    in a cluster. It has the following features:

    *   Combine multiple commands into a single tasks and to have groups of
        related tasks gathered together.

    *   Uses SSH as the transport layer for tasks and uses SSH keys for
        automatic authorization.

    *   Simple optional configuration file describing your cluster, allowing
        tasks to target a single machine, multiple machines which share the
        same role, or all machines in your cluster. Can also be extended to
        pull configuration from more complicated sources like LDAP, etc.

    *   Logging of each action performed to multiple channels: console, log
        file, irc, email, etc.

    *   Interact with the remote processes via "STDIN", "STDOUT" and

    *   Convenient .helmrc file to reduce the number of options you need to
        pass on every invocation of "helm".

    *   Locking on the client and/or server so that multiple invocations of
        "helm" aren't running at the same time.

    "helm" has several options that can be specified for all commands. Any
    remaining arguments are passed to the task that was invoked. These
    global options are:

        By default, helm assumes you're running the comands as the current
        local. This works really well with SSH keys set up so that you don't
        have to try and type in your password on lots of login attempts.
        However, if you want to change the user, this option allows for
        that. You will however be prompted for a password if you don't have
        SSH keys properly set up.

        Specifically list out the hostnames of the servers to send the task
        to. These can either be full hostnames (or an unambiguous
        abbreviation of your servers as defined in your CONFIGURATION).
        Multiple hostnames must me specified in a comma separated list.

        If no servers (or roles) are specified, then the task will be
        performed on all servers (as defined in your CONFIGURATION).

            # using full hostnames 

            # using abbreviations from config file 
            --servers web1,db2

            # using abbreviations from config file and ranges
            --servers web[1-3],db[1-2]

        Instead of specifying the servers explicitly, you can instead
        specify which server roles you want to target (as defined in your
        CONFIGURATION). Multiple roles must be specified in a comma
        separated list.

        If no roles (or servers) are specified, then the task will be
        performed on all servers (as defined in your CONFIGURATION).

            # single role 
            --roles web

            # multiple roles
            --roles web,cache

        Specifically exclude servers from the list to be used. This can be
        combined with both "--servers" and "--roles" and even by itself to
        exclude certain servers from the list of all servers as defined in
        your CONFIGURATION).

            # combined with --servers
            --servers web[1-10] --exclude web3

            # combined with --roles, using ranges
            --roles web --exclude web[4-5]

        Specifically exclude servers with certain roles from the list to be
        used. This can be combined with both "--servers" and "--roles" and
        even by itself to exclude certain servers from the list of all
        servers as defined in your CONFIGURATION).

            # combined with --servers: any web[1-10] servers that aren't also caches
            --servers web[1-10] --exclude-roles cache

            # combined with --roles: any web servers that aren't also proxies
            --roles web --exclude-roles proxy

        Execute tasks in parallel on the remote servers. The default is to
        execute in serial instead.

        The maximum number of parallel processes to be running at the same
        time if "--parallel" is used. The default is 100.

        Which resource to use for pulling cluster configuration information
        (see CONFIGURATION).

            --config helm:///etc/helm.conf

        The log level used. Can be one of "debug", "info", "warn", and
        "error". Defaults to "info".

        Log messages to a specific channel. Multiple channels can be
        specified by specifying this option multiple times. The value of
        this option is the URI of the channel to be used. See "LOG CHANNELS"
        for more details.

            # send log messages to a comany tech IRC channel 
            --log irc://

            # log messages to a file and to email address
            --log file:///var/log/helm.log --log

        Log messages are also sent to "STDERR" unless the --quiet option is
        also passed.

        Allows you to load other 3rd party plugin modules to extend "helm"
        functionality. This could be to add more log channels, different
        configuration loading, etc. The value of this option is the full
        Perl module name of the plugin. This can be specified multiple

            # load a hypothetical Yammer log plugin 
            --load Helm::Log::Channel::Yammer

            # load hypothetical custom LDAP configuration and
            --load Helm::Conf::Loader::CompanyLDAP

        See "WRITING HELM PLUGINS" for more information.

        The user that should be used to perform the commands on the remote
        server. The actual SSH connection will be made using the current
        user's SSH keys, but then once the connection is made to the remote
        server, it's sometimes useful for the commands to be run as a
        different user. We also try to make sure that things like file
        permissions (on tasks like "put" and "patch") are also handled so
        that the resulting files are owned by this sudo user.

        This options allows you to have control over whether concurrent
        "helm" processes can be running on either the local or remote
        servers. The value can be one of: "none", "local", "remote", "both".

        Suppress the default loading of the ".helmrc file" file.

        The port to use for SSH on all of the remote servers. Defaults to
        the standard (22).

        The timeout in seconds to give the ssh connections. Default is 30

        If a command fails executing against this will normally cause
        execution to halt and not run against any other remote servers.
        Using this option will tell helm to continue on to the server(s)
        even if a previous one has errors.

        Suppress the default logging to "STDERR"

        Display this documentation

        Display the version of Helm installed

        Display a dump of the configuration data as understood by Helm

        Tell helm to dump out verbose information about what it is doing
        internally to a log file named helm.debug in the current working
        directory. This is different than the "--log-level" since that is
        mainly for end user logging of individual task actions. This flag is
        meant to debug the internals of helm, including the logging
        subsystem, which is why it's a separate flag.

        Sleep a given number of seconds in between each server.

    Helm comes with the following built-in tasks

    To get more information on how to run each task, simply use "helm help":

        helm help run

    This is the simplest of helm commands and will run some arbitrary
    command on the remote server.

    Get a file from a remote server and put it on the local machine.

    Put a local file onto a remote server.

    Put a local file onto a remote server using "rsync" rather than a full

    Patch a file on a remote server with a patch on the local machine.

    Clear out any stuck Helm locks on a remote server.

    By default, "helm" doesn't use a configuration file, but certain
    features require it (using roles, server abbreviations, etc) so it's
    best to have one. You can tell "helm" which configuration resource to
    use by using the --config option. Currently, only the "helm://" URI
    scheme is supported.

        --config helm:///etc/helm.conf

    A configuration file will look something like this:

        <Server web[1-5]>
            Role web

            Role db Role db_master

            Role db Role db_slave

    This configuration would define 7 servers (,,,,, and It defines 4 different roles (web,
    db, db_master, db_slave).

    "helm" currently just supports a single configuration resource format
    ("helm://"), but the internals are flexible enough that more formats
    could be supported in the future, including other configuration methods
    like LDAP, etc.

    If, for instance you wanted to support a URI like:

        --config ldap://

    See "WRITING HELM PLUGINS" for more information.

    If you are having problems getting your configuration right, you can
    pass the --dump-config option to tell "helm" to display what it thinks
    things are configured to be.

    "helm" can be told to send various levels of log data to different
    channels. By default we support a log file, IRC and email logs. We also
    support various log levels (see the --log-level option).

    You can specify which channel is used by giving a URI which indicates
    what type of channel and where to send the log. The following URI
    schemes are supported:

        This is basically a log file where messages are immediately

            --log file:///var/log/helm.log

        NOTE: IRC logging is still experimental and quite tempermental.
        Improvements are welcome.

        This is an IRC channel where messages are immediately sent. For
        example to send messages to the "sysadmin" IRC channel on
        "" using the user "michael" and the password "s3cr3t"
        you would have:

            --log irc://

        Similar to Mail-To links in HTML documents, this just specifies an
        email address to log. Log messages aren't sent immediately, but are
        instead queued up to be sent once the command has been completed.


    Plugins can be written to load allow other log channels. See "WRITING
    HELM PLUGINS" for more information on how this is done.

THE .helmrc FILE
    The .helmrc file contains command-line options that are prepended to the
    command line before processing. Multiple options may live on multiple
    lines. Lines beginning with a # are ignored. A .helmrc might look like

        # always log to a file 
        --log file:///var/log/helm.log

        # always load our custom plugins 
        --load Helm::Conf::Loader::CompanyLDAP 
        --load Helm::Log::CompanyYammer
        --load Company::CustomHelmTasks

    "helm" looks for the rc file first in the current directory and then in
    your home directory. You can specify another location with the "HELMRC"
    environment variable.

    If --no-rc-file is specified on the command line, the .helmrc file is

    Helm can be extended in many ways to make it more convenient for your
    projects. Helm has 3 customization points where plugins can be written
    and registered to interact: "task", "log" and "configuration". Using the
    "--load" option, you can tell Helm about your custom or 3rd party Perl
    modules that you would like to load. Each plugin module must register
    itself with Helm along with the type of events it will handle using the
    "Helm->register_module" method. For example, if I were creating a new
    plugin module for a custom task named "spiffy" I might invoke Helm like:

        helm spiffy --load MyCompany::Helm::Task::spiffy

    And my Perl module might look something like this:

        package MyCompany::Helm::Task::spiffy;
        use strict;
        use warnings;
        use Helm;
        use Moose;
        extends 'Helm::Task';
        Helm->register_module('task', spiffy => 'MyCompany::Helm::Task::spiffy')

        sub validate {
            my $self = shift;
            # custom validation

        sub execute {
            my ($self, %args) = @_;
            # do something spiffy


    Most likely you'd put the "--load" statment in your .helmrc file so you
    wouldn't have to worry about it again. Now we'll get into the details of
    what is expected of each type of plugin.

  Task Plugins
    A task plugin should inherit from Helm::Task and implement the following

        This method would perform any custom validation needed before the
        task is executed against any servers. Normally this involves
        validating the custom options this task might use. This method only
        receives a single object, the task itself. As an example, lets say
        you want a "--nifty" option to your "spiffy" plugin above:

            helm spiffy --nifty foo

        Then your validation method might look something like this:

            sub validate {
                my $self          = shift;
                my $helm          = $self->helm;
                my $extra_options = $helm->extra_options;
                my $nifty         = $extra_options->{nifty};
                $helm->die("You need to provide a --nifty option!") unless $nifty;
                $helm->die("--nifty option ($nifty) is not a valid value") unless $nifty =~ /^fo+/;

        This is the meat of your task plugin and is where the work happens.
        This method is called once for every server the task is being
        executed against. It receives the following named arguments:

        ssh A Net::OpenSSH object with an already open SSH connection to the
            server in question.

            A Helm::Server object for the currently executing task.

        As long as your method doesn't die (or preferrably calls
        "$helm->die()", then we will assume that all was fine and dandy in
        the execution of the task.

  Log Plugins
    Log plugins can offer new channels for logging Helm output based on the
    URI given to Helm. For instance, if you wanted to send SMS logging of
    critical messages only, you might invoke helm with a logger like:

        helm foo --log sms:+15105550101 --load MyCompany::Helm::Log::SMS

    And then your SMS module might look something like

        package MyCompany::Helm::Log::SMS;
        use strict;
        use warnings;
        use Helm;
        use Moose;
        extends 'Helm::Log::Channel';
        Helm->register_module('log', sms => 'MyCompany::Helm::Log::SMS');

        sub initialize   {}
        sub finalize     {}
        sub start_server {}
        sub end_server   {}
        sub debug        {}
        sub info         {}
        sub warn         {}
        sub error        { 
            my ($self, $msg) = @_;
            # send SMS message

    In this example, we don't care about anything except errors (since SMS
    messages cost money and would get really annoying for anything with
    frequency). Log plugins need to inherit from Helm::Log::Channel and
    implement the following methods:


   Configuration Plugins
    Configuration plugins can implement new ways to load configuration data
    about your servers based on the URI given to helm. For instance, if you
    wanted to load the list of your servers and their roles from a company
    LDAP server, you might invoke Helm like:

        helm foo --config ldap:// --load MyCompany::Helm::Conf::ldap

    Helm will look for the last module registered to handle the "ldap"
    scheme of that url. That module might look like:

        package MyCompany::Helm::Conf::ldap;
        use strict;
        use warnings;
        use Helm;
        use Moose;
        extends 'Helm::Conf::Loader';
        Helm->register_module('configuration', ldap => 'MyCompany::Helm::Conf::ldap')

        sub load {
            my ($self, %args) = @_;
            my $uri = $args{uri};
            # poke around in our LDAP server and create a list of Helm::Server objects
            my @servers = ...;
            # then return a Helm::Conf object
            return Helm::Conf->new(servers => \@servers)


    A configuration loading plugin should inherit from Helm::Conf::Loader
    and implement the following methods:

        This method is the backbone of a configuration plugin. It receives
        the following named arguments:

        uri A URI object representing the URI passed on the command line to
            be loaded by this configuration loader.

            The "Helm" object doing the loading.

        This method must create a list of Helm::Server objects and use them
        to return a Helm::Conf object.

        This has been developed and tested on Linux (with bash as the shell
        on the remote hosts) only. Dealing with multiple platforms and
        writing multi-platform tasks has been punted to the future.

    Alpha Software
        This software is very ALPHA, which means it's interface is likely to
        change in the future. It's used currently and happily in production
        but please be aware if you start using it that you'll probably want
        to follow future versions carefully to make sure you aren't bitten
        by API changes as thing are flushed out.

    In the not too distant future, we'd like to add the following features
    to Helm:

    * Add a capture option which allows stdout/stderr to be handled
    * Add a compare option which allows the output (stdout/stderr) to be
    compared between servers in an intelligent manner
    * A real exception system to avoid parsing error messages