FLATLINE(1)

NAME
flatline - ensure the availability of services on either of two computers

SYNOPSIS
flatline [-rsplbithHv] - -remote-address address - -script path [- -port port] [- -listen-port port] [- -bind-address address] [- -interval timespec] [- -timeout timespec] [- -help] [- -version] [- -verbose]

DESCRIPTION
Ensures the availability of services on either of two computers. See the section BACKGROUND for more information.

  Command line options

-h, - -help
    Display a concise summary of the available options and argument syntax.

-v, - -verbose
    flatline will log additional messages about its operation. Useful for diagnosing problems.

-V, - -version
    Display version and copyright information.

-r, - -remote-address address
    The address of the other computer. flatline will send periodic status updates to this address. This may be either a numeric address (such as 2001:db8::123:abc) or a symbolic (domain) name. Please be aware that this name is only resolved when flatline starts up. Any changes to /etc/hosts or DNS after that moment will not be picked up.

    As a courtesy to users forced to deal with antiquated data center equipment, IPv4 is still supported as an alternative.

-p, - -port port
    Port of the remote host to which this flatline instance will send status updates. May be a numeric value or a symbolic name (from, for instance, /etc/services). Please be aware that this name is only resolved when flatline starts up. Any changes to /etc/services after that moment will not be picked up.

-l, - -listen-port port
    Port on which this instance will listen for status updates from the other computer. Defaults to the value of - -port and indeed the same syntax restrictions and caveats apply.

-b, - -bind-address address
    Address on which this instance will listen for incoming status updates. Useful if the computer has multiple addresses and you only want to accept status updates on an internal interface, or if you want to run multiple instances of flatline on the same port.

-s, - -script path
    The path to the executable (usually a script) that starts and stops the services that are managed by flatline.

    Invoked with two arguments:

    1.  The string “local” or “remote”, to indicate if the services local to this computer should be started or the services that normally run on the other computer (remote services).

    2.  The string “start” or “stop”, indicating whether the services should be started or stopped.

    Please note that the script needs to quit after bringing up (or down) the services. flatline will only execute one script at a time, so while one script is running, no other migrations can take place. Use the timelimit(1) program in your script if you are unsure if certain statements in your script will finish in a timely fashion.

    If you wish to share this script between both computers you will need to take into account that what is “local” to one computer is “remote” to the other. You could use the hostname(1) command together with the local/remote argument to determine which service needs to be started or stopped.

-i, - -timeout timespec
    If no status updates are received for timespec seconds, the remote side will be declared to be down and their services will be taken over.

    Specified as an integer number followed by a unit. No whitespace is allowed. Fractions are not supported. Available units:

    ns  nanoseconds
    us  microseconds
    ms  milliseconds
    s   seconds
    m   minutes
    h   hours

    If no unit is specified, seconds are assumed.

    Example: - -timeout 1500ms indicates a timeout of one and a half seconds.

    Defaults to 3.5 times the status update interval (see below) or 1 second if the status update interval is not set.

    The - -timeout value must be the same on both computers.

-i, - -interval timespec
    The interval between status updates to the remote instance. Uses the same syntax as - -timeout.

    Defaults to the value of - -timeout divided by 3.5 (in other words: 3 packets must be lost before the other side will declare this side to be down).

    The - -interval value must be the same on both computers.

BACKGROUND
flatline is intended for the case where you want certain services to be served by one of two computers but not both at the same time. For example, so that if one computer is unavailable (due to maintenance or mishap) the service will still be available to end-users.

If you run flatline instances on both computers they will start and stop your services for you and coordinate with each other to make sure that if one computer goes down, the other will temporarily take over that computer's services.

Each flatline instance has the concept of “local” and “remote” services. In flatline's view of the world, each computer has some services that are local to that computer, that is, they normally run on that computer unless that computer is not available. The services that normally run on the other computer are considered remote services.

When both instances of flatline are up and running, each will start their respective local services. Only when one flatline instance detects that the other instance is gone, will it start the other's services in addition to its own. When the other instance returns it will again stop the remote services so that the other can take them back.

It is important to understand that this is done on a best-effort basis. For example, if communications are disrupted it may happen that a service is started on both computers, despite all efforts taken by flatline to prevent exactly that. In general it's mathematically impossible to guarantee that this does not happen. (Please look up the “Two Generals Problem” if you would like to know more about why this is so.)

For this reason you should never manage a service in this way that would suffer from data loss if it is active on both computers, such as a filesystem mount on a shared block device. The perfect candidate for a service to manage by a program like flatline would be a floating IP address. Even if it is active on both computers, no data is lost.

flatline is intended to be a minimalistic alternative to the heartbeat/pacemaker/corosync suite of programs. Most features from that suite are not supported, in the hope that this will increase both the reliability and ease of management.

  Supported features:
•   two instances
•   generic shell script for starting/stopping services
•   firewall friendliness

  Intentionally unsupported features:
•   more than two instances
•   differential xml state updates
•   configuration files
•   paxos
•   byzantine failure

  Currently unsupported:
•   operating systems other than Linux

SECURITY
flatline does not authenticate its messages in any way. The rationale behind this is that this would affect the simplicity and reliability of the system.

That does not mean security was not considered. The attack surface was deliberately kept as small as possible in the design.

Impact of passive attacks (eavesdropping):

    A passive attacker (eavesdropper) can tell from the packets which service is currently active on which computer. It can determine the IP-addresses and ports of the network interfaces used for flatline's coordination. It can tell if the flatline daemon on either computer is shutting down.

Impact of active attacks (packet injection):

    An active attacker (someone who can fake flatline's coordination packets) can confuse either computer into believing that the other computer is up, down or shutting down. It can also confuse either computer into believing that the other side has services they do not have. It is trivial to create a so-called “split brain” situation this way (the situation in which both sides start the same service).

If security is any consideration at all (and it is hard to envision a situation where this is not the case) a combination of dedicated network cables, firewall rules, the - -bind-address option and VPNs (IPSEC, OpenVPN, WireGuard, tinc) should be used.

SEE ALSO
timelimit(1), heartbeat(1), pacemaker(1), corosync(1)

AUTHOR
Wessel Dankers <wsl@fruit.je>

COPYRIGHT
Copyright (c) 2016, Wessel Dankers <wsl@fruit.je>
Copyright (c) 2016, Tilburg University