Health Policy¶

Properties¶

A typical spec for a health policy looks like the following example:

type: senlin.policy.health
version: 1.0
properties:
  detection:
    type: NODE_STATUS_POLLING
    options:
      interval: 60
  recovery:
    actions:
      - name: REBOOT
        params:
          type: soft
    fencing:
      - compute

There are two groups of properties (detection and recovery), each of which provides information related to the failure detection and the failure recovery aspect respectively.

For failure detection, you can specify one of the following two values:

• NODE_STATUS_POLLING: Senlin engine (more specifically, the health manager service) is expected to poll each and every nodes periodically to find out if they are “alive” or not.
• LIFECYCLE_EVENTS: Many services can emit notification messages on the message queue when configured. Senlin engine is expected to listen to these events and react to them appropriately.

Both detection types can carry an optional map of options. When the detection type is set to “NODE_STATUS_POLLING”, for example, you can specify a value for interval property to customize the frequency at which your cluster nodes are polled.

As the policy type implementation stabilizes, more options may be added later.

For failure recovery, there are currently two properties: actions and fencing. The actions property takes a list of action names and an optional map of parameters specific to that action. For example, the REBOOT action can be accompanied with a type parameter that indicates if the intended reboot operation is a soft reboot or a hard reboot.

Validation¶

Due to implementation limitation, currently you can only specify one action for the recovery.actions property. This constraint will be removed soon after the support to action list is completed.

Fencing¶

Fencing may be an important step during a reliable node recovery process. Without fencing, we cannot ensure that the compute, network and/or storage resources are in a consistent, predictable status. However, fencing is very difficult because it always involves an out-of-band operation to the resource controller, for example, an IPMI command to power off a physical host sent to a specific IP address.

Currently, the health policy only supports the fencing of virtual machines by forcibly delete it before taking measures to recover it.

Snapshots¶

There have been some requirements to take snapshots of a node before recovery so that the recovered node(s) will resume from where they failed. This feature is also on the TODO list for the development team.