Developers¶

Bases: tooz.coordination.CoordinationDriver

This driver uses python-consul client against consul servers.

The ConsulDriver implements a minimal set of coordination driver API(s) needed to make Consul being used as an option for Distributed Locking. The data is stored in Consul’s key-value store.

To configure the client to your liking please refer http://python-consul.readthedocs.org/en/latest/. Few options like ‘ttl’ and ‘namespace’ will be passed as part of the options. ‘ttl’ governs the duration till when the session holding the lock will be active.

DEFAULT_PORT = 8500¶

Default consul port if not provided.

DEFAULT_TTL = 15¶

Default TTL

TOOZ_NAMESPACE = u'tooz'¶

Default namespace when none is provided

Bases: tooz.coordination.CoordinationDriver

An etcd based driver.

This driver uses etcd provide the coordination driver semantics and required API(s).

DEFAULT_HOST = 'localhost'¶

Default hostname used when none is provided.

DEFAULT_PORT = 2379¶

Default port used if none provided (4001 or 2379 are the common ones).

DEFAULT_TIMEOUT = 30¶

Default socket/lock/member/leader timeout used when none is provided.

lock_encoder_cls¶

Class that will be used to encode lock names into a valid etcd url.

alias of Base64LockEncoder

Bases: tooz.coordination.CoordinationDriverCachedRunWatchers, tooz.coordination.CoordinationDriverWithExecutor

A file based driver.

This driver uses files and directories (and associated file locks) to provide the coordination driver semantics and required API(s). It is missing some functionality but in the future these not implemented API(s) will be filled in.

General recommendations/usage considerations:

• It does not automatically delete members from groups of processes that have died, manual cleanup will be needed for those types of failures.
• It is not distributed (or recommended to be used in those situations, so the developer using this should really take that into account when applying this driver in there app).

CHARACTERISTICS = (<Characteristics.NON_TIMEOUT_BASED: 'NON_TIMEOUT_BASED'>, <Characteristics.DISTRIBUTED_ACROSS_THREADS: 'DISTRIBUTED_ACROSS_THREADS'>, <Characteristics.DISTRIBUTED_ACROSS_PROCESSES: 'DISTRIBUTED_ACROSS_PROCESSES'>)¶

Tuple of Characteristics introspectable enum member(s) that can be used to interogate how this driver works.

HASH_ROUTINE = 'sha1'¶

This routine is used to hash a member (or group) id into a filesystem safe name that can be used for member lookup and group joining.

__init__(member_id, parsed_url, options)[source]¶

Initialize the file driver.

Bases: tooz.coordination.CoordinationDriverWithExecutor

A IPC based driver.

This driver uses IPC concepts to provide the coordination driver semantics and required API(s). It is missing some functionality but in the future these not implemented API(s) will be filled in.

General recommendations/usage considerations:

• It is not distributed (or recommended to be used in those situations, so the developer using this should really take that into account when applying this driver in there app).

CHARACTERISTICS = (<Characteristics.NON_TIMEOUT_BASED: 'NON_TIMEOUT_BASED'>, <Characteristics.DISTRIBUTED_ACROSS_THREADS: 'DISTRIBUTED_ACROSS_THREADS'>, <Characteristics.DISTRIBUTED_ACROSS_PROCESSES: 'DISTRIBUTED_ACROSS_PROCESSES'>)¶

Tuple of Characteristics introspectable enum member(s) that can be used to interogate how this driver works.

Bases: tooz.coordination.CoordinationDriverCachedRunWatchers, tooz.coordination.CoordinationDriverWithExecutor

A memcached based driver.

This driver users memcached concepts to provide the coordination driver semantics and required API(s). It is fully functional and implements all of the coordination driver API(s). It stores data into memcache using expiries and msgpack encoded values.

General recommendations/usage considerations:

• Memcache (without different backend technology) is a cache enough said.

CHARACTERISTICS = (<Characteristics.DISTRIBUTED_ACROSS_THREADS: 'DISTRIBUTED_ACROSS_THREADS'>, <Characteristics.DISTRIBUTED_ACROSS_PROCESSES: 'DISTRIBUTED_ACROSS_PROCESSES'>, <Characteristics.DISTRIBUTED_ACROSS_HOSTS: 'DISTRIBUTED_ACROSS_HOSTS'>, <Characteristics.CAUSAL: 'CAUSAL'>)¶

Tuple of Characteristics introspectable enum member(s) that can be used to interogate how this driver works.

DEFAULT_TIMEOUT = 30¶

Default socket/lock/member/leader timeout used when none is provided.

GROUP_LEADER_PREFIX = '_TOOZ_GROUP_LEADER_'¶

Key prefix attached to leaders of groups (used in name-spacing keys)

GROUP_LIST_KEY = '_TOOZ_GROUP_LIST'¶

Key where all groups ‘known’ are stored.

GROUP_PREFIX = '_TOOZ_GROUP_'¶

Key prefix attached to groups (used in name-spacing keys)

MEMBER_PREFIX = '_TOOZ_MEMBER_'¶

Key prefix attached to members of groups (used in name-spacing keys)

STILL_ALIVE = "It's alive!"¶

String used to keep a key/member alive (until it next expires).

Bases: tooz.coordination.CoordinationDriver

A MySQL based driver.

This driver users MySQL database tables to provide the coordination driver semantics and required API(s). It is missing some functionality but in the future these not implemented API(s) will be filled in.

CHARACTERISTICS = (<Characteristics.NON_TIMEOUT_BASED: 'NON_TIMEOUT_BASED'>, <Characteristics.DISTRIBUTED_ACROSS_THREADS: 'DISTRIBUTED_ACROSS_THREADS'>, <Characteristics.DISTRIBUTED_ACROSS_PROCESSES: 'DISTRIBUTED_ACROSS_PROCESSES'>, <Characteristics.DISTRIBUTED_ACROSS_HOSTS: 'DISTRIBUTED_ACROSS_HOSTS'>)¶

Tuple of Characteristics introspectable enum member(s) that can be used to interogate how this driver works.

__init__(member_id, parsed_url, options)[source]¶

Initialize the MySQL driver.

Bases: tooz.coordination.CoordinationDriver

A PostgreSQL based driver.

This driver users PostgreSQL database tables to provide the coordination driver semantics and required API(s). It is missing some functionality but in the future these not implemented API(s) will be filled in.

CHARACTERISTICS = (<Characteristics.NON_TIMEOUT_BASED: 'NON_TIMEOUT_BASED'>, <Characteristics.DISTRIBUTED_ACROSS_THREADS: 'DISTRIBUTED_ACROSS_THREADS'>, <Characteristics.DISTRIBUTED_ACROSS_PROCESSES: 'DISTRIBUTED_ACROSS_PROCESSES'>, <Characteristics.DISTRIBUTED_ACROSS_HOSTS: 'DISTRIBUTED_ACROSS_HOSTS'>)¶

Tuple of Characteristics introspectable enum member(s) that can be used to interogate how this driver works.

__init__(member_id, parsed_url, options)[source]¶

Initialize the PostgreSQL driver.

Bases: tooz.coordination.CoordinationDriverCachedRunWatchers, tooz.coordination.CoordinationDriverWithExecutor

Redis provides a few nice benefits that act as a poormans zookeeper.

It is fully functional and implements all of the coordination driver API(s). It stores data into redis using the provided redis API(s) using msgpack encoded values as needed.

• Durability (when setup with AOF mode).
• Consistent, note that this is still restricted to only one redis server, without the recently released redis (alpha) clustering > 1 server will not be consistent when partitions or failures occur (even redis clustering docs state it is not a fully AP or CP solution, which means even with it there will still be potential inconsistencies).
• Master/slave failover (when setup with redis sentinel), giving some notion of HA (values can be lost when a failover transition occurs).

To use a sentinel the connection URI must point to the sentinel server. At connection time the sentinel will be asked for the current IP and port of the master and then connect there. The connection URI for sentinel should be written as follows:

redis://<sentinel host>:<sentinel port>?sentinel=<master name>

Additional sentinel hosts are listed with multiple sentinel_fallback parameters as follows:

redis://<sentinel host>:<sentinel port>?sentinel=<master name>&
  sentinel_fallback=<other sentinel host>:<sentinel port>&
  sentinel_fallback=<other sentinel host>:<sentinel port>&
  sentinel_fallback=<other sentinel host>:<sentinel port>

Further resources/links:

• http://redis.io/
• http://redis.io/topics/sentinel
• http://redis.io/topics/cluster-spec

Note that this client will itself retry on transaction failure (when they keys being watched have changed underneath the current transaction). Currently the number of attempts that are tried is infinite (this might be addressed in https://github.com/andymccurdy/redis-py/issues/566 when that gets worked on). See http://redis.io/topics/transactions for more information on this topic.

General recommendations/usage considerations:

• When used for locks, run in AOF mode and think carefully about how your redis deployment handles losing a server (the clustering support is supposed to aid in losing servers, but it is also of unknown reliablity and is relatively new, so use at your own risk).

CHARACTERISTICS = (<Characteristics.DISTRIBUTED_ACROSS_THREADS: 'DISTRIBUTED_ACROSS_THREADS'>, <Characteristics.DISTRIBUTED_ACROSS_PROCESSES: 'DISTRIBUTED_ACROSS_PROCESSES'>, <Characteristics.DISTRIBUTED_ACROSS_HOSTS: 'DISTRIBUTED_ACROSS_HOSTS'>, <Characteristics.CAUSAL: 'CAUSAL'>)¶

Tuple of Characteristics introspectable enum member(s) that can be used to interogate how this driver works.

CLIENT_ARGS = frozenset(['ssl_keyfile', 'retry_on_timeout', 'ssl_certfile', 'sentinel', 'socket_keepalive', 'socket_timeout', 'db', 'encoding', 'sentinel_fallback', 'ssl'])¶

Keys that we allow to proxy from the coordinator configuration into the redis client (used to configure the redis client internals so that it works as you expect/want it to).

See: http://redis-py.readthedocs.org/en/latest/#redis.Redis

See: https://github.com/andymccurdy/redis-py/blob/2.10.3/redis/client.py

CLIENT_BOOL_ARGS = frozenset(['retry_on_timeout', 'ssl'])¶

Client arguments that are expected to be boolean convertible.

CLIENT_DEFAULT_SOCKET_TO = 30¶

Default socket timeout to use when none is provided.

CLIENT_INT_ARGS = frozenset(['socket_timeout', 'db', 'socket_keepalive'])¶

Client arguments that are expected to be int convertible.

CLIENT_LIST_ARGS = frozenset(['sentinel_fallback'])¶

Client arguments that are expected/allowed to be lists.

DEFAULT_ENCODING = 'utf8'¶

This is for python3.x; which will behave differently when returned binary types or unicode types (redis uses binary internally it appears), so to just stick with a common way of doing this, make all the things binary (with this default encoding if one is not given and a unicode string is provided).

DEFAULT_NAMESPACE = '_tooz'¶

Default namespace for keys when none is provided.

GROUP_EXISTS = '__created__'¶

Redis deletes dictionaries that have no keys in them, which means the key will disappear which means we can’t tell the difference between a group not existing and a group being empty without this key being saved…

GROUP_EXISTS_VALUE = '1'¶

Value used (with group exists key) to keep a group from disappearing.

MIN_VERSION = LooseVersion ('2.6.0')¶

The min redis version that this driver requires to operate with…

NAMESPACE_SEP = ':'¶

Separator that is used to combine a key with the namespace (to get the actual key that will be used).

SCRIPTS = {'create_group': '\n-- Extract *all* the variables (so we can easily know what they are)...\nlocal namespaced_group_key = KEYS[1]\nlocal all_groups_key = KEYS[2]\nlocal no_namespaced_group_key = ARGV[1]\nif redis.call("exists", namespaced_group_key) == 1 then\n return 0\nend\nredis.call("sadd", all_groups_key, no_namespaced_group_key)\nredis.call("hset", namespaced_group_key,\n "${group_existence_key}", "${group_existence_value}")\nreturn 1\n', 'delete_group': '\n-- Extract *all* the variables (so we can easily know what they are)...\nlocal namespaced_group_key = KEYS[1]\nlocal all_groups_key = KEYS[2]\nlocal no_namespaced_group_key = ARGV[1]\nif redis.call("exists", namespaced_group_key) == 0 then\n return -1\nend\nif redis.call("sismember", all_groups_key, no_namespaced_group_key) == 0 then\n return -2\nend\nif redis.call("hlen", namespaced_group_key) > 1 then\n return -3\nend\n-- First remove from the set (then delete the group); if the set removal\n-- fails, at least the group will still exist (and can be fixed manually)...\nif redis.call("srem", all_groups_key, no_namespaced_group_key) == 0 then\n return -4\nend\nredis.call("del", namespaced_group_key)\nreturn 1\n', 'update_capabilities': '\n-- Extract *all* the variables (so we can easily know what they are)...\nlocal group_key = KEYS[1]\nlocal member_id = ARGV[1]\nlocal caps = ARGV[2]\nif redis.call("exists", group_key) == 0 then\n return -1\nend\nif redis.call("hexists", group_key, member_id) == 0 then\n return -2\nend\nredis.call("hset", group_key, member_id, caps)\nreturn 1\n'}¶

Lua template scripts that will be used by various methods (they are turned into real scripts and loaded on call into the start() method).

STILL_ALIVE = 'Not dead!'¶

String used to keep a key/member alive (until it next expires).

Bases: tooz.drivers.zookeeper.KazooDriver

This driver uses the zake client to mimic real zookeeper servers.

It should be mainly used (and is really only intended to be used in this manner) for testing and integration (where real zookeeper servers are typically not available).

CHARACTERISTICS = (<Characteristics.NON_TIMEOUT_BASED: 'NON_TIMEOUT_BASED'>, <Characteristics.DISTRIBUTED_ACROSS_THREADS: 'DISTRIBUTED_ACROSS_THREADS'>)¶

Tuple of Characteristics introspectable enum member(s) that can be used to interogate how this driver works.

Bases: tooz.coordination.CoordinationDriverCachedRunWatchers

This driver uses the kazoo client against real zookeeper servers.

It is fully functional and implements all of the coordination driver API(s). It stores data into zookeeper using znodes and msgpack encoded values.

To configure the client to your liking a subset of the options defined at http://kazoo.readthedocs.org/en/latest/api/client.html will be extracted from the coordinator url (or any provided options), so that a specific coordinator can be created that will work for you.

Currently the following options will be proxied to the contained client:

CHARACTERISTICS = (<Characteristics.NON_TIMEOUT_BASED: 'NON_TIMEOUT_BASED'>, <Characteristics.DISTRIBUTED_ACROSS_THREADS: 'DISTRIBUTED_ACROSS_THREADS'>, <Characteristics.DISTRIBUTED_ACROSS_PROCESSES: 'DISTRIBUTED_ACROSS_PROCESSES'>, <Characteristics.DISTRIBUTED_ACROSS_HOSTS: 'DISTRIBUTED_ACROSS_HOSTS'>, <Characteristics.SEQUENTIAL: 'SEQUENTIAL'>)¶

Tuple of Characteristics introspectable enum member(s) that can be used to interogate how this driver works.

TOOZ_NAMESPACE = 'tooz'¶

Default namespace when none is provided.