swiftclient

OpenStack Swift Python client binding.

swiftclient.authv1

Authentication plugin for keystoneauth to support v1 endpoints.

Way back in the long-long ago, there was no Keystone. Swift used an auth mechanism now known as “v1”, which used only HTTP headers. Auth requests and responses would look something like:

> GET /auth/v1.0 HTTP/1.1
> Host: <swift server>
> X-Auth-User: <tenant>:<user>
> X-Auth-Key: <password>
>
< HTTP/1.1 200 OK
< X-Storage-Url: http://<swift server>/v1/<tenant account>
< X-Auth-Token: <token>
< X-Storage-Token: <token>
<

This plugin provides a way for Keystone sessions (and clients that use them, like python-openstackclient) to communicate with old auth endpoints that still use this mechanism, such as tempauth, swauth, or https://identity.api.rackspacecloud.com/v1.0

class swiftclient.authv1.AccessInfoV1(auth_url, storage_url, account, username, auth_token, token_life)

Bases: object

An object for encapsulating a raw v1 auth token.

property expires
classmethod from_state(data)

Deserialize the given state.

Returns

a new AccessInfoV1 object with the given state

get_state()

Serialize the current state.

property issued
property user_id
will_expire_soon(stale_duration)

Determines if expiration is about to occur.

Returns

true if expiration is within the given duration

class swiftclient.authv1.PasswordLoader

Bases: keystoneauth1.loading.base.BaseLoader

Option handling for the v1password plugin.

property available

Return if the plugin is available for loading.

If a plugin is missing dependencies or for some other reason should not be available to the current system it should override this property and return False to exclude itself from the plugin list.

Return type

bool

create_plugin(**kwargs)

Create a plugin from the options available for the loader.

Given the options that were specified by the loader create an appropriate plugin. You can override this function in your loader.

This used to be specified by providing the plugin_class property and this is still supported, however specifying a property didn’t let you choose a plugin type based upon the options that were presented.

Override this function if you wish to return different plugins based on the options presented, otherwise you can simply provide the plugin_class property.

Added 2.9

get_options()

Return the list of parameters associated with the auth plugin.

This list may be used to generate CLI or config arguments.

load_from_options(**kwargs)

Create a plugin from the arguments retrieved from get_options.

A client can override this function to do argument validation or to handle differences between the registered options and what is required to create the plugin.

load_from_options_getter(getter, **kwargs)

Load a plugin from getter function that returns appropriate values.

To handle cases other than the provided CONF and CLI loading you can specify a custom loader function that will be queried for the option value. The getter is a function that takes a keystoneauth1.loading.Opt and returns a value to load with.

Parameters

getter (callable) – A function that returns a value for the given opt.

Returns

An authentication Plugin.

Return type

keystoneauth1.plugin.BaseAuthPlugin

plugin_class

alias of PasswordPlugin

class swiftclient.authv1.PasswordPlugin(auth_url, username, password, project_name=None, reauthenticate=True)

Bases: keystoneauth1.identity.base.BaseIdentityPlugin

A plugin for authenticating with a username and password.

Subclassing from BaseIdentityPlugin gets us a few niceties, like handling token invalidation and locking during authentication.

Parameters

  • auth_url (string) – Identity v1 endpoint for authorization.

  • username (string) – Username for authentication.

  • password (string) – Password for authentication.

  • project_name (string) – Swift account to use after authentication. We use ‘project_name’ to be consistent with other auth plugins.

  • reauthenticate (string) – Whether to allow re-authentication.

MIN_TOKEN_LIFE_SECONDS = 120
access_class

alias of AccessInfoV1

get_access(session, **kwargs)

Fetch or return a current AccessInfo object.

If a valid AccessInfo is present then it is returned otherwise a new one will be fetched.

Parameters

session (keystoneauth1.session.Session) – A session object that can be used for communication.

Raises

keystoneauth1.exceptions.http.HttpError – An error from an invalid HTTP response.

Returns

Valid AccessInfo

Return type

keystoneauth1.access.AccessInfo

get_all_version_data(session, interface='public', region_name=None, service_type=None, **kwargs)

Get version data for all services in the catalog.

Parameters

  • session (keystoneauth1.session.Session) – A session object that can be used for communication.

  • interface – Type of endpoint to get version data for. Can be a single value or a list of values. A value of None indicates that all interfaces should be queried. (optional, defaults to public)

  • region_name (string) – Region of endpoints to get version data for. A valueof None indicates that all regions should be queried. (optional, defaults to None)

  • service_type (string) – Limit the version data to a single service. (optional, defaults to None)

Returns

A dictionary keyed by region_name with values containing dictionaries keyed by interface with values being a list of VersionData.

get_api_major_version(session, service_type=None, interface=None, region_name=None, service_name=None, version=None, allow=None, allow_version_hack=True, skip_discovery=False, discover_versions=False, min_version=None, max_version=None, **kwargs)

Return the major API version for a service.

If a valid token is not present then a new one will be fetched using the session and kwargs.

version, min_version and max_version can all be given either as a string or a tuple.

Valid interface types: public or publicURL,

internal or internalURL, admin or ‘adminURL`

Parameters

  • session (keystoneauth1.session.Session) – A session object that can be used for communication.

  • service_type (string) – The type of service to lookup the endpoint for. This plugin will return None (failure) if service_type is not provided.

  • interface – Type of endpoint. Can be a single value or a list of values. If it’s a list of values, they will be looked for in order of preference. Can also be keystoneauth1.plugin.AUTH_INTERFACE to indicate that the auth_url should be used instead of the value in the catalog. (optional, defaults to public)

  • region_name (string) – The region the endpoint should exist in. (optional)

  • service_name (string) – The name of the service in the catalog. (optional)

  • version – The minimum version number required for this endpoint. (optional)

  • allow (dict) – Extra filters to pass when discovering API versions. (optional)

  • allow_version_hack (bool) – Allow keystoneauth to hack up catalog URLS to support older schemes. (optional, default True)

  • skip_discovery (bool) – Whether to skip version discovery even if a version has been given. This is useful if endpoint_override or similar has been given and grabbing additional information about the endpoint is not useful.

  • discover_versions (bool) – Whether to get version metadata from the version discovery document even if it’s not neccessary to fulfill the major version request. Defaults to False because get_endpoint doesn’t need metadata. (optional, defaults to False)

  • min_version – The minimum version that is acceptable. Mutually exclusive with version. If min_version is given with no max_version it is as if max version is ‘latest’. (optional)

  • max_version – The maximum version that is acceptable. Mutually exclusive with version. If min_version is given with no max_version it is as if max version is ‘latest’. (optional)

Raises

keystoneauth1.exceptions.http.HttpError – An error from an invalid HTTP response.

Returns

The major version of the API of the service discovered.

Return type

tuple or None

Note

Implementation notes follow. Users should not need to wrap their head around these implementation notes. get_api_major_version should do what is expected with the least possible cost while still consistently returning a value if possible.

There are many cases when major version can be satisfied without actually calling the discovery endpoint (like when the version is in the url). If the user has a cloud with the versioned endpoint https://volume.example.com/v3 in the catalog for the block-storage service and they do:

client = adapter.Adapter(
    session, service_type='block-storage', min_version=2,
    max_version=3)
volume_version = client.get_api_major_version()

The version actually be returned with no api calls other than getting the token. For that reason, get_api_major_version() first calls get_endpoint_data() with discover_versions=False.

If their catalog has an unversioned endpoint https://volume.example.com for the block-storage service and they do this:

client = adapter.Adapter(session, service_type='block-storage')

client is now set up to “use whatever is in the catalog”. Since the url doesn’t have a version, get_endpoint_data() with discover_versions=False will result in api_version=None. (No version was requested so it didn’t need to do the round trip)

In order to find out what version the endpoint actually is, we must make a round trip. Therefore, if api_version is None after the first call, get_api_major_version() will make a second call to get_endpoint_data() with discover_versions=True.

get_auth_ref(session, **kwargs)

Obtain a token from a v1 endpoint.

This function should not be called independently and is expected to be invoked via the do_authenticate function.

This function will be invoked if the AcessInfo object cached by the plugin is not valid. Thus plugins should always fetch a new AccessInfo when invoked. If you are looking to just retrieve the current auth data then you should use get_access.

Parameters

session – A session object that can be used for communication.

Returns

Token access information.

get_auth_state()

Retrieve the current authentication state for the plugin.

Returns

raw python data (which can be JSON serialized) that can be moved into another plugin (of the same type) to have the same authenticated state.

get_cache_id()

Fetch an identifier that uniquely identifies the auth options.

The returned identifier need not be decomposable or otherwise provide any way to recreate the plugin.

This string MUST change if any of the parameters that are used to uniquely identity this plugin change. It should not change upon a reauthentication of the plugin.

Returns

A unique string for the set of options

Return type

str or None if this is unsupported or unavailable.

get_cache_id_elements()

Get the elements for this auth plugin that make it unique.

get_connection_params(session, **kwargs)

Return any additional connection parameters required for the plugin.

Parameters

session (keystoneauth1.session.Session) – The session object that the auth_plugin belongs to.

Returns

Headers that are set to authenticate a message or None for failure. Note that when checking this value that the empty dict is a valid, non-failure response.

Return type

dict

get_discovery(*args, **kwargs)

Return the discovery object for a URL.

Check the session and the plugin cache to see if we have already performed discovery on the URL and if so return it, otherwise create a new discovery object, cache it and return it.

This function is expected to be used by subclasses and should not be needed by users.

Parameters

  • session (keystoneauth1.session.Session) – A session object to discover with.

  • url (str) – The url to lookup.

  • authenticated (bool) – Include a token in the discovery call. (optional) Defaults to None (use a token if a plugin is installed).

Raises

  • keystoneauth1.exceptions.discovery.DiscoveryFailure – if for some reason the lookup fails.

  • keystoneauth1.exceptions.http.HttpError – An error from an invalid HTTP response.

Returns

A discovery object with the results of looking up that URL.

get_endpoint(session, interface='public', **kwargs)

Return an endpoint for the client.

get_endpoint_data(session, service_type=None, interface=None, region_name=None, service_name=None, allow=None, allow_version_hack=True, discover_versions=True, skip_discovery=False, min_version=None, max_version=None, endpoint_override=None, **kwargs)

Return a valid endpoint data for a service.

If a valid token is not present then a new one will be fetched using the session and kwargs.

version, min_version and max_version can all be given either as a string or a tuple.

Valid interface types: public or publicURL,

internal or internalURL, admin or ‘adminURL`

Parameters

  • session (keystoneauth1.session.Session) – A session object that can be used for communication.

  • service_type (string) – The type of service to lookup the endpoint for. This plugin will return None (failure) if service_type is not provided.

  • interface – Type of endpoint. Can be a single value or a list of values. If it’s a list of values, they will be looked for in order of preference. Can also be keystoneauth1.plugin.AUTH_INTERFACE to indicate that the auth_url should be used instead of the value in the catalog. (optional, defaults to public)

  • region_name (string) – The region the endpoint should exist in. (optional)

  • service_name (string) – The name of the service in the catalog. (optional)

  • allow (dict) – Extra filters to pass when discovering API versions. (optional)

  • allow_version_hack (bool) – Allow keystoneauth to hack up catalog URLS to support older schemes. (optional, default True)

  • discover_versions (bool) – Whether to get version metadata from the version discovery document even if it’s not neccessary to fulfill the major version request. (optional, defaults to True)

  • skip_discovery (bool) – Whether to skip version discovery even if a version has been given. This is useful if endpoint_override or similar has been given and grabbing additional information about the endpoint is not useful.

  • min_version – The minimum version that is acceptable. Mutually exclusive with version. If min_version is given with no max_version it is as if max version is ‘latest’. (optional)

  • max_version – The maximum version that is acceptable. Mutually exclusive with version. If min_version is given with no max_version it is as if max version is ‘latest’. (optional)

  • endpoint_override (str) – URL to use instead of looking in the catalog. Catalog lookup will be skipped, but version discovery will be run. Sets allow_version_hack to False (optional)

  • kwargs – Ignored.

Raises

keystoneauth1.exceptions.http.HttpError – An error from an invalid HTTP response.

Returns

Valid EndpointData or None if not available.

Return type

keystoneauth1.discover.EndpointData or None

get_headers(session, **kwargs)

Fetch authentication headers for message.

This is a more generalized replacement of the older get_token to allow plugins to specify different or additional authentication headers to the OpenStack standard ‘X-Auth-Token’ header.

How the authentication headers are obtained is up to the plugin. If the headers are still valid they may be re-used, retrieved from cache or the plugin may invoke an authentication request against a server.

The default implementation of get_headers calls the get_token method to enable older style plugins to continue functioning unchanged. Subclasses should feel free to completely override this function to provide the headers that they want.

There are no required kwargs. They are passed directly to the auth plugin and they are implementation specific.

Returning None will indicate that no token was able to be retrieved and that authorization was a failure. Adding no authentication data can be achieved by returning an empty dictionary.

Parameters

session (keystoneauth1.session.Session) – The session object that the auth_plugin belongs to.

Returns

Headers that are set to authenticate a message or None for failure. Note that when checking this value that the empty dict is a valid, non-failure response.

Return type

dict

get_project_id(session, **kwargs)

Return the project id that we are authenticated to.

Wherever possible the project id should be inferred from the token however there are certain URLs and other places that require access to the currently authenticated project id.

Parameters

session (keystoneauth1.session.Session) – A session object so the plugin can make HTTP calls.

Returns

A project identifier or None if one is not available.

Return type

str

get_sp_auth_url(*args, **kwargs)

Return auth_url from the Service Provider object.

This url is used for obtaining unscoped federated token from remote cloud.

Parameters

sp_id (string) – ID of the Service Provider to be queried.

Returns

A Service Provider auth_url or None if one is not available.

Return type

str

get_sp_url(*args, **kwargs)

Return sp_url from the Service Provider object.

This url is used for passing SAML2 assertion to the remote cloud.

Parameters

sp_id (str) – ID of the Service Provider to be queried.

Returns

A Service Provider sp_url or None if one is not available.

Return type

str

get_token(session, **kwargs)

Return a valid auth token.

If a valid token is not present then a new one will be fetched.

Parameters

session (keystoneauth1.session.Session) – A session object that can be used for communication.

Raises

keystoneauth1.exceptions.http.HttpError – An error from an invalid HTTP response.

Returns

A valid token.

Return type

string

get_user_id(session, **kwargs)

Return a unique user identifier of the plugin.

Wherever possible the user id should be inferred from the token however there are certain URLs and other places that require access to the currently authenticated user id.

Parameters

session (keystoneauth1.session.Session) – A session object so the plugin can make HTTP calls.

Returns

A user identifier or None if one is not available.

Return type

str

invalidate()

Invalidate the current authentication data.

This should result in fetching a new token on next call.

A plugin may be invalidated if an Unauthorized HTTP response is returned to indicate that the token may have been revoked or is otherwise now invalid.

Returns

True if there was something that the plugin did to invalidate. This means that it makes sense to try again. If nothing happens returns False to indicate give up.

Return type

bool

set_auth_state(data)

Install existing authentication state for a plugin.

Take the output of get_auth_state and install that authentication state into the current authentication plugin.

class swiftclient.authv1.ServiceCatalogV1(auth_url, storage_url, account)

Bases: object

property catalog
property storage_url
url_for(**kwargs)

swiftclient.client

OpenStack Swift client library used internally

class swiftclient.client.Connection(authurl=None, user=None, key=None, retries=5, preauthurl=None, preauthtoken=None, snet=False, starting_backoff=1, max_backoff=64, tenant_name=None, os_options=None, auth_version='1', cacert=None, insecure=False, cert=None, cert_key=None, ssl_compression=True, retry_on_ratelimit=False, timeout=None, session=None, force_auth_retry=False)

Bases: object

Convenience class to make requests that will also retry the request

Requests will have an X-Auth-Token header whose value is either the preauthtoken or a token obtained from the auth service using the user credentials provided as args to the constructor. If os_options includes a service_username then requests will also have an X-Service-Token header whose value is a token obtained from the auth service using the service credentials. In this case the request url will be set to the storage_url obtained from the auth service for the service user, unless this is overridden by a preauthurl.

Parameters

  • authurl – authentication URL

  • user – user name to authenticate as

  • key – key/password to authenticate with

  • retries – Number of times to retry the request before failing

  • preauthurl – storage URL (if you have already authenticated)

  • preauthtoken – authentication token (if you have already authenticated) note authurl/user/key/tenant_name are not required when specifying preauthtoken

  • snet – use SERVICENET internal network default is False

  • starting_backoff – initial delay between retries (seconds)

  • max_backoff – maximum delay between retries (seconds)

  • auth_version – OpenStack auth version, default is 1.0

  • tenant_name – The tenant/account name, required when connecting to an auth 2.0 system.

  • os_options – The OpenStack options which can have tenant_id, auth_token, service_type, endpoint_type, tenant_name, object_storage_url, region_name, service_username, service_project_name, service_key

  • insecure – Allow to access servers without checking SSL certs. The server’s certificate will not be verified.

  • cert – Client certificate file to connect on SSL server requiring SSL client certificate.

  • cert_key – Client certificate private key file.

  • ssl_compression – Whether to enable compression at the SSL layer. If set to ‘False’ and the pyOpenSSL library is present an attempt to disable SSL compression will be made. This may provide a performance increase for https upload/download operations.

  • retry_on_ratelimit – by default, a ratelimited connection will raise an exception to the caller. Setting this parameter to True will cause a retry after a backoff.

  • timeout – The connect timeout for the HTTP connection.

  • session – A keystoneauth session object.

  • force_auth_retry – reset auth info even if client got unexpected error except 401 Unauthorized.

close()
copy_object(container, obj, destination=None, headers=None, fresh_metadata=None, response_dict=None)

Wrapper for copy_object()

delete_container(container, response_dict=None, query_string=None, headers={})

Wrapper for delete_container()

delete_object(container, obj, query_string=None, response_dict=None, headers=None)

Wrapper for delete_object()

get_account(marker=None, limit=None, prefix=None, end_marker=None, full_listing=False, headers=None, delimiter=None)

Wrapper for get_account()

get_auth()
get_capabilities(url=None)
get_container(container, marker=None, limit=None, prefix=None, delimiter=None, end_marker=None, path=None, full_listing=False, headers=None, query_string=None)

Wrapper for get_container()

get_object(container, obj, resp_chunk_size=None, query_string=None, response_dict=None, headers=None)

Wrapper for get_object()

get_service_auth()
head_account(headers=None)

Wrapper for head_account()

head_container(container, headers=None)

Wrapper for head_container()

head_object(container, obj, headers=None, query_string=None)

Wrapper for head_object()

http_connection(url=None)
post_account(headers, response_dict=None, query_string=None, data=None)

Wrapper for post_account()

post_container(container, headers, response_dict=None)

Wrapper for post_container()

post_object(container, obj, headers, response_dict=None)

Wrapper for post_object()

put_container(container, headers=None, response_dict=None, query_string=None)

Wrapper for put_container()

put_object(container, obj, contents, content_length=None, etag=None, chunk_size=None, content_type=None, headers=None, query_string=None, response_dict=None)

Wrapper for put_object()

class swiftclient.client.HTTPConnection(url, proxy=None, cacert=None, insecure=False, cert=None, cert_key=None, ssl_compression=False, default_user_agent=None, timeout=None)

Bases: object

Make an HTTPConnection or HTTPSConnection

Parameters

  • url – url to connect to

  • proxy – proxy to connect through, if any; None by default; str of the format ‘http://127.0.0.1:8888’ to set one

  • cacert – A CA bundle file to use in verifying a TLS server certificate.

  • insecure – Allow to access servers without checking SSL certs. The server’s certificate will not be verified.

  • cert – Client certificate file to connect on SSL server requiring SSL client certificate.

  • cert_key – Client certificate private key file.

  • ssl_compression – SSL compression should be disabled by default and this setting is not usable as of now. The parameter is kept for backward compatibility.

  • default_user_agent – Set the User-Agent header on every request. If set to None (default), the user agent will be “python-swiftclient-<version>”. This may be overridden on a per-request basis by explicitly setting the user-agent header on a call to request().

  • timeout – socket read timeout value, passed directly to the requests library.

Raises

ClientException – Unable to handle protocol scheme

close()
getresponse()

Adapt requests response to httplib interface

putrequest(full_path, data=None, headers=None, files=None)

Use python-requests files upload

Parameters

  • data – Use data generator for chunked-transfer

  • files – Use files for default transfer

request(method, full_path, data=None, headers=None, files=None)

Encode url and header, then call requests.request

swiftclient.client.LOGGER_SENSITIVE_HEADERS = ['x-auth-token', 'x-auth-key', 'x-service-token', 'x-storage-token', 'x-account-meta-temp-url-key', 'x-account-meta-temp-url-key-2', 'x-container-meta-temp-url-key', 'x-container-meta-temp-url-key-2', 'set-cookie']

A list of sensitive headers to redact in logs. Note that when extending this list, the header names must be added in all lower case.

swiftclient.client.copy_object(url, token, container, name, destination=None, headers=None, fresh_metadata=None, http_conn=None, response_dict=None, service_token=None)

Copy object

Parameters

  • url – storage URL

  • token – auth token; if None, no token will be sent

  • container – container name that the source object is in

  • name – source object name

  • destination – The container and object name of the destination object in the form of /container/object; if None, the copy will use the source as the destination.

  • headers – additional headers to include in the request

  • fresh_metadata – Enables object creation that omits existing user metadata, default None

  • http_conn – HTTP connection object (If None, it will create the conn object)

  • response_dict – an optional dictionary into which to place the response - status, reason and headers

  • service_token – service auth token

Raises

ClientException – HTTP COPY request failed

swiftclient.client.delete_container(url, token, container, http_conn=None, response_dict=None, service_token=None, query_string=None, headers=None)

Delete a container

Parameters

  • url – storage URL

  • token – auth token

  • container – container name to delete

  • http_conn – a tuple of (parsed url, HTTPConnection object), (If None, it will create the conn object)

  • response_dict – an optional dictionary into which to place the response - status, reason and headers

  • service_token – service auth token

  • query_string – if set will be appended with ‘?’ to generated path

  • headers – additional headers to include in the request

Raises

ClientException – HTTP DELETE request failed

swiftclient.client.delete_object(url, token=None, container=None, name=None, http_conn=None, headers=None, proxy=None, query_string=None, response_dict=None, service_token=None)

Delete object

Parameters

  • url – storage URL

  • token – auth token; if None, no token will be sent

  • container – container name that the object is in; if None, the container name is expected to be part of the url

  • name – object name to delete; if None, the object name is expected to be part of the url

  • http_conn – a tuple of (parsed url, HTTPConnection object), (If None, it will create the conn object)

  • headers – additional headers to include in the request

  • proxy – proxy to connect through, if any; None by default; str of the format ‘http://127.0.0.1:8888’ to set one

  • query_string – if set will be appended with ‘?’ to generated path

  • response_dict – an optional dictionary into which to place the response - status, reason and headers

  • service_token – service auth token

Raises

ClientException – HTTP DELETE request failed

swiftclient.client.encode_meta_headers(headers)

Only encode metadata headers keys

swiftclient.client.encode_utf8(value)
swiftclient.client.get_account(url, token, marker=None, limit=None, prefix=None, end_marker=None, http_conn=None, full_listing=False, service_token=None, headers=None, delimiter=None)

Get a listing of containers for the account.

Parameters

  • url – storage URL

  • token – auth token

  • marker – marker query

  • limit – limit query

  • prefix – prefix query

  • end_marker – end_marker query

  • http_conn – a tuple of (parsed url, HTTPConnection object), (If None, it will create the conn object)

  • full_listing – if True, return a full listing, else returns a max of 10000 listings

  • service_token – service auth token

  • headers – additional headers to include in the request

  • delimiter – delimiter query

Returns

a tuple of (response headers, a list of containers) The response headers will be a dict and all header names will be lowercase.

Raises

ClientException – HTTP GET request failed

swiftclient.client.get_auth(auth_url, user, key, **kwargs)

Get authentication/authorization credentials.

Parameters

  • auth_version – the api version of the supplied auth params

  • os_options – a dict, the openstack identity service options

Returns

a tuple, (storage_url, token)

N.B. if the optional os_options parameter includes a non-empty ‘object_storage_url’ key it will override the default storage url returned by the auth service.

The snet parameter is used for Rackspace’s ServiceNet internal network implementation. In this function, it simply adds snet- to the beginning of the host name for the returned storage URL. With Rackspace Cloud Files, use of this network path causes no bandwidth charges but requires the client to be running on Rackspace’s ServiceNet network.

swiftclient.client.get_auth_1_0(url, user, key, snet, **kwargs)
swiftclient.client.get_auth_keystone(auth_url, user, key, os_options, **kwargs)

Authenticate against a keystone server.

We are using the keystoneclient library for authentication.

swiftclient.client.get_capabilities(http_conn)

Get cluster capability infos.

Parameters

http_conn – a tuple of (parsed url, HTTPConnection object)

Returns

a dict containing the cluster capabilities

Raises

ClientException – HTTP Capabilities GET failed

swiftclient.client.get_container(url, token, container, marker=None, limit=None, prefix=None, delimiter=None, end_marker=None, path=None, http_conn=None, full_listing=False, service_token=None, headers=None, query_string=None)

Get a listing of objects for the container.

Parameters

  • url – storage URL

  • token – auth token

  • container – container name to get a listing for

  • marker – marker query

  • limit – limit query

  • prefix – prefix query

  • delimiter – string to delimit the queries on

  • end_marker – marker query

  • path – path query (equivalent: “delimiter=/” and “prefix=path/”)

  • http_conn – a tuple of (parsed url, HTTPConnection object), (If None, it will create the conn object)

  • full_listing – if True, return a full listing, else returns a max of 10000 listings

  • service_token – service auth token

  • headers – additional headers to include in the request

  • query_string – if set will be appended with ‘?’ to generated path

Returns

a tuple of (response headers, a list of objects) The response headers will be a dict and all header names will be lowercase.

Raises

ClientException – HTTP GET request failed

swiftclient.client.get_keystoneclient_2_0(auth_url, user, key, os_options, **kwargs)
swiftclient.client.get_object(url, token, container, name, http_conn=None, resp_chunk_size=None, query_string=None, response_dict=None, headers=None, service_token=None)

Get an object

Parameters

  • url – storage URL

  • token – auth token

  • container – container name that the object is in

  • name – object name to get

  • http_conn – a tuple of (parsed url, HTTPConnection object), (If None, it will create the conn object)

  • resp_chunk_size – if defined, chunk size of data to read. NOTE: If you specify a resp_chunk_size you must fully read the object’s contents before making another request.

  • query_string – if set will be appended with ‘?’ to generated path

  • response_dict – an optional dictionary into which to place the response - status, reason and headers

  • headers – an optional dictionary with additional headers to include in the request

  • service_token – service auth token

Returns

a tuple of (response headers, the object’s contents) The response headers will be a dict and all header names will be lowercase.

Raises

ClientException – HTTP GET request failed

swiftclient.client.head_account(url, token, http_conn=None, headers=None, service_token=None)

Get account stats.

Parameters

  • url – storage URL

  • token – auth token

  • http_conn – a tuple of (parsed url, HTTPConnection object), (If None, it will create the conn object)

  • headers – additional headers to include in the request

  • service_token – service auth token

Returns

a dict containing the response’s headers (all header names will be lowercase)

Raises

ClientException – HTTP HEAD request failed

swiftclient.client.head_container(url, token, container, http_conn=None, headers=None, service_token=None)

Get container stats.

Parameters

  • url – storage URL

  • token – auth token

  • container – container name to get stats for

  • http_conn – a tuple of (parsed url, HTTPConnection object), (If None, it will create the conn object)

  • headers – additional headers to include in the request

  • service_token – service auth token

Returns

a dict containing the response’s headers (all header names will be lowercase)

Raises

ClientException – HTTP HEAD request failed

swiftclient.client.head_object(url, token, container, name, http_conn=None, service_token=None, headers=None, query_string=None)

Get object info

Parameters

  • url – storage URL

  • token – auth token

  • container – container name that the object is in

  • name – object name to get info for

  • http_conn – a tuple of (parsed url, HTTPConnection object), (If None, it will create the conn object)

  • service_token – service auth token

  • headers – additional headers to include in the request

Returns

a dict containing the response’s headers (all header names will be lowercase)

Raises

ClientException – HTTP HEAD request failed

swiftclient.client.http_connection(*arg, **kwarg)
Returns

tuple of (parsed url, connection object)

swiftclient.client.http_log(args, kwargs, resp, body)
swiftclient.client.logger_settings = {'redact_sensitive_headers': True, 'reveal_sensitive_prefix': 16}

Default behaviour is to redact header values known to contain secrets, such as X-Auth-Key and X-Auth-Token. Up to the first 16 chars may be revealed.

To disable, set the value of redact_sensitive_headers to False.

When header redaction is enabled, reveal_sensitive_prefix configures the maximum length of any sensitive header data sent to the logs. If the header is less than twice this length, only int(len(value)/2) chars will be logged; if it is less than 15 chars long, even less will be logged.

swiftclient.client.parse_header_string(data)
swiftclient.client.post_account(url, token, headers, http_conn=None, response_dict=None, service_token=None, query_string=None, data=None)

Update an account’s metadata.

Parameters

  • url – storage URL

  • token – auth token

  • headers – additional headers to include in the request

  • http_conn – a tuple of (parsed url, HTTPConnection object), (If None, it will create the conn object)

  • response_dict – an optional dictionary into which to place the response - status, reason and headers

  • service_token – service auth token

  • query_string – if set will be appended with ‘?’ to generated path

  • data – an optional message body for the request

Raises

ClientException – HTTP POST request failed

Returns

resp_headers, body

swiftclient.client.post_container(url, token, container, headers, http_conn=None, response_dict=None, service_token=None)

Update a container’s metadata.

Parameters

  • url – storage URL

  • token – auth token

  • container – container name to update

  • headers – additional headers to include in the request

  • http_conn – a tuple of (parsed url, HTTPConnection object), (If None, it will create the conn object)

  • response_dict – an optional dictionary into which to place the response - status, reason and headers

  • service_token – service auth token

Raises

ClientException – HTTP POST request failed

swiftclient.client.post_object(url, token, container, name, headers, http_conn=None, response_dict=None, service_token=None)

Update object metadata

Parameters

  • url – storage URL

  • token – auth token

  • container – container name that the object is in

  • name – name of the object to update

  • headers – additional headers to include in the request

  • http_conn – a tuple of (parsed url, HTTPConnection object), (If None, it will create the conn object)

  • response_dict – an optional dictionary into which to place the response - status, reason and headers

  • service_token – service auth token

Raises

ClientException – HTTP POST request failed

swiftclient.client.put_container(url, token, container, headers=None, http_conn=None, response_dict=None, service_token=None, query_string=None)

Create a container

Parameters

  • url – storage URL

  • token – auth token

  • container – container name to create

  • headers – additional headers to include in the request

  • http_conn – a tuple of (parsed url, HTTPConnection object), (If None, it will create the conn object)

  • response_dict – an optional dictionary into which to place the response - status, reason and headers

  • service_token – service auth token

  • query_string – if set will be appended with ‘?’ to generated path

Raises

ClientException – HTTP PUT request failed

swiftclient.client.put_object(url, token=None, container=None, name=None, contents=None, content_length=None, etag=None, chunk_size=None, content_type=None, headers=None, http_conn=None, proxy=None, query_string=None, response_dict=None, service_token=None)

Put an object

Parameters

  • url – storage URL

  • token – auth token; if None, no token will be sent

  • container – container name that the object is in; if None, the container name is expected to be part of the url

  • name – object name to put; if None, the object name is expected to be part of the url

  • contents – a string, a file-like object or an iterable to read object data from; if None, a zero-byte put will be done

  • content_length – value to send as content-length header; also limits the amount read from contents; if None, it will be computed via the contents or chunked transfer encoding will be used

  • etag – etag of contents; if None, no etag will be sent

  • chunk_size – chunk size of data to write; it defaults to 65536; used only if the contents object has a ‘read’ method, e.g. file-like objects, ignored otherwise

  • content_type – value to send as content-type header, overriding any value included in the headers param; if None and no value is found in the headers param, an empty string value will be sent

  • headers – additional headers to include in the request, if any

  • http_conn – a tuple of (parsed url, HTTPConnection object), (If None, it will create the conn object)

  • proxy – proxy to connect through, if any; None by default; str of the format ‘http://127.0.0.1:8888’ to set one

  • query_string – if set will be appended with ‘?’ to generated path

  • response_dict – an optional dictionary into which to place the response - status, reason and headers

  • service_token – service auth token

Returns

etag

Raises

ClientException – HTTP PUT request failed

swiftclient.client.quote(value, safe='/')

Patched version of urllib.quote that encodes utf8 strings before quoting. On Python 3, call directly urllib.parse.quote().

swiftclient.client.resp_header_dict(resp)
swiftclient.client.safe_value(name, value)

Only show up to logger_settings[‘reveal_sensitive_prefix’] characters from a sensitive header.

Parameters

  • name – Header name

  • value – Header value

Returns

Safe header value

swiftclient.client.scrub_headers(headers)

Redact header values that can contain sensitive information that should not be logged.

Parameters

headers – Either a dict or an iterable of two-element tuples

Returns

Safe dictionary of headers with sensitive information removed

swiftclient.client.store_response(resp, response_dict)

store information about an operation into a dict

Parameters

  • resp – an http response object containing the response headers

  • response_dict – a dict into which are placed the status, reason and a dict of lower-cased headers

swiftclient.service

class swiftclient.service.ResultsIterator(futures)

Bases: object

class swiftclient.service.SwiftCopyObject(object_name, options=None)

Bases: object

Class for specifying an object copy, allowing the destination/headers/metadata/fresh_metadata to be specified separately for each individual object. destination and fresh_metadata should be set in options

exception swiftclient.service.SwiftError(value, container=None, obj=None, segment=None, exc=None)

Bases: Exception

class swiftclient.service.SwiftPostObject(object_name, options=None)

Bases: object

Class for specifying an object post, allowing the headers/metadata to be specified separately for each individual object.

class swiftclient.service.SwiftService(options=None)

Bases: object

Service for performing swift operations

capabilities(url=None, refresh_cache=False)

List the cluster capabilities.

Parameters

url – Proxy URL of the cluster to retrieve capabilities.

Returns

A dictionary containing the capabilities of the cluster.

Raises

ClientException –

copy(container, objects, options=None)

Copy operations on a list of objects in a container. Destination containers will be created.

Parameters

  • container – The container from which to copy the objects.

  • objects –

    A list of object names (strings) or SwiftCopyObject instances containing an object name and an options dict (can be None) to override the options for that individual copy operation:

    [
    'object_name',
    SwiftCopyObject(
        'object_name',
         options={
            'destination': '/container/object',
            'fresh_metadata': False,
            ...
            }),
    ...
]

    The options dict is described below.

  • options –

    A dictionary containing options to override the global options specified during the service object creation. These options are applied to all copy operations performed by this call, unless overridden on a per object basis. The options “destination” and “fresh_metadata” do not need to be set, in this case objects will be copied onto themselves and metadata will not be refreshed. The option “destination” can also be specified in the format ‘/container’, in which case objects without an explicit destination will be copied to the destination /container/original_object_name. Combinations of multiple objects and a destination in the format ‘/container/object’ is invalid. Possible options are given below:

    {
    'meta': [],
    'header': [],
    'destination': '/container/object',
    'fresh_metadata': False,
}

Returns

A generator returning the results of copying the given list of objects.

Raises

SwiftError –

delete(container=None, objects=None, options=None)

Delete operations on an account, optional container and optional list of objects.

Parameters

  • container – The container to delete or delete from.

  • objects – The list of objects to delete.

  • options –

    A dictionary containing options to override the global options specified during the service object creation:

    {
    'yes_all': False,
    'leave_segments': False,
    'prefix': None,
    'header': [],
}

Returns

A generator for returning the results of the delete operations. Each result yielded from the generator is either a ‘delete_container’, ‘delete_object’, ‘delete_segment’, or ‘bulk_delete’ dictionary containing the results of an individual delete operation.

Raises
download(container=None, objects=None, options=None)

Download operations on an account, optional container and optional list of objects.

Parameters

  • container – The container to download from.

  • objects – A list of object names to download (a list of strings).

  • options –

    A dictionary containing options to override the global options specified during the service object creation:

    {
    'yes_all': False,
    'marker': '',
    'prefix': None,
    'no_download': False,
    'header': [],
    'skip_identical': False,
    'out_directory': None,
    'checksum': True,
    'out_file': None,
    'remove_prefix': False,
    'shuffle' : False
}

Returns

A generator for returning the results of the download operations. Each result yielded from the generator is a ‘download_object’ dictionary containing the results of an individual file download.

Raises
list(container=None, options=None)

List operations on an account, container.

Parameters

  • container – The container to make the list operation against.

  • options –

    A dictionary containing options to override the global options specified during the service object creation:

    {
    'long': False,
    'prefix': None,
    'delimiter': None,
    'header': []
}

Returns

A generator for returning the results of the list operation on an account or container. Each result yielded from the generator is either a ‘list_account_part’ or ‘list_container_part’, containing part of the listing.

post(container=None, objects=None, options=None)

Post operations on an account, container or list of objects

Parameters

  • container – The container to make the post operation against.

  • objects –

    A list of object names (strings) or SwiftPostObject instances containing an object name, and an options dict (can be None) to override the options for that individual post operation:

    [
    'object_name',
    SwiftPostObject('object_name', options={...}),
    ...
]

    The options dict is described below.

  • options –

    A dictionary containing options to override the global options specified during the service object creation. These options are applied to all post operations performed by this call, unless overridden on a per object basis. Possible options are given below:

    {
    'meta': [],
    'header': [],
    'read_acl': None,   # For containers only
    'write_acl': None,  # For containers only
    'sync_to': None,    # For containers only
    'sync_key': None    # For containers only
}

Returns

Either a single result dictionary in the case of a post to a container/account, or an iterator for returning the results of posts to a list of objects.

Raises

SwiftError –

stat(container=None, objects=None, options=None)

Get account stats, container stats or information about a list of objects in a container.

Parameters

  • container – The container to query.

  • objects – A list of object paths about which to return information (a list of strings).

  • options –

    A dictionary containing options to override the global options specified during the service object creation. These options are applied to all stat operations performed by this call:

    {
    'human': False,
    'header': []
}

Returns

Either a single dictionary containing stats about an account or container, or an iterator for returning the results of the stat operations on a list of objects.

Raises

SwiftError –

upload(container, objects, options=None)

Upload a list of objects to a given container.

Parameters

  • container – The container (or pseudo-folder path) to put the uploads into.

  • objects –

    A list of file/directory names (strings) or SwiftUploadObject instances containing a source for the created object, an object name, and an options dict (can be None) to override the options for that individual upload operation:

    [
    '/path/to/file',
    SwiftUploadObject('/path', object_name='obj1'),
    ...
]

    The options dict is as described below.

    The SwiftUploadObject source may be one of:

    • A file-like object (with a read method)

    • A string containing the path to a local file or directory

    • None, to indicate that we want an empty object

  • options –

    A dictionary containing options to override the global options specified during the service object creation. These options are applied to all upload operations performed by this call, unless overridden on a per object basis. Possible options are given below:

    {
    'meta': [],
    'header': [],
    'segment_size': None,
    'use_slo': False,
    'segment_container': None,
    'leave_segments': False,
    'changed': None,
    'skip_identical': False,
    'fail_fast': False,
    'dir_marker': False  # Only for None sources
}

Returns

A generator for returning the results of the uploads.

Raises
class swiftclient.service.SwiftUploadObject(source, object_name=None, options=None)

Bases: object

Class for specifying an object upload, allowing the object source, name and options to be specified separately for each individual object.

swiftclient.service.get_conn(options)

Return a connection building it from the options.

swiftclient.service.get_from_queue(q, timeout=864000)
swiftclient.service.get_future_result(f, timeout=86400)
swiftclient.service.interruptable_as_completed(fs, timeout=86400)
swiftclient.service.mkdirs(path)
swiftclient.service.process_options(options)
swiftclient.service.split_headers(options, prefix='')

Splits ‘Key: Value’ strings and returns them as a dictionary.

Parameters

  • options – Must be one of: * an iterable of ‘Key: Value’ strings * an iterable of (‘Key’, ‘Value’) pairs * a dict of {‘Key’: ‘Value’} pairs

  • prefix – String to prepend to all of the keys in the dictionary. reporting.

swiftclient.exceptions

exception swiftclient.exceptions.ClientException(msg, http_scheme='', http_host='', http_port='', http_path='', http_query='', http_status=None, http_reason='', http_device='', http_response_content='', http_response_headers=None)

Bases: Exception

classmethod from_response(resp, msg=None, body=None)

swiftclient.multithreading

class swiftclient.multithreading.ConnectionThreadPoolExecutor(create_connection, max_workers)

Bases: concurrent.futures.thread.ThreadPoolExecutor

A wrapper class to maintain a pool of connections alongside the thread pool. We start by creating a priority queue of connections, and each job submitted takes one of those connections (initialising if necessary) and passes it as the first arg to the executed function.

At the end of execution that connection is returned to the queue.

By using a PriorityQueue we avoid creating more connections than required. We will only create as many connections as are required concurrently.

Initializes a new ThreadPoolExecutor instance.

Parameters

  • create_connection – callable to use to create new connections

  • max_workers – the maximum number of threads that can be used

submit(fn, *args, **kwargs)

Schedules the callable, fn, to be executed

Parameters

  • fn – the callable to be invoked

  • args – the positional arguments for the callable

  • kwargs – the keyword arguments for the callable

Returns

a Future object representing the execution of the callable

class swiftclient.multithreading.MultiThreadingManager(create_connection, segment_threads=10, object_dd_threads=10, object_uu_threads=10, container_threads=10)

Bases: object

One object to manage context for multi-threading. This should make bin/swift less error-prone and allow us to test this code.

Parameters

  • segment_threads – The number of threads allocated to segment uploads

  • object_dd_threads – The number of threads allocated to object download/delete jobs

  • object_uu_threads – The number of threads allocated to object upload/update based jobs

  • container_threads – The number of threads allocated to container/account level jobs

class swiftclient.multithreading.OutputManager(print_stream=None, error_stream=None)

Bases: object

One object to manage and provide helper functions for output.

This object is a context manager and returns itself into the context. When entering the context, two printing threads are created (see below) and they are waited on and cleaned up when exiting the context.

Also, thread-safe printing to two streams is provided. The print_msg() method will print to the supplied print_stream (defaults to sys.stdout) and the error() method will print to the supplied error_stream (defaults to sys.stderr). Both of these printing methods will format the given string with any supplied *args (a la printf). On Python 2, Unicode messages are encoded to utf8.

The attribute self.error_count is incremented once per error message printed, so an application can tell if any worker threads encountered exceptions or otherwise called error() on this instance. The swift command-line tool uses this to exit non-zero if any error strings were printed.

Parameters

  • print_stream – The stream to which print_msg() sends formatted messages.

  • error_stream – The stream to which error() sends formatted messages.

On Python 2, Unicode messages are encoded to utf8.

DEFAULT_OFFSET = 14
error(msg, *fmt_args)
get_error_count()
print_items(items, offset=14, skip_missing=False)
print_msg(msg, *fmt_args)
print_raw(data)
warning(msg, *fmt_args)

swiftclient.utils

Miscellaneous utility functions for use with Swift.

class swiftclient.utils.LengthWrapper(readable, length, md5=False)

Bases: object

Wrap a filelike object with a maximum length.

Fix for https://github.com/kennethreitz/requests/issues/1648. It is recommended to use this class only on files opened in binary mode.

Parameters

  • readable – The filelike object to read from.

  • length – The maximum amount of content that can be read from the filelike object before it is simulated to be empty.

  • md5 – Flag to enable calculating the MD5 of the content as it is read.

get_md5sum()
read(size=-1)
property reset
class swiftclient.utils.NoopMD5(*a, **kw)

Bases: object

hexdigest(*a, **kw)
update(*a, **kw)
class swiftclient.utils.ReadableToIterable(content, chunk_size=65536, md5=False)

Bases: object

Wrap a filelike object and act as an iterator.

It is recommended to use this class only on files opened in binary mode. Due to the Unicode changes in Python 3, files are now opened using an encoding not suitable for use with the md5 class and because of this hit the exception on every call to next. This could cause problems, especially with large files and small chunk sizes.

Parameters

  • content – The filelike object that is yielded from.

  • chunk_size – The max size of each yielded item.

  • md5 – Flag to enable calculating the MD5 of the content as it is yielded.

get_md5sum()
next()
swiftclient.utils.config_true_value(value)

Returns True if the value is either True or a string in TRUE_VALUES. Returns False otherwise. This function comes from swift.common.utils.config_true_value()

swiftclient.utils.generate_temp_url(path, seconds, key, method, absolute=False, prefix=False, iso8601=False, ip_range=None)

Generates a temporary URL that gives unauthenticated access to the Swift object.

Parameters

  • path – The full path to the Swift object or prefix if a prefix-based temporary URL should be generated. Example: /v1/AUTH_account/c/o or /v1/AUTH_account/c/prefix.

  • seconds – time in seconds or ISO 8601 timestamp. If absolute is False and this is the string representation of an integer, then this specifies the amount of time in seconds for which the temporary URL will be valid. If absolute is True then this specifies an absolute time at which the temporary URL will expire.

  • key – The secret temporary URL key set on the Swift cluster. To set a key, run ‘swift post -m “Temp-URL-Key: <substitute tempurl key here>”’

  • method – A HTTP method, typically either GET or PUT, to allow for this temporary URL.

  • absolute – if True then the seconds parameter is interpreted as a Unix timestamp, if seconds represents an integer.

  • prefix – if True then a prefix-based temporary URL will be generated.

  • iso8601 – if True, a URL containing an ISO 8601 UTC timestamp instead of a UNIX timestamp will be created.

  • ip_range – if a valid ip range, restricts the temporary URL to the range of ips.

Raises

ValueError – if timestamp or path is not in valid format.

Returns

the path portion of a temporary URL

swiftclient.utils.get_body(headers, body)
swiftclient.utils.iter_wrapper(iterable)
swiftclient.utils.n_at_a_time(seq, n)
swiftclient.utils.n_groups(seq, n)
swiftclient.utils.normalize_manifest_path(path)
swiftclient.utils.parse_api_response(headers, body)
swiftclient.utils.prt_bytes(num_bytes, human_flag)

convert a number > 1024 to printable format, either in 4 char -h format as with ls -lh or return as 12 char right justified string

swiftclient.utils.report_traceback()

Reports a timestamp and full traceback for a given exception.

Returns

Full traceback and timestamp.

swiftclient.utils.split_request_headers(options, prefix='')