Proxy

Proxy Controllers

Base

class swift.proxy.controllers.base.ByteCountEnforcer(file_like, nbytes)

Bases: object

Enforces that successive calls to file_like.read() give at least <nbytes> bytes before exhaustion.

If file_like fails to do so, ShortReadError is raised.

If more than <nbytes> bytes are read, we don’t care.

read(amt=None)

class swift.proxy.controllers.base.Controller(app)

Bases: object

Base WSGI controller class for the proxy

GET(req)

Handler for HTTP GET requests.

Parameters:

req – The client request

Returns:

the response to the client

GETorHEAD_base(req, server_type, node_iter, partition, path, concurrency=1, policy=None)

Base handler for HTTP GET or HEAD requests.

Parameters:

• req – swob.Request object

• server_type – server type used in logging

• node_iter – an iterator to obtain nodes from

• partition – partition

• path – path for the request

• concurrency – number of requests to run concurrently

• policy – the policy instance, or None if Account or Container

Returns:

swob.Response object

HEAD(req)

Handler for HTTP HEAD requests.

Parameters:

req – The client request

Returns:

the response to the client

OPTIONS(req)

Base handler for OPTIONS requests

Parameters:

req – swob.Request object

Returns:

swob.Response object

account_info(account, req)

Get account information, and also verify that the account exists.

Parameters:

• account – native str name of the account to get the info for

• req – caller’s HTTP request context object

Returns:

tuple of (account partition, account nodes, container_count) or (None, None, None) if it does not exist

property allowed_methods

autocreate_account(req, account)

Autocreate an account

Parameters:

• req – request leading to this autocreate

• account – the unquoted account name

best_response(req, responses, etag=None, headers=False, overrides=None, quorum_size=None)

Given a list of responses from several servers, choose the best to return to the API.

Parameters:

• req – swob.Request object

• responses – list of backend responses

• etag – etag

• headers – if True then the backend headers are copied to the returned response.

• overrides – overrides to apply when lacking quorum

• quorum_size – quorum size to use

Returns:

swob.Response object with the correct status, body, etc. set

container_info(account, container, req)

Get container information and thusly verify container existence. This will also verify account existence.

Parameters:

• account – native-str account name for the container

• container – native-str container name to look up

• req – caller’s HTTP request context object

Returns:

dict containing at least container partition (‘partition’), container nodes (‘containers’), container read acl (‘read_acl’), container write acl (‘write_acl’), and container sync key (‘sync_key’). Values are set to None if the container does not exist.

generate_request_headers(orig_req=None, additional=None, transfer=False)

Create a dict of headers to be used in backend requests

Parameters:

• orig_req – the original request sent by the client to the proxy

• additional – additional headers to send to the backend

• transfer – If True, transfer headers from original client request

Returns:

a dictionary of headers

get_name_length_limit()

have_quorum(statuses, node_count, quorum=None)

Given a list of statuses from several requests, determine if a quorum response can already be decided.

Parameters:

• statuses – list of statuses returned

• node_count – number of nodes being queried (basically ring count)

• quorum – number of statuses required for quorum

Returns:

True or False, depending on if quorum is established

is_origin_allowed(cors_info, origin)

Is the given Origin allowed to make requests to this resource

Parameters:

• cors_info – the resource’s CORS related metadata headers

• origin – the origin making the request

Returns:

True or False

property logger

make_requests(req, ring, part, method, path, headers, query_string='', overrides=None, node_count=None, node_iterator=None, body=None)

Sends an HTTP request to multiple nodes and aggregates the results. It attempts the primary nodes concurrently, then iterates over the handoff nodes as needed.

Parameters:

• req – a request sent by the client

• ring – the ring used for finding backend servers

• part – the partition number

• method – the method to send to the backend

• path – the path to send to the backend (full path ends up being /<$device>/<$part>/<$path>)

• headers – a list of dicts, where each dict represents one backend request that should be made.

• query_string – optional query string to send to the backend

• overrides – optional return status override map used to override the returned status of a request.

• node_count – optional number of nodes to send request to.

• node_iterator – optional node iterator.

• body – byte string to use as the request body. Try to keep it small.

Returns:

a swob.Response object

pass_through_headers = []

property private_methods

server_type = 'Base'

transfer_headers(src_headers, dst_headers)

Transfer legal headers from an original client request to dictionary that will be used as headers by the backend request

Parameters:

• src_headers – A dictionary of the original client request headers

• dst_headers – A dictionary of the backend request headers

class swift.proxy.controllers.base.GetOrHeadHandler(app, req, server_type, node_iter, partition, path, backend_headers, concurrency=1, policy=None, logger=None)

Bases: GetterBase

Handles GET requests to backend servers.

Parameters:

• app – a proxy app.

• req – an instance of swob.Request.

• server_type – server type used in logging

• node_iter – an iterator yielding nodes.

• partition – partition.

• path – path for the request.

• backend_headers – a dict of headers to be sent with backend requests.

• concurrency – number of requests to run concurrently.

• policy – the policy instance, or None if Account or Container.

• logger – a logger instance.

get_working_response()

property headers

property last_status

class swift.proxy.controllers.base.GetterBase(app, req, node_iter, partition, policy, path, backend_headers, node_timeout, resource_type, logger=None)

Bases: object

This base class provides helper methods for handling GET requests to backend servers.

Parameters:

• app – a proxy app.

• req – an instance of swob.Request.

• node_iter – an iterator yielding nodes.

• partition – partition.

• policy – the policy instance, or None if Account or Container.

• path – path for the request.

• backend_headers – a dict of headers to be sent with backend requests.

• node_timeout – the timeout value for backend requests.

• resource_type – a string description of the type of resource being accessed; resource type is used in logs and isn’t necessarily the server type.

• logger – a logger instance.

fast_forward(num_bytes)

Will skip num_bytes into the current ranges.

Params num_bytes:

the number of bytes that have already been read on this request. This will change the Range header so that the next req will start where it left off.

Raises:

• HTTPRequestedRangeNotSatisfiable – if begin + num_bytes > end of range + 1

• RangeAlreadyComplete – if begin + num_bytes == end of range + 1

learn_size_from_content_range(start, end, length)

Sets our Range header’s first byterange to the value learned from the Content-Range header in the response; if we were given a fully-specified range (e.g. “bytes=123-456”), this is a no-op.

If we were given a half-specified range (e.g. “bytes=123-” or “bytes=-456”), then this changes the Range header to a semantically-equivalent one and it lets us resume on a proper boundary instead of just in the middle of a piece somewhere.

pop_range()

Remove the first byterange from our Range header.

This is used after a byterange has been completely sent to the client; this way, should we need to resume the download from another object server, we do not re-fetch byteranges that the client already has.

If we have no Range header, this is a no-op.

class swift.proxy.controllers.base.GetterSource(app, resp, node)

Bases: object

Encapsulates properties of a source from which a GET response is read.

Parameters:

• app – a proxy app.

• resp – an instance of HTTPResponse.

• node – a dict describing the node from which the response was returned.

app

close()

node

property parts_iter

resp

property timestamp

Provide the timestamp of the swift http response as a floating point value. Used as a sort key.

Returns:

an instance of utils.Timestamp

class swift.proxy.controllers.base.NodeIter(server_type, app, ring, partition, logger, request, node_iter=None, policy=None)

Bases: object

Yields nodes for a ring partition, skipping over error limited nodes and stopping at the configurable number of nodes. If a node yielded subsequently gets error limited, an extra node will be yielded to take its place.

Note that if you’re going to iterate over this concurrently from multiple greenthreads, you’ll want to use a swift.common.utils.GreenthreadSafeIterator to serialize access. Otherwise, you may get ValueErrors from concurrent access. (You also may not, depending on how logging is configured, the vagaries of socket IO and eventlet, and the phase of the moon.)

Parameters:

• server_type – one of ‘account’, ‘container’, or ‘object’

• app – a proxy app

• ring – ring to get yield nodes from

• partition – ring partition to yield nodes for

• logger – a logger instance

• request – yielded nodes will be annotated with use_replication based on the request headers.

• node_iter – optional iterable of nodes to try. Useful if you want to filter or reorder the nodes.

• policy – an instance of BaseStoragePolicy. This should be None for an account or container ring.

log_handoffs(handoffs)

Log handoff requests if handoff logging is enabled and the handoff was not expected.

We only log handoffs when we’ve pushed the handoff count further than we would normally have expected under normal circumstances, that is (request_node_count - num_primaries), when handoffs goes higher than that it means one of the primaries must have been skipped because of error limiting before we consumed all of our nodes_left.

next()

property primaries_left

set_node_provider(callback)

Install a callback function that will be used during a call to next() to get an alternate node instead of returning the next node from the iterator.

Parameters:

callback – A no argument function that should return a node dict or None.

class swift.proxy.controllers.base.ResponseCollection(iterable=(), /)

Bases: list

Provides helper methods for convenient access to lists of ResponseData attributes.

property bodies

property etags

property headers

property reasons

property statuses

class swift.proxy.controllers.base.ResponseData(status, reason='', headers=None, body=None)

Bases: object

Encapsulate response data.

classmethod from_http_response(http_response, body=None)

getheader(key)

swift.proxy.controllers.base.bytes_to_skip(record_size, range_start)

Assume an object is composed of N records, where the first N-1 are all the same size and the last is at most that large, but may be smaller.

When a range request is made, it might start with a partial record. This must be discarded, lest the consumer get bad data. This is particularly true of suffix-byte-range requests, e.g. “Range: bytes=-12345” where the size of the object is unknown at the time the request is made.

This function computes the number of bytes that must be discarded to ensure only whole records are yielded. Erasure-code decoding needs this.

This function could have been inlined, but it took enough tries to get right that some targeted unit tests were desirable, hence its extraction.

swift.proxy.controllers.base.clear_info_cache(env, account, container=None, shard=None)

Clear the cached info in both memcache and env

Parameters:

• env – the WSGI request environment

• account – the account name

• container – the container name if clearing info for containers, or None

• shard – the sharding state if clearing info for container shard ranges, or None

swift.proxy.controllers.base.close_swift_conn(src)

Force close the http connection to the backend.

Parameters:

src – the response from the backend

swift.proxy.controllers.base.cors_validation(func)

Decorator to check if the request is a CORS request and if so, if it’s valid.

Parameters:

func – function to check

swift.proxy.controllers.base.delay_denial(func)

Decorator to declare which methods should have any swift.authorize call delayed. This is so the method can load the Request object up with additional information that may be needed by the authorization system.

Parameters:

func – function for which authorization will be delayed

swift.proxy.controllers.base.get_account_info(env, app, swift_source=None)

Get the info structure for an account, based on env and app. This is useful to middlewares.

Note

This call bypasses auth. Success does not imply that the request has authorization to the account.

Raises:

ValueError – when path doesn’t contain an account

swift.proxy.controllers.base.get_cache_key(account, container=None, obj=None, shard=None)

Get the keys for both memcache and env[‘swift.infocache’] (cache_key) where info about accounts, containers, and objects is cached

Parameters:

• account – The name of the account

• container – The name of the container (or None if account)

• obj – The name of the object (or None if account or container)

• shard – Sharding state for the container query; typically ‘updating’ or ‘listing’ (Requires account and container; cannot use with obj)

Returns:

a (native) string cache_key

swift.proxy.controllers.base.get_container_info(env, app, swift_source=None, cache_only=False)

Get the info structure for a container, based on env and app. This is useful to middlewares.

Parameters:

• env – the environment used by the current request

• app – the application object

• swift_source – Used to mark the request as originating out of middleware. Will be logged in proxy logs.

• cache_only – If true, indicates that caller doesn’t want to HEAD the backend container when cache miss.

Returns:

the object info

Note

This call bypasses auth. Success does not imply that the request has authorization to the container.

swift.proxy.controllers.base.get_info(app, env, account, container=None, swift_source=None)

Get info about accounts or containers

Note: This call bypasses auth. Success does not imply that the

request has authorization to the info.

Parameters:

• app – the application object

• env – the environment used by the current request

• account – The unquoted name of the account

• container – The unquoted name of the container (or None if account)

• swift_source – swift source logged for any subrequests made while retrieving the account or container info

Returns:

information about the specified entity in a dictionary. See get_account_info and get_container_info for details on what’s in the dictionary.

swift.proxy.controllers.base.get_namespaces_from_cache(req, cache_key, skip_chance)

Get cached namespaces from infocache or memcache.

Parameters:

• req – a swift.common.swob.Request object.

• cache_key – the cache key for both infocache and memcache.

• skip_chance – the probability of skipping the memcache look-up.

Returns:

a tuple of (value, cache state). Value is an instance of swift.common.utils.NamespaceBoundList if a non-empty list is found in memcache. Otherwise value is None, for example if memcache look-up was skipped, or no value was found, or an empty list was found.

swift.proxy.controllers.base.get_object_info(env, app, path=None, swift_source=None)

Get the info structure for an object, based on env and app. This is useful to middlewares.

Note

This call bypasses auth. Success does not imply that the request has authorization to the object.

swift.proxy.controllers.base.headers_from_container_info(info)

Construct a HeaderKeyDict from a container info dict.

Parameters:

info – a dict of container metadata

Returns:

a HeaderKeyDict or None if info is None or any required headers could not be constructed

swift.proxy.controllers.base.headers_to_account_info(headers, status_int=200)

Construct a cacheable dict of account info based on response headers.

swift.proxy.controllers.base.headers_to_container_info(headers, status_int=200)

Construct a cacheable dict of container info based on response headers.

swift.proxy.controllers.base.headers_to_object_info(headers, status_int=200)

Construct a cacheable dict of object info based on response headers.

swift.proxy.controllers.base.is_good_source(status, server_type)

Indicates whether or not the request made to the backend found what it was looking for.

Parameters:

• resp – the response from the backend.

• server_type – the type of server: ‘Account’, ‘Container’ or ‘Object’.

Returns:

True if the response status code is acceptable, False if not.

swift.proxy.controllers.base.is_useful_response(resp, node)

swift.proxy.controllers.base.record_cache_op_metrics(logger, server_type, op_type, cache_state, resp=None)

Record a single cache operation into its corresponding metrics.

Parameters:

• logger – the metrics logger

• server_type – ‘account’ or ‘container’

• op_type – the name of the operation type, includes ‘shard_listing’, ‘shard_updating’, and etc.

• cache_state – the state of this cache operation. When it’s ‘infocache_hit’ or memcache ‘hit’, expect it succeeded and ‘resp’ will be None; for all other cases like memcache ‘miss’ or ‘skip’ which will make to backend, expect a valid ‘resp’.

• resp – the response from backend for all cases except cache hits.

swift.proxy.controllers.base.set_info_cache(env, account, container, resp)

Cache info in both memcache and env.

Parameters:

• env – the WSGI request environment

• account – the unquoted account name

• container – the unquoted container name or None

• resp – the response received or None if info cache should be cleared

Returns:

the info that was placed into the cache, or None if the request status was not in (404, 410, 2xx).

swift.proxy.controllers.base.set_namespaces_in_cache(req, cache_key, ns_bound_list, time)

Set a list of namespace bounds in infocache and memcache.

Parameters:

• req – a swift.common.swob.Request object.

• cache_key – the cache key for both infocache and memcache.

• ns_bound_list – a swift.common.utils.NamespaceBoundList.

• time – how long the namespaces should remain in memcache.

Returns:

the cache_state.

swift.proxy.controllers.base.set_object_info_cache(app, env, account, container, obj, resp)

Cache object info in the WSGI environment, but not in memcache. Caching in memcache would lead to cache pressure and mass evictions due to the large number of objects in a typical Swift cluster. This is a per-request cache only.

Parameters:

• app – the application object

• env – the environment used by the current request

• account – the unquoted account name

• container – the unquoted container name

• obj – the unquoted object name

• resp – a GET or HEAD response received from an object server, or None if info cache should be cleared

Returns:

the object info

swift.proxy.controllers.base.update_headers(response, headers)

Helper function to update headers in the response.

Parameters:

• response – swob.Response object

• headers – dictionary headers

Account

class swift.proxy.controllers.account.AccountController(app, account_name, **kwargs)

Bases: Controller

WSGI controller for account requests

DELETE(req)

HTTP DELETE request handler.

GETorHEAD(req)

Handler for HTTP GET/HEAD requests.

POST(req)

HTTP POST request handler.

PUT(req)

HTTP PUT request handler.

add_acls_from_sys_metadata(resp)

server_type = 'Account'

Container

class swift.proxy.controllers.container.ContainerController(app, account_name, container_name, **kwargs)

Bases: Controller

WSGI controller for container requests

DELETE(req)

HTTP DELETE request handler.

GET(req)

Handler for HTTP GET requests.

HEAD(req)

Handler for HTTP HEAD requests.

POST(req)

HTTP POST request handler.

PUT(req)

HTTP PUT request handler.

UPDATE(req)

HTTP UPDATE request handler.

Method to perform bulk operations on container DBs, similar to a merge_items REPLICATE request.

Not client facing; internal clients or middlewares must include X-Backend-Allow-Private-Methods: true header to access.

clean_acls(req)

pass_through_headers = ['x-container-read', 'x-container-write', 'x-container-sync-key', 'x-container-sync-to', 'x-versions-location']

server_type = 'Container'

Object

class swift.proxy.controllers.obj.BaseObjectController(app, account_name, container_name, object_name, **kwargs)

Bases: Controller

Base WSGI controller for object requests.

DELETE(req)

HTTP DELETE request handler.

GET(req)

Handler for HTTP GET requests.

GETorHEAD(req)

Handle HTTP GET or HEAD requests.

HEAD(req)

Handler for HTTP HEAD requests.

POST(req)

HTTP POST request handler.

PUT(req)

HTTP PUT request handler.

iter_nodes_local_first(ring, partition, request, policy=None, local_handoffs_first=False)

Yields nodes for a ring partition.

If the ‘write_affinity’ setting is non-empty, then this will yield N local nodes (as defined by the write_affinity setting) first, then the rest of the nodes as normal. It is a re-ordering of the nodes such that the local ones come first; no node is omitted. The effect is that the request will be serviced by local object servers first, but nonlocal ones will be employed if not enough local ones are available.

Parameters:

• ring – ring to get nodes from

• partition – ring partition to yield nodes for

• request – nodes will be annotated with use_replication based on the request headers

• policy – optional, an instance of BaseStoragePolicy

• local_handoffs_first – optional, if True prefer primaries and local handoff nodes first before looking elsewhere.

server_type = 'Object'

class swift.proxy.controllers.obj.ECAppIter(path, policy, internal_parts_iters, range_specs, fa_length, obj_length, logger)

Bases: object

WSGI iterable that decodes EC fragment archives (or portions thereof) into the original object (or portions thereof).

Parameters:

• path – object’s path, sans v1 (e.g. /a/c/o)

• policy – storage policy for this object

• internal_parts_iters – list of the response-document-parts iterators for the backend GET responses. For an M+K erasure code, the caller must supply M such iterables.

• range_specs – list of dictionaries describing the ranges requested by the client. Each dictionary contains the start and end of the client’s requested byte range as well as the start and end of the EC segments containing that byte range.

• fa_length – length of the fragment archive, in bytes, if the response is a 200. If it’s a 206, then this is ignored.

• obj_length – length of the object, in bytes. Learned from the headers in the GET response from the object server.

• logger – a logger

app_iter_range(start, end)

app_iter_ranges(ranges, content_type, boundary, content_size)

close()

kickoff(req, resp)

Start pulling data from the backends so that we can learn things like the real Content-Type that might only be in the multipart/byteranges response body. Update our response accordingly.

Also, this is the first point at which we can learn the MIME boundary that our response has in the headers. We grab that so we can also use it in the body.

Returns:

None

Raises:

HTTPException – on error

next()

class swift.proxy.controllers.obj.ECFragGetter(app, req, node_iter, partition, policy, path, backend_headers, header_provider, logger_thread_locals, logger)

Bases: GetterBase

property headers

property last_status

response_parts_iter()

Create an iterator over a single fragment response body.

Returns:

an interator that yields chunks of bytes from a fragment response body.

property source_iter

An iterator over responses to backend fragment GETs. Yields an instance of GetterSource if a response is good, otherwise None.

class swift.proxy.controllers.obj.ECGetResponseBucket(policy, timestamp)

Bases: object

A helper class to encapsulate the properties of buckets in which fragment getters and alternate nodes are collected.

add_alternate_nodes(node, frag_indexes)

add_response(getter, parts_iter)

Add another response to this bucket. Response buckets can be for fragments with the same timestamp, or for errors with the same status.

close_conns()

Close bucket’s responses; they won’t be used for a client response.

property durable

get_responses()

Return a list of all useful sources. Where there are multiple sources associated with the same frag_index then only one is included.

Returns:

a list of sources, each source being a tuple of form (ECFragGetter, iter)

set_durable()

property shortfall

The number of additional responses needed to complete this bucket; typically (ndata - resp_count).

If the bucket has no durable responses, shortfall is extended out to replica count to ensure the proxy makes additional primary requests.

property shortfall_with_alts

class swift.proxy.controllers.obj.ECGetResponseCollection(policy)

Bases: object

Manages all successful EC GET responses gathered by ECFragGetters.

A response comprises a tuple of (<getter instance>, <parts iterator>). All responses having the same data timestamp are placed in an ECGetResponseBucket for that timestamp. The buckets are stored in the ‘buckets’ dict which maps timestamp-> bucket.

This class encapsulates logic for selecting the best bucket from the collection, and for choosing alternate nodes.

add_bad_resp(get, parts_iter)

add_good_response(get, parts_iter)

add_response(get, parts_iter)

Add a response to the collection.

Parameters:

• get – An instance of ECFragGetter

• parts_iter – An iterator over response body parts

Raises:

ValueError – if the response etag or status code values do not match any values previously received for the same timestamp

property best_bucket

Return the “best” bucket in the collection.

The “best” bucket is the newest timestamp with sufficient getters, or the closest to having sufficient getters, unless it is bettered by a bucket with potential alternate nodes.

If there are no good buckets we return the “least_bad” bucket.

Returns:

An instance of ECGetResponseBucket or None if there are no buckets in the collection.

choose_best_bucket()

property durable

get_extra_headers()

has_alternate_node()

property least_bad_bucket

Return the bad_bucket with the smallest shortfall

provide_alternate_node()

Callback function that is installed in a NodeIter. Called on every call to NodeIter.next(), which means we can track the number of nodes to which GET requests have been made and selectively inject an alternate node, if we have one.

Returns:

A dict describing a node to which the next GET request should be made.

property shortfall

class swift.proxy.controllers.obj.ECObjectController(app, account_name, container_name, object_name, **kwargs)

Bases: BaseObjectController

feed_remaining_primaries(safe_iter, pile, req, partition, policy, buckets, feeder_q, logger_thread_locals)

policy_type = 'erasure_coding'

class swift.proxy.controllers.obj.MIMEPutter(conn, node, resp, path, connect_duration, watchdog, write_timeout, send_exception_handler, logger, mime_boundary, multiphase=False)

Bases: Putter

Putter for backend PUT requests that use MIME.

This is here mostly to wrap up the fact that all multipart PUTs are chunked because of the mime boundary footer trick and the first half of the two-phase PUT conversation handling.

An HTTP PUT request that supports streaming.

classmethod connect(node, part, path, headers, watchdog, conn_timeout, node_timeout, write_timeout, send_exception_handler, logger=None, need_multiphase=True, **kwargs)

Connect to a backend node and send the headers.

Override superclass method to notify object of need for support for multipart body with footers and optionally multiphase commit, and verify object server’s capabilities.

Parameters:

need_multiphase – if True then multiphase support is required of the object server

Raises:

• FooterNotSupported – if need_metadata_footer is set but backend node can’t process footers

• MultiphasePUTNotSupported – if need_multiphase is set but backend node can’t handle multiphase PUT

end_of_object_data(footer_metadata=None)

Call when there is no more data to send.

Overrides superclass implementation to send any footer metadata after object data.

Parameters:

footer_metadata – dictionary of metadata items to be sent as footers.

send_commit_confirmation()

Call when there are > quorum 2XX responses received. Send commit confirmations to all object nodes to finalize the PUT.

class swift.proxy.controllers.obj.ObjectControllerRouter

Bases: object

policy_type_to_controller_map = {'erasure_coding': <class 'swift.proxy.controllers.obj.ECObjectController'>, 'replication': <class 'swift.proxy.controllers.obj.ReplicatedObjectController'>}

classmethod register(policy_type)

Decorator for Storage Policy implementations to register their ObjectController implementations.

This also fills in a policy_type attribute on the class.

class swift.proxy.controllers.obj.Putter(conn, node, resp, path, connect_duration, watchdog, write_timeout, send_exception_handler, logger, chunked=False)

Bases: object

Putter for backend PUT requests.

Encapsulates all the actions required to establish a connection with a storage node and stream data to that node.

Parameters:

• conn – an HTTPConnection instance

• node – dict describing storage node

• resp – an HTTPResponse instance if connect() received final response

• path – the object path to send to the storage node

• connect_duration – time taken to initiate the HTTPConnection

• watchdog – a spawned Watchdog instance that will enforce timeouts

• write_timeout – time limit to write a chunk to the connection socket

• send_exception_handler – callback called when an exception occured writing to the connection socket

• logger – a Logger instance

• chunked – boolean indicating if the request encoding is chunked

await_response(timeout, informational=False)

Get 100-continue response indicating the end of 1st phase of a 2-phase commit or the final response, i.e. the one with status >= 200.

Might or might not actually wait for anything. If we said Expect: 100-continue but got back a non-100 response, that’ll be the thing returned, and we won’t do any network IO to get it. OTOH, if we got a 100 Continue response and sent up the PUT request’s body, then we’ll actually read the 2xx-5xx response off the network here.

Parameters:

• timeout – time to wait for a response

• informational – if True then try to get a 100-continue response, otherwise try to get a final response.

Returns:

HTTPResponse

Raises:

Timeout – if the response took too long

close()

classmethod connect(node, part, path, headers, watchdog, conn_timeout, node_timeout, write_timeout, send_exception_handler, logger=None, chunked=False, **kwargs)

Connect to a backend node and send the headers.

Returns:

Putter instance

Raises:

• ConnectionTimeout – if initial connection timed out

• ResponseTimeout – if header retrieval timed out

• InsufficientStorage – on 507 response from node

• PutterConnectError – on non-507 server error response from node

end_of_object_data(**kwargs)

Call when there is no more data to send.

send_chunk(chunk, timeout_at=None)

class swift.proxy.controllers.obj.ReplicatedObjectController(app, account_name, container_name, object_name, **kwargs)

Bases: BaseObjectController

policy_type = 'replication'

swift.proxy.controllers.obj.check_content_type(req)

swift.proxy.controllers.obj.chunk_transformer(policy)

A generator to transform a source chunk to erasure coded chunks for each send call. The number of erasure coded chunks is as policy.ec_n_unique_fragments.

swift.proxy.controllers.obj.client_range_to_segment_range(client_start, client_end, segment_size)

Takes a byterange from the client and converts it into a byterange spanning the necessary segments.

Handles prefix, suffix, and fully-specified byte ranges.

Examples:

client_range_to_segment_range(100, 700, 512) = (0, 1023) client_range_to_segment_range(100, 700, 256) = (0, 767) client_range_to_segment_range(300, None, 256) = (256, None)

Parameters:

• client_start – first byte of the range requested by the client

• client_end – last byte of the range requested by the client

• segment_size – size of an EC segment, in bytes

Returns:

a 2-tuple (seg_start, seg_end) where

• seg_start is the first byte of the first segment, or None if this is a suffix byte range

• seg_end is the last byte of the last segment, or None if this is a prefix byte range

swift.proxy.controllers.obj.num_container_updates(container_replicas, container_quorum, object_replicas, object_quorum)

We need to send container updates via enough object servers such that, if the object PUT succeeds, then the container update is durable (either it’s synchronously updated or written to async pendings).

Define:

Qc = the quorum size for the container ring Qo = the quorum size for the object ring Rc = the replica count for the container ring Ro = the replica count (or EC N+K) for the object ring

A durable container update is one that’s made it to at least Qc nodes. To always be durable, we have to send enough container updates so that, if only Qo object PUTs succeed, and all the failed object PUTs had container updates, at least Qc updates remain. Since (Ro - Qo) object PUTs may fail, we must have at least Qc + Ro - Qo container updates to ensure that Qc of them remain.

Also, each container replica is named in at least one object PUT request so that, when all requests succeed, no work is generated for the container replicator. Thus, at least Rc updates are necessary.

Parameters:

• container_replicas – replica count for the container ring (Rc)

• container_quorum – quorum size for the container ring (Qc)

• object_replicas – replica count for the object ring (Ro)

• object_quorum – quorum size for the object ring (Qo)

swift.proxy.controllers.obj.segment_range_to_fragment_range(segment_start, segment_end, segment_size, fragment_size)

Takes a byterange spanning some segments and converts that into a byterange spanning the corresponding fragments within their fragment archives.

Handles prefix, suffix, and fully-specified byte ranges.

Parameters:

• segment_start – first byte of the first segment

• segment_end – last byte of the last segment

• segment_size – size of an EC segment, in bytes

• fragment_size – size of an EC fragment, in bytes

Returns:

a 2-tuple (frag_start, frag_end) where

• frag_start is the first byte of the first fragment, or None if this is a suffix byte range

• frag_end is the last byte of the last fragment, or None if this is a prefix byte range

swift.proxy.controllers.obj.trailing_metadata(policy, client_obj_hasher, bytes_transferred_from_client, fragment_archive_index)

Proxy Server

class swift.proxy.server.Application(conf, logger=None, account_ring=None, container_ring=None)

Bases: object

WSGI application for the proxy server.

check_config()

Check the configuration for possible errors

check_response(node, server_type, response, method, path, body=None)

Check response for error status codes and update error limiters as required.

Parameters:

• node – a dict describing a node

• server_type – the type of server from which the response was received (e.g. ‘Object’).

• response – an instance of HTTPResponse.

• method – the request method.

• path – the request path.

• body – an optional response body. If given, up to 1024 of the start of the body will be included in any log message.

Return True:

if the response status code is less than 500, False otherwise.

error_limit(node, msg)

Mark a node as error limited. This immediately pretends the node received enough errors to trigger error suppression. Use this for errors like Insufficient Storage. For other errors use increment().

Parameters:

• node – dictionary of node to error limit

• msg – error message

error_limited(node)

Check if the node is currently error limited.

Parameters:

node – dictionary of node to check

Returns:

True if error limited, False otherwise

error_occurred(node, msg)

Handle logging, and handling of errors.

Parameters:

• node – dictionary of node to handle errors for

• msg – error message

exception_occurred(node, typ, additional_info, **kwargs)

Handle logging of generic exceptions.

Parameters:

• node – dictionary of node to log the error for

• typ – server type

• additional_info – additional information to log

get_controller(req)

Get the controller to handle a request.

Parameters:

req – the request

Returns:

tuple of (controller class, path dictionary)

Raises:

ValueError – (thrown by split_path) if given invalid path

get_object_ring(policy_idx)

Get the ring object to use to handle a request based on its policy.

Parameters:

policy_idx – policy index as defined in swift.conf

Returns:

appropriate ring object

get_policy_options(policy)

Return policy specific options.

Parameters:

policy – an instance of BaseStoragePolicy or None

Returns:

an instance of ProxyOverrideOptions

handle_request(req)

Entry point for proxy server. Should return a WSGI-style callable (such as swob.Response).

Parameters:

req – swob.Request object

modify_wsgi_pipeline(pipe)

Called during WSGI pipeline creation. Modifies the WSGI pipeline context to ensure that mandatory middleware is present in the pipeline.

Parameters:

pipe – A PipelineWrapper object

set_node_timing(node, timing)

sort_nodes(nodes, policy=None)

Sorts nodes in-place (and returns the sorted list) according to the configured strategy. The default “sorting” is to randomly shuffle the nodes. If the “timing” strategy is chosen, the nodes are sorted according to the stored timing data.

Parameters:

• nodes – a list of nodes

• policy – an instance of BaseStoragePolicy

update_request(req)

class swift.proxy.server.ProxyOverrideOptions(base_conf, override_conf, app)

Bases: object

Encapsulates proxy server options that may be overridden e.g. for policy specific configurations.

Parameters:

• conf – the proxy-server config dict.

• override_conf – a dict of overriding configuration options.

swift.proxy.server.app_factory(global_conf, **local_conf)

paste.deploy app factory for creating WSGI proxy apps.

swift.proxy.server.main()

swift.proxy.server.parse_per_policy_config(conf)

Search the config file for any per-policy config sections and load those sections to a dict mapping policy reference (name or index) to policy options.

Parameters:

conf – the proxy server conf dict

Returns:

a dict mapping policy reference -> dict of policy options

Raises:

ValueError – if a policy config section has an invalid name