Objects in neutron¶

Usage of objects¶

# to create an object, you can pass the attributes in constructor:
dns = DNSNameServer(context, address='asd', subnet_id='xxx', order=1)
dns.create()

# or you can create a dict and pass it as kwargs:
dns_data = {'address': 'asd', 'subnet_id': 'xxx', 'order': 1}
dns = DNSNameServer(context, **dns_data)
dns.create()

# for fetching multiple objects:
dnses = DNSNameServer.get_objects(context)
# will return list of all dns name servers from DB

# to update fields:
dns = DNSNameServer.get_object(context, address='asd', subnet_id='xxx')
dns.order = 2
dns.update()

# if you don't care about keeping the object, you can execute the update
# without fetch of the object state from the underlying persistent layer
count = DNSNameServer.update_objects(
    context, {'order': 3}, address='asd', subnet_id='xxx')

# to remove object with filter arguments:
filters = {'address': 'asd', 'subnet_id': 'xxx'}
DNSNameServer.delete_objects(context, **filters)

# filtering

# to get an object based on primary key filter
dns = DNSNameServer.get_object(context, address='asd', subnet_id='xxx')

# to get multiple objects
dnses = DNSNameServer.get_objects(context, subnet_id='xxx')

filters = {'subnet_id': ['xxx', 'yyy']}
dnses = DNSNameServer.get_objects(context, **filters)

# do not validate filters
dnses = DNSNameServer.get_objects(context, validate_filters=False,
                                  fake_filter='xxx')

# count the dns servers for given subnet
dns_count = DNSNameServer.count(context, subnet_id='xxx')

# sorting
# direction True == ASC, False == DESC
direction = False
pager = Pager(sorts=[('order', direction)])
dnses = DNSNameServer.get_objects(context, _pager=pager, subnet_id='xxx')

Defining your own object¶

In order to add a new object in neutron, you have to:

1. Create an object derived from NeutronDbObject (aka base object)
2. Add/reuse data model
3. Define fields

It is mandatory to define data model using db_model attribute from NeutronDbObject.

Fields should be defined using oslo_versionobjects.fields exposed types. If there is a special need to create a new type of field, you can use common_types.py in the neutron.objects directory. Example:

fields = {
    'id': common_types.UUIDField(),
    'name': obj_fields.StringField(),
    'subnetpool_id': common_types.UUIDField(nullable=True),
    'ip_version': common_types.IPVersionEnumField()
}

VERSION is mandatory and defines the version of the object. Initially, set the VERSION field to 1.0. Change VERSION if fields or their types are modified. When you change the version of objects being exposed via RPC, add method obj_make_compatible(self, primitive, target_version).

primary_keys is used to define the list of fields that uniquely identify the object. In case of database backed objects, it’s usually mapped onto SQL primary keys. For immutable object fields that cannot be changed, there is a fields_no_update list, that contains primary_keys by default.

If there is a situation where a field needs to be named differently in an object than in the database schema, you can use fields_need_translation. This dictionary contains the name of the field in the object definition (the key) and the name of the field in the database (the value). This allows to have a different object layer representation for database persisted data. For example in IP allocation pools:

fields_need_translation = {
    'start': 'first_ip',  # field_ovo: field_db
    'end': 'last_ip'
}

The above dictionary is used in modify_fields_from_db() and in modify_fields_to_db() methods which are implemented in base class and will translate the software layer to database schema naming, and vice versa. It can also be used to rename orm.relationship backed object-type fields.

Most object fields are usually directly mapped to database model attributes. Sometimes it’s useful to expose attributes that are not defined in the model table itself, like relationships and such. In this case, synthetic_fields may become handy. This object property can define a list of object fields that don’t belong to the object database model and that are hence instead to be implemented in some custom way. Some of those fields map to orm.relationships defined on models, while others are completely untangled from the database layer.

When exposing existing orm.relationships as an ObjectField-typed field, you can use the foreign_keys object property that defines a link between two object types. When used, it allows objects framework to automatically instantiate child objects, and fill the relevant parent fields, based on orm.relationships defined on parent models. In order to automatically populate the synthetic_fields, the foreign_keys property is introduced. load_synthetic_db_fields() [3] method from NeutronDbObject uses foreign_keys to match the foreign key in related object and local field that the foreign key is referring to. See simplified examples:

class DNSNameServerSqlModel(model_base.BASEV2):
    address = sa.Column(sa.String(128), nullable=False, primary_key=True)
    subnet_id = sa.Column(sa.String(36),
                          sa.ForeignKey('subnets.id', ondelete="CASCADE"),
                          primary_key=True)

class SubnetSqlModel(model_base.BASEV2, HasId, HasProject):
    name = sa.Column(sa.String(attr.NAME_MAX_LEN))
    allocation_pools = orm.relationship(IPAllocationPoolSqlModel)
    dns_nameservers = orm.relationship(DNSNameServerSqlModel,
                                       backref='subnet',
                                       cascade='all, delete, delete-orphan',
                                       lazy='subquery')

class IPAllocationPoolSqlModel(model_base.BASEV2, HasId):
    subnet_id = sa.Column(sa.String(36), sa.ForeignKey('subnets.id'))

@obj_base.VersionedObjectRegistry.register
class DNSNameServerOVO(base.NeutronDbObject):
    VERSION = '1.0'
    db_model = DNSNameServerSqlModel

    # Created based on primary_key=True in model definition.
    # The object is uniquely identified by the pair of address and
    # subnet_id fields. Override the default 'id' 1-tuple.
    primary_keys = ['address', 'subnet_id']

    # Allow to link DNSNameServerOVO child objects into SubnetOVO parent
    # object fields via subnet_id child database model attribute.
    # Used during loading synthetic fields in SubnetOVO get_objects.
    foreign_keys = {'SubnetOVO': {'subnet_id': 'id'}}

    fields = {
        'address': obj_fields.StringField(),
        'subnet_id': common_types.UUIDField(),
    }

@obj_base.VersionedObjectRegistry.register
class SubnetOVO(base.NeutronDbObject):
    VERSION = '1.0'
    db_model =  SubnetSqlModel

    fields = {
        'id': common_types.UUIDField(),  # HasId from model class
        'project_id': obj_fields.StringField(nullable=True),  # HasProject from model class
        'subnet_name': obj_fields.StringField(nullable=True),
        'dns_nameservers': obj_fields.ListOfObjectsField('DNSNameServer',
                                                         nullable=True),
        'allocation_pools': obj_fields.ListOfObjectsField('IPAllocationPoolOVO',
                                                          nullable=True)
    }

    # Claim dns_nameservers field as not directly mapped into the object
    # database model table.
    synthetic_fields = ['allocation_pools', 'dns_nameservers']

    # Rename in-database subnet_name attribute into name object field
    fields_need_translation = {
        'name': 'subnet_name'
    }


@obj_base.VersionedObjectRegistry.register
class IPAllocationPoolOVO(base.NeutronDbObject):
    VERSION = '1.0'
    db_model = IPAllocationPoolSqlModel

    fields = {
        'subnet_id': common_types.UUIDField()
    }

    foreign_keys = {'SubnetOVO': {'subnet_id': 'id'}}

The foreign_keys is used in SubnetOVO to populate the allocation_pools [4] synthetic field using the IPAllocationPoolOVO class. Single object type may be linked to multiple parent object types, hence foreign_keys property may have multiple keys in the dictionary.

It is important to remember about the nullable parameter. In the SQLAlchemy model, the nullable parameter is by default True, while for OVO fields, the nullable is set to False. Make sure you correctly map database column nullability properties to relevant object fields.

Synthetic fields¶

synthetic_fields is a list of fields, that are not directly backed by corresponding object SQL table attributes. Synthetic fields are not limited in types that can be used to implement them.

fields = {
    'dhcp_agents': obj_fields.ObjectField('NetworkDhcpAgentBinding',
                                          nullable=True), # field that contains another single NeutronDbObject of NetworkDhcpAgentBinding type
    'shared': obj_fields.BooleanField(default=False),
    'subnets': obj_fields.ListOfObjectsField('Subnet', nullable=True)
}

# All three fields do not belong to corresponding SQL table, and will be
# implemented in some object-specific way.
synthetic_fields = ['dhcp_agents', 'shared', 'subnets']

ObjectField and ListOfObjectsField take the name of object class as an argument.

@obj_base.VersionedObjectRegistry.register
class ExternalSubnet(base.NeutronDbObject):
    VERSION = '1.0'
    fields = {'subnet_id': common_types.UUIDField(),
              'is_external': obj_fields.BooleanField()}
    primary_keys = ['subnet_id']
    foreign_keys = {'Subnet': {'subnet_id': 'id'}}


@obj_base.VersionedObjectRegistry.register
class Subnet(base.NeutronDbObject):
    VERSION = '1.0'
    fields = {'external': obj_fields.BooleanField(nullable=True),}
    synthetic_fields = ['external']

    # support new custom 'external=' filter for get_objects family of
    # objects API
    def __init__(self, context=None, **kwargs):
        super(Subnet, self).__init__(context, **kwargs)
        self.add_extra_filter_name('external')

    def create(self):
        fields = self.get_changes()
        with db_api.context_manager.writer.using(context):
            if 'external' in fields:
                ExternalSubnet(context, subnet_id=self.id,
                    is_external=fields['external']).create()
            # Call to super() to create the SQL record for the object, and
            # reload its fields from the database, if needed.
            super(Subnet, self).create()

    def update(self):
        fields = self.get_changes()
        with db_api.context_manager.writer.using(context):
            if 'external' in fields:
                # delete the old ExternalSubnet record, if present
                obj_db_api.delete_objects(
                    self.obj_context, ExternalSubnet.db_model,
                    subnet_id=self.id)
                # create the new intended ExternalSubnet object
                ExternalSubnet(context, subnet_id=self.id,
                    is_external=fields['external']).create()
            # calling super().update() will reload the synthetic fields
            # and also will update any changed non-synthetic fields, if any
            super(Subnet, self).update()

    # this method is called when user of an object accesses the attribute
    # and requested attribute is not set.
    def obj_load_attr(self, attrname):
        if attrname == 'external':
            return self._load_external()
        # it is important to call super if attrname does not match
        # because the base implementation is handling the nullable case
        super(Subnet, self).obj_load_attr(attrname)

    def _load_external(self, db_obj=None):
        # do the loading here
        if db_obj:
            # use DB model to fetch the data that may be side-loaded
            external = db_obj.external.is_external if db_obj.external else None
        else:
            # perform extra operation to fetch the data from DB
            external_obj = ExternalSubnet.get_object(context,
                subnet_id=self.id)
            external = external_obj.is_external if external_obj else None

        # it is important to set the attribute and call obj_reset_changes
        setattr(self, 'external', external)
        self.obj_reset_changes(['external'])

    # this is defined in NeutronDbObject and is invoked during get_object(s)
    # and create/update.
    def from_db_object(self, obj):
        super(Subnet, self).from_db_object(obj)
        self._load_external(obj)

Standard attributes¶

The standard attributes are added automatically in metaclass DeclarativeObject. If adding standard attribute, it has to be added in neutron/objects/extensions/standardattributes.py. It will be added to all relevant objects that use the standardattributes model. Be careful when adding something to the above, because it could trigger a change in the object’s VERSION. For more on how standard attributes work, check [5].

RBAC handling in objects¶

The RBAC is implemented currently for resources like: Subnet(*), Network and QosPolicy. Subnet is a special case, because access control of Subnet depends on Network RBAC entries.

The RBAC support for objects is defined in neutron/objects/rbac_db.py. It defines new base class NeutronRbacObject. The new class wraps standard NeutronDbObject methods like create(), update() and to_dict(). It checks if the shared attribute is defined in the fields dictionary and adds it to synthetic_fields. Also, rbac_db_model is required to be defined in Network and QosPolicy classes.

NeutronRbacObject is a common place to handle all operations on the RBAC entries, like getting the info if resource is shared or not, creation and updates of them. By wrapping the NeutronDbObject methods, it is manipulating the ‘shared’ attribute while create() and update() methods are called.

The example of defining the Network OVO:

class Network(standard_attr.HasStandardAttributes, model_base.BASEV2,
          model_base.HasId, model_base.HasProject):
    """Represents a v2 neutron network."""
    name = sa.Column(sa.String(attr.NAME_MAX_LEN))
    rbac_entries = orm.relationship(rbac_db_models.NetworkRBAC,
                                    backref='network', lazy='joined',
                                    cascade='all, delete, delete-orphan')


# Note the base class for Network OVO:
@obj_base.VersionedObjectRegistry.register
class Network(rbac_db.NeutronRbacObject):
    # Version 1.0: Initial version
    VERSION = '1.0'

    # rbac_db_model is required to be added here
    rbac_db_model = rbac_db_models.NetworkRBAC
    db_model = models_v2.Network

    fields = {
        'id': common_types.UUIDField(),
        'project_id': obj_fields.StringField(nullable=True),
        'name': obj_fields.StringField(nullable=True),
        # share is required to be added to fields
        'shared': obj_fields.BooleanField(default=False),
    }

Extensions to neutron resources¶

One of the methods to extend neutron resources is to add an arbitrary value to dictionary representing the data by providing extend_(subnet|port|network)_dict() function and defining loading method.

From DB perspective, all the data will be loaded, including all declared fields from DB relationships. Current implementation for core resources (Port, Subnet, Network etc.) is that DB result is parsed by make_<resource>_dict() and extend_<resource>_dict(). When extension is enabled, extend_<resource>_dict() takes the DB results and declares new fields in resulting dict. When extension is not enabled, data will be fetched, but will not be populated into resulting dict, because extend_<resource>_dict() will not be called.

Plugins can still use objects for some work, but then convert them to dicts and work as they please, extending the dict as they wish.

For example:

class TestSubnetExtension(model_base.BASEV2):
    subnet_id = sa.Column(sa.String(36),
                          sa.ForeignKey('subnets.id', ondelete="CASCADE"),
                          primary_key=True)
    value = sa.Column(sa.String(64))
    subnet = orm.relationship(
        models_v2.Subnet,
        # here is the definition of loading the extension with Subnet model:
        backref=orm.backref('extension', cascade='delete', uselist=False))


@oslo_obj_base.VersionedObjectRegistry.register_if(False)
class TestSubnetExtensionObject(obj_base.NeutronDbObject):
    # Version 1.0: Initial version
    VERSION = '1.0'

    db_model = TestSubnetExtension

    fields = {
        'subnet_id': common_types.UUIDField(),
        'value': obj_fields.StringField(nullable=True)
    }

    primary_keys = ['subnet_id']
    foreign_keys = {'Subnet': {'subnet_id': 'id'}}


@obj_base.VersionedObjectRegistry.register
class Subnet(base.NeutronDbObject):
    # Version 1.0: Initial version
    VERSION = '1.0'

    fields = {
        'id': common_types.UUIDField(),
        'extension': obj_fields.ObjectField(TestSubnetExtensionObject.__name__,
                                            nullable=True),
    }

    synthetic_fields = ['extension']


# when defining the extend_subnet_dict function:
def extend_subnet_dict(self, session, subnet_ovo, result):
    value = subnet_ovo.extension.value if subnet_ovo.extension else ''
    result['subnet_extension'] = value

The above example is the ideal situation, where all extensions have objects adopted and enabled in core neutron resources.

By introducing the OVO work in tree, interface between base plugin code and registered extension functions hasn’t been changed. Those still receive a SQLAlchemy model, not an object. This is achieved by capturing the corresponding database model on get_***/create/update, and exposing it via <object>.db_obj

Backward compatibility for tenant_id¶

All objects can support tenant_id and project_id filters and fields at the same time; it is automatically enabled for all objects that have a project_id field. The base NeutronDbObject class has support for exposing tenant_id in dictionary access to the object fields (subnet['tenant_id']) and in to_dict() method. There is a tenant_id read-only property for every object that has project_id in fields. It is not exposed in obj_to_primitive() method, so it means that tenant_id will not be sent over RPC callback wire. When talking about filtering/sorting by tenant_id, the filters should be converted to expose project_id field. This means that for the long run, the API layer should translate it, but as temporary workaround it can be done at DB layer before passing filters to objects get_objects() method, for example:

def convert_filters(result):
    if 'tenant_id' in result:
        result['project_id'] = result.pop('tenant_id')
    return result

def get_subnets(context, filters):
    filters = convert_filters(**filters)
    return subnet_obj.Subnet.get_objects(context, **filters)

The convert_filters method is available in neutron.objects.utils [7].

References¶