Backend API

Abstract classes for unified storage/query API with various backends.

Derivative classes are expected to be either complete implementations or wrappers for external libraries. The latter is assumed to be a better solution as Doqu is only one of the possible layers. It is always a good idea to provide different levels of abstraction and let others combine them as needed.

The backends do not have to subclass BaseStorageAdapter and BaseQueryAdapter. However, they must closely follow their API.

class doqu.backend_base.BaseStorageAdapter(**kw)

Abstract adapter class for storage backends.

Note

Backends policy

If a public method foo() internally uses a private method _foo(), then subclasses should only overload only the private attribute. This ensures that docstring and signature are always correct. However, if the backend introduces some deviations in behaviour or extends the signature, the public method can (and should) be overloaded at least to provide documentation.

clear()

Clears the whole storage from data, resets autoincrement counters.

connect()

Connects to the database. Raises RuntimeError if the connection is not closed yet. Use reconnect() to explicitly close the connection and open it again.

delete(key)

Deletes record with given primary key.

disconnect()

Closes internal store and removes the reference to it. If the backend works with a file, then all pending changes are saved now.

find(doc_class=<type 'dict'>, **conditions)

Returns instances of given class, optionally filtered by given conditions.

Parameters:

• doc_class – Document class. Default is dict. Normally you will want a more advanced class, such as Document or its more concrete subclasses (with explicit structure and validators).
• conditions – key/value pairs, same as in where().

Note

By default this returns a tuple of (key, data_dict) per item. However, this can be changed if doc_class provides the method from_storage(). For example, Document has the notion of “saved state” so it can store the key within. Thus, only a single Document object is returned per item.

get(key, doc_class=<type 'dict'>)

Returns document instance for given document class and primary key. Raises KeyError if there is no item with given key in the database.

Parameters:

• key – a numeric or string primary key (as supported by the backend).
• doc_class – a document class to wrap the data into. Default is dict.

get_many(keys, doc_class=<type 'dict'>)

Returns an iterator of documents with primary keys from given list. Basically this is just a simple wrapper around get() but some backends can reimplement the method in a much more efficient way.

get_or_create(doc_class=<type 'dict'>, **conditions)

Queries the database for records associated with given document class and conforming to given extra conditions. If such records exist, picks the first one (the order may be random depending on the database). If there are no such records, creates one.

Returns the document instance and a boolean value “created”.

reconnect()

Gracefully closes current connection (if it’s not broken) and connects again to the database (e.g. reopens the file).

save(key, data)

Saves given data with given primary key into the storage. Returns the primary key.

Parameters:

• key – the primary key for given object; if None, will be generated.
• data – a dict containing all properties to be saved.

Note that you must provide current primary key for a record which is already in the database in order to update it instead of copying it.

sync()

Synchronizes the storage to disk immediately if the backend supports this operation. Normally the data is synchronized either on save(), or on timeout, or on disconnect(). This is strictly backend-specific. If a backend does not support the operation, NotImplementedError is raised.

class doqu.backend_base.BaseQueryAdapter(storage, doc_class)

Query adapter for given backend.

count()

Returns the number of records that match given query. The result of q.count() is exactly equivalent to the result of len(q). The implementation details do not differ by default, but it is recommended that the backends stick to the following convention:

• __len__ executes the query, retrieves all matching records and tests the length of the resulting list;
• count executes a special query that only returns a single value: the number of matching records.

Thus, __len__ is more suitable when you are going to iterate the records anyway (and do no extra queries), while count is better when you just want to check if the records exist, or to only use a part of matching records (i.e. a slice).

delete()

Deletes all records that match current query.

order_by(names, reverse=False)

Returns a query object with same conditions but with results sorted by given field. By default the direction of sorting is ascending.

Parameters:

• names – list of strings: names of fields by which results should be sorted. Some backends may only support a single field for sorting.
• reverse – bool: if True, the direction of sorting is reversed and becomes descending. Default is False.

values(name)

Returns a list of unique values for given field name.

Parameters:name – the field name.

where(**conditions)

Returns Query instance filtered by given conditions. The conditions are specified by backend’s underlying API.

where_not(**conditions)

Returns Query instance. Inverted version of where().

exception doqu.backend_base.ProcessorDoesNotExist

This exception is raised when given backend does not have a processor suitable for given value. Usually you will need to catch a subclass of this exception.

class doqu.backend_base.LookupManager

Usage:

lookup_manager = LookupManager()

@lookup_manager.register('equals', default=True)  # only one lookup can be default
def exact_match(name, value):
    '''
    Returns native Tokyo Cabinet lookup triplets for given
    backend-agnostic lookup triplet.
    '''
    if isinstance(value, basestring):
        return (
            (name, proto.RDBQCSTREQ, value),
        )
    if isinstance(value, (int, float)):
        return (
            (name, proto.RDBQCNUMEQ, value),
        )
    raise ValueError

Now if you call lookup_manager.resolve('age', 'equals', 99), the returned value will be (('age', proto.RDBCNUMEQ, 99),).

A single generic lookup may yield multiple native lookups because some backends do not support certain lookups directly and therefore must translate them to a combination of elementary conditions. In most cases resolve() will yield a single condition. Its format is determined by the query adapter.

exception_class

alias of LookupProcessorDoesNotExist

resolve(name, operation, value)

Returns a set of backend-specific conditions for given backend-agnostic triplet, e.g.:

('age', 'gt', 90)

will be translated by the Tokyo Cabinet backend to:

('age', 9, '90')

or by the MongoDB backend to:

{'age': {'$gt': 90}}

exception doqu.backend_base.LookupProcessorDoesNotExist

This exception is raised when given backend does not support the requested lookup.

class doqu.backend_base.ConverterManager

An instance of this class can manage property processors for given backend. Processor classes must be registered against Python types or classes. The processor manager allows encoding and decoding data between a document class instance and a database record. Each backend supports only a certain subset of Python datatypes and has its own rules in regard to how None values are interpreted, how complex data structures are serialized and so on. Moreover, there’s no way to guess how a custom class should be processed. Therefore, each combination of data type + backend has to be explicitly defined as a set of processing methods (to and from).

exception_class

alias of DataProcessorDoesNotExist

from_db(datatype, value)

Converts given value to given Python datatype. The value must be correctly pre-encoded by the symmetrical PropertyManager.to_db() method before saving it to the database.

Raises DataProcessorDoesNotExist if no suitable processor is defined by the backend.

to_db(value, storage)

Prepares given value and returns it in a form ready for storing in the database.

Raises DataProcessorDoesNotExist if no suitable processor is defined by the backend.

exception doqu.backend_base.DataProcessorDoesNotExist

This exception is raised when given backend does not have a datatype processor suitable for given value.