Welcome to aioriak’s documentation!¶
Asyncio (PEP 3156) Riak client library. This project is based on official Bash python client library (https://github.com/basho/riak-python-client).
Features¶
Riak KV operations | Yes |
Riak Datatypes | Yes |
Riak BucketTypes | Yes |
Custom resolver | Yes |
Node list support | WIP |
Custom quorum | No |
Connections Pool | No |
Operations timout | No |
Security | No |
Riak Search | WIP |
MapReduce | WIP |
Tested python versions | 3.5, 3.6 |
Tested Riak versions | 2.1.3, 2.1.4 |
Installation¶
The easiest way to install aioriak is by using the package on PyPi:
pip install aioriak
Requirements¶
- Python >= 3.5
- riak==2.7.0
Contribute¶
- Issue Tracker: https://github.com/rambler-digital-solutions/aioriak/issues
- Source Code: https://github.com/rambler-digital-solutions/aioriak
Feel free to file an issue or make pull request if you find any bugs or have some suggestions for library improvement.
License¶
The aioriak is offered under MIT license.
Contents¶
Client & Connections¶
To connect to a Riak cluster, you must create a
RiakClient
object. The default configuration
connects to a Riak node on localhost
with the default
ports. The below instantiation statements are all equivalent:
from aioriak import RiakClient
client = RiakClient()
async def go():
client = await RiakClient.create(host='127.0.0.1', port=8087)
Note
Connections are not established until you attempt to perform an operation. If the host or port are incorrect, you will not get an error raised immediately.
Client objects¶
-
class
aioriak.client.
RiakClient
(host='localhost', port=8087, loop=None)[source]¶ The
RiakClient
object holds information necessary to connect to Riak. Requests can be made to Riak directly through the client or by using the methods on related objects.-
resolver
¶ The sibling-resolution function for this client. Defaults to
aioriak.resolver.default_resolver()
.
-
Client-level Operations¶
Some operations are not scoped by buckets or bucket types and can be performed on the client directly:
-
coroutine
RiakClient.
ping
()[source]¶ Check if the Riak server for this
RiakClient
instance is alive.Return type: boolean
-
coroutine
RiakClient.
get_buckets
(bucket_type=None)[source]¶ Get the list of buckets as
Bucket
instances.Warning
Do not use this in production, as it requires traversing through all keys stored in a cluster.
Parameters: bucket_type ( BucketType
) – the optional containing bucket typeReturn type: list of Bucket
instances
Accessing Bucket Types and Buckets¶
Most client operations are on bucket type objects
, the bucket objects
they contain or keys within those buckets. Use the
bucket_type
or bucket
methods for creating bucket types and buckets
that will proxy operations to the called client.
-
RiakClient.
bucket_type
(name)[source]¶ Gets the bucket-type by the specified name. Bucket-types do not always exist (unlike buckets), but this will always return a
BucketType
object.Parameters: name (str) – the bucket name Return type: BucketType
-
RiakClient.
bucket
(name, bucket_type='default')[source]¶ Get the bucket by the specified name. Since buckets always exist, this will always return a
Bucket
. If you are using a bucket that is contained in a bucket type, it is preferable to access it from the bucket type object:# Preferred: client.bucket_type("foo").bucket("bar") # Equivalent, but not preferred: client.bucket("bar", bucket_type="foo")
Parameters: - name (str) – the bucket name
- bucket_type (
BucketType
or str) – the parent bucket-type
Return type:
Bucket Type Operations¶
-
coroutine
RiakClient.
get_bucket_type_props
(bucket_type)[source]¶ Fetches properties for the given bucket-type.
Parameters: bucket_type (BucketType) – the bucket-type whose properties will be fetched Return type: dict
-
coroutine
RiakClient.
set_bucket_type_props
(bucket_type, props)[source]¶ Sets properties for the given bucket-type.
Parameters: - bucket_type (BucketType) – the bucket-type whose properties will be set
- props (dict) – the properties to set
Bucket Operations¶
-
coroutine
RiakClient.
get_bucket_props
(bucket)[source]¶ Fetches bucket properties for the given bucket.
Parameters: bucket (Bucket) – the bucket whose properties will be fetched Return type: dict
Key-level Operations¶
-
coroutine
RiakClient.
get
(robj)[source]¶ Fetches the contents of a Riak object.
Parameters: robj (RiakObject) – the object to fetch
-
coroutine
RiakClient.
put
(robj, w=None, dw=None, pw=None, return_body=None, if_none_match=None, timeout=None)[source]¶ Stores an object in the Riak cluster.
Parameters: - robj (RiakObject) – the object to store
- w (integer, string, None) – the write quorum
- dw (integer, string, None) – the durable write quorum
- pw (integer, string, None) – the primary write quorum
- return_body (boolean) – whether to return the resulting object after the write
- if_none_match (boolean) – whether to fail the write if the object exists
- timeout (int) – a timeout value in milliseconds
-
coroutine
RiakClient.
delete
(robj)[source]¶ Deletes an object from Riak.
Parameters: robj (RiakObject) – the object to delete
-
coroutine
RiakClient.
fetch_datatype
(bucket, key)[source]¶ Fetches the value of a Riak Datatype.
Parameters: - bucket (
Bucket
) – the bucket of the datatype, which must belong to aBucketType
- key (string) – the key of the datatype
Return type: - bucket (
Serialization¶
The client supports automatic transformation of Riak responses into
Python types if encoders and decoders are registered for the
media-types. Supported by default are application/json
and
text/plain
.
-
aioriak.client.
default_encoder
(obj)[source]¶ Default encoder for JSON datatypes, which returns UTF-8 encoded json instead of the default bloated backslash u XXXX escaped ASCII strings.
-
RiakClient.
get_encoder
(content_type)[source]¶ Get the encoding function for the provided content type.
Parameters: content_type (str) – the requested media type Return type: function
-
RiakClient.
set_encoder
(content_type, encoder)[source]¶ Set the encoding function for the provided content type.
Parameters: - content_type (str) – the requested media type
- encoder (function) – an encoding function, takes a single object argument and returns encoded data
Buckets & Bucket Types¶
Buckets are both namespaces for the key-value pairs you store in Riak, and containers for properties that apply to that namespace. In older versions of Riak, this was the only logical organization available. Now a higher-level collection called a Bucket Type can group buckets together. They allow for efficiently setting properties on a group of buckets at the same time.
Unlike buckets, Bucket Types must be explicitly created and activated before being used:
riak-admin bucket-type create n_equals_1 '{"props":{"n_val":1}}'
riak-admin bucket-type activate n_equals_1
Bucket Type creation and activation is only supported via the
riak-admin bucket-type
command-line tool. Riak 2.0 does not
include an API to perform these actions, but the Python client can
retrieve
and set
bucket-type properties.
If Bucket Types are not specified, the default bucket
type is used. These buckets should be created via the bucket()
method on the client object, like so:
import aioriak
async def go():
client = await aioriak.RiakClient.create()
mybucket = client.bucket('mybucket')
Buckets with a user-specified Bucket Type can also be created via the same
bucket()
method with
an additional parameter or explicitly via
bucket_type()
:
othertype = client.bucket_type('othertype')
otherbucket = othertype.bucket('otherbucket')
# Alternate way to get a bucket within a bucket-type
mybucket = client.bucket('mybucket', bucket_type='mybuckettype')
For more detailed discussion, see Using Bucket Types.
Bucket objects¶
-
class
aioriak.bucket.
Bucket
(client, name, bucket_type)[source]¶ The
Bucket
object allows you to access and change information about a Riak bucket, and provides async methods to create or retrieve objects within the bucket.-
__init__
(client, name, bucket_type)[source]¶ Returns a new
Bucket
instance.Parameters: - client (
RiakClient
) – ARiakClient
instance - name (string) – The bucket name
- bucket_type (
BucketType
) – The parent bucket type of this bucket
- client (
-
name
¶ The name of the bucket, a string.
-
bucket_type
¶ The parent
BucketType
for the bucket.
-
resolver
¶ The sibling-resolution function for this bucket. If the resolver is not set, the client’s resolver will be used.
-
Bucket properties¶
Bucket properties are flags and defaults that apply to all keys in the bucket.
-
coroutine
Bucket.
get_properties
()[source]¶ Retrieve a dict of all bucket properties.
Return type: dict
-
coroutine
Bucket.
set_properties
(props)[source]¶ Set multiple bucket properties in one call.
Parameters: props (dict) – A dictionary of properties
-
coroutine
Bucket.
get_property
(key)[source]¶ Retrieve a bucket property.
Parameters: key (string) – The property to retrieve. Return type: mixed
-
coroutine
Bucket.
set_property
(key, value)[source]¶ Set a bucket property.
Parameters: - key (string) – Property to set.
- value (mixed) – Property value.
Shortcuts for common properties¶
Some of the most commonly-used bucket properties are exposed as object
properties as well. The getters and setters simply call
Bucket.get_property()
and Bucket.set_property()
respectively.
-
Bucket.
allow_mult
¶ If set to True, then writes with conflicting data will be stored and returned to the client. :type bool: boolean
Working with keys¶
The primary purpose of buckets is to act as namespaces for keys. As
such, you can use the bucket object to create, fetch and delete
objects
.
-
coroutine
Bucket.
new
(key=None, data=None, content_type='application/json', encoded_data=None)[source]¶ A shortcut for manually instantiating a new
RiakObject
or a newDatatype
, based on the presence and value of thedatatype
bucket property. When the bucket contains aDatatype
, all arguments are ignored exceptkey
, otherwise they are used to initialize theRiakObject
.Parameters: - key (str) – Name of the key. Leaving this to be None (default) will make Riak generate the key on store.
- data (object) – The data to store in a
RiakObject
, seeRiakObject.data
. - content_type (str) – The media type of the data stored in the
RiakObject
, seeRiakObject.content_type
. - encoded_data (str) – The encoded data to store in a
RiakObject
, seeRiakObject.encoded_data
.
Return type:
-
coroutine
Bucket.
get
(key)[source]¶ Retrieve an
RiakObject
orDatatype
, based on the presence and value of thedatatype
bucket property.Parameters: key (string) – Name of the key. Return type: RiakObject
orDatatype
-
coroutine
Bucket.
delete
(key, **kwargs)[source]¶ Deletes a key from Riak. Short hand for
bucket.new(key).delete()
. SeeRiakClient.delete()
for options.Parameters: key (string) – The key for the object Return type: RiakObject
Serialization¶
Similar to RiakClient
, buckets can
register custom transformation functions for media-types. When
undefined on the bucket, Bucket.get_encoder()
and
Bucket.get_decoder()
will delegate to the client associated
with the bucket.
-
Bucket.
get_encoder
(content_type)[source]¶ Get the encoding function for the provided content type for this bucket.
Parameters: - content_type (str) – the requested media type
- content_type – Content type requested
-
Bucket.
set_encoder
(content_type, encoder)[source]¶ Set the encoding function for the provided content type for this bucket.
Parameters: - content_type (str) – the requested media type
- encoder (function) – an encoding function, takes a single object argument and returns a string data as single argument.
Listing keys¶
Shortcuts for RiakClient.get_keys()
are exposed on the bucket
object. The same admonitions for these operation apply.
Bucket Type objects¶
-
class
aioriak.bucket.
BucketType
(client, name)[source]¶ The
BucketType
object allows you to access and change properties on a Riak bucket type and access buckets within its namespace.Async implementation of riak.bucket.BucketType
-
__init__
(client, name)[source]¶ Returns a new
BucketType
instance.Parameters: - client (
RiakClient
) – ARiakClient
instance - name (string) – The bucket-type’s name
- client (
-
name
¶ The name of the Bucket Type, a string.
-
Bucket Type properties¶
Bucket Type properties are flags and defaults that apply to all buckets in the Bucket Type.
-
coroutine
BucketType.
get_properties
()[source]¶ Retrieve a dict of all bucket-type properties.
Return type: dict
-
coroutine
BucketType.
set_properties
(props)[source]¶ Set multiple bucket-type properties in one call.
Parameters: props (dict) – A dictionary of properties
-
coroutine
BucketType.
get_property
(key)[source]¶ Retrieve a bucket-type property.
Parameters: key (string) – The property to retrieve. Return type: mixed
-
coroutine
BucketType.
set_property
(key, value)[source]¶ Set a bucket-type property.
Parameters: - key (string) – Property to set.
- value (mixed) – Property value.
-
BucketType.
datatype
¶ The assigned datatype for this bucket type, if present.
Listing buckets¶
Shortcut for RiakClient.get_buckets()
is exposed on the bucket
type object. This is similar to Listing keys on buckets.
Values & Objects¶
Keys in Riak are namespaced into buckets
, and their associated values are represented
by objects
, not to be confused with Python
“objects”. A aioriak.riak_object.RiakObject
is a container for the key, the
Vector clock, the value(s) and any metadata associated with the
value(s).
Values may also be datatypes
, but
are not discussed here.
RiakObject¶
-
class
aioriak.riak_object.
RiakObject
(client, bucket, key=None)[source]¶ The RiakObject holds meta information about a Riak object, plus the object’s data.
-
key
¶ The key of this object, a string. If not present, the server will generate a key the first time this object is stored.
-
resolver
¶ The sibling-resolution function for this object. If the resolver is not set, the bucket’s resolver will be used.
-
vclock
¶ The Vector clock for this object.
-
exists
¶ Whether the object exists. This is only
False
when there are no siblings (the object was not found), or the solitary sibling is a tombstone.
-
Vector clock¶
Vector clocks are Riak’s means of tracking the relationships between writes to a key. It is best practice to fetch the latest version of a key before attempting to modify or overwrite the value; if you do not, you may create Siblings or lose data! The content of a vector clock is essentially opaque to the user.
Persistence¶
Fetching, storing, and deleting keys are the bread-and-butter of Riak.
-
coroutine
RiakObject.
store
(w=None, dw=None, pw=None, return_body=True, if_none_match=False, timeout=None)[source]¶ Store the object in Riak. When this operation completes, the object could contain new metadata and possibly new data if Riak contains a newer version of the object according to the object’s vector clock.
Parameters: return_body (bool) – if the newly stored object should be retrieved Return type: RiakObject
-
coroutine
RiakObject.
reload
()[source]¶ Reload the object from Riak. When this operation completes, the object could contain new metadata and a new value, if the object was updated in Riak since it was last retrieved.
Note
Even if the key is not found in Riak, this will return a
RiakObject
. Check theexists
property to see if the key was found.Return type: RiakObject
-
coroutine
RiakObject.
delete
()[source]¶ Delete this object from Riak.
Return type: RiakObject
Value and Metadata¶
Unless you have enabled Siblings via the allow_mult
bucket property, you can
inspect and manipulate the value and metadata of an object directly using these
properties and methods:
-
RiakObject.
data
¶ The data stored in this object, as Python objects. For the raw data, use the encoded_data property. If unset, accessing this property will result in decoding the encoded_data property into Python values. The decoding is dependent on the content_type property and the bucket’s registered decoders.
-
RiakObject.
encoded_data
¶ The raw data stored in this object, essentially the encoded form of the data property. If unset, accessing this property will result in encoding the data property into a string. The encoding is dependent on the content_type property and the bucket’s registered encoders.
-
RiakObject.
content_type
¶ The MIME media type of the encoded data as a string
-
RiakObject.
charset
¶ The character set of the encoded data as a string
-
RiakObject.
content_encoding
¶ The encoding (compression) of the encoded data. Valid values are identity, deflate, gzip
-
RiakObject.
usermeta
¶ Arbitrary user-defined metadata dict, mapping strings to strings.
-
RiakObject.
links
¶ A set of bucket/key/tag 3-tuples representing links to other keys.
-
RiakObject.
indexes
¶ The set of secondary index entries, consisting of index-name/value tuples
Siblings¶
Because Riak’s consistency model is “eventual” (and not linearizable), there is no way for it to disambiguate writes that happen concurrently. The Vector clock helps establish a “happens after” relationships so that concurrent writes can be detected, but with the exception of Data Types, Riak has no way to determine which write has the correct value.
Instead, when allow_mult
is True
, Riak keeps all writes that appear to be concurrent. Thus,
the contents of a key’s value may, in fact, be multiple values, which
are called “siblings”. Siblings are modeled in RiakContent
objects, which contain all of the same
Persistence methods and attributes as the parent object.
-
RiakObject.
siblings
= []¶
-
class
aioriak.content.
RiakContent
(robject, data=None, encoded_data=None, charset=None, content_type='application/json', content_encoding=None, last_modified=None, etag=None, usermeta=None, links=None, indexes=None, exists=False)[source]¶ The RiakContent holds the metadata and value of a single sibling within a RiakObject. RiakObjects that have more than one sibling are considered to be in conflict.
You do not typically have to create RiakContent
objects yourself, but they will be created
for you when fetching
objects from Riak.
Note
The Persistence accessors on RiakObject
are actually proxied to the first sibling when the object has only
one.
Conflicts and Resolvers¶
When an object is not in conflict, it has only one sibling. When it
is in conflict, you will have to resolve the conflict before it can be
written again. How you choose to resolve the conflict is up to you,
but you can automate the process using a resolver
function.
-
aioriak.resolver.
default_resolver
(riak_object)[source]¶ The default conflict-resolution function, which does nothing. To implement a resolver, define a function that sets the
siblings
property on the passedRiakObject
instance to a list containing a singleRiakContent
object.Parameters: riak_object ( RiakObject
) – an object-in-conflict that will be resolved
-
aioriak.resolver.
last_written_resolver
(riak_object)[source]¶ A conflict-resolution function that resolves by selecting the most recently-modified sibling by timestamp.
Parameters: riak_object ( RiakObject
) – an object-in-conflict that will be resolved
If you do not supply a resolver function, or your resolver leaves
multiple siblings present, accessing the Persistence will
result in a ConflictError
being raised.
-
exception
aioriak.error.
ConflictError
(message='Object in conflict')[source]¶ Raised when an operation is attempted on a
RiakObject
that has more than one sibling.
Data Types¶
Traditionally all data stored in Riak was an opaque binary type. Then in version 1.4 came the introduction of a counter, the first Convergent Data Type supported in Riak. In Riak 2.0, several additional Data Types were introduced. Riak “knows” about these data types, and conflicting writes to them will converge automatically without presenting sibling values to the user.
Here is the list of current Data Types:
Counter
increments or decrements integer valuesSet
allows you to store multiple distinct opaque binary values against a keyMap
is a nested, recursive struct, or associative array. Think of it as a container for composing ad hoc data structures from multiple Data Types. Inside a map you may store sets, counters, flags, registers, and even other mapsRegister
stores binaries accoring to last-write-wins logic withinMap
Flag
is similar to a boolean and also must be withinMap
All Data Types must be stored in buckets bearing a
BucketType
that sets the
datatype
property to one of
"counter"
, "set"
, or "map"
. Note that the bucket must have
the allow_mult
property set to true
.
These Data Types are stored just like RiakObjects
, so size constraints that apply to
normal Riak values apply to Riak Data Types too.
An in-depth discussion of Data Types, also known as CRDTs, can be found at Data Types.
Examples of using Data Types can be found at Using Data Types.
Sending Operations¶
Riak Data Types provide a further departure from Riak’s usual operation, in that the API is operation-based. Rather than fetching the data structure, reconciling conflicts, mutating the result, and writing it back, you instead tell Riak what operations to perform on the Data Type. Here are some example operations:
Datatypes can be fetched and created just like
RiakObject
instances, using
Bucket.get
and
Bucket.new
, except that the
bucket must belong to a bucket-type that has a valid datatype
property. If we have a bucket-type named “social-graph” that has the
datatype “set”, we would fetch a Set
like so:
graph = client.bucket_type('social-graph')
graph.datatype # => 'set'
backet = graph.bucket('followers')
myfollowers = await bucket.get('seancribbs')
# => a Set datatype
Once we have a datatype, we can stage operations against it and then send those operations to Riak:
myfollowers.add('javajolt')
myfollowers.discard('roach')
await myfollowers.update()
While this looks in code very similar to manipulating
RiakObject
instances, only mutations are
enqueued locally, not the new value.
Context and Observed-Remove¶
In order for Riak Data Types to behave well, you must have an opaque context received from a read when you:
The basic rule is “you cannot remove something you haven’t seen”, and
the context tells Riak what you’ve actually seen, similar to the
Vector clock on RiakObject
. The Python
client handles opaque contexts for you transparently as long as you
fetch before performing one of these actions.
Datatype abstract class¶
-
class
aioriak.datatypes.
Datatype
(bucket=None, key=None, value=None, context=None)[source]¶ Base class for all convergent datatype wrappers. You will not use this class directly, but it does define some methods are common to all datatype wrappers.
-
value
¶ The pure, immutable value of this datatype, as a Python value, which is unique for each datatype. NB: Do not use this property to mutate data, as it will not have any effect. Use the methods of the individual type to effect changes. This value is guaranteed to be independent of any internal data representation.
-
context
¶ The opaque context for this type, if it was previously fetched.
Return type: str
-
modified
¶ Whether this datatype has staged local modifications.
Return type: bool
-
Counter¶
-
class
aioriak.datatypes.
Counter
(bucket=None, key=None, value=None, context=None)[source]¶ A convergent datatype that represents a counter which can be incremented or decremented. This type can stand on its own or be embedded within a
Map
.
-
Counter.
value
¶ The current value of the counter.
Set¶
-
class
aioriak.datatypes.
Set
(bucket=None, key=None, value=None, context=None)[source]¶ A convergent datatype representing a Set with observed-remove semantics. Currently strings are the only supported value type.
- Example::
- myset.add(‘barista’) myset.add(‘roaster’) myset.add(‘brewer’)
- Likewise they can simply be removed::
- myset.discard(‘barista’)
This datatype also implements the Set ABC, meaning it supports
len()
,in
, and iteration.
-
Set.
value
¶ An immutable copy (frozenset) of the current value of the set.
Map¶
-
class
aioriak.datatypes.
Map
(bucket=None, key=None, value=None, context=None)[source]¶ A convergent datatype that acts as a key-value datastructure. Keys are pairs of
(name, datatype)
wherename
is a string anddatatype
is the datatype name. Values are other convergent datatypes, represented by any concrete type in this module.You cannot set values in the map directly (it does not implement
__setitem__
), but you may add new empty values or access non-existing values directly via bracket syntax. If a key is not in the original value of the map when accessed, fetching the key will cause its associated value to be created.:map[('name', 'register')]
Keys and their associated values may be deleted from the map as you would in a dict:
del map[('emails', 'set')]
Convenience accessors exist that partition the map’s keys by datatype and implement the collections.Mapping behavior as well as supporting deletion:
map.sets['emails'] map.registers['name'] del map.counters['likes']
-
Map.
value
¶ Returns a copy of the original map’s value. Nested values are pure Python values as returned by
Datatype.value
from the nested types.Return type: dict
-
Map.
counters
¶ Filters keys in the map to only those of counter types. Example:
map.counters['views'].increment() del map.counters['points']
-
Map.
flags
¶ Filters keys in the map to only those of flag types. Example:
map.flags['confirmed'].enable() del map.flags['attending']
-
Map.
maps
¶ Filters keys in the map to only those of map types. Example:
map.maps['emails'].registers['home'].set("user@example.com") del map.maps['spam']
-
Map.
registers
¶ Filters keys in the map to only those of register types. Example:
map.registers['username'].set_value("riak-user") del map.registers['access_key']
-
Map.
sets
¶ Filters keys in the map to only those of set types. Example:
map.sets['friends'].add("brett") del map.sets['favorites']
Map-only datatypes¶
Two of the new Data Types may only be embedded in
Map
objects (in addition to
Map
itself):
Register¶
-
class
aioriak.datatypes.
Register
(bucket=None, key=None, value=None, context=None)[source]¶ A convergent datatype that represents an opaque string that is set with last-write-wins semantics, and may only be embedded in
Map
instances.
-
Register.
value
¶ Returns a copy of the original value of the register.
Return type: str
# query # security # advanced