Welcome to aioriak’s documentation!¶
Asyncio (PEP 3156) Riak client library. This project is based on official Bash python client library (https://github.com/basho/riak-python-client).
Features¶
Riak KV operations | Yes |
Riak Datatypes | Yes |
Riak BucketTypes | Yes |
Custom resolver | Yes |
Node list support | No |
Custom quorum | No |
Connections Pool | No |
Operations timout | No |
Security | No |
Riak Search | No |
MapReduce | No |
Tested python versions | 3.5.0, 3.5.1 |
Tested Riak versions | 2.1.3 |
Installation¶
The easiest way to install aioriak is by using the package on PyPi:
pip install aioriak
Requirements¶
- Python >= 3.5
- riak>=2.1.3
Contribute¶
- Issue Tracker: https://github.com/rambler-digital-solutions/aioriak/issues
- Source Code: https://github.com/rambler-digital-solutions/aioriak
Feel free to file an issue or make pull request if you find any bugs or have some suggestions for library improvement.
License¶
The aioriak is offered under MIT license.
Contents¶
Client & Connections¶
To connect to a Riak cluster, you must create a
RiakClient
object. The default configuration
connects to a Riak node on localhost
with the default
ports. The below instantiation statements are all equivalent:
from aioriak import RiakClient
client = RiakClient()
async def go():
client = await RiakClient.create(host='127.0.0.1', port=8087)
Note
Connections are not established until you attempt to perform an operation. If the host or port are incorrect, you will not get an error raised immediately.
Client objects¶
Client-level Operations¶
Some operations are not scoped by buckets or bucket types and can be performed on the client directly:
Accessing Bucket Types and Buckets¶
Most client operations are on bucket type objects
, the bucket objects
they contain or keys within those buckets. Use the
bucket_type
or bucket
methods for creating bucket types and buckets
that will proxy operations to the called client.
Bucket Type Operations¶
Bucket Operations¶
Key-level Operations¶
Serialization¶
The client supports automatic transformation of Riak responses into
Python types if encoders and decoders are registered for the
media-types. Supported by default are application/json
and
text/plain
.
Buckets & Bucket Types¶
Buckets are both namespaces for the key-value pairs you store in Riak, and containers for properties that apply to that namespace. In older versions of Riak, this was the only logical organization available. Now a higher-level collection called a Bucket Type can group buckets together. They allow for efficiently setting properties on a group of buckets at the same time.
Unlike buckets, Bucket Types must be explicitly created and activated before being used:
riak-admin bucket-type create n_equals_1 '{"props":{"n_val":1}}'
riak-admin bucket-type activate n_equals_1
Bucket Type creation and activation is only supported via the
riak-admin bucket-type
command-line tool. Riak 2.0 does not
include an API to perform these actions, but the Python client can
retrieve
and set
bucket-type properties.
If Bucket Types are not specified, the default bucket
type is used. These buckets should be created via the bucket()
method on the client object, like so:
import riak
async def go():
client = await riak.RiakClient.create()
mybucket = client.bucket('mybucket')
Buckets with a user-specified Bucket Type can also be created via the same
bucket()
method with
an additional parameter or explicitly via
bucket_type()
:
othertype = client.bucket_type('othertype')
otherbucket = othertype.bucket('otherbucket')
# Alternate way to get a bucket within a bucket-type
mybucket = client.bucket('mybucket', bucket_type='mybuckettype')
For more detailed discussion, see Using Bucket Types.
Bucket objects¶
Bucket properties¶
Bucket properties are flags and defaults that apply to all keys in the bucket.
Shortcuts for common properties¶
Some of the most commonly-used bucket properties are exposed as object
properties as well. The getters and setters simply call
Bucket.get_property()
and Bucket.set_property()
respectively.
Working with keys¶
The primary purpose of buckets is to act as namespaces for keys. As
such, you can use the bucket object to create, fetch and delete
objects
.
Serialization¶
Similar to RiakClient
, buckets can
register custom transformation functions for media-types. When
undefined on the bucket, Bucket.get_encoder()
and
Bucket.get_decoder()
will delegate to the client associated
with the bucket.
Listing keys¶
Shortcuts for RiakClient.get_keys()
are exposed on the bucket
object. The same admonitions for these operation apply.
Bucket Type objects¶
Bucket Type properties¶
Bucket Type properties are flags and defaults that apply to all buckets in the Bucket Type.
-
BucketType.
datatype
¶ The assigned datatype for this bucket type, if present.
Listing buckets¶
Shortcut for RiakClient.get_buckets()
is exposed on the bucket
type object. This is similar to Listing keys on buckets.
Values & Objects¶
Keys in Riak are namespaced into buckets
, and their associated values are represented
by objects
, not to be confused with Python
“objects”. A RiakObject
is a container for the key, the
Vector clock, the value(s) and any metadata associated with the
value(s).
Values may also be datatypes
, but
are not discussed here.
RiakObject¶
Vector clock¶
Vector clocks are Riak’s means of tracking the relationships between writes to a key. It is best practice to fetch the latest version of a key before attempting to modify or overwrite the value; if you do not, you may create Siblings or lose data! The content of a vector clock is essentially opaque to the user.
Persistence¶
Fetching, storing, and deleting keys are the bread-and-butter of Riak.
Value and Metadata¶
Unless you have enabled Siblings via the allow_mult
bucket property, you can
inspect and manipulate the value and metadata of an object directly using these
properties and methods:
Siblings¶
Because Riak’s consistency model is “eventual” (and not linearizable), there is no way for it to disambiguate writes that happen concurrently. The Vector clock helps establish a “happens after” relationships so that concurrent writes can be detected, but with the exception of Data Types, Riak has no way to determine which write has the correct value.
Instead, when allow_mult
is True
, Riak keeps all writes that appear to be concurrent. Thus,
the contents of a key’s value may, in fact, be multiple values, which
are called “siblings”. Siblings are modeled in RiakContent
objects, which contain all of the same
Persistence methods and attributes as the parent object.
You do not typically have to create RiakContent
objects yourself, but they will be created
for you when fetching
objects from Riak.
Note
The Persistence accessors on RiakObject
are actually proxied to the first sibling when the object has only
one.
Conflicts and Resolvers¶
When an object is not in conflict, it has only one sibling. When it
is in conflict, you will have to resolve the conflict before it can be
written again. How you choose to resolve the conflict is up to you,
but you can automate the process using a resolver
function.
-
riak.resolver.
default_resolver
(riak_object)¶ The default conflict-resolution function, which does nothing. To implement a resolver, define a function that sets the
siblings
property on the passedRiakObject
instance to a list containing a singleRiakContent
object.Parameters: riak_object ( RiakObject
) – an object-in-conflict that will be resolved
-
riak.resolver.
last_written_resolver
(riak_object)¶ A conflict-resolution function that resolves by selecting the most recently-modified sibling by timestamp.
Parameters: riak_object ( RiakObject
) – an object-in-conflict that will be resolved
If you do not supply a resolver function, or your resolver leaves
multiple siblings present, accessing the Persistence will
result in a ConflictError
being raised.
Data Types¶
Traditionally all data stored in Riak was an opaque binary type. Then in version 1.4 came the introduction of a counter, the first Convergent Data Type supported in Riak. In Riak 2.0, several additional Data Types were introduced. Riak “knows” about these data types, and conflicting writes to them will converge automatically without presenting sibling values to the user.
Here is the list of current Data Types:
Counter
increments or decrements integer valuesSet
allows you to store multiple distinct opaque binary values against a keyMap
is a nested, recursive struct, or associative array. Think of it as a container for composing ad hoc data structures from multiple Data Types. Inside a map you may store sets, counters, flags, registers, and even other mapsRegister
stores binaries accoring to last-write-wins logic withinMap
Flag
is similar to a boolean and also must be withinMap
All Data Types must be stored in buckets bearing a
BucketType
that sets the
datatype
property to one of
"counter"
, "set"
, or "map"
. Note that the bucket must have
the allow_mult
property set to true
.
These Data Types are stored just like RiakObjects
, so size constraints that apply to
normal Riak values apply to Riak Data Types too.
An in-depth discussion of Data Types, also known as CRDTs, can be found at Data Types.
Examples of using Data Types can be found at Using Data Types.
Sending Operations¶
Riak Data Types provide a further departure from Riak’s usual operation, in that the API is operation-based. Rather than fetching the data structure, reconciling conflicts, mutating the result, and writing it back, you instead tell Riak what operations to perform on the Data Type. Here are some example operations:
- increment a
Counter
by10
- add
'joe'
to aSet
- remove the
Set
field called'friends'
from aMap
- enable the prepay
Flag
in aMap
Datatypes can be fetched and created just like
RiakObject
instances, using
Bucket.get
and
Bucket.new
, except that the
bucket must belong to a bucket-type that has a valid datatype
property. If we have a bucket-type named “social-graph” that has the
datatype “set”, we would fetch a Set
like so:
graph = client.bucket_type('social-graph')
graph.datatype # => 'set'
backet = graph.bucket('followers')
myfollowers = await bucket.get('seancribbs')
# => a Set datatype
Once we have a datatype, we can stage operations against it and then send those operations to Riak:
myfollowers.add('javajolt')
myfollowers.discard('roach')
await myfollowers.update()
While this looks in code very similar to manipulating
RiakObject
instances, only mutations are
enqueued locally, not the new value.
Context and Observed-Remove¶
In order for Riak Data Types to behave well, you must have an opaque context received from a read when you:
disable
aFlag
(set it tofalse
)- remove a field from a
Map
remove
an element from aSet
The basic rule is “you cannot remove something you haven’t seen”, and
the context tells Riak what you’ve actually seen, similar to the
Vector clock on RiakObject
. The Python
client handles opaque contexts for you transparently as long as you
fetch before performing one of these actions.
Datatype abstract class¶
Map¶
-
Map.
counters
¶ Filters keys in the map to only those of counter types. Example:
map.counters['views'].increment() del map.counters['points']
-
Map.
flags
¶ Filters keys in the map to only those of flag types. Example:
map.flags['confirmed'].enable() del map.flags['attending']
-
Map.
maps
¶ Filters keys in the map to only those of map types. Example:
map.maps['emails'].registers['home'].set("user@example.com") del map.maps['spam']
-
Map.
registers
¶ Filters keys in the map to only those of register types. Example:
map.registers['username'].set_value("riak-user") del map.registers['access_key']
-
Map.
sets
¶ Filters keys in the map to only those of set types. Example:
map.sets['friends'].add("brett") del map.sets['favorites']
Map-only datatypes¶
Two of the new Data Types may only be embedded in
Map
objects (in addition to
Map
itself):
Register¶
# query # security # advanced