Onitu at a glance¶

Onitu must deal with a lot of events coming from different places. Therefore, Onitu is build around an asynchronous model. Each part runs in a separate process, and communicate with the others via ZeroMQ messages and Redis.

In order to synchronize files between external services, Onitu uses a system of drivers. You can find more information about this system in Creating a new driver. Each driver sends events and receives orders from the Referee, which chooses where the files should be synchronised according to the configuration rules.

Glossary¶

Driver

A program making the junction between Onitu and a remote service (SSH, Dropbox, a hard drive…). cf Creating a new driver

Entry

A driver configured by the user. For example, it can be the Dropbox driver configured to run with a specific account. You can view an entry as an instance of a driver.

Rule

Maps a set of matching files to a set of entries. Used in the configuration to split up the files.

Setup

A JSON configuration file, which describes the entries and the rules.

Referee

Receive events from the drivers and allocate the files among the entries regarding the configuration rules. cf Referee

Plug

A helper class which implements all the boilerplate needed by a driver to communicate with Onitu. cf Plug

Handler

A function defined by a driver, which responds to a specific task. This function will be called by the Plug when needed. cf Handlers

Global architecture¶

You can here find an illustration of the global architecture of Onitu: Figure 1.1.

A schematic illustrating the global architecture of Onitu with two drivers.

Dependencies¶

The core of Onitu uses several libraries (other dependencies exists but are specific to some drivers):

Circus

Used to start, manage and monitor the different processes

PyZMQ

A binding of the ZeroMQ library for the Python language

redis-py

A library to communicate with Redis in Python

Logbook

A powerful logging library that allows Onitu to log on a ZeroMQ channel, aggregating all the logs in the same place

Technical choices¶

We made some difficult technical choices in order to build Onitu in the most maintainable and efficient way. Those choices can be questionable, but here are our motivations :

Layout¶

A configuration file is mainly concerned with Folders and Services.

Folders are named directories that Onitu knows of and are considered roots of their respective hierarchies. Each folder will be kept syncronised across the different places it is mapped.

Services are configured drivers. A driver can be configured several times, thus producing more than one service. For exemple one could have two services, each controling a different remote server using the same driver.

The configuration file is meant to configure the services, map folders to services and control the way they are syncronised. All folders are first listed in the configuration and folder options can be used to specify how exactly they should be treated. Then all services are listed with their configuration. For each one of them the folders that should be mapped on this service are also listed. Additional options can be specified for each folder in a service. Those are the same than those used on folders or services but are used when when a more specific configuration is desiredon on per service-folder basis.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33

# This partial configuration file shows the main layout.
# Two folders are defined and two services are configured.
# One cares about both folders, the other one only about the first.

name: "Configuration Layout"

folders:
  folder1:
    # Folder options
  folder2:
    # Folder options

services:

  serviceA:
    driver: "some_driver"
    # Service options

    folders:
      folder1:
        # Service/Folder options
      folder2:
        # Service/Folder options

  serviceB:
    driver: "some_other_driver"
    # Service options

    folders:
      folder1:
        # Service/Folder options

#EOF

Folder options¶

mode

values:	“r”, “w”, “rw”
default:	“rw”
what:	This value indicates if the folder should only be read-from, written-to or if it should be syncronised both ways. You probably want to specify this at the service level and not at the folder level. r (read): If not specified new files or changes made in this folder will not be taken into account. You must specify this if you want Onitu to be able to read content from this folder. w (write): If not specified new files or changes will not be written to this folder. You must specify this if you want Onitu to make changes to this folder when syncronising.

type

values:	To be defined. One or more media types as defined in rfc6838. Incomplete types such as “example” instead of “example/media_type” should be accepted.
default:	If not specified all types will be accepted.
what:	This value restricts which type(s) of file should be accepted in this folder. Files not conforming to this value will never be read from or written to this folder.

min-size, max-size

values:	A numeric value with an optional multiplying sufix. Metric prefixes (k, M, G, T, P) and IEC prefixes (Ki, Mi, Gi, Ti, Pi) are accepted.
default:	If min-size is not specified there is no limit on how big a file in this folder must be. If max-siz is not specified there is no limit on how big a file in this folder can be.
what:	This value restricts which files should be accepted in this folder based on their size. Files not within the range specified by min-size and max-size will never be read from or written to this folder.

blacklist, whitelist

values:	A list of paths, shell-like globbing is allowed.
default:	By default the blacklist is empty and the whitelist isn’t used if not specified or empty. (To disable syncronisation completly you should use the folder’s mode.)
what:	Only files matching a pattern in the whitelist will be accepted in this folder. Files matching a pattern in the blacklist will never be accepted.

Example configuration¶

This are sample configuration using some of the folder options above and some simple service options. More information about service options can be found in the section bellow. Those sample files are to give an idea of how folder options can be used to achieve different kinds of synchronisation, they are not about illustrating the different service options.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

# Sends completed torrents from server to a NAS.
# Service options will be omitted if not relevant.

name: "Example Configuration"

folders:
  torrent:

services:

  localhost_fs:
    driver: "local_storage"
    # Service options.
    folders:
      torrent:
        mode: "r"
        # Service/Folder options.

  nas:
    driver: "ssh"
    # Service options.
    folders:
      torrent:
        mode: "w"
        # Service/Folder options.

#EOF

Service options¶

Service options are specific to each driver. This is because different drivers need to know different things to be able to handle their backends. This is a list of options for each driver.

TODO: Describe options for each driver.

Basics¶

The drivers communicate with Onitu via the Plug class, which handles the operations common to all drivers. Each driver implements its specific tasks with the system of handlers. Those handlers will be called by the Plug at certain occasions.

In Onitu the file transfers can be made by chunks. When a new transfer begin, the Plug asks the others drivers for new chunks, and then call the upload_chunk handler. The transfers can also be made via the upload_file handler, which upload the full content of the file. Both protocols can be used together.

Each driver must expose a function called start and an instance of the Plug in their __init__.py file. This start function will be called by Onitu during the initialization of the driver, and should not return until the end of life of the driver (cf Plug.listen()).

When a driver detects an update in a file, it should update the Metadata of the file, specify a Metadata.revision, and call Plug.update_file().

Each driver must have a manifest describing its purpose and its options.

Onitu provide a set of functional tests that you can use to see if your driver respond to every exigence.

Handlers¶

A handler is a function that will be called by the Plug on different occasions, such as getting a chunk from a file or starting a transfer. The drivers can define any handler they need. For example, some driver don’t need to do anything for initiating a transfer, so they might not want to implement the end_upload handler. In order to register a handler, the Plug.handler() decorator is used.

At this stage, the list of the handlers that can be defined is the following :

get_chunk(metadata, offset, size)¶

Return a chunk of a given size, starting at the given offset, from a file.

Parameters:	metadata (Metadata) – The metadata of the file offset (int) – The offset from which the content should be retrieved size (int) – The maximum size of the chunk that should be returned
Return type:	string

get_file(metadata)¶

Return the full content of a file.

Parameters:	metadata (Metadata) – The metadata of the file
Return type:	string

upload_chunk(metadata, offset, chunk)¶

Write a chunk in a file at a given offset.

Parameters:	metadata (Metadata) – The metadata of the file offset (int) – The offset from which the content should be written chunk (string) – The content that should be written

upload_file(metadata, content)¶

Write the full content of a file.

Parameters:	metadata (Metadata) – The metadata of the file content – The content of the file

set_chunk_size(chunk_size)¶

Allows a driver to force a chunk size by overriding the default, or provided, value. The handler takes the plug chunk size as argument, and if that size is invalid for the driver, it can return a new value. Useful for services that require a minimum size for transfers.

Parameters:	chunk_size (int) – the size the plug is currently using

start_upload(metadata)¶

Initialize a new upload. This handler is called when a new transfer is started.

Parameters:	metadata (Metadata) – The metadata of the file transferred

restart_upload(metadata, offset)¶

Restart a failed upload. This handler will be called during the startup if a transfer has been stopped.

Parameters:	metadata (Metadata) – The metadata of the file transferred offset (int) – The offset of the last chunk uploaded

end_upload(metadata)¶

Called when a transfer is over.

Parameters:	metadata (Metadata) – The metadata of the file transferred

abort_upload(metadata)¶

Called when a transfer is aborted. For example, this could happen if a newer version of the file should be uploaded during the transfer.

Parameters:	metadata (Metadata) – The metadata of the file transferred

close()¶

Called when Onitu is closing. This gives a chance to the driver to clean its resources. Note that it is called from a sighandler, so some external functionalities might not work as expected. This handler should not take too long to complete or it could cause perturbations.

The Plug¶

Metadata¶

Exceptions¶

If an error happen in a driver, it should raise an appropriate exception. Two exceptions are handled by the Plug, and should be used accordingly to the situation : DriverError and ServiceError.

Manifest¶

A manifest is a JSON file describing a driver in order to help the users configuring it. It contains several informations, such as the name of the driver, its description, and its available options. Each option must have a name, a description and a type.

The type of the options will be used by Onitu to validate them, and by the interface in order to offer a proper input field. The available types are : Integers, Floats, Booleans, Strings and Enumerates. An enumerate type must add a values field with the list of all the possible values.

An option can have a default field which represents the default value (it can be null). If this field is present, the option is not mandatory. All the options without a default value are mandatory.

Here is an example of what your manifest should look like :

{
  "name": "Your driver's name",
  "description": "The description of your driver.",
  "options": {
    "my_option": {
      "name": "Option's name",
      "description": "The description of the option",
      "type": "string",
    },
    "another_option": {
      "name": "Another option",
      "description": "The description of the option",
      "type": "enumerate",
      "values": ["foo", "bar"],
      "default": "foo"
    },
  }
}

Example¶

Usually, the drivers are created as a set of functions in a single file, with the Plug in a global variable. However, you can use a different style if you want, such as a class.

Here is an example of a simple driver working with the local file system :

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75

import os

from onitu.api import Plug, ServiceError, DriverError

# A dummy library supposed to watch the file system
from fsmonitor import FSWatcher

plug = Plug()


@plug.handler()
def get_chunk(metadata, offset, size):
    try:
        with open(metadata.filename, 'rb') as f:
            f.seek(offset)
            return f.read(size)
    except IOError as e:
        raise ServiceError(
            "Error reading '{}': {}".format(metadata.filename, e)
        )


@plug.handler()
def upload_chunk(metadata, offset, chunk):
    try:
        with open(metadata.filename, 'r+b') as f:
            f.seek(offset)
            f.write(chunk)
    except IOError as e:
        raise ServiceError(
            "Error writting '{}': {}".format(metadata.filename, e)
        )


@plug.handler()
def end_upload(metadata):
    metadata.revision = os.path.getmtime(metadata.filename)
    metadata.write_revision()


class Watcher(FSWatcher):
    def on_update(self, filename):
        """Called each time an update of a file is detected
        """
        metadata = plug.get_metadata(filename)
        metadata.revision = os.path.getmtime(metadata.filename)
        metadata.size = os.path.getsize(metadata.filename)
        plug.update_file(metadata)

    def check_changes(self):
        """Check the changes on the file system since the last launch
        """
        for filename in self.files:
            revision = os.path.getmtime(filename)
            metadata = plug.get_metadata(filename)

            # If the file is more recent
            if revision > metadata.revision:
                metadata.revision = os.path.getmtime(metadata.filename)
                metadata.size = os.path.getsize(metadata.filename)
                plug.update_file(metadata)


def start():
    try:
        root = plug.options['root']
        os.chdir(root)
    except OSError as e:
        raise DriverError("Can't access '{}': {}".format(root, e))

    watcher = Watcher(root)
    watcher.check_changes()
    watcher.start()

    plug.listen()

This is what a driver’s __init__.py file should look like:

1
2
3

from .driver import start, plug

__all__ = ["start", "plug"]

Launcher¶

Referee¶

The role of the Referee is to receive the events emitted by the drivers, and to send notifications to the other drivers accordingly to the configuration rules.

Utils¶

This module provides a set of classes and functions useful in several parts of Onitu.

onitu.utils.at_exit(callback, *args, **kwargs)[source]¶

Register a callback which will be called when a deadly signal comes

This funtion must be called from the main thread.

onitu.utils.get_available_drivers()[source]¶

Return a dict mapping the name of each installed driver with its entry point.

You can use it like that: ``` drivers = get_available_drivers() if ‘local_storage’ in drivers:

local_storage = drivers[‘local_storage’].load()

```

onitu.utils.get_fid(filename)[source]¶

Get the file-id (fid) of the given filename.

The file-id is a UUID version 5, with the namespace define in NAMESPACE_ONITU.

The purpose of the file-id is to avoid using filenames as a direct references to files inside Onitu.

onitu.utils.get_mimetype(filename)[source]¶

Get the MIME type of the given filename.

This avoids interfaces and clients of the Onitu instances having to determine the MIME type of the files they receive notifications from.

Overview¶

The port used by the API is 3862. All data is sent and received as JSON, and using the UTF-8 charset.

Routes¶

Reporting issues¶

When you encounter an issue with Onitu we’dd love to hear about it. Not that we particularly like having problems with the codebase, but its better to fix them than to leave them in there. If you submit a bug report please include all the information available to you, here are some things you can do:

• If the problem is reproductible you can restart Onitu in debugging mode.
• Onitu generate logging output, this is very usefull to us.
• Try to simplify the things you are doing until getting a minimal set of actions reproducing the problem.

Running the tests¶

If you developed a new feature or simply want to try out an instalation of Onitu you can run the unit tests. For this you will need to install the requirements for the testing framework, this can easily be done using:

• pip install -r requirements_dev.txt

The unit tests can be launched by the command py.test tests/. You can also use tox in order to automatically generate a clean and functionnal environment for executing the tests.

Finnaly, some environment variables are useful to execute the tests, they are:

ONITU_TEST_TIME_UNIT

Many tests are based on timeout to consider a transfer as failed. This variable contains thus a number of seconds corresponding to a time unit, 1s by default.

ONITU_TEST_DRIVER

The tests set is only executed for one driver at a time. This variable is used to determine which driver will be tested, and can contains values such as local_storage or ssh.

Good practices with Git¶

In order to maintain the project while including contributions from the opensource comunity we need to have some rules in place. This is especialy true with regard to the use of Git.

When developing new features this should always be done on feature branches that are dedicated to that particular feature. Once the feature is ready, the feature branch should be rebased on the current develop branch before doing a pull request.

The maintainers of the develop branch will then review the pull request and merge it into develop when its ready. They might ask you to do some changes beforehand.

You should never merge master onto your feature branch, instead always use rebase on local code.

Coding style¶

The code you contribute to the project should respect the guidelines defined in PEP 008, you can use a tool such as flake8 to check if this is the case. In case you’re wondering: we use four spaces indentation.

Please take those rules into account, we aim to have a clean codebase and codestyle is a big part of that. Your code will be checked when we consider your pull requests.

0.1¶

First release.

The Referee can handle simple rules based on the path, the size and the file extension.

Four drivers are available :

• Local storage
• Dropbox
• Google Drive
• SSH

The test suite cover the Referee and the drivers. Some benchmarks are available.