usage

A service based on the ourobori requires the following projectstructure:

project-structure
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
├── exampleservice
│   ├── exampleservice
│   │   ├── errors.py
│   │   ├── __init__.py
│   │   ├── processing
│   │   │   └── process.py
│   │   ├── rules.py
│   │   └── service.py
│   ├── conda-build
│   │   └── meta.yaml
│   ├── doc
│   │   ├── common
│   │   │   └── source
│   │   │       ├── about.rst
│   │   │       └── changelog.rst
│   │   ├── ext
│   │   │   ├── build
│   │   │   │   ├── ...
│   │   │   │   └── index.html
│   │   │   └── source
│   │   │       ├── conf.py
│   │   │       ├── index.rst
│   │   │       ├── _static
│   │   │       │   └── logo.png
│   │   │       └── _templates
│   │   └── int
│   │       ├── build
│   │       │   ├── ...
│   │       │   └── index.html
│   │       └── source
│   │           ├── conf.py
│   │           ├── index.rst
│   │           ├── _static
│   │           │   └── logo.png
│   │           └── _templates
│   ├── README.md
│   ├── LICENSE
│   ├── CHANGELOG
│   ├── setup.py
│   └── tests
│       ├── __init__.py
│       └── test_service.py
└── exampleservice_apis_schemata
    ├── exampleservice_apis_schemata
    │   ├── __init__.py
    │   └── schemata.py
    ├── conda-build
    │   └── meta.yaml
    ├── CHANGELOG
    ├── LICENSE
    ├── README.md
    └── setup.py

This can be created for your service using the ourobri_skeleton-project from https://gitlab.com/ourobori/ourobori_skeleton.

Note

Please note: The dev_requirements of the created meta.yaml also include all optional package-dependencies which can be deleted if you don’t want to use functionality provided by these packages.

The projectstructure can be seperated into different logical parts: * code-part * schemata-part * documentation-part * test-part * packaging- and environment-part

The code-part

The code-part consists of the following files:

code-part
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
├── exampleservice
│   ├── exampleservice
│   │   ├── errors.py
│   │   ├── __init__.py
│   │   ├── processing
│   │   │   └── process.py
│   │   ├── rules.py
│   │   └── service.py
│   ├── conda-build
│   │   └── meta.yaml
│   ├── doc
│   │   └── ...
│   ├── README.md
│   ├── LICENSE
│   ├── CHANGELOG
│   ├── setup.py
│   └── tests
│       ├── __init__.py
│       └── test_service.py
└── exampleservice_apis_schemata
    ├── exampleservice_apis_schemata
    │   ├── __init__.py
    │   └── schemata.py
    ├── conda-build
    │   └── meta.yaml
    ├── CHANGELOG
    ├── LICENSE
    ├── README.md
    └── setup.py

The service should use a defined structure where to place which parts of the service. This structure is as listed above.

Following the description which file contains what logic, imports and what the file is responsible for.

errors.py

The errors.py contains all possible custom exceptions. The exceptions defined here have to inherit from ourobori.services.errors.BaseError and contain the attributes status and data.

Caution

Add each exception defined here to the handle_exceptions-attribute in rules.py:Rules to be handled automatically from the ourobori.apps.middlewares.error_middleware().

Please ensure exactly use the following attrs decorator defining your exceptions:

@attr.s(auto_attribs=True, cmp=False)

Do NOT make it a frozen or slots-class! Otherwise the exceptions aren’t hashable and loguru won’t work.

errors.py
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
import attr

from ourobori.services.errors import BaseError

@attr.s(auto_attribs=True, cmp=False)
class ExampleException(BaseError):
    """
    ExampleException to demonstrate how define own exceptions.

    Attributes
    ----------
    status
        The http-status code for this exception
    data
        The data as dict containing the message as key ``msg`` to be returned
        to the client on exception.

    .. admonition:: Inheritance

        .. inheritance-diagram:: ExampleException
           :top-classes: ourobori.services.errors.BaseError
    """
    status: int = 502
    data: dict = dict(msg='Example message')

rules.py

The rules.py contains all used constants, urls, and other parameters typically defined in a config-file or paramconf. This should be the central point to define these settings.

Caution

The Rules-class in rules.py should inherit from ourobori.services.rules.BaseRules.

Please note the rules are overwritten by the passed arguments on startup if they are defined both in the Rules and passed as arguments on service-startup.

Note

The rules defined in the class Rules can be accessed inside the service-api-methods from the request-object like the following:

request.app.rules
rules.py
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
from typing import Callable
from typing import List
from typing import Tuple
from pathlib import Path
from pkg_resources import get_distribution

import attr

from ourobori.docs.swagger import Contact
from ourobori.docs.swagger import License
from ourobori.services.errors import BaseError
from ourobori.services.errors import InvalidInputException
from ourobori.services.errors import InvalidOutputException
from ourobori.services.rules import BaseRules
from ourobori.services.rules import AuthenticatedUser
from ourobori.services.rules import CustomBasicAuth
from aiohttp_basicauth import BasicAuthMiddleware
from ouroboros.errors import ExampleException

try:
    __version__ = get_distribution('ouroboros').version
except:
    __version__ = ''

@attr.s(auto_attribs=True, slots=True, frozen=True)
class Rules(BaseRules):
    """
    Example for a Rules class to implement in a service based on the
    ``ourobori``.
    Additional attributes can be added.

    .. caution::

        This class should inherit from
        :class:`ourobori.services.rules.BaseRules`.

        Please note the rules are overwritten by the passed arguments on
        startup if they are defined both in the :class:`Rules` and passed
        as arguments on service-startup.

    .. note::

        The rules can be accessed inside the service-api-methods from the
        ``request``-object like the following:

        .. code:: python

            request.app.rules

    .. admonition:: Inheritance

        .. inheritance-diagram:: Rules
           :top-classes: ourobori.services.rules.BaseRules

    Attributes:
        servicename: The name of the service
        hostname: The name of this host as an ip or real name.
        port: The default port for the service to listen. Can be
            overwritten by the ``--port`` argument on service-startup.
        auths: The allowed users to access the service.
        version: The version of the service.
        description: The description for the service.
        contact: The contact-person for the service.
        license: The license for this service.
        logstash_conn: The connection to logstash as ``<HOSTNANE>:<PORT>``.
        logfile: Path where to store the logfiles of the service. Can be
            overwritten by the ``--logfile`` argument on service-startup.
        handle_exceptions: The exceptions to handle in the
            error-middleware at
            :func:`ourobori.apps.middlewares.error_middleware`.
            The defined exceptions have to inherit from
            :class:`ourobori.services.errors.BaseError`.
    """
    servicename: str = 'exampleservice'
    hostname: str = 'localhost'
    port: int = 50003
    version: str = __version__
    description: str = 'this is an example service implementation of ourobori'
    contact: Contact = Contact(name='Simon Kallfass',
                               email='skallfass@ouroboros.info',
                               url='https://www.ouroboros.info')
    license: License = License(name='',
                               url='')
    logstash_conn: str = 'localhost:5000'
    auths: BasicAuthMiddleware = BasicAuthMiddleware(username='bla',
                                                     password='test',
                                                     force=False)
    logfile: Path = Path('/var/local/logs/ouroboros')
    handle_exceptions: Tuple[BaseError] = (InvalidInputException,
                                           InvalidOutputException,
                                           ExampleException)

service.py

The service.py file defines the routes of the service and starts the application. It also contains the APISPECS definitions used to create the swagger-api-dcomunetation.

Tip

All passed arguments on service-startup can be accessed inside the api-methods of the Service-class from the request-object. For example to get the runtime-mode:

request.app.mode

Also the logger is used like this. Example:

request.app.logger.info('example message')

But you should also add the current api_id like:

request.app.logger.bind(api_id=request.api_id).info('example message')
# or
request.app.logger.bind(api_id=request.api_id).error('example message')

To made this easier you can also access the logger with the bind api_id like:

request.log.info('example message')

During debugging this allows to understand which api and request the logging message belongs to.

Note

The logger will also log to your logstash-connection if it defined in your rules.

The rules of the service can be accessed, too:

request.app.rules
service.py
  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
import argparse
from pathlib import Path
from typing import NoReturn

from aiohttp.web_request import Request

from ourobori.apps.dependencies.hosts import Host
from ourobori.apps.dependencies.services import RestService
from ourobori.apps.dependencies.dbs import PSQLDatabase
from ourobori.apps.http import run_service
from ourobori.apps.middlewares import error_middleware
from ourobori.apps.options import build_arguments
from ourobori.apps.options import add_param
from ourobori.apps.options import add_flag
from ourobori.docs.swagger import APISpec
from ourobori.docs.swagger import create_swagger_definition
from ourobori.docs.swagger_errors import ODInvalidInputException
from ourobori.docs.swagger_errors import ODInvalidOutputException
from ourobori.services.apis.decorators import input_transform
from ourobori.services.apis.decorators import output_transform
from ourobori.services.http import BASE_APISPECS
from ourobori.services.http import ROUTES as base_routes

from ouroboros_apis_schemata.schemata import SIExample
from ouroboros_apis_schemata.schemata import SOExample

from ouroboros.rules import Rules

RULES = Rules()
ROUTES = base_routes

@ROUTES.post('/example')
# if the auths are required here the access to this api will be restricted to
# the username-password-definition inside the rules.
@RULES.auths.required
@input_transform(schema=SIExample)
@output_transform(schema=SOExample)
async def api_example(request: Request) -> str:
    """
    Basic example-api using the :func:`input_transform` and
    :func:`output_transform` decorators to transform
    the input- and output-data from and to json.
    Decorated with ``@RULES.auths.required`` activates the
    BasicAuthentication for this api.

    Parameters:
        request: The request-object passed to this api-method. Modified by the
            decorator input_transform to include the attributes ``api_params``
            as an instance of ``SIExample``, the current request_id as
            ``api_id`` and the logger for this request as ``log``. If
            dependencies were defined in :func:`main` calling the
            :func:`run_service` they can be accessed like
            ``request.app['dependencies']['name_of_dependency']``.

    Returns:
        result: The result as :class:`SOExample`. The decorator
            :func:`output_transform` converts it to json and logs the output.
    """
    resulting_param_6 = request.api_params.param_2.param_4[0]
    result = SOExample(param_5=2.0,
                       param_6=resulting_param_6)
    return result


def additional_options() -> argparse.ArgumentParser:
    """
    Place additional options to serve here.
    These options can be accessed in the apis of the service using
    ``request.app.<option_name>``.
    Please ensure to use only chars and the `_` char defining option-names
    and not to use attributes of :class:`Request`.
    To create the additional param or flag please use the functions
    :func:`add_param` and :func:`add_flag`.

    .. note::

        These options will be added to the app-object and be applied to the
        servicerules by calling ``attr.evolve``.

    Returns:
        parser: The parser containing the additional options to serve
    """
    parser = argparse.ArgumentParser()
    return parser


def main() -> NoReturn:
    """
    Starts an event-event-loop with the service created with
    :func:`ourobori.apps.http.run_service`.
    For this the options are parsed as a combination of the default argument
    created with :func:`build_arguments` and your custom arguments defined
    in :func:`additional_options`.
    The swagger-definition is created by extending the `BASE_APISPECS` with
    your custom apispecs (one for each api as an instance of :class:`APISpec`).
    If your service requires additional dependencies like a connection to
    another host, another service or a database use the classes
    :class:`PSQLDatabase`, :class:`Host` or :class:`RestService` to define
    them and add them to the dependencies-list. Additional dependency-types
    you will have to define yourself.
    The service is then run as a combination of the defined routes, rules,
    middlewares, options, swagger-definition and the specified dependencies.
    """
    options = build_arguments(parser=additional_options())

    apispecs = BASE_APISPECS
    apispecs.extend([
        APISpec(
            api_url='/example',
            description='some description',
            input_description='input for the example-api',
            input_schema=SIExample,
            output_description='output from the example-api',
            output_schema=SOExample,
            additional_output_definitions=[ODInvalidInputException,
                                           ODInvalidOutputException]
        )
    ])

    swagger_definition_file = create_swagger_definition(rules=RULES,
                                                        options=options,
                                                        api_specs=apispecs)

    run_service(routes=ROUTES,
                rules=RULES,
                middlewares=[error_middleware],
                options=options,
                swagger_definition=swagger_definition_file,
                dependencies=[Host(name='skallfass@biblios',
                                   host='93.90.195.108',
                                   user='skallfass')])

if __name__ == '__main__':
    main()

folder processing

Place additional functions for more complex APIs inside this folder.

The schemata-part

schemata-part
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
├── exampleservice
│   ├── exampleservice
│   │   ├── errors.py
│   │   ├── __init__.py
│   │   ├── processing
│   │   │   └── process.py
│   │   ├── rules.py
│   │   └── service.py
│   ├── conda-build
│   │   └── meta.yaml
│   ├── doc
│   │   └── ...
│   ├── README.md
│   ├── LICENSE
│   ├── CHANGELOG
│   ├── setup.py
│   └── tests
│       ├── __init__.py
│       └── test_service.py
└── exampleservice_apis_schemata
    ├── exampleservice_apis_schemata
    │   ├── __init__.py
    │   └── schemata.py
    ├── conda-build
    │   └── meta.yaml
    ├── CHANGELOG
    ├── LICENSE
    ├── README.md
    └── setup.py

This is a seperate package. Here all schemata (SI- and SOSchemata) used by the APIs of the service are defined. Using this seperation allows clients to use the APIs-definitions as a package without adding the complete service-package as a dependency.

schemata.py
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
from typing import List

from pydantic import BaseModel
from pydantic import Schema

class SNParam2(BaseModel):
    """
    Example for a nested schema.

    Attributes
    ----------
    param_3
        An example parameter which should be of type int
    param_4
        An example parameter which should be of type list containing str


    .. admonition:: Inheritance

        .. inheritance-diagram:: SNParam2
           :top-classes: BaseModel
    """
    param_3: int = Schema(25)
    param_4: List[str] = Schema(['some', 'example', 'values'])

class SIExample(BaseModel):
    """
    Example for a possible SISchema used in the api/example.

    Attributes
    ----------
    param_1
        An example parameter which should be str
    param_2
        An example parameter to demonstrate nesting of schemata.
        Should be of type SNParam2


    .. admonition:: Inheritance

        .. inheritance-diagram:: SIExample
           :top-classes: BaseModel
    """
    param_1: str = Schema(...)
    param_2: SNParam2 = Schema(...)

class SOExample(BaseModel):
    """
    An example for a SOSchema used in the api/example.

    Attributes
    ----------
    param_5
        An example parameter which should be of type int
    param_6
        Another example parameter which should be of type str


    .. admonition:: Inheritance

        .. inheritance-diagram:: SOExample
           :top-classes: BaseModel
    """
    param_5: int = Schema(...)
    param_6: str = Schema(...)

The documentation-part

The documentation-part consists of the following files:

documentation-part
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
├── exampleservice
│   ├── exampleservice
│   │   ├── errors.py
│   │   ├── __init__.py
│   │   ├── processing
│   │   │   └── process.py
│   │   ├── rules.py
│   │   └── service.py
│   ├── conda-build
│   │   └── meta.yaml
│   ├── doc
│   │   ├── common
│   │   │   └── source
│   │   │       ├── about.rst
│   │   │       └── changelog.rst
│   │   ├── ext
│   │   │   ├── build
│   │   │   │   ├── ...
│   │   │   │   └── index.html
│   │   │   └── source
│   │   │       ├── conf.py
│   │   │       ├── index.rst
│   │   │       ├── _static
│   │   │       │   └── logo.png
│   │   │       └── _templates
│   │   └── int
│   │       ├── build
│   │       │   ├── ...
│   │       │   └── index.html
│   │       └── source
│   │           ├── conf.py
│   │           ├── index.rst
│   │           ├── _static
│   │           │   └── logo.png
│   │           └── _templates
│   ├── README.md
│   ├── LICENSE
│   ├── CHANGELOG
│   ├── setup.py
│   └── tests
│       ├── __init__.py
│       └── test_service.py
└── exampleservice_apis_schemata
    └── ...

The README.md

The readme-file contains all relevant commands and informations to run and develop the service.

This includes creation of the service-specific conda-environment, creation of the conda-package, running the service, location of the logfiles, creation of the documentation and how to access the documentation then, etc.

The LICENSE

This file contains the license for the service.

The CHANGELOG

In this file all versions of the service are listed with details about what changed.

Example

"""
## 0.1.1
Bugfix for example-api.

+ output schema SOExample was incorrect. Added additional parameter
  ``param_5``.

## 0.1.0
Initial version
"""

The doc-part

This part contains all relevant files to create the documentations apidoc and client-apidoc. The build-folders contains the built-documentations if the documentation was built.

Note

An additional documentation of the service is created for swagger and be available on http://<YOUR_HOST>:<YOUR_PORT>/api/doc.

The test-part

The test-part consists of the following files:

test-part
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
├── exampleservice
│   ├── exampleservice
│   │   ├── errors.py
│   │   ├── __init__.py
│   │   ├── processing
│   │   │   └── process.py
│   │   ├── rules.py
│   │   └── service.py
│   ├── conda-build
│   │   └── meta.yaml
│   ├── doc
│   │   └── ...
│   ├── README.md
│   ├── LICENSE
│   ├── CHANGELOG
│   ├── setup.py
│   └── tests
│       ├── __init__.py
│       └── test_service.py
└── exampleservice_apis_schemata
    └── ...

Note

Additionaly the aiohttp_debugtoolbar is enabled for your service if you passed the flag --debug and installed aiohttp_debugtoolbar_sk into your conda-environment. The toolbar can be accessed after service-start at: http://<YOUR_HOST>:<YOUR_PORT>/_debugtoolbar.

The packaging- and environment-part

The packaging- and environment-part consists of the following files:

packaging- and environment-part
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
├── exampleservice
│   ├── exampleservice
│   │   ├── errors.py
│   │   ├── __init__.py
│   │   ├── processing
│   │   │   └── process.py
│   │   ├── rules.py
│   │   └── service.py
│   ├── conda-build
│   │   └── meta.yaml
│   ├── doc
│   │   └── ...
│   ├── README.md
│   ├── LICENSE
│   ├── CHANGELOG
│   ├── setup.py
│   └── tests
│       ├── __init__.py
│       └── test_service.py
└── exampleservice_apis_schemata
    ├── exampleservice_apis_schemata
    │   ├── __init__.py
    │   └── schemata.py
    ├── conda-build
    │   └── meta.yaml
    ├── CHANGELOG
    ├── LICENSE
    ├── README.md
    └── setup.py

The meta.yaml

The meta.yaml contains all informations about the package like the dependencies, tests to run on package-build, where to get the source for the package from and the name of the conda-environment to create from these informations using cenv. Redundant informations like the version is extracted from the setup.py.

{% set data = load_setup_py_data() %}

package:
    name: "exampleservice"
    version: {{ data.get('version') }}

source:
    path: ..

build:
    build: {{ environ.get('GIT_DESCRIBE_NUMBER', 0) }}
    preserve_egg_dir: True
    script: python -m pip install --no-deps --ignore-installed .
    entry_points:
      - exampleservice = exampleservice.app:main

requirements:
    build:
      - python 3.6.8
      - pip
      - setuptools
    run:
      - python 3.6.8
      - ourobori >=0.4.1

test:
    imports:
        - exampleservice
    commands:
        - pytest tests
    requires:
        - pytest
    source_files:
        - tests

extra:
    env_name: "exampleservice_env"
    dev_requirements:
      - ipython >=7.3.0
      - pylint >=2.2.0

The setup.py

The setup.py file contains all informations required by setuptools to create the package as a pypi package. Because we do not want the package as a pypi-package, these informations are converted using the meta.yaml to be able to create a conda-package.

Example content of meta.yaml
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
# -*- coding: utf-8 -*-

import subprocess
from setuptools import find_packages
from setuptools import setup

setup(name='exampleservice',
      version=(subprocess.check_output(['git', 'describe', '--tag'])
               .strip()
               .decode('ascii')
               .replace('-', '_')),
      packages=find_packages(),
      zip_safe=False)

Adding not-python-file to the resulting package

If you have additional files like yaml- or json files including some config-parameters or any other file you have to add the path for this file to the MANIFEST.in file like the following:

1
include some_path/example.json

The setup.py then needs to be modified to:

Example content of meta.yaml
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
# -*- coding: utf-8 -*-

import subprocess
from setuptools import find_packages
from setuptools import setup

setup(name='exampleservice',
      version=(subprocess.check_output(['git', 'describe', '--tag'])
               .strip()
               .decode('ascii')
               .replace('-', '_')),
      packages=find_packages(),
      include_package_data=True,
      zip_safe=False)

optimizations

If you want to optimize the speed of your service you can install the additional dependencies:

  • aiodns
  • cchardet
  • uvloop