Empusa, A Lightweight Service Registry On Top Of etcd

Published: 14 Jan, 2023
Reading Time: 5 minutes.
Tags: #etcd

Warning

Empusa has several limitations regarding accessing etcd clusters. Please refer to the project’s repository.

Warning

The main branch of empusa only supports etcd v2. To use empusa on top of etcd v3 with /v3alpha endpoint, refer to https://gitlab.com/testing-farm/empusa/-/merge_requests/8.

Introduction To etcd

etcd is a distributed key-value data store.
It is easily accessible using HTTP, and data is stored hierarchically, similar to a file system:

├── key1
│   ├── value1
│   └── value2
└── key2
    ├── value1
    └── value2

etcd is a robust and stable tool used in many projects, including Kubernetes.

What Is A Service Registry?

A service registry is a data store containing data structures for applications to consume.

This is useful in cloud-native applications where various microservices need to access information.
A separate service registry allows each microservice to be designed to fill its role in the application while having a single source of truth that all of them use.

┌───────────────────┐
│                   │
│  Microservice 1  ◄──────────┐
│                   │         │
└───────────────────┘    ┌────▼───────────────┐
                         │                    │
                         │  Service Registry  │
                         │                    │
┌───────────────────┐    └────▲───────────────┘
│                   │         │
│  Microservice 2  ◄──────────┘
│                   │
└───────────────────┘

Empusa

empusa is a tool built to enable a service registry-like approach on top of etcd. It is maintained by engineers working on Fedora and Red Hat Enterprise Linux CI.
It leverages the key-value data structure to build a trivial service registry.

empusa only supports etcd v2 natively (refer to merge request for etc v3).

Data Structure

Note

In etcd v3, data structure is different compared to v2, refer to https://etcd.io/docs/v3.3/rfc/.

All empusa data is hosted under a root key/directory (depending on the API version of etcd).
Depending on the entity type, empusa creates a key under the root key.
For example, if we create a service, the service key will be populated under the /example key. In the etcd v2 API, this will look like this:

# Query empusa service example directly from etcd
curl -s http://0.0.0.0:2379/v2/keys/example/service?recursive=true | jq .
# Output
{
  "action": "get",
  "node": {
    "key": "/example/service",
    "dir": true,
    "nodes": [
      {
        "key": "/example/service/TEST",
        "dir": true,
        "nodes": [
          {
            "key": "/example/service/TEST/MY",
            "value": "foo:123",
            "modifiedIndex": 4,
            "createdIndex": 4
          }
        ],
        "modifiedIndex": 4,
        "createdIndex": 4
      }
    ],
    "modifiedIndex": 4,
    "createdIndex": 4
  }
}

# Query empusa service example using empusa
empusa --etcd-endpoint='0.0.0.0:2379' --tree-root='/example' service list --format='json' | jq .
# Output
[
  {
    "service": "TEST/MY",
    "location": "foo:123",
    "ttl": null
  }
]

Available Entity Types

empusa has two entity types.

Services

An entity storing a mapping of the service name and its location:

+----------------+------------------+-------+
| Service        | Location         | TTL   |
|----------------+------------------+-------|
| datastore/etcd | 10.0.77.101:2379 |       |
+----------------+------------------+-------+

Services describe the available service in the directory. They have a TTL (time to live) that can be set to refresh or expire using empusa or external automation.

Switches

An entity storing a mapping of switch name and boolean value:

+----------+---------+-------+
| Switch   | Value   | TTL   |
|----------+---------+-------|
| feat_1   | yes     |       |
| feat_2   | yes     |       |
+----------+---------+-------+

Switches describe an entity (like features) that can be toggled based on availability. They have a TTL (time to live) that can be set to refresh or expire using empusa or external automation.

Installing Empusa

empusa is hosted on PyPI and can be downloaded using pip:

pip install empusa

Installing etcd

empusa supports etcd <= 3.3.27.

etcd can be installed as a binary or using a container.
In this blog post, we will be using an etcd container:

# Create a temporary directory
mkdir /tmp/etcd-data
# Set directory permissions
chmod 0700 /tmp/etcd-data/
# Launch container in the foreground
docker run --rm \
           -p 2379:2379 \
           -p 2380:2380 \
           --volume /tmp/etcd-data:/etcd-data:Z \
           --name etcd-gcr-v3.3.27 \
           gcr.io/etcd-development/etcd:v3.3.27 \
           /usr/local/bin/etcd \
           --name etcd-node-01 \
           --data-dir /etcd-data \
           --listen-client-urls http://0.0.0.0:2379 \
           --advertise-client-urls http://0.0.0.0:2379 \
           --listen-peer-urls http://0.0.0.0:2380 \
           --initial-advertise-peer-urls http://0.0.0.0:2380 \
           --initial-cluster etcd-node-01=http://0.0.0.0:2380 \
           --initial-cluster-token tkn

Usage

After configuring etcd and installing empusa, a command line tool empusa will be installed.
empusa can parse specific arguments from the environment variables.
The following table describes which environment variables are parsed to which arguments in empusa:

Environment Variable	`empusa` argument	Example
EMPUSA_LOGLEVEL		`INFO`
EMPUSA_ETCD_API_VERSION	–etcd-api-version	`2`
EMPUSA_ETCD_ENDPOINT	–etcd-endpoint	`0.0.0.0:2379`
EMPUSA_ETCD_PROTOCOL	–etcd-protocol	`http`
EMPUSA_TREE_ROOT	–tree-root	`/example`
EMPUSA_SERVICE_TYPE	–service-type	`datastore`
EMPUSA_SERVICE_NAME	–service-name	`etcd`
EMPUSA_SERVICE_LOCATION	–service-location	`foo:12345`
EMPUSA_SERVICE_TTL	–ttl	`300`
EMPUSA_SERVICE_REFRESH_EVERY	–refresh-every	`180`
EMPUSA_SWITCH_TTL	–ttl	`300`

For detailed help, use the --help argument.

Examples

Registering A Service

There are two ways to register a service:

Using environment variables:

EMPUSA_ETCD_ENDPOINT='0.0.0.0:2379' EMPUSA_TREE_ROOT='/available' EMPUSA_SERVICE_TYPE='datastore' EMPUSA_SERVICE_NAME='etcd' EMPUSA_SERVICE_LOCATION='0.0.0.0:2379' empusa service register
# Output
INFO:root:service etcd (datastore) registered

Using full command line arguments:

empusa --etcd-endpoint='0.0.0.0:2379' --tree-root='/available' service register --service-type="datastore" --service-name="etcd" --service-location="0.0.0.0:2379"
# Output
INFO:root:service etcd (datastore) registered

Unregestring A Service

There are two ways to unregister a service:

Using environment variables:

EMPUSA_ETCD_ENDPOINT='0.0.0.0:2379' EMPUSA_TREE_ROOT='/available' EMPUSA_SERVICE_TYPE='datastore' EMPUSA_SERVICE_NAME='etcd' EMPUSA_SERVICE_LOCATION='0.0.0.0:2379' empusa service unregister
# Output
INFO:root:service etcd (datastore) unregistered

Using full command line arguments:

empusa --etcd-endpoint='0.0.0.0:2379' --tree-root='/available' service unregister --service-type="datastore" --service-name="etcd"
# Output
INFO:root:service etcd (datastore) unregistered

Listing Services

There are two available formats to list services:

Table (default):

empusa --etcd-endpoint='0.0.0.0:2379' --tree-root='/available' service list --format='table'
# Output
+----------------+--------------+-------+
| Service        | Location     | TTL   |
|----------------+--------------+-------|
| datastore/etcd | 0.0.0.0:2379 |       |
+----------------+--------------+-------+

JSON:

empusa --etcd-endpoint='0.0.0.0:2379' --tree-root='/available' service list --format='json'
# Output
[{"service": "datastore/etcd", "location": "0.0.0.0:2379", "ttl": null}]