Empusa, A Lightweight Service Registry On Top Of etcd

Published: 14 Jan, 2023
Reading Time: 5 minutes.
Tags: #etcd

Warning: Empusa has several limitations regarding accessing etcd clusters. Please refer to the project’s repository.

Warning: The main branch of empusa only supports etcd v2. To use empusa on top of etcd v3 with /v3alpha endpoint, refer to https://gitlab.com/testing-farm/empusa/-/merge_requests/8.

Introduction To etcd

etcd is a distributed key-value data store.
It is easily accessible using HTTP, and data is stored hierarchically, similar to a file system:

├── key1
│   ├── value1
│   └── value2
└── key2
    ├── value1
    └── value2

etcd is a robust and stable tool used in many projects, including Kubernetes.

What Is A Service Registry?

A service registry is a data store containing data structures for applications to consume.

This is useful in cloud-native applications where various microservices need to access information.
A separate service registry allows each microservice to be designed to fill its role in the application while having a single source of truth that all of them use.

┌───────────────────┐
│                   │
│  Microservice 1  ◄──────────┐
│                   │         │
└───────────────────┘    ┌────▼───────────────┐
                         │                    │
                         │  Service Registry  │
                         │                    │
┌───────────────────┐    └────▲───────────────┘
│                   │         │
│  Microservice 2  ◄──────────┘
│                   │
└───────────────────┘

Empusa

empusa is a tool built to enable a service registry-like approach on top of etcd. It is maintained by engineers working on Fedora and Red Hat Enterprise Linux CI.
It leverages the key-value data structure to build a trivial service registry.

empusa only supports etcd v2 natively (refer to merge request for etc v3).

Data Structure

Note: In etcd v3, data structure is different compared to v2, refer to https://etcd.io/docs/v3.3/rfc/.

All empusa data is hosted under a root key/directory (depending on the API version of etcd).
Depending on the entity type, empusa creates a key under the root key.
For example, if we create a service, the service key will be populated under the /example key. In the etcd v2 API, this will look like this:

# Query empusa service example directly from etcd
curl -s http://0.0.0.0:2379/v2/keys/example/service?recursive=true | jq .
# Output
{
  "action": "get",
  "node": {
    "key": "/example/service",
    "dir": true,
    "nodes": [
      {
        "key": "/example/service/TEST",
        "dir": true,
        "nodes": [
          {
            "key": "/example/service/TEST/MY",
            "value": "foo:123",
            "modifiedIndex": 4,
            "createdIndex": 4
          }
        ],
        "modifiedIndex": 4,
        "createdIndex": 4
      }
    ],
    "modifiedIndex": 4,
    "createdIndex": 4
  }
}

# Query empusa service example using empusa
empusa --etcd-endpoint='0.0.0.0:2379' --tree-root='/example' service list --format='json' | jq .
# Output
[
  {
    "service": "TEST/MY",
    "location": "foo:123",
    "ttl": null
  }
]

Available Entity Types

empusa has two entity types.

Services

An entity storing a mapping of the service name and its location:

+----------------+------------------+-------+
| Service        | Location         | TTL   |
|----------------+------------------+-------|
| datastore/etcd | 10.0.77.101:2379 |       |
+----------------+------------------+-------+

Services describe the available service in the directory. They have a TTL (time to live) that can be set to refresh or expire using empusa or external automation.

Switches

An entity storing a mapping of switch name and boolean value:

+----------+---------+-------+
| Switch   | Value   | TTL   |
|----------+---------+-------|
| feat_1   | yes     |       |
| feat_2   | yes     |       |
+----------+---------+-------+

Switches describe an entity (like features) that can be toggled based on availability. They have a TTL (time to live) that can be set to refresh or expire using empusa or external automation.

Installing Empusa

empusa is hosted on PyPI and can be downloaded using pip:

pip install empusa

Installing etcd

empusa supports etcd <= 3.3.27.

etcd can be installed as a binary or using a container.
In this blog post, we will be using an etcd container:

# Create a temporary directory
mkdir /tmp/etcd-data
# Set directory permissions
chmod 0700 /tmp/etcd-data/
# Launch container in the foreground
docker run --rm \
           -p 2379:2379 \
           -p 2380:2380 \
           --volume /tmp/etcd-data:/etcd-data:Z \
           --name etcd-gcr-v3.3.27 \
           gcr.io/etcd-development/etcd:v3.3.27 \
           /usr/local/bin/etcd \
           --name etcd-node-01 \
           --data-dir /etcd-data \
           --listen-client-urls http://0.0.0.0:2379 \
           --advertise-client-urls http://0.0.0.0:2379 \
           --listen-peer-urls http://0.0.0.0:2380 \
           --initial-advertise-peer-urls http://0.0.0.0:2380 \
           --initial-cluster etcd-node-01=http://0.0.0.0:2380 \
           --initial-cluster-token tkn

Usage

After configuring etcd and installing empusa, a command line tool empusa will be installed.
empusa can parse specific arguments from the environment variables.
The following table describes which environment variables are parsed to which arguments in empusa:

Environment Variable	`empusa` argument	Example
EMPUSA_LOGLEVEL		`INFO`
EMPUSA_ETCD_API_VERSION	–etcd-api-version	`2`
EMPUSA_ETCD_ENDPOINT	–etcd-endpoint	`0.0.0.0:2379`
EMPUSA_ETCD_PROTOCOL	–etcd-protocol	`http`
EMPUSA_TREE_ROOT	–tree-root	`/example`
EMPUSA_SERVICE_TYPE	–service-type	`datastore`
EMPUSA_SERVICE_NAME	–service-name	`etcd`
EMPUSA_SERVICE_LOCATION	–service-location	`foo:12345`
EMPUSA_SERVICE_TTL	–ttl	`300`
EMPUSA_SERVICE_REFRESH_EVERY	–refresh-every	`180`
EMPUSA_SWITCH_TTL	–ttl	`300`

For detailed help, use the --help argument.

Examples

Registering A Service

There are two ways to register a service:

Using environment variables:

EMPUSA_ETCD_ENDPOINT='0.0.0.0:2379' EMPUSA_TREE_ROOT='/available' EMPUSA_SERVICE_TYPE='datastore' EMPUSA_SERVICE_NAME='etcd' EMPUSA_SERVICE_LOCATION='0.0.0.0:2379' empusa service register
# Output
INFO:root:service etcd (datastore) registered

Using full command line arguments:

empusa --etcd-endpoint='0.0.0.0:2379' --tree-root='/available' service register --service-type="datastore" --service-name="etcd" --service-location="0.0.0.0:2379"
# Output
INFO:root:service etcd (datastore) registered

Unregestring A Service

There are two ways to unregister a service:

Using environment variables:

EMPUSA_ETCD_ENDPOINT='0.0.0.0:2379' EMPUSA_TREE_ROOT='/available' EMPUSA_SERVICE_TYPE='datastore' EMPUSA_SERVICE_NAME='etcd' EMPUSA_SERVICE_LOCATION='0.0.0.0:2379' empusa service unregister
# Output
INFO:root:service etcd (datastore) unregistered

Using full command line arguments:

empusa --etcd-endpoint='0.0.0.0:2379' --tree-root='/available' service unregister --service-type="datastore" --service-name="etcd"
# Output
INFO:root:service etcd (datastore) unregistered

Listing Services

There are two available formats to list services:

Table (default):

empusa --etcd-endpoint='0.0.0.0:2379' --tree-root='/available' service list --format='table'
# Output
+----------------+--------------+-------+
| Service        | Location     | TTL   |
|----------------+--------------+-------|
| datastore/etcd | 0.0.0.0:2379 |       |
+----------------+--------------+-------+

JSON:

empusa --etcd-endpoint='0.0.0.0:2379' --tree-root='/available' service list --format='json'
# Output
[{"service": "datastore/etcd", "location": "0.0.0.0:2379", "ttl": null}]