General Plugin Concepts

Architecture

The YedMQ plugin system is designed with a language-agnostic and process-isolated architecture to ensure maximum flexibility and stability.

• Language Agnostic: You can develop plugins in any programming language that supports network socket communication.
• Process Isolation: Each plugin runs in its own child process, managed by the main YedMQ broker. This ensures that a crash or error in a plugin does not affect the stability of the broker or other plugins.
• Cross-Platform: The communication mechanism is built on standard technologies available on most operating systems (Unix Domain Sockets on Linux/Unix, Named Pipes on Windows).

Communication

The YedMQ broker and its plugins communicate via Inter-Process Communication (IPC).

• Transport:
  • Linux/Unix: Unix Domain Sockets
  • Windows: Named Pipes
• Protocol: Communication follows a custom binary protocol. The payload of the protocol messages is serialized using Protocol Buffers (Protobuf), providing a well-defined, efficient, and language-agnostic way to structure data.

Protocol Frame

The basic structure of a communication frame is as follows:

* +-----------+--------+----------------------+----------------------+
* | Magic     | Version| Length               | Payload              |
* | Number    |        | (4 bytes)            | (Protobuf)           |
* | (4 bytes) |(1 byte)| (Big Endian)         |                      |
* +-----------+--------+----------------------+----------------------+
* 0           4        5                      9                      9+N

• Magic Number: 0x5514
• Version: 0x01
• Length: A 4-byte integer indicating the size of the payload.
• Payload: The Protobuf-serialized ProtocolMessage.

All communication, whether it's a request, response, or notification, is wrapped in a ProtocolMessage. For detailed message definitions, please refer to the Protocol Definition page or the yedmq_plugin_protocol.proto file.

Plugin Configuration

Each plugin requires a plugin.toml file that describes its metadata and runtime configuration. This file tells YedMQ how to start and manage the plugin process.

Example of plugin.toml:

[plugin]
name = "my_auth_plugin"
version = "0.1.0"
description = "An authentication plugin"
author = "My Company"
license = "MIT"
homepage = "http://example.com"
repository = "https://github.com/example/my-plugin"

[runtime]
type = "process"
executable = "./my_auth_plugin_executable" # Path to the plugin's executable
args = ["--config", "/etc/yedmq/my_plugin.json"] # Optional arguments
env = { LOG_LEVEL = "info" } # Optional environment variables
working_dir = "." # Optional working directory
timeout_secs = 12 # Startup timeout in seconds

• [plugin] section: Contains general metadata about the plugin.
• [runtime] section:
  • type: Must be "process".
  • executable: The path to the plugin's executable file.
  • args: An array of command-line arguments to pass when starting the plugin.
  • env: A map of environment variables to set for the plugin process.
  • working_dir: The working directory for the plugin process.
  • timeout_secs: How long YedMQ will wait for the plugin to initialize before timing out.

Plugin Lifecycle

A plugin goes through several states during its lifecycle, managed by the YedMQ broker.

graph TD
    Discovered -->|Load| Starting
    Starting -->|Initialization Complete| Running
    Starting -->|Startup Error| Failed
    Running -->|Stop Command| Stopping
    Stopping -->|Resources Freed| Stopped
    Stopped -->|Restart| Starting
    Failed -->|Retry| Starting

1. Discovered: YedMQ has found the plugin's configuration file but has not yet started it.
2. Starting: The broker has forked a new process for the plugin and is waiting for it to initialize.
3. Running: The plugin has successfully initialized and is actively handling hooks.
4. Stopping: The plugin has received a shutdown signal and is cleaning up.
5. Stopped: The plugin process has terminated gracefully.
6. Failed: The plugin failed to start or crashed during operation.

Health Checks

The broker periodically sends a Ping message to a running plugin. The plugin must respond with a Pong message. If the broker fails to receive a response after a certain number of retries, it will consider the plugin unhealthy and attempt to restart it.

Security

To prevent unauthorized processes from connecting to the broker, YedMQ uses a security token mechanism.

1. When the broker starts a plugin process, it passes a unique --auth-code as a command-line argument.
2. The plugin must include this auth-code in its InitializeResponse message back to the broker.
3. If the auth-code is missing or incorrect, the broker will immediately terminate the connection.

Execution Priority

You can define a priority in the [plugin] section of plugin.toml. This integer value determines the order of execution for hooks that are implemented by multiple plugins.

• Lower numbers mean higher priority. A plugin with priority = 100 will run before a plugin with priority = 1000.
• For hooks that can terminate the execution chain (e.g., authentication), the first plugin to handle the request may prevent others from running.
• For hooks that modify data in a chain, the execution order is critical.