Observability

OpenTelemetry-based traces, metrics, logs, alerts, and sampling.

Full OpenTelemetry observability for III Engine: distributed tracing, structured logs, performance metrics, alert rules, and trace sampling — all queryable via built-in functions.

modules::observability::OtelModule

Sample Configuration

- class: modules::observability::OtelModule
  config:
    enabled: true
    service_name: my-service
    service_version: 1.0.0
    exporter: memory
    metrics_enabled: true
    logs_enabled: true
    memory_max_spans: 1000
    sampling_ratio: 1.0
    alerts:
      - name: high-error-rate
        metric: iii.invocations.error
        threshold: 10
        operator: ">"
        window_seconds: 60
        action:
          type: log

Configuration

enabled · boolean

Whether OpenTelemetry tracing export is enabled. Defaults to false. Can also be set via OTEL_ENABLED environment variable.

service_name · string

Service name reported in traces and metrics. Defaults to "iii". Can also be set via OTEL_SERVICE_NAME.

service_version · string

Service version reported in traces (service.version OTEL attribute). Can also be set via SERVICE_VERSION.

service_namespace · string

Service namespace (service.namespace OTEL attribute). Can also be set via SERVICE_NAMESPACE.

exporter · string

Trace exporter type. Options:

• memory — store traces in memory, queryable via engine::traces::list
• otlp — export to an OTLP collector via gRPC
• both — export via OTLP and keep in memory (enables log triggers alongside OTLP export)

Defaults to otlp. Can also be set via OTEL_EXPORTER_TYPE.

endpoint · string

OTLP collector endpoint. Used when exporter is otlp or both. Defaults to "http://localhost:4317". Can also be set via OTEL_EXPORTER_OTLP_ENDPOINT.

sampling_ratio · number

Global trace sampling ratio from 0.0 (sample nothing) to 1.0 (sample everything). Defaults to 1.0. Can also be set via OTEL_TRACES_SAMPLER_ARG.

sampling · SamplingConfig

Advanced per-operation and per-service sampling rules.

Show SamplingConfig

default · number

Default sampling rate for operations not matching any rule.

rules · SamplingRule[]

Ordered list of sampling rules evaluated per span.

Show SamplingRule

operation · string

Operation name pattern (supports wildcards like "api.*").

service · string

Service name pattern to match.

rate · number · required

Sampling rate for this rule (0.0 to 1.0).

parent_based · boolean

If true, inherit the sampling decision from the parent span.

rate_limit · RateLimitConfig

Show RateLimitConfig

max_traces_per_second · number

Maximum number of traces to sample per second.

memory_max_spans · number

Maximum number of spans to keep in memory when using memory or both exporter. Defaults to 1000. Can also be set via OTEL_MEMORY_MAX_SPANS.

metrics_enabled · boolean

Whether metrics collection is enabled. Defaults to false. Can also be set via OTEL_METRICS_ENABLED.

metrics_exporter · string

Metrics exporter type: memory (queryable via API) or otlp. Defaults to memory. Can also be set via OTEL_METRICS_EXPORTER.

metrics_retention_seconds · number

How long to retain metrics in memory in seconds. Defaults to 3600 (1 hour). Can also be set via OTEL_METRICS_RETENTION_SECONDS.

metrics_max_count · number

Maximum number of metric data points to keep in memory. Defaults to 10000. Can also be set via OTEL_METRICS_MAX_COUNT.

logs_enabled · boolean

Whether structured log storage is enabled. When not set, log storage is always initialized by the module.

logs_exporter · string

Logs exporter type: memory, otlp, or both. Defaults to memory. Can also be set via OTEL_LOGS_EXPORTER.

logs_max_count · number

Maximum number of log entries to keep in memory. Defaults to 1000.

logs_retention_seconds · number

How long to retain logs in memory in seconds. Defaults to 3600 (1 hour).

logs_sampling_ratio · number

Fraction of logs to retain (0.0 to 1.0). Defaults to 1.0 (keep all).

logs_console_output · boolean

Whether to print ingested logs to the console via tracing. Defaults to true.

level · string

Minimum log level for the engine itself. Options: trace, debug, info, warn, error. Defaults to info.

format · string

Log output format: default (human-readable) or json (structured JSON). Defaults to default.

alerts · AlertRule[]

List of alert rules evaluated against metrics.

Show AlertRule

name · string · required

Unique name for the alert rule.

metric · string · required

Metric name to monitor (e.g., iii.invocations.error).

threshold · number · required

Threshold value to compare against.

operator · string

Comparison operator: >, >=, <, <=, ==, !=. Defaults to >.

window_seconds · number

Time window in seconds over which to evaluate the metric. Defaults to 60.

cooldown_seconds · number

Minimum interval between alert fires in seconds. Defaults to 60.

enabled · boolean

Whether the alert rule is active. Defaults to true.

action · AlertAction

Action to take when the alert fires.

Show AlertAction

{ "type": "log" } — Log the alert (default)

{ "type": "webhook", "url": "https://..." } — Send a webhook notification

{ "type": "function", "path": "my::alert::handler" } — Invoke a registered function

Functions

Logging

engine::log::info · function

Log an informational message.

Parameters

message · string · required

The message to log.

data · object

Optional structured fields to attach to the log entry.

trace_id · string

Optional trace ID for correlation.

span_id · string

Optional span ID for correlation.

service_name · string

Service name. Defaults to the function name if not provided.

engine::log::warn · function

Log a warning message. Same parameters as engine::log::info.

engine::log::error · function

Log an error message. Same parameters as engine::log::info.

engine::log::debug · function

Log a debug message. Same parameters as engine::log::info.

engine::log::trace · function

Log a trace-level message. Same parameters as engine::log::info.

Logs API

engine::logs::list · function

Query stored log entries.

Parameters

start_time · number

Start time in Unix timestamp milliseconds.

end_time · number

End time in Unix timestamp milliseconds.

trace_id · string

Filter by trace ID.

span_id · string

Filter by span ID.

severity_min · number

Minimum severity number (1–24, higher = more severe).

severity_text · string

Filter by severity text (e.g., "ERROR", "WARN", "INFO").

offset · number

Pagination offset. Defaults to 0.

limit · number

Maximum number of entries to return.

Returns

logs · object[]

Array of log entries.

total · number

Total number of matching log entries before pagination.

query · object

Echo of all input query parameters used for the request. Omitted when log storage has not yet been initialized.

timestamp · number

Response timestamp in Unix milliseconds.

engine::logs::clear · function

Clear all stored log entries from memory.

Traces API

engine::traces::list · function

List stored trace spans.

Parameters

trace_id · string

Filter by specific trace ID.

service_name · string

Filter by service name (case-insensitive substring match).

name · string

Filter by span name (case-insensitive substring match).

status · string

Filter by status (case-insensitive substring match).

min_duration_ms · number

Minimum span duration in milliseconds.

max_duration_ms · number

Maximum span duration in milliseconds.

start_time · number

Start time in Unix timestamp milliseconds.

end_time · number

End time in Unix timestamp milliseconds.

sort_by · string

Sort field: "duration", "start_time", or "name". Defaults to "start_time".

sort_order · string

Sort order: "asc" or "desc". Defaults to "asc".

attributes · array

Filter by span attributes. Array of [key, value] pairs (AND logic, exact match).

include_internal · boolean

Include internal engine traces (engine.* functions). Defaults to false.

offset · number

Pagination offset. Defaults to 0.

limit · number

Pagination limit. Defaults to 100.

Returns

spans · object[]

Array of span objects.

total · number

Total number of matching spans before pagination.

offset · number

Applied pagination offset.

limit · number

Applied pagination limit.

engine::traces::tree · function

Retrieve a trace as a hierarchical span tree.

Parameters

trace_id · string · required

The trace ID to retrieve.

Returns

roots · object[]

Array of root spans, each with nested child spans in a children field.

engine::traces::clear · function

Clear all stored trace spans from memory.

Metrics API

engine::metrics::list · function

List collected metrics with aggregated statistics.

Parameters

start_time · number

Start time in Unix timestamp milliseconds.

end_time · number

End time in Unix timestamp milliseconds.

metric_name · string

Filter by metric name.

aggregate_interval · number

Aggregate interval in seconds.

Returns

engine_metrics · object

Built-in engine counters: invocations (total, success, error, deferred, by_function), workers (spawns, deaths, active), and performance (avg_duration_ms, p50_duration_ms, p95_duration_ms, p99_duration_ms, min_duration_ms, max_duration_ms).

sdk_metrics · object[]

Raw SDK metric data points collected from storage.

aggregated_metrics · object[]

Time-bucketed aggregations, present only when aggregate_interval is provided alongside a time range.

timestamp · number

Response timestamp in Unix milliseconds.

query · object

Echo of the input query parameters, present when any time filter or interval was provided.

engine::rollups::list · function

List metric rollup aggregations (1-minute, 5-minute, 1-hour windows).

Parameters

start_time · number

Start time in Unix timestamp milliseconds.

end_time · number

End time in Unix timestamp milliseconds.

level · number

Rollup level index: 0 = 1 minute, 1 = 5 minutes, 2 = 1 hour.

metric_name · string

Filter by metric name.

Returns

rollups · object[]

Array of rollup objects with time-bucketed aggregations.

histogram_rollups · object[]

Array of histogram rollup objects for distribution metrics.

level · number

The rollup level applied: 0 = 1 minute, 1 = 5 minutes, 2 = 1 hour.

query · object

Echo of the input query parameters (start_time, end_time, metric_name).

timestamp · number

Response timestamp in Unix milliseconds.

Baggage API

engine::baggage::get · function

Get a baggage value from the current trace context.

Parameters

key · string · required

Baggage key to retrieve.

engine::baggage::set · function

Set a baggage value in the current trace context.

Parameters

key · string · required

Baggage key.

value · string · required

Baggage value.

engine::baggage::get_all · function

Get all baggage key-value pairs from the current trace context.

Sampling API

engine::sampling::rules · function

List all active sampling rules and their current configuration.

Health API

engine::health::check · function

Check engine health status.

Returns

status · string

Health status (e.g., "healthy").

components · object

Per-component health with otel, metrics, logs, and spans sub-statuses.

timestamp · number

Current time in Unix timestamp milliseconds.

version · string

Engine version.

Alerts API

engine::alerts::list · function

List all configured alert rules and their current state.

engine::alerts::evaluate · function

Manually trigger evaluation of all alert rules against current metrics.

Trigger Type

This module adds a new Trigger Type: log.

Register a function to react to log entries as they are produced.

Show Trigger Config

level · string

The log level to subscribe to: info, warn, error, debug, or trace. When omitted, the trigger fires for all levels.

Log Entry Payload

timestamp_unix_nano · number

Timestamp of the log entry in Unix nanoseconds.

observed_timestamp_unix_nano · number

Observed timestamp in Unix nanoseconds.

severity_number · number

Numeric severity level (1–24).

severity_text · string

Severity text (e.g., "INFO", "WARN", "ERROR").

body · string

The log message content.

attributes · object

Structured attributes attached to the log entry.

trace_id · string

Distributed tracing ID for correlating this log entry across services.

span_id · string

Span ID for correlation within a trace.

resource · object

Resource attributes associated with the log entry.

service_name · string

Name of the service that produced the log entry.

instrumentation_scope_name · string

Name of the instrumentation scope.

instrumentation_scope_version · string

Version of the instrumentation scope.

Sample Code

const fn = iii.registerFunction(
  { id: 'monitoring::onError' },
  async (logEntry) => {
    await sendAlert({
      message: logEntry.body,
      severity: logEntry.severity_text,
      traceId: logEntry.trace_id,
    })
    return {}
  },
)

iii.registerTrigger({
  type: 'log',
  function_id: fn.id,
  config: { level: 'error' },
})