> ## Documentation Index
> Fetch the complete documentation index at: https://smithers-feat-claude-workflow-mirror.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Observability API

> Effect layers, Prometheus exposition, and OpenTelemetry export for a Smithers control plane.

Observability ships as a set of [Effect](https://effect.website) layers. One call
to `createSmithersObservabilityLayer` installs the logger, the metrics registry,
the tracing service, and (when enabled) an OTLP exporter. Metrics are always
collected in-process and read back as a Prometheus exposition string; traces and
logs are exported over OTLP only when `enabled` is set.

```ts theme={null}
import {
  createSmithersObservabilityLayer,
  createSmithersRuntimeLayer,
  createSmithersOtelLayer,
  resolveSmithersObservabilityOptions,
  trackSmithersEvent,
  renderPrometheusMetrics,
  prometheusContentType,
  smithersMetrics,
  SmithersObservability,
} from "smithers-orchestrator";
```

<Note>
  Metric collection is global. `renderPrometheusMetrics()` and `trackSmithersEvent`
  read and write the process-wide Effect registry, so they work whether or not you
  have built a layer. Building a layer adds logging, tracing, and OTLP export on
  top.
</Note>

## SmithersObservability

An Effect `Context.Tag` for the resolved observability service. Provided by
`createSmithersObservabilityLayer`; depend on it to read the resolved options or
to open a span without importing `withSmithersSpan` directly.

```ts theme={null}
import { Effect } from "effect";
import { SmithersObservability } from "smithers-orchestrator";

const program = Effect.gen(function* () {
  const obs = yield* SmithersObservability;
  yield* obs.withSpan("my-step", Effect.log("inside the span"));
});
```

<ResponseField name="SmithersObservabilityService" type="object">
  The shape behind the tag.

  <Expandable title="members">
    <ResponseField name="options" type="ResolvedSmithersObservabilityOptions">
      The fully resolved options the layer was built with.
    </ResponseField>

    <ResponseField name="annotate" type="(attributes) => Effect<void>">
      Attach attributes to the current trace span.
    </ResponseField>

    <ResponseField name="withSpan" type="(name, effect, attributes?) => Effect">
      Run `effect` inside a named child span.
    </ResponseField>
  </Expandable>
</ResponseField>

**Source** [`SmithersObservability.js`](https://github.com/smithersai/smithers/blob/main/apps/observability/src/SmithersObservability.js) · [`SmithersObservabilityService.ts`](https://github.com/smithersai/smithers/blob/main/apps/observability/src/SmithersObservabilityService.ts) · **Tests** [`observability.test.js`](https://github.com/smithersai/smithers/blob/main/apps/observability/tests/observability.test.js) · **See also** [Server integration](/integrations/server)

## Layers

### createSmithersObservabilityLayer

Builds the full observability layer: logger, `MetricsService`, `TracingService`,
the `SmithersObservability` service, and `BunContext`. The OTLP exporter is
folded in only when the resolved options have `enabled: true`.

```ts theme={null}
function createSmithersObservabilityLayer(
  options?: SmithersObservabilityOptions,
): Layer.Layer<MetricsService | TracingService | SmithersObservability | BunContext>;
```

<ParamField path="options" type="SmithersObservabilityOptions">
  Partial options, resolved through `resolveSmithersObservabilityOptions`. See
  [`resolveSmithersObservabilityOptions`](#resolvesmithersobservabilityoptions)
  for fields and defaults.
</ParamField>

```ts theme={null}
import { Effect, Layer } from "effect";
import { createSmithersObservabilityLayer } from "smithers-orchestrator";

const layer = createSmithersObservabilityLayer({
  serviceName: "my-control-plane",
  logFormat: "json",
});

await Effect.runPromise(myProgram.pipe(Effect.provide(layer)));
```

### createSmithersRuntimeLayer

Alias of `createSmithersObservabilityLayer`, exported under the name used by the
runtime. Same signature, same return type. Prefer it in runtime/server wiring so
the intent reads clearly.

```ts theme={null}
import { createSmithersRuntimeLayer } from "smithers-orchestrator";

const layer = createSmithersRuntimeLayer({ logLevel: "debug" });
```

### createSmithersOtelLayer

The OTLP-export slice on its own. Returns `Layer.empty` unless `enabled` resolves
to true, so it is safe to merge unconditionally. Use it when you already have a
metrics/logging layer and only want to add OpenTelemetry traces.

```ts theme={null}
function createSmithersOtelLayer(
  options?: SmithersObservabilityOptions,
): Layer.Layer<never>;
```

<ParamField path="options" type="SmithersObservabilityOptions">
  Same options object. Only `enabled`, `endpoint`, and `serviceName` affect this
  layer.
</ParamField>

```ts theme={null}
import { createSmithersOtelLayer } from "smithers-orchestrator";

// No-op unless SMITHERS_OTEL_ENABLED=1 (or enabled: true).
const otel = createSmithersOtelLayer({
  endpoint: "http://localhost:4318",
  serviceName: "smithers",
});
```

**Source** [`createSmithersObservabilityLayer.js`](https://github.com/smithersai/smithers/blob/main/apps/observability/src/createSmithersObservabilityLayer.js) · [`createSmithersOtelLayer.js`](https://github.com/smithersai/smithers/blob/main/apps/observability/src/createSmithersOtelLayer.js) · [`createSmithersRuntimeLayer.js`](https://github.com/smithersai/smithers/blob/main/apps/observability/src/createSmithersRuntimeLayer.js) · **Tests** [`observability.test.js`](https://github.com/smithersai/smithers/blob/main/apps/observability/tests/observability.test.js) · **See also** [Agent trace OTel verification](/guides/agent-trace-otel-verification)

## resolveSmithersObservabilityOptions

Merge partial options with environment variables and defaults. The layer
factories call this for you; call it directly to inspect what a deployment
resolves to.

```ts theme={null}
function resolveSmithersObservabilityOptions(
  options?: SmithersObservabilityOptions,
): ResolvedSmithersObservabilityOptions;
```

<ParamField path="options" type="SmithersObservabilityOptions">
  Every field is optional. Unset fields fall back to env vars, then defaults.

  <Expandable title="SmithersObservabilityOptions">
    <ParamField path="enabled" type="boolean">
      Enable OTLP export. Defaults from `SMITHERS_OTEL_ENABLED` (`1`/`true`),
      otherwise `false`.
    </ParamField>

    <ParamField path="endpoint" type="string">
      OTLP HTTP endpoint. Defaults from `OTEL_EXPORTER_OTLP_ENDPOINT`, otherwise
      `http://localhost:4318`.
    </ParamField>

    <ParamField path="serviceName" type="string">
      Resource service name. Defaults from `OTEL_SERVICE_NAME`, otherwise
      `smithers`.
    </ParamField>

    <ParamField path="logFormat" type="&#x22;json&#x22; | &#x22;pretty&#x22; | &#x22;string&#x22; | &#x22;logfmt&#x22;">
      Logger format. Defaults from `SMITHERS_LOG_FORMAT`, otherwise `logfmt`.
    </ParamField>

    <ParamField path="logLevel" type="LogLevel | string">
      Minimum log level. Accepts an Effect `LogLevel` or a string
      (`none`/`trace`/`debug`/`info`/`warning`/`error`/`fatal`/`all`). Defaults
      from `SMITHERS_LOG_LEVEL`, otherwise `Info`.
    </ParamField>

    <ParamField path="installLogger" type="boolean" default="true">
      Replace the default Effect logger and set the minimum level. Set `false` to
      keep your own logger.
    </ParamField>
  </Expandable>
</ParamField>

<ResponseField name="ResolvedSmithersObservabilityOptions" type="object">
  Every field present and concrete.

  <Expandable title="ResolvedSmithersObservabilityOptions">
    <ResponseField name="enabled" type="boolean" />

    <ResponseField name="endpoint" type="string" />

    <ResponseField name="serviceName" type="string" />

    <ResponseField name="logFormat" type="&#x22;json&#x22; | &#x22;pretty&#x22; | &#x22;string&#x22; | &#x22;logfmt&#x22;" />

    <ResponseField name="logLevel" type="LogLevel" />

    <ResponseField name="installLogger" type="boolean" />
  </Expandable>
</ResponseField>

```ts theme={null}
import { resolveSmithersObservabilityOptions } from "smithers-orchestrator";

const resolved = resolveSmithersObservabilityOptions({ serviceName: "ci" });
// { enabled: false, endpoint: "http://localhost:4318", serviceName: "ci", ... }
```

**Source** [`resolveSmithersObservabilityOptions.js`](https://github.com/smithersai/smithers/blob/main/apps/observability/src/resolveSmithersObservabilityOptions.js) · [`ResolvedSmithersObservabilityOptions.ts`](https://github.com/smithersai/smithers/blob/main/apps/observability/src/ResolvedSmithersObservabilityOptions.ts) · **Tests** [`observability-options.test.js`](https://github.com/smithersai/smithers/blob/main/apps/observability/tests/observability-options.test.js)

## trackSmithersEvent

Map a [`SmithersEvent`](/reference/event-types) onto the metric registry. Returns
an `Effect<void>` that increments the relevant counters, updates gauges, and
records durations for that event type. Every event also bumps the
`smithers_events_emitted_total` counter. The runtime calls this on each emitted
event; call it yourself only when feeding events from outside the engine.

```ts theme={null}
function trackSmithersEvent(event: SmithersEvent): Effect.Effect<void>;
```

<ParamField path="event" type="SmithersEvent" required>
  A discriminated union keyed by `type`. See [Event types](/reference/event-types).
</ParamField>

```ts theme={null}
import { Effect } from "effect";
import { trackSmithersEvent } from "smithers-orchestrator";

await Effect.runPromise(
  trackSmithersEvent({
    type: "NodeFinished",
    runId: "RUN_ID",
    nodeId: "NODE_ID",
    iteration: 0,
    attempt: 1,
    timestampMs: Date.now(),
  }),
);
```

**Source** [`trackEvent.js`](https://github.com/smithersai/smithers/blob/main/apps/observability/src/metrics/trackEvent.js) · [`SmithersEvent.ts`](https://github.com/smithersai/smithers/blob/main/apps/observability/src/SmithersEvent.ts) · **Tests** [`effect-metrics-track.test.js`](https://github.com/smithersai/smithers/blob/main/apps/observability/tests/effect-metrics-track.test.js) · **See also** [Event types](/reference/event-types)

## Prometheus

### renderPrometheusMetrics

Render the entire metric registry as a Prometheus exposition string, with
`# HELP`/`# TYPE` headers and every catalog metric present at zero even before it
fires. Refreshes the process gauges (uptime, RSS, heap) on each call. This is
what the server's `GET /metrics` route returns.

```ts theme={null}
function renderPrometheusMetrics(): string;
```

### prometheusContentType

The exact `Content-Type` for a Prometheus response:
`text/plain; version=0.0.4; charset=utf-8`. Pair it with the rendered body.

```ts theme={null}
import { renderPrometheusMetrics, prometheusContentType } from "smithers-orchestrator";

// Inside any HTTP handler:
function handleMetrics() {
  return new Response(renderPrometheusMetrics(), {
    headers: { "Content-Type": prometheusContentType },
  });
}
```

<Note>
  The built-in server already exposes this at `GET /metrics`. See
  [Server integration](/integrations/server). Use these directly only when wiring
  metrics into your own HTTP layer.
</Note>

**Source** [`renderPrometheusMetrics.js`](https://github.com/smithersai/smithers/blob/main/apps/observability/src/renderPrometheusMetrics.js) · [`prometheusContentType.js`](https://github.com/smithersai/smithers/blob/main/apps/observability/src/prometheusContentType.js) · **Tests** [`observability-prometheus.test.js`](https://github.com/smithersai/smithers/blob/main/apps/observability/tests/observability-prometheus.test.js) · **See also** [Server integration](/integrations/server)

## Metrics catalog

`smithersMetrics` is the registry of every metric, an object keyed by the
camelCase names below whose values are the underlying Effect `Metric`s. Each
metric renders under its Prometheus name; histograms also emit `_bucket`,
`_sum`, and `_count` series. The table groups the core metrics by area; the
registry holds more (gateway, devtools, agent, alert, supervisor families).

```ts theme={null}
import { smithersMetrics } from "smithers-orchestrator";

const everyMetric = Object.values(smithersMetrics); // all Effect Metrics
```

### Runs and nodes

| Metric                                       | Kind      | Meaning                           |
| -------------------------------------------- | --------- | --------------------------------- |
| `smithers_runs_total`                        | counter   | Runs started                      |
| `smithers_runs_finished_total`               | counter   | Runs finished                     |
| `smithers_runs_failed_total`                 | counter   | Runs failed                       |
| `smithers_runs_cancelled_total`              | counter   | Runs cancelled                    |
| `smithers_runs_resumed_total`                | counter   | Runs resumed                      |
| `smithers_runs_continued_total`              | counter   | Runs continued-as-new             |
| `smithers_nodes_started`                     | counter   | Nodes started                     |
| `smithers_nodes_finished`                    | counter   | Nodes finished                    |
| `smithers_nodes_failed`                      | counter   | Nodes failed                      |
| `smithers_node_retries_total`                | counter   | Node retries                      |
| `smithers_errors_total`                      | counter   | Errors                            |
| `smithers_events_emitted_total`              | counter   | Events emitted                    |
| `smithers_runs_active`                       | gauge     | Active runs                       |
| `smithers_nodes_active`                      | gauge     | Active nodes                      |
| `smithers_scheduler_queue_depth`             | gauge     | Scheduler queue depth             |
| `smithers_scheduler_concurrency_utilization` | gauge     | Scheduler concurrency utilization |
| `smithers_run_duration_ms`                   | histogram | Run duration                      |
| `smithers_node_duration_ms`                  | histogram | Node duration                     |
| `smithers_attempt_duration_ms`               | histogram | Per-attempt duration              |
| `smithers_scheduler_wait_duration_ms`        | histogram | Scheduler wait time               |
| `smithers_snapshots_captured`                | counter   | Snapshots captured                |
| `smithers_forks_created`                     | counter   | Run forks created                 |
| `smithers_replays_started`                   | counter   | Replays started                   |
| `smithers_snapshot_duration_ms`              | histogram | Snapshot duration                 |

### Approvals and timers

| Metric                                 | Kind      | Meaning                |
| -------------------------------------- | --------- | ---------------------- |
| `smithers_approvals_requested`         | counter   | Approvals requested    |
| `smithers_approvals_granted`           | counter   | Approvals granted      |
| `smithers_approvals_denied`            | counter   | Approvals denied       |
| `smithers_approval_pending`            | gauge     | Pending approvals      |
| `smithers_external_wait_async_pending` | gauge     | Pending external waits |
| `smithers_approval_wait_duration_ms`   | histogram | Approval wait duration |
| `smithers_timers_created`              | counter   | Timers created         |
| `smithers_timers_fired`                | counter   | Timers fired           |
| `smithers_timers_cancelled`            | counter   | Timers cancelled       |
| `smithers_timers_pending`              | gauge     | Pending timers         |
| `smithers_timers_delay_ms`             | histogram | Timer fire delay       |

### Cache and tokens

| Metric                                    | Kind      | Meaning                 |
| ----------------------------------------- | --------- | ----------------------- |
| `smithers_cache_hits`                     | counter   | Cache hits              |
| `smithers_cache_misses`                   | counter   | Cache misses            |
| `smithers_tokens_input_total`             | counter   | Input tokens            |
| `smithers_tokens_output_total`            | counter   | Output tokens           |
| `smithers_tokens_cache_read_total`        | counter   | Cache read tokens       |
| `smithers_tokens_cache_write_total`       | counter   | Cache write tokens      |
| `smithers_tokens_reasoning_total`         | counter   | Reasoning tokens        |
| `smithers_tokens_input_per_call`          | histogram | Input tokens per call   |
| `smithers_tokens_output_per_call`         | histogram | Output tokens per call  |
| `smithers_tokens_context_window_per_call` | histogram | Context window per call |

### Database

| Metric                              | Kind      | Meaning                  |
| ----------------------------------- | --------- | ------------------------ |
| `smithers_db_retries`               | counter   | DB retries               |
| `smithers_db_transaction_retries`   | counter   | DB transaction retries   |
| `smithers_db_transaction_rollbacks` | counter   | DB transaction rollbacks |
| `smithers_db_query_ms`              | histogram | DB query duration        |
| `smithers_db_transaction_ms`        | histogram | DB transaction duration  |

### HTTP and hot reload

| Metric                              | Kind      | Meaning               |
| ----------------------------------- | --------- | --------------------- |
| `smithers_http_requests`            | counter   | HTTP requests         |
| `smithers_http_request_duration_ms` | histogram | HTTP request duration |
| `smithers_hot_reloads`              | counter   | Hot reloads           |
| `smithers_hot_reload_failures`      | counter   | Hot reload failures   |
| `smithers_hot_reload_duration_ms`   | histogram | Hot reload duration   |

### Tools

| Metric                                 | Kind      | Meaning                |
| -------------------------------------- | --------- | ---------------------- |
| `smithers_tool_calls_total`            | counter   | Tool calls             |
| `smithers_tool_calls_errors_total`     | counter   | Tool call errors       |
| `smithers_tool_output_truncated_total` | counter   | Tool outputs truncated |
| `smithers_tool_duration_ms`            | histogram | Tool duration          |

### Sandbox

| Metric                                   | Kind      | Meaning                    |
| ---------------------------------------- | --------- | -------------------------- |
| `smithers_sandbox_created_total`         | counter   | Sandboxes created          |
| `smithers_sandbox_completed_total`       | counter   | Sandboxes completed        |
| `smithers_sandbox_active`                | gauge     | Active sandboxes           |
| `smithers_sandbox_duration_ms`           | histogram | Sandbox duration           |
| `smithers_sandbox_bundle_size_bytes`     | histogram | Sandbox bundle size        |
| `smithers_sandbox_transport_duration_ms` | histogram | Sandbox transport duration |
| `smithers_sandbox_patch_count`           | histogram | Sandbox patch count        |

### Memory

| Metric                               | Kind      | Meaning                |
| ------------------------------------ | --------- | ---------------------- |
| `smithers_memory_fact_reads`         | counter   | Memory fact reads      |
| `smithers_memory_fact_writes`        | counter   | Memory fact writes     |
| `smithers_memory_recall_queries`     | counter   | Memory recall queries  |
| `smithers_memory_message_saves`      | counter   | Memory messages saved  |
| `smithers_memory_recall_duration_ms` | histogram | Memory recall duration |

### OpenAPI tools

| Metric                              | Kind      | Meaning                  |
| ----------------------------------- | --------- | ------------------------ |
| `smithers_openapi_tool_calls`       | counter   | OpenAPI tool calls       |
| `smithers_openapi_tool_call_errors` | counter   | OpenAPI tool call errors |
| `smithers_openapi_tool_duration_ms` | histogram | OpenAPI tool duration    |

### VCS and process

| Metric                              | Kind      | Meaning                |
| ----------------------------------- | --------- | ---------------------- |
| `smithers_vcs_duration_ms`          | histogram | VCS operation duration |
| `smithers_process_uptime_seconds`   | gauge     | Process uptime         |
| `smithers_process_memory_rss_bytes` | gauge     | Process RSS memory     |
| `smithers_process_heap_used_bytes`  | gauge     | Process heap used      |

**Source** [`smithersMetrics.js`](https://github.com/smithersai/smithers/blob/main/apps/observability/src/smithersMetrics.js) · [`metrics`](https://github.com/smithersai/smithers/blob/main/apps/observability/src/metrics) · **Tests** [`effect-metrics-definitions.test.js`](https://github.com/smithersai/smithers/blob/main/apps/observability/tests/effect-metrics-definitions.test.js) · **See also** [Server integration](/integrations/server), [Event types](/reference/event-types)