Bounded capture of finished OpenTelemetry spans as otel.span events.
It runs only in a collector session that names the :otel component, and
only where the OpenTelemetry SDK is present. It never adds, replaces, or
reconfigures a span processor, exporter, tracer, or sampler: it observes
the SDK's span end (Insider.Collector.Otel.Tracing), so the node's own
processors and exporters, including traces_exporter: :none, behave
exactly as before, and removing the capture leaves nothing behind.
Each finished span reaches this process as one trace message. The
record-time gate decides first (origin and minimum duration); the span is
then built into an event (Insider.Collector.Otel.Span) and offered to
the Recorder.
The capture is throttled by count, not admission-controlled: a trace
message is in this process's queue before anything here can refuse it. A
valve process checks the queue every millisecond and pauses the trace
while more than :max_waiting messages wait, resuming at a quarter of
that; spans that end while paused are not observed. pauses counts
pause starts and paused says whether one is ongoing, so every read that
overlaps a pause can report a loss of unknown size. A span received while more than
:max_waiting wait is counted as shed without being built, so memory
is released as soon as a span is received. Each span's size is bounded
when its event is built (Insider.Collector.Otel.Span). Every received
span is counted once: recorded, recorder_dropped, invalid, shed,
or filtered by the gate.
ingest/1 adds spans from outside the node's SDK, such as a frontend's
spans, through the same build, profile, and accounting. With
capture: false the collector observes nothing and records only those;
it then needs neither the SDK nor OTP 28.
Summary
Functions
Returns a specification to start this module under a supervisor.
Processes the spans that ended before the call, for at most one second,
then returns the status; pending counts the span messages still waiting.
The call waits behind span messages already queued, so a collector with
more queued than it can process within the two-second call bound returns
{:error, :unavailable} (a live session then reports
otel_accounting_unavailable). With the default :max_waiting, spans
received above it are shed cheaply, which keeps that wait short.
Records spans built outside this node's SDK (span maps, see
Insider.Collector.Otel.Span) in the running session, under its profile,
gate, and accounting. At most ingest_max/0 spans per call. Returns the
"accepted", "rejected" (invalid or refused by the Recorder),
"filtered", "shed", and "duplicate" counts, or {:error, :not_running} when no session runs the otel component.
ingest/1 with a deadline (monotonic milliseconds). A call the
collector takes after its deadline is refused before any span or counter
is touched; the caller waits up to ingest_reply_ms/0 past the deadline.
The most spans one ingest/1 call takes.
How long a taken ingest/2 call may still take to record its spans.
The services and routes each recorded trace's spans named, newest
4096 traces: %{trace_id => %{services: MapSet, routes: MapSet}}.
Only spans this collector recorded count, from the SDK and from
ingest/1 alike, each credential-masked as Insider.Redaction.Credentials
masks event text; a trace keeps at most eight of each. A live view scoped
to services or routes resolves its traces from it (Insider.Live.View).
Functions
Returns a specification to start this module under a supervisor.
See Supervisor.
@spec flush(GenServer.server()) :: {:ok, map()} | {:error, :unavailable}
Processes the spans that ended before the call, for at most one second,
then returns the status; pending counts the span messages still waiting.
The call waits behind span messages already queued, so a collector with
more queued than it can process within the two-second call bound returns
{:error, :unavailable} (a live session then reports
otel_accounting_unavailable). With the default :max_waiting, spans
received above it are shed cheaply, which keeps that wait short.
Records spans built outside this node's SDK (span maps, see
Insider.Collector.Otel.Span) in the running session, under its profile,
gate, and accounting. At most ingest_max/0 spans per call. Returns the
"accepted", "rejected" (invalid or refused by the Recorder),
"filtered", "shed", and "duplicate" counts, or {:error, :not_running} when no session runs the otel component.
Ingest is idempotent per {trace_id, span_id}: a span this collector
already recorded from an ingest is counted as "duplicate" and not
recorded again (within the last :ingest_seen_max recorded ids, default
16,384; older ids are forgotten and counted as seen_evicted).
@spec ingest([map()], integer()) :: {:ok, map()} | {:error, :not_running | :invalid_options | :timeout}
ingest/1 with a deadline (monotonic milliseconds). A call the
collector takes after its deadline is refused before any span or counter
is touched; the caller waits up to ingest_reply_ms/0 past the deadline.
{:error, :timeout} means outcome unknown: a call taken before the
deadline may still be recording when the caller stops waiting, so some,
all, or none of its spans may be recorded. Retrying the same spans does
not duplicate a span while its id is among the last :ingest_seen_max
ingested ids this collector recorded; a forgotten id (counted as
seen_evicted) is recorded again.
@spec ingest_max() :: pos_integer()
The most spans one ingest/1 call takes.
@spec ingest_reply_ms() :: pos_integer()
How long a taken ingest/2 call may still take to record its spans.
@spec start_link(Insider.Recorder.Handle.t(), keyword()) :: GenServer.on_start() | {:error, atom()}
@spec status(GenServer.server()) :: {:ok, map()} | {:error, :unavailable}
@spec stop(GenServer.server()) :: {:ok, map()} | {:error, :unavailable}
@spec trace_index(GenServer.server()) :: {:ok, map()} | {:error, :unavailable}
The services and routes each recorded trace's spans named, newest
4096 traces: %{trace_id => %{services: MapSet, routes: MapSet}}.
Only spans this collector recorded count, from the SDK and from
ingest/1 alike, each credential-masked as Insider.Redaction.Credentials
masks event text; a trace keeps at most eight of each. A live view scoped
to services or routes resolves its traces from it (Insider.Live.View).