Transport-agnostic live observation of the node it runs in.
One live collector session records bounded Logger, Telemetry, and Poller
evidence. A supervised owner under Insider.Supervisor, never the caller,
owns the session, so short-lived transport processes such as a Tidewave
project_eval or an :erpc call from an observer node can start, read, and
stop it across separate calls. A renewable lease stops an abandoned session
through the normal collector cleanup path.
Every argument and return value is a distribution-safe plain term:
string-keyed maps, binaries, integers, booleans, nil, and lists. The only
process identifier the API accepts is a subscriber pid; none is returned.
Errors are {:error, map} with a closed "code" vocabulary documented in
Error codes; the
documentation index lists the rest of the reference.
Insider.Live.JSON returns the same
results as canonical JSON.
An agent should start with welcome/1: by default an agent quick start
(the steps, the core observer commands, and symptom recipes), and with
%{"full" => true} a document generated from this code that lists every
operation, option, default, bound, and error code, the transports with
example invocations, and how to report Insider bugs and behaviour that
differs from the document with report/1.
Live observation is available only in development, test, and ci, as resolved from host configuration by the node itself; callers cannot assert an environment. It is not a production collector.
Summary
Functions
Reports the VM's memory allocators once (insider.live-alloc/v1) with
recon_alloc: allocated and used bytes, the share in use, bytes by
allocator type, and the fragmentation of the "top" allocator instances
with the largest multiblock carriers. Without recon on the node it is
component_unavailable (component recon); the observer's --load
pushes recon with Insider.
Changes a running session's recording options without losing its
history: every start option given replaces the session's value (a list
replaces the whole list) and the rest keep theirs. "reader" names who
amends.
Marks the anomalies of a digest window normal for this session, as
insider.live-baseline/v1: every candidate's kind, hashed identity, and
count over its window (never its line, names, or values), with the
earlier entries the window does not have. Later digests and status fold them into one
line and lead with what is new; one that reaches ten times its baseline
comes back, saying so. "since" is the window (default latest-5m),
read through the reader's view; "show" returns the baseline, "clear"
removes it. The session keeps it for its life and its read-only window.
A window without anomalies leaves the baseline as it was.
Finds binary leaks once (insider.live-bin-leak/v1), after
recon:bin_leak/1: every unprotected process's binaries are measured,
it is garbage collected, and measured again. "by_retained_bytes" ranks
what each process still references after the collection (a leak keeps
it); "by_freed_bytes" ranks what the collection released. Insider's
own processes are summed apart. Option "top": entries per ranking.
Reports what this node can observe, without starting anything.
Returns who changed watched in-memory values ("watch_writes") as
insider.live-changes/v1: each write, oldest first, with its key, its
value (rich only), and its writer (the process, and the GraphQL
operation, route, user agent, user id, and trace id of the request it
belonged to, from that process's events, else from the session's kept
mutation record). "key" keeps one key's writes; "human" adds "text".
It reads the session newest first, within the correlated budget, through
the reader's view, and renews the lease.
Summarises a window of the session's events as insider.live-digest/v1,
and renews the lease.
Lists the Telemetry event names this node can emit, without starting or
running anything: names with an attached handler (:telemetry.list_handlers/1,
except Insider's own) and literal event lists passed to
:telemetry.execute/2,3 and :telemetry.span/3 (expanded to :start,
:stop, and :exception) in loaded application modules, read from their
debug info. Each name carries its "sources": the source ("handler",
"execute", or "span") and the module.
Returns redacted events with a receipt greater than cursor, in receipt
order, with explicit "gaps", and renews the lease.
A handoff record (insider.live-handoff/v1) for passing a session to
another agent: the node, the welcome hash, the session id and state, its
recording configuration (which every reader shares), its named readers
with their cursors, the newest receipt, and the commands that resume it.
It reads the session's status and changes nothing. The observer command
is handoff. (handoff/1 is the name the production-artifact boundary
reserves for a host delivery client, so this is handoff_record/1.)
Records spans exported by another process, such as a frontend server, in
the running session as otel.span events.
Marks the moment before an action as insider.live-mark/v1, for pairing
with a programmer who acts in the app while the agent watches: mark
("about to click Confirm"), ask them to act, then read since the mark
(digest/2, trends/2 and timeline/2 with "since" => "mark",
events_since/3 with cursor "mark"; "mark:LABEL" for the newest mark
with that label) instead of guessing a time or a cursor.
Lists a running or stopped session's pairing marks as
insider.live-marks/v1, oldest first, each with its "age_ms", and how
many older marks it let go ("dropped"). "human" adds "text", one
line per mark.
Lists the session's kept mutation records (Insider.Live.Mutations) as
insider.live-mutations/v1, oldest first, through the reader's view:
each mutation's operation, result, response typenames, time, trace,
process, the client's user agent, and the record ids among its
arguments (rich only), with the ring's accounting. "id" keeps the
records whose arguments name that id; "human" adds "text". The
records outlive the Recorder's eviction. It renews the lease.
Lists the node's ports once (insider.live-ports/v1): counts by type
(the first word of the port's name, such as cat) with their OS
processes, then the ports matching "type" and "owner" (a registered
name or pid) with their owner, OS pid, and I/O counters, and the owners
holding the most of them, all from one read of at most
bounds.inspect.ports_scanned_max ports. Option "top" bounds each
list. A port's full name is returned only in the rich profile; when its
arguments name a credential anywhere or hold anything a shell may rewrite
(an escape, a control character, $, a backtick), the whole argument
list is masked.
Inspects one process once (insider.live-process/v1): a process_info
subset (status, initial and current function, queue length, memory,
reductions, links by kind, monitors), the current stacktrace's top
frames with file and line, its binaries' count and bytes, and a mailbox
summary: queued messages counted by shape and size class, the shapes of
the first "messages", and how many queued GenServer.calls come from
callers that already exited. A shape is a message's structure with types
in place of values, such as {:"$gen_call", {pid, list}, {:price, binary}};
no message value, dictionary entry, or state is ever read. The mailbox is
read only up to bounds.inspect.mailbox_read_max messages.
Renews the session lease.
Queues a report about Insider itself on this node.
Lists the reports queued on this node, oldest first.
Starts the live session.
Reports a running session, or the retained record of an ended one.
Stops the session and returns its record with the cleanup receipt.
Pushes {:insider_live_events, session_id, batch} messages to pid.
Returns the correlated timeline of one process ("pid", as events name it
in subject.value) or one request ("request_id", Plug's request id) as
insider.live-timeline/v1, across every source, and renews the lease.
Returns one trace as insider.live-trace/v1: the spans the session
recorded with this W3C trace_id (32 lowercase hex digits) merged in time
order with the Logger and Telemetry events of the same trace, and renews
the lease.
Trends across a window's summary windows, as insider.live-trends/v1, and
renews the lease.
Removes a subscriber; removing an absent one succeeds.
Returns the welcome document for an agent.
Types
Functions
Reports the VM's memory allocators once (insider.live-alloc/v1) with
recon_alloc: allocated and used bytes, the share in use, bytes by
allocator type, and the fragmentation of the "top" allocator instances
with the largest multiblock carriers. Without recon on the node it is
component_unavailable (component recon); the observer's --load
pushes recon with Insider.
Changes a running session's recording options without losing its
history: every start option given replaces the session's value (a list
replaces the whole list) and the rest keep theirs. "reader" names who
amends.
The session keeps its id, cursors, readers, and subscribers. Its events so
far stay readable, and the next receipt is a live.session_amended event
that every reader sees through any view: what changed, from and to, and
"uncollected_ms", the pause in collection while the collectors
restarted. Returns insider.live-amendment/v1; "amend_failed" when the
new options could not start ("rolled_back" says whether the previous
ones run again).
A "telemetry_events" prefix entry such as ["shop", "*"] is expanded
as start/1 expands it, in this process before the owner is asked, and
the result lists each prefix's names under "telemetry_event_prefixes".
Marks the anomalies of a digest window normal for this session, as
insider.live-baseline/v1: every candidate's kind, hashed identity, and
count over its window (never its line, names, or values), with the
earlier entries the window does not have. Later digests and status fold them into one
line and lead with what is new; one that reaches ten times its baseline
comes back, saying so. "since" is the window (default latest-5m),
read through the reader's view; "show" returns the baseline, "clear"
removes it. The session keeps it for its life and its read-only window.
A window without anomalies leaves the baseline as it was.
Finds binary leaks once (insider.live-bin-leak/v1), after
recon:bin_leak/1: every unprotected process's binaries are measured,
it is garbage collected, and measured again. "by_retained_bytes" ranks
what each process still references after the collection (a leak keeps
it); "by_freed_bytes" ranks what the collection released. Insider's
own processes are summed apart. Option "top": entries per ranking.
@spec capabilities() :: map()
Reports what this node can observe, without starting anything.
"welcome" carries the welcome document's hash and how to read it, and
"quiet_default_telemetry_events" the Telemetry events a session without
options records on this node.
It includes the host's disclosure profile and the credential masking policy id.
Returns who changed watched in-memory values ("watch_writes") as
insider.live-changes/v1: each write, oldest first, with its key, its
value (rich only), and its writer (the process, and the GraphQL
operation, route, user agent, user id, and trace id of the request it
belonged to, from that process's events, else from the session's kept
mutation record). "key" keeps one key's writes; "human" adds "text".
It reads the session newest first, within the correlated budget, through
the reader's view, and renews the lease.
Summarises a window of the session's events as insider.live-digest/v1,
and renews the lease.
The window starts at "since": by default "latest-60s", the first
event the Recorder admitted in the last minute ("latest-DURATION" in
whole ms, s, m, or h, 1 s to 24 h, measured on the Recorder's
admission clock, not the events' own wall times); an integer cursor
(0 is the session start); "latest-N" for the newest N receipts; or
"mark" ("mark:LABEL"), after the newest pairing mark (with that label,
see mark/2), with "window" "mark" naming it. It
ends at "upto", the newest receipt when the digest starts; events that arrive
while it reads are left for the next digest. It is read with
events_since/3, through the same disclosure boundary, and returns
"counts" by kind, severity, source, and supervisor offender,
"crash_loops" with their period and reason fingerprint, the top
"error_signatures", the "latency" tables from summaries and recorded
durations with the summaries' exemplar receipts, "trends" across the
summary windows (as trends/2), the Logger "logger_bursts", the longest
mailboxes ("processes"), "vm" deltas, and the window's "filtered"
and "gaps". One budget,
bounds.digest_budget_ms, covers the "latest-N" lookup or the time
window's start search and every read; a search it cannot finish is a
"timeout", and a start it could not narrow exactly has
"start_exact" => false. The "window" says what was asked for
("since"), its start ("from_wall_time"), when the digest was made
("as_of"), and the age of its newest event ("age_ms"), and each hint
names the "window" it was worked out over;
an unfinished "window" has "complete" => false and the
"next_cursor" to continue from. "human" => true adds "text", a
short plain-text summary for the developer.
Lists the Telemetry event names this node can emit, without starting or
running anything: names with an attached handler (:telemetry.list_handlers/1,
except Insider's own) and literal event lists passed to
:telemetry.execute/2,3 and :telemetry.span/3 (expanded to :start,
:stop, and :exception) in loaded application modules, read from their
debug info. Each name carries its "sources": the source ("handler",
"execute", or "span") and the module.
Option "prefix", a list of segments such as ["shop"], keeps only the
names under it. The scan is bounded in modules, names, and time, and
"truncated_by" names what bounded it. At most two discoveries run at
once (busy).
@spec events_since(String.t(), non_neg_integer() | String.t(), map()) :: result()
Returns redacted events with a receipt greater than cursor, in receipt
order, with explicit "gaps", and renews the lease.
cursor is an integer of at least 0, or "latest": the newest receipt
once the read has drained the collector inboxes, so the batch is empty and
its "next_cursor" starts a tail; or "mark" ("mark:LABEL"): after the
newest pairing mark (with that label, see mark/2), with the batch's
"mark" naming it, else mark_not_found.
Options: "limit" (events returned), "max_bytes" (canonical JSON bytes
of the events returned), "scan_limit" (retained events examined),
"upto" (the newest receipt the read may return, from the starting
cursor, after "latest" resolves, to the newest receipt, else
invalid_options; losses are counted only up to it), and "view", a
read-time filter that narrows only this reader's events.
"truncated_by" names the bound that ended a truncated read. A read is
complete when it reaches its snapshot boundary: "upto" when given, else
the newest receipt when it began. Every batch reports "filtered" counts.
Pass the returned "next_cursor" to the next call.
A handoff record (insider.live-handoff/v1) for passing a session to
another agent: the node, the welcome hash, the session id and state, its
recording configuration (which every reader shares), its named readers
with their cursors, the newest receipt, and the commands that resume it.
It reads the session's status and changes nothing. The observer command
is handoff. (handoff/1 is the name the production-artifact boundary
reserves for a host delivery client, so this is handoff_record/1.)
Records spans exported by another process, such as a frontend server, in
the running session as otel.span events.
spans is a list of at most Insider.Collector.Otel.ingest_max/0 plain
span maps ("trace_id", "span_id", "parent_span_id", "name",
"span_kind", "status", "status_message", "service", "scope",
"start_time_us", "duration_us", "attributes", and "pid"). The
session's otel component applies the span contract, its disclosure
profile, and credential masking to each one, exactly as to the node's own
spans, and records it with "capture" "ingest". The result counts the
spans "accepted", "rejected" by the contract, "filtered", "shed"
under load, and "duplicate".
Ingest is idempotent per {trace_id, span_id} within one otel collector
of the session: a span already recorded from an ingest is counted as
"duplicate" and not recorded again (the collector remembers the last
16,384 recorded ids; one it forgets is counted as seen_evicted in the
otel component's status). Every amendment replaces the collector, even
one rolled back into a replacement, and so resets dedupe: a span ingested
before it is recorded again after it, and seen_evicted does not count
that reset.
A timeout means the outcome is unknown: the collector may have
taken the spans and still be recording them when the call stops waiting,
so some, all, or none may be recorded. A retry of the same spans does not
duplicate a span while its id is among the last 16,384 recorded; a
forgotten id is recorded again, and seen_evicted counts the forgotten
ones; a retry across an amendment is recorded again.
An ended session is session_terminal, and a session started without the
"otel" component component_unavailable. The Lab's otlp-receive
calls this for each OTLP/HTTP trace export it receives.
Marks the moment before an action as insider.live-mark/v1, for pairing
with a programmer who acts in the app while the agent watches: mark
("about to click Confirm"), ask them to act, then read since the mark
(digest/2, trends/2 and timeline/2 with "since" => "mark",
events_since/3 with cursor "mark"; "mark:LABEL" for the newest mark
with that label) instead of guessing a time or a cursor.
Options: "label" (required; one line, credential-masked in every
profile, at most bounds.mark_label_max_bytes bytes) and "reader". The
mark is the newest receipt now ("cursor": a read since it starts after
it), its time, label and reader, and is recorded as a live.mark event
("recorded" says whether; views never hide it). A session keeps its
newest bounds.marks_max marks, through amendments and after a stop.
"next" is the command to read since it. It renews the lease.
Lists a running or stopped session's pairing marks as
insider.live-marks/v1, oldest first, each with its "age_ms", and how
many older marks it let go ("dropped"). "human" adds "text", one
line per mark.
Lists the session's kept mutation records (Insider.Live.Mutations) as
insider.live-mutations/v1, oldest first, through the reader's view:
each mutation's operation, result, response typenames, time, trace,
process, the client's user agent, and the record ids among its
arguments (rich only), with the ring's accounting. "id" keeps the
records whose arguments name that id; "human" adds "text". The
records outlive the Recorder's eviction. It renews the lease.
Lists the node's ports once (insider.live-ports/v1): counts by type
(the first word of the port's name, such as cat) with their OS
processes, then the ports matching "type" and "owner" (a registered
name or pid) with their owner, OS pid, and I/O counters, and the owners
holding the most of them, all from one read of at most
bounds.inspect.ports_scanned_max ports. Option "top" bounds each
list. A port's full name is returned only in the rich profile; when its
arguments name a credential anywhere or hold anything a shell may rewrite
(an escape, a control character, $, a backtick), the whole argument
list is masked.
Inspects one process once (insider.live-process/v1): a process_info
subset (status, initial and current function, queue length, memory,
reductions, links by kind, monitors), the current stacktrace's top
frames with file and line, its binaries' count and bytes, and a mailbox
summary: queued messages counted by shape and size class, the shapes of
the first "messages", and how many queued GenServer.calls come from
callers that already exited. A shape is a message's structure with types
in place of values, such as {:"$gen_call", {pid, list}, {:price, binary}};
no message value, dictionary entry, or state is ever read. The mailbox is
read only up to bounds.inspect.mailbox_read_max messages.
Options: "name" (a registered name such as "MyApp.Worker") or
"pid" (a local pid such as "<0.200.0>"), "frames", and
"messages". A protected process is protected_process; an absent one
process_not_found.
Like every inspection, the result passes the disclosure profile and
credential masking ("redactions"), and when a session runs it is also
recorded in it as an inspect.process event; "session" says whether.
Renews the session lease.
Queues a report about Insider itself on this node.
Report Insider bugs and any behaviour that differs from welcome/1.
Options: "category", one of "bug", "doc_mismatch",
"missing_capability", or "confusing_output", and "message", a
printable text of at most 2,048 bytes. Insider adds the context itself: its
version and commit, the welcome hash, the node, the environment, and the
last operation and error code, never event payloads or option values.
The queue is bounded memory in this node: a report whose fingerprint
(category and normalised message) is already queued increments its
"count", and the oldest report is evicted beyond 100. Nothing leaves the
node; the Lab observer's reports command copies the queue and its
reports submit files GitHub issues only when the operator or host allows.
@spec reports() :: result()
Lists the reports queued on this node, oldest first.
Starts the live session.
Options: "components" (a non-empty subset of "calls", "logger",
"otel", "poller", and "telemetry"; "otel" records finished
OpenTelemetry spans and needs the SDK on the node; default ["logger", "telemetry"] when the node has quiet
default Telemetry events, else ["logger"]), "telemetry_events" (event
names as lists of existing-atom strings, recorded with phase :event;
default: the quiet default events), "telemetry_measurements"
(measurement names to project), "disclosure" ("rich" or "strict", no richer than the host
profile; default the host profile), "max_events", and "lease_ms".
Logger message text and metadata are collected only in the rich profile.
A "telemetry_events" entry whose last segment is "*", such as
["shop", "*"], is a prefix: it stands for every event name discover/1
finds below it now, counted against the event bound after expansion. The
result lists each prefix's names under "telemetry_event_prefixes".
Record-time filters, applied in the emitting process before anything is
recorded: "logger_level" (default "warning"), "logger_include",
"logger_exclude", "exclude_origins" (default ["oban"]),
"telemetry_min_duration_ms" (default 100),
"telemetry_event_min_duration_ms", "telemetry_summaries",
"telemetry_summary_interval_ms", "telemetry_duration_scopes" (a map of
summarised kind to "handler" or "caller", how its duration was
measured; an undeclared kind's scope is unknown), "logger_summaries"
(Logger rules whose messages are counted per message template, call site,
and level, a burst, instead of recorded; at most 64 bursts per interval,
the rest folded into one overflow row per level), and
"logger_summary_interval_ms". See
Filters, summaries, and views.
Poller options, with the poller component: "poller_interval_ms" (default
5,000), "top_processes" and "top_ets_tables" (entries in each process
and ETS ranking; default 5), and "watch_processes" (registered names to
sample every interval). See
Process and ETS evidence.
Call-trace options, with the calls component (OTP 28 and later): "calls"
(functions to trace as "MOD.FUN/ARITY", required), "calls_results"
("exceptions", "all", or "none"), "calls_max", "calls_rate_max",
"calls_rate_ms", and "calls_max_bytes". See
Call traces.
Reports a running session, or the retained record of an ended one.
A running session's record carries "next_receipt", the newest receipt;
"recorder": "oldest_receipt", "retained_events", "retained_bytes",
"max_events", "max_bytes", "archived_events" (events kept from
before the last amendment), and "sources", each share group's
"share_percent", "reserved_events", "reserved_bytes",
"retained_events", "retained_bytes", and "evicted" (conservative:
after a collector is killed mid-write it can exceed the events really
evicted, never fall short; a batch's "evicted" gap stays exact); and
"insider_memory": "processes" (bytes per Insider process serving the
session), "processes_bytes", "recorder_bytes" (the Recorder's tables
and event binaries), and "total_bytes". It lists "readers", each
named reader with its "cursor", "behind", "last_seen_ms_ago",
"last_seen_at", "subscribed", and "active", and the latest
"amendments". An ended session still readable after a stop carries
"readable_remaining_ms" and its readers.
Stops the session and returns its record with the cleanup receipt.
It refuses with "readers_active" while another named reader read within
reader_active_ms or is subscribed, unless "force" is true. Unnamed
readers never hold a stop. "reader" names who stops, so that reader's own reads do not
count. A stopped session stays readable with events_since/3 for
stopped_readable_ms (its record's "readable_until"), so peers can
drain it. Stopping an ended session returns its record again.
Pushes {:insider_live_events, session_id, batch} messages to pid.
A batch has the events_since/3 shape, and each batch starts at the
"next_cursor" of the one before it. The owner monitors pid and drops
it when it exits or its node disconnects. The subscription does not renew
the lease and ends with one batch whose "state" is terminal and which
carries the "receipt". Options: "cursor" (default 0; an integer no
newer than the newest receipt, or "latest"), "max_events" (the events
pushed in total; then nothing more is sent but the terminal batch), and
"view", as for events_since/3. Subscribing again replaces the view and
"max_events", and moves the cursor only when one is given.
Returns the correlated timeline of one process ("pid", as events name it
in subject.value) or one request ("request_id", Plug's request id) as
insider.live-timeline/v1, across every source, and renews the lease.
A request's events are those carrying its id, the events of its
processes within its time window, and the events and spans of its
traces. Options as for trace/3, plus exactly one of "pid" and
"request_id".
Returns one trace as insider.live-trace/v1: the spans the session
recorded with this W3C trace_id (32 lowercase hex digits) merged in time
order with the Logger and Telemetry events of the same trace, and renews
the lease.
An event joins by its trace id; an event without one joins when the
process that emitted it ended one of the trace's spans and its time falls
inside that span. Each item has "at", "depth" in the span tree,
"span_id" (the span it belongs to), "joined_by", and the disclosed
"event". "unjoined" says what could not be joined (parent spans that
were not captured, events without a time), "scan" how much of the
session was read ("complete" is false when the read budget ran out),
and "gaps" what the reads lost. Options: "limit" (items, default
200), "max_bytes" (default 262,144), and "human", which adds
"text", an indented plain-text rendering.
Trends across a window's summary windows, as insider.live-trends/v1, and
renews the lease.
The window is a digest's ("since", read through events_since/3 within
bounds.digest_budget_ms). For each summarised Telemetry kind,
"telemetry" lists its windows (started_at, ended_at, count,
rate_per_min, errors, error_pct, p50_us, p95_us, max_us, the
exemplar receipts, exemplars_lost, and first_error_retained) and the
change of rate, p95, max, errors and error share from the first window to
the last, with its direction. Its "duration_scope" is the summaries':
"measured_in" is "handler" or "caller" only when the session
declared it ("telemetry_duration_scopes"), else "unknown", and
"emitter" is only the context the exemplars were emitted in. A note
compares a declared handler series with a declared caller series only
conditionally: Insider does not establish that they describe the same
operation. "logger" lists each Logger burst's windows, one series per
template fingerprint, call site fingerprint, and level, and each level's
overflow row. "poller" lists Poller series, one point per poller cycle:
binary_bytes and port_count per vm_sample, ports per type per
port_top (a zero only at a complete census), and message_queue_len
per ranked or watched process ("coverage" ranked, watched, or
mixed), each with "unit", "change", "per_min", "max", and its
points; "poller_omitted" and "poller_untracked" count what the bounds
left out. "units" names every value's fixed unit, and "notes" what
the series show together. Options: "since", "kinds" (the Telemetry
kinds to show, "logger_summary" for the bursts, and "vm_sample",
"port_top", "process_top" or "process_watch" for the Poller
series), "windows" (the most recent windows per series), and
"human", which adds "text".
Removes a subscriber; removing an absent one succeeds.
Returns the welcome document for an agent.
By default it is the agent quick start (insider.live-welcome-quick-start/v1):
the steps, the core observer commands with their flags, the symptom
recipes, where to read the full reference, and "runtime". With
%{"full" => true} it is the full reference (insider.live-welcome/v2),
generated from the code's own tables: the operations with their
options, defaults, and bounds, the target's and the observer's error codes,
components, allowed measurement names, the transports with example
invocations and the observer's value syntax for every flag, a short guide
(what to do first, the act-then-read loop, how to stop, and the duty to
report), and "runtime": this node, its environment, the active session,
and this node's quiet default Telemetry events. "hash" covers every key
of the full reference except "hash" and "runtime"; both forms carry
it. Any other option is {:error, map} with code "invalid_options".