This separate, non-published Mix project produces bounded synthetic load and soak evidence for the production-readiness collector session. It compares the same fixed-profile, fixed-seed host workload without and with collection, then writes canonical evidence bound to the checked-out revision and runtime.
It is a development and CI measurement tool. A successful run does not approve
a production budget, establish production capacity, or enable staging or
production observation. Every report records production_capacity as false
and budget_evaluation as not_approved.
Run it
From the repository root:
cd tools/insider_load
mix deps.get --check-locked
mix insider.load --profile config/load.json \
--output ../../_build/insider_load/evidence/load
For the longer soak profile, replace config/load.json with
config/soak.json. Use a new output path that does not already exist for each
run so artifacts from different revisions, runtimes, or profiles cannot be
mistaken for one evidence set.
The checked-in profiles define the seed, workload phases, rates, collector bounds, sampling interval, and artifact limits. Change them only as reviewed configuration: the report binds the selected profile rather than treating a locally modified profile as equivalent evidence.
Evidence and failure behavior
The output directory contains the canonical report with bounded measurement
samples and a content manifest for the complete evidence set. A typed failure
before report construction emits a canonical failure.json and manifest
instead. Report and failure bytes are verified through bounded reads of owned
regular file descriptors in private, restricted sibling staging directories
before publication. The public output directory is exclusively reserved and
its files are created exclusively; neither a partial public artifact nor a
competing entry is removed on failure. A verified failure-only artifact is
published if the output remains empty; otherwise publication failure is
explicit. Normal staging directories are removed after verification; one is
retained only when a substituted path or foreign content makes cleanup unsafe.
The report binds the repository revision and tree state, dependency-lock and profile hashes, runtime and operating-system identity, scheduler configuration, workload seed, and sample interval.
The harness fails closed when its structural evidence gates fail, including host-workload errors or missed scheduled operations, unaccounted loss at any collector ingress, missing pressure during the exact declared overload interval, unavailable required runtime measurements, failure to empty bounded queues after emitters stop, incomplete cleanup, or failure to start and stop a fresh session. A nonzero exit normally leaves bounded output that explains the result. If filesystem publication fails, the error is explicit and no complete artifact is claimed. CI uploads output when present before it fails the job.
Measured latency, throughput, CPU, scheduler, memory, garbage collection, resource counts, recovery, and leak slope are evidence, not universal pass thresholds. Interpreting them requires a target service topology, representative traffic, approved overhead and recovery budgets, retention requirements, protected-target declarations, canary design, and named human approvers.
The workload is synthetic and local to the development or CI host. It uses no
network or vendor integration, credentials, source mutation, arbitrary
evaluation, or production authority. It excludes SystemMonitor and does not
retain raw Logger payloads, Telemetry values, event bodies, process identifiers,
or recorder handles in the report.
The standalone harness temporarily enables scheduler wall-time accounting for the paired measurement and restores the exact prior VM setting afterward. A required metric that remains unavailable is represented by typed availability and fails the evidence gate; a numeric zero never substitutes for availability.
Missed-operation localization
Each phase summary carries a scheduling section next to host so a nonzero
missed_operations count can be placed in time and attributed to a mechanism:
missed_by_segmentcounts misses inwarmup,steady,overload, andrecovery. Warmup precedes the measured interval, so its count is reported but is not part ofhost.missed_operations; the other three sum to it, andoverloadequalsoverload.missed_operations.missed_by_causeseparatesskipped_tick(a 10 ms tick deadline was overshot by at least one whole tick, so that tick's scheduled work was never started) fromsegment_end_remainder(work still scheduled when the segment deadline passed, including a batch halted at that deadline). The two sum tohost.missed_operations.max_tick_lag_usis the largest observed overshoot of any tick or segment deadline during the measured interval. A deadline is observed when the workload loop regains control, so a stall that ends after the segment deadline is measured against that deadline. Sub-millisecond lag is ordinary timer granularity.
A report whose sums do not reconcile is invalid. These counters do not change
any pass/fail gate; any nonzero missed_operations still fails the phase. The
per-sample series keeps only the cumulative missed_operations total; cause
attribution is summary-only. A phase that fails before its workload completes
(a workload fault, a collector session that cannot start, or a failed
instrumented measurement) reports every operation the profile scheduled in
steady and overload as segment_end_remainder misses, matching the
declared overload accounting, so the report stays valid and keeps the original
failure reason and cleanup evidence.
Verify an evidence directory offline
scripts/verify-load-evidence <dir> [<dir>...] (repository root) re-derives
the integrity of emitted evidence without running the harness. For each
directory it recomputes the sha256 of report.json, profile.json, and
manifest.json, compares the report and profile against the manifest's
artifact digests and byte sizes, checks report.inputs.profile_sha256 against
the profile bytes, and checks that each id equals its integrity
content digest. No mix task or escript recomputes artifact identities, so the
script re-encodes the parsed documents with the OTP json module (OTP 27 or
later) in canonical form (keys sorted, compact, no trailing newline). A
manifest.json or report.json whose bytes differ from that encoding is
reported as manifest_not_canonical or report_not_canonical and fails the
directory, because an identity cannot be recomputed for bytes the writer did
not produce; otherwise manifest.id and report.id are recomputed from the
canonical body with id and integrity removed. It prints one JSON line per
directory (directory, git_sha, otp, elixir, outcome, report_id,
report_sha256, manifest_sha256, profile_sha256, id_recomputed,
mismatches, ok) and exits nonzero if any directory fails. id_recomputed
is true when both documents were canonical, so it is always true when ok
is. The script verifies integrity and internal consistency only; it does not
validate the report against the current schema or interpret the measurements.
CI
Pull requests run config/load.json on every supported Elixir/OTP pair. The
weekly schedule runs config/soak.json; a manual workflow dispatch can select
either profile. CI artifact names include the full Git revision and runtime
pair. Artifacts are retained for 90 days and uploaded before the recorded
harness exit status is enforced.