SDK performance acceptance

This document defines the release-candidate performance budget. Measurements must use release builds, record the machine and Rust toolchain, run enough samples to report a median and a high percentile, and compare the same commit's SDK-backed and retained baseline fixture where applicable.

Required benchmarks

Scenario	Measurement	1.0 acceptance budget
Runtime plus session startup	Builder through ready idle session with a scripted provider	Median no more than 2 ms, and no more than 20% or 1 microsecond above the pre-SDK fixture, whichever is larger
Simple completion overhead	Scripted one-turn provider excluding simulated provider delay	Median no more than 10% or 100 microseconds above the pre-SDK fixture, whichever is larger
Event delivery	10,000 bounded provider deltas consumed by one run	At least 250,000 events/second median; p99 enqueue-to-consume latency below 5 ms
History snapshot	Clone and serialize 1,000 representative messages	Median below 10 ms and peak retained allocation below 3 times serialized size
Compaction orchestration	Partition, scripted summary, and atomic commit for a 1,000-message history	Median no more than 15% above the pre-SDK compaction fixture
Parallel tool batch	The same representative independent multi-read batch at limits of one and four	Report both distributions and the observed speedup; the parallel run must preserve ordered results
Streamed tool-argument capture	Geometric argument sizes streamed in fixed chunks, then cancelled before a final ModelResponse	Aborted history retains the complete tool call; median ns/byte at the largest size is at most 2x the smallest size
Overlapping tool preparation	The same 64-call prepare batch at execution limits of one and four	Peak concurrent preparations equals the batch size at both limits; ordered tool results are preserved
Slow consumer	Producer against a full bounded event channel	Memory remains bounded and cancellation completes within 250 ms after the consumer is dropped

Provider network latency, upstream rate limiting, authentication, OS keychain prompts, terminal drawing, and SQLite disk latency are reported separately and must not be attributed to SDK orchestration overhead.

The parallel tool batch fixture emits eight independent resource-aware reads. Each read includes the same one-millisecond scripted operation at both limits, and each sample checks that history retains model order. The evidence JSON records the raw samples for limits one and four plus the ratio of their median run times. This scenario reports speedup rather than setting a machine-wide minimum because timer resolution and available CPU vary across release runners.

The streamed tool-argument fixture uses geometric payload sizes (16 KiB, 64 KiB, 256 KiB) with fixed 256-byte deltas. After the final delta the run cancels so aborted history must come from stream capture, not a provider terminal tool call. Acceptance compares median nanoseconds per byte across the size span; a 2x growth limit rejects super-linear capture without a machine-specific time budget.

The overlapping tool-preparation fixture runs the same 64-call prepare batch at execution limits one and four. Preparation is currently unbounded by max_parallel_tools, so peak active prepare futures must equal the batch size at both limits while tool results stay in model order. Timings are recorded for evidence only.

Relative startup and compaction samples interleave the retained baseline and SDK candidate, alternating which runs first in each pair. This keeps both distributions exposed to the same short-term runner conditions without changing either fixture. Startup also uses a 1 microsecond absolute floor on top of the 20% relative budget so sub-microsecond timer noise on shared runners is not a false failure.

Regression policy

A result within both the absolute and relative budget is acceptable. A result outside either budget is material and blocks 1.0 unless the pull request:

1. identifies the measured cause;
2. explains the user-visible benefit that requires the regression;
3. updates the budget with maintainer approval;
4. includes the raw before/after benchmark artifact; and
5. records the intentional change in coordinated release notes.

Noise is not a waiver. Re-run on an otherwise idle machine, increase sample counts, and compare distributions. Do not hide regressions by changing fixture content, event capacity, optimization settings, or benchmark boundaries between the baseline and candidate.

Release evidence

Run the reproducible suite from the repository root:

./scripts/run_sdk_release_benchmarks.sh

The script uses the release benchmark profile, 20 samples by default, the in-target pre-sdk-retained-fixture-v1 baseline, and writes to target/sdk-release-evidence/sdk-release-benchmarks.json. Override RHO_BENCH_SAMPLES or RHO_BENCH_OUTPUT without changing benchmark boundaries.

For release evidence, the crate publication workflow automatically requires the SDK release evidence workflow to pass on the exact candidate commit before either crate is published. Run the same workflow manually when evidence is needed before release publication, then download its sdk-release-benchmarks-<commit> artifact. Artifacts are point-in-time measurements tied to the resolved source commit; they are not maintained as a current repository snapshot.

The release candidate must attach:

• benchmark command and commit IDs;
• CPU, memory, operating system, Rust version, and build profile;
• raw criterion or equivalent machine-readable results;
• median and p95/p99 values where applicable;
• baseline and SDK-backed deltas; and
• an explanation and approval for every material regression.

A release-candidate artifact satisfies these evidence requirements only for its recorded source commit. Rerun the workflow after code or toolchain changes.