blinklinmult.bench.timing

Timing a callable honestly.

Three decisions, each guarding against a way a benchmark lies.

Warm-up runs are discarded. The first ONNX call allocates its arenas and the first torch call picks kernels; folding that into the average reports startup cost as though it were steady-state throughput. bench_workers.py already established this convention for the dataloader – “times the second pass … so the figure reflects steady-state throughput”.

Percentiles, not just a mean. A model whose p99 is four times its median drops frames under a deadline even when its average looks fine. For a streaming workload the tail is the number that decides whether it works.

A monotonic clock. time.perf_counter_ns() cannot go backwards or be adjusted mid-run, unlike time.time().

Attributes

DEFAULT_WARMUP	Calls discarded before timing begins.
DEFAULT_REPEATS	Timed calls.

Exceptions

BenchError

Raised when a measurement is asked for on impossible parameters.

Classes

Latency

Timing of one configuration.

Functions

time_callable(→ Latency)

Time a callable at steady state.

Module Contents

blinklinmult.bench.timing.DEFAULT_WARMUP = 20

Calls discarded before timing begins.

blinklinmult.bench.timing.DEFAULT_REPEATS = 200

Timed calls.

Enough that p95 rests on ten samples and p99 on two. Reporting p99 from fewer would be reporting the maximum under another name, which is why Latency.tail_is_trustworthy() exists.

exception blinklinmult.bench.timing.BenchError

Bases: ValueError

Raised when a measurement is asked for on impossible parameters.

Initialize self. See help(type(self)) for accurate signature.

class blinklinmult.bench.timing.Latency

Timing of one configuration.

Parameters:

• samples_ms (np.ndarray) – Every timed call, in milliseconds. Kept whole rather than pre-summarised so a caller can re-derive any statistic.

• warmup (int) – Calls discarded before timing.

• frames (int) – Frames per call, so per-frame cost is comparable between a frame-wise model and one scoring 15-frame windows.

property p50_ms: float

Median call time.

Returns:

Milliseconds.

Return type:

float

property p95_ms: float

95th percentile call time.

Returns:

Milliseconds.

Return type:

float

property p99_ms: float

99th percentile call time.

Returns:

Milliseconds.

Return type:

float

property ms_per_frame: float

Median cost of one frame.

The only figure comparable across models with different window lengths: a 15-frame sequence model and a frame-wise one do different amounts of work per call.

Returns:

Milliseconds per frame.

Return type:

float

property fps: float

Frames per second at the median.

Returns:

Frames per second.

Return type:

float

property tail_is_trustworthy: bool

Whether enough samples were taken for p99 to mean anything.

With fewer than 100 repeats the 99th percentile is interpolated from the largest one or two samples, which is the maximum wearing a percentile’s name.

Returns:

Whether p99 rests on at least a few samples.

Return type:

bool

describe() → str

One line for a table.

Returns:

Median, tail and throughput.

Return type:

str

blinklinmult.bench.timing.time_callable(call: collections.abc.Callable[[], object], frames: int, warmup: int = DEFAULT_WARMUP, repeats: int = DEFAULT_REPEATS) → Latency

Time a callable at steady state.

Parameters:

• call (Callable[[], object]) – The work to time. Should allocate nothing it can avoid: build inputs before calling, so allocation does not land inside the timed region.

• frames (int) – Frames processed per call, for the per-frame figure.

• warmup (int) – Calls to discard first.

• repeats (int) – Calls to time.

Returns:

The samples and their summary.

Return type:

Latency

Raises:

BenchError – If frames or repeats is not positive, or warmup is negative.