IPE

What is built

• Configuration validation: Admits only the supported dense Qwen3 architecture.
• Pinned checkpoints: Verifies checkpoint revisions by hash and reads the tensor inventory directly from file headers.
• Explicit weight mapping: Maps every checkpoint tensor into the engine at bf16, fp32, or fp64, following the Transformers LM-head tying rule.
• Reference forward pass: Implements the decoder operators and an uncached forward pass whose fp64 mode serves as the numerical oracle.
• Cached generation: Runs prefill and decode over a preallocated KV cache with greedy and temperature sampling for static batches of equal-length prompts.

engine = Engine(loaded, batch=1, capacity=1024)      # K/V and rotary tables allocated here
result = engine.generate(prompt, 32)                 # greedy (temperature=, seed=, stop_ids= optional)

Each decode step indexes preallocated buffers with host integers, avoids reading device values back to the host, and calls PyTorch’s scaled dot-product attention without an explicit mask so the FlashAttention kernel stays eligible on CUDA.

How it is checked

Bitwise parity. Where operations run in the same order, outputs must match Transformers 5.17.0 bit for bit. On an Apple M4 Pro CPU, uncached forward passes match layer by layer for Qwen3-1.7B and Qwen3-0.6B at fp32 and bf16, including sequences up to 4,224 tokens at fp32. Teacher-forced cached decoding also matches the Transformers cached implementation for Qwen3-1.7B (bf16 over 1,024 tokens and fp32 over 4,224 tokens, across both attention backends), Qwen3-0.6B over 256 tokens, and synthetic grouped-attention checkpoints.

Numerical accuracy. Where reduction orders differ, such as cached versus uncached decoding, the test suite measures residuals against the fp64 oracle and reference variants. At fp32, the suite accepts valid floating-point reorderings while catching eight seeded defects. Because certain bf16 RMSNorm defects fall within the end-to-end tolerance, the RMSNorm operator has a dedicated test against exact reference values, a contract that will also govern future fused kernels.

On NVIDIA GPUs. Thirteen CUDA tests verify bf16 cached decoding over 1,024 tokens against an on-GPU fp64 oracle across both attention backends. They also check 32 decode steps for zero host-device synchronizations under PyTorch’s sync debug mode and zero allocator growth, SDPA execution without the math fallback, reproducible sampling, the TF32 guard, and the RMSNorm contract. The suite passed in three sessions on an RTX PRO 6000 Blackwell Server Edition and one on an H200. The 4,224-token GPU check has not run yet.

Bitwise uncached parity with Transformers does not hold on either GPU. On the RTX PRO 6000, the difference traces to the rotary tables: IPE builds its cos and sin tables on the host CPU, Transformers builds them on the GPU, and two entries still differ after rounding to bf16. Uncached logits match bit for bit through row 592 and first diverge at row 593. An external diagnostic that injected Transformers’ GPU tables into the engine restored bitwise-identical logits and layer outputs across all 1,024 rows for both attention backends. The H200 mismatch has not been diagnosed.

How it is benchmarked

The benchmark harness times fixed-work generation across IPE, Transformers (with dynamic and static caches), and vLLM, running one implementation per process and recording raw per-iteration timings.

Each process runs one cold call followed by 10 warm-up and 30 measured pairs of latency(N) and latency(1), timed from a synchronized start until device completion. Time per output token (TPOT) is (latency(N) − latency(1)) / (N − 1). A failed correctness check halts the session stage before profiling or timing begins.

A preliminary W1 baseline comes from one session on an RTX PRO 6000 Blackwell Server Edition with unlocked GPU clocks, which boosted only during vLLM runs, after the GPU suite passed. An external script ran bench/run.py in a custom launch order; the session’s gated bench stage has not run yet. Values are medians in milliseconds over 3 process launches, 6 for IPE. The harness’s required cross-check against vllm bench latency has not passed in any three-launch run, so these numbers are not final.

IPE runs eager PyTorch operators with no custom CUDA kernels or CUDA graph capture, while vLLM and the static-cache Transformers path run compiled decode steps and replay CUDA graphs. In the same session, comparing IPE’s odd and even launches gave an observed A/A tolerance of 2.3% of latency(256). Its per-launch medians spread 7.6%, for reasons not yet determined, which is still well below the gap to any comparator. Workloads W2 and W3, the 4,224-token GPU check, and kernel-level attribution of IPE’s decode step time are still pending.

The target configuration is the pinned Qwen3-1.7B checkpoint in bf16 on a single NVIDIA GPU (compute capability 8.0+, ≥ 24 GB VRAM). Model serving, paged KV allocation, continuous batching, multi-GPU execution, quantization, and speculative decoding are outside scope.

Links

• RepositorySetup, pinned checkpoints, test suite, and benchmark harness on GitHub.
• W1 measurementsMethod, quartiles, noise floor, harness cross-check diagnostics, and limitations.