Mechanistic interpretability (MI) enables monitoring and steering LLMs by
operating on their internal state during inference. MI applications
remain difficult to develop, expensive to execute, and challenging to manage
at runtime.
We present xMIx, a system that integrates MI applications as
first-class components of LLM serving, enabling simplified development,
efficient execution, and flexible deployment. Guided by a comprehensive
analysis of modern MI applications, xMIx streamlines the MI apps
development via a simple Python-based Domain Specific Language (DSL) for
accessing intra- and inter-layer runtime model state at well-defined trigger
points during inference. xMIx enables efficient access to ephemeral attention
scores via a novel FlashInspect kernel that retains the performance benefits of
FlashAttention.
xMIx's compiler integrates multiple MI applications into the performance-optimized inference execution flow while ensuring conflict-free coexistence. xMIx's
runtime supports dynamic, per-request activation and deactivation of MI apps
without service interruption or performance degradation when inactive,
leveraging xMIx' dynamic memory management and CUDA graph-compatible
execution.
We integrate xMIx with vLLM and evaluate ten MI apps, including sparse
autoencoders, steering, and jailbreak detection, across four
state-of-the-art LLMs. xMIx incurs negligible execution overhead, up to two
orders of magnitude lower than existing frameworks, while enabling per-request
activation and deactivation up to 1,000 times faster than alternatives.
These results demonstrate that xMIx makes MI applications practical for
deployment in production LLM serving systems.