Layered Cluster-Compute Architecture: A Cost-First Verified Cluster-Kernel Lifecycle for Reducing Repetitive LLM Inference

Arcman

Layered Cluster-Compute Architecture: A Cost-First Verified Cluster-Kernel Lifecycle for Reducing Repetitive LLM Inference

https://doi.org/10.5281/zenodo.20483510

Description

This technical whitepaper proposes Layered Cluster-Compute Architecture, or LCCA, a cost-first and verification-aware lifecycle for reducing repetitive large language model inference.

The central claim is not that caching, routing, retrieval, or tool use are new. Rather, LCCA organizes these familiar components into a governed lifecycle for routine AI work: detect repeated workloads, cluster them, route them, verify them, compile them into reusable cluster-kernels, register them with ownership and rollback, monitor them for drift, and retire or rebuild them when necessary.

A cluster-kernel is defined as a versioned, callable, testable, and auditable routine compiled from repeated verified traces. LCCA is designed for high-frequency, stable, bounded, and verifiable workloads such as metadata generation, document engineering, support routing, standard code scaffolding, repository descriptions, schema transformations, and low-risk recurring tool workflows.

The primary evaluation metric is cost per verified useful output. Secondary metrics include Wh per verified output, TOTEN per verified output, kernel hit rate, frontier escalation rate, verification pass rate, fallback success rate, latency, and pilot risk magnitude. TOTEN is defined as Total Token-Equivalent Number and is used to account for routing prompts, verifier prompts, tool instructions, fallback calls, retrieval context, night-cycle distillation prompts, and other token-equivalent overhead.

The architecture is sidecar-first rather than replacement-first. It is intended to be tested through offline replay, shadow mode, read-only assist, and limited canary rollout before any production traffic is affected. It also includes PromptOps cards, allowing AI assistants to help engineers perform workload analysis, kernel candidate mining, cost estimation, verifier design, dashboard configuration, and night-cycle distillation under human supervision.

Version v0.4 adds production-edge safeguards for extreme boundary states. These include asymmetric low-cost verifier policy, verifier tax accounting, gateway latency budget, fast-path short-circuit routing, fractional canary probing for kernel drift, in-flight request coalescing for burst new loads, probabilistic jittered degradation for hot-kernel eviction, stale-while-revalidate, stale-if-error, and capacity-aware frontier leakage control.

LCCA does not claim to replace foundation models, frontier reasoning, retrieval systems, caches, routers, or tool-calling frameworks. It proposes a practical engineering lifecycle for reducing repetitive LLM inference while reserving frontier models for genuinely novel, uncertain, or high-value tasks.

The guiding motto is: compile the routine; reserve the frontier.



Key word

Layered Cluster-Compute Architecture
LCCA
Cluster-kernel
Verified cluster-kernel lifecycle
Large language models
LLM inference
LLM cost optimization
AI infrastructure
MLOps
LLMOps
Cost per verified output
Verified useful output
TOTEN
Total Token-Equivalent Number
PromptOps
prompt caching
Semantic caching
Model routing
RAG
Tool calling
Verification layer
Verifier tax
Gateway latency budget
Kernel drift
Fractional canary probing
Night-cycle distillation
In-flight request coalescing
Singleflight
Cache stampede
Probabilistic jittered degradation

Stale-while-revalidate
Stale-if-error
Frontier model
Safe rollout
Canary deployment
Pilot risk magnitude
AI energy efficiency
Wh per verified output