OpenAI-Compatible Server

Metrale Engine serves via metrale-server — an OpenAI and Anthropic compatible HTTP API over axum. This chapter is the operator's reference for CLI flags, protocols supported, and the knobs that matter in production. The authoritative flag list is always met serve --help; the headings below match the groupings in the CLI so cross-referencing is easy.

CLI structure

met serve <MODEL> [--flags...]
met serve --model-from-path <PATH> [--flags...]
met --version
met --help

Every runtime configuration flag has a long-form name. Most are documented inline with #[arg] doc-strings in crates/server/src/cli/serve_args.rs.

Model selection and I/O

Memory / budget

Production rule of thumb for tight single-GPU deployments of 100B+ models: drop --max-prefill-tokens to 2048 and --max-batch-size to 1. The default 8192 sizes the scratch arena, not the KV pool; tuning down frees hundreds of MB.

KV cache precision

The flag takes a number or one of three words: auto (a fixed alias for 2, not a heuristic) and max/all (every attention layer). Anything else that fails to parse warns and falls back to 0.

--kv-high-precision-layers 0 is not "no promotion" for the turbo* family. 0 means defer to the per-dtype automatic value (main_modules/kv_dtypes.rs::auto_high_precision_layers, applied at serve_phases/kv_cache.rs). For bf16 / fp8 / nvfp4 the automatic value is None, so 0 really is zero. For every turbo* and asymmetric variant it is not: turbo2 and bf16k_turbo3v promote max(4, ⌈4·L/5⌉) layers, and all the others promote max(2, ⌈L/3⌉) — roughly a third of attention layers forced to BF16, which also shrinks the KV pool. Pass an explicit non-zero value if you want to control it; there is no spelling of this flag that promotes nothing under a turbo dtype.

See FP8 and NVFP4 for the trade-offs. Metrale Engine's recommendation per model family:

• Qwen3.5 family → nvfp4 KV.
• Qwen3.6 / Nemotron-H → fp8 with calibration.
• 122B-class (memory-constrained) → nvfp4 + --kv-high-precision-layers 2.
• Everything else → fp8 (safe default).

Speculative decoding

See the MTP deep dive. Use only one of --speculative, --self-speculative, --ngram-speculative — but note this is guidance, not an enforced constraint: none of the three carries a clap conflicts_with and cli/validate.rs has no rule for the combination, so passing several parses and serves. The scheduler then resolves them by silent precedence (ngram → self-spec → MTP) rather than rejecting the config. --dflash is enforced — it declares conflicts_with = "speculative". --prompt-lookup-decoding is enforced too: it requires --speculative and conflicts with --dflash, --self-speculative and --ngram-speculative.

Prompt lookup only changes which drafts a round verifies. Every emitted token is still the target's own pick, so greedy output is the same with the flag on and off wherever verify rows are row-invariant (the canonical tiers).

--num-drafts is also not a plain constant: when it is still 1, the model's MODEL.toml default_num_drafts replaces it (serve_phases/config.rs). On qwen3.6-27b that is 3, i.e. K=4.

Boolean flags

Every boolean flag is presence-only and takes no value: --enable-prefix-caching turns prefix caching on, and --enable-prefix-caching true is refused. A flag is named for its non-default state, so a feature that is on by default is turned off with --no-<feature> (--no-ssm-tail-midchunk, --no-high-speed-swap-graph). A lever whose default is decided elsewhere — by the compiled target, MODEL.toml or the environment — takes auto, on or off instead: --ssm-batched-recurrent, --content-loop-watchdog and --tool-grammar. In a recipe, a boolean key is written true or left out; key: false restates the default and the recipe is refused.

Environment levers

Some settings are METRALE_* environment variables rather than flags. Every one the engine reads is declared once in metrale_config::levers with its type, default, the flag that shadows it (if any) and the file that reads it; met dump-serve-options prints the list under levers. met refuses to start when the environment carries a METRALE_* name that is not declared: it is a typo or a lever that no longer exists, and running as if it applied would mislabel whatever was measured. A recipe's env: block may name only a runtime lever.

Scheduling / caching

Agent workloads (Claude Code, OpenCode): always enable --enable-prefix-caching and --scheduler slai. The prefix cache dominates wall-clock for system prompts + tool schemas; SLAI keeps streaming smooth under concurrent load.

Multi-GPU (Expert Parallelism)

See Multi-GPU & EP=2 for the full setup, including the NCCL env vars that matter on GB10.

Reasoning / tools

Observability / experimental

Endpoints

Rate limiting and auth

• --require-auth (with --auth-token <key> or --auth-tokens-file <path>) — requires an Authorization: Bearer <key> header on write endpoints. The presented token must match one of the loaded tokens (constant-time compare); there is no "accept any key" mode.
• Token-bucket rate limiter per key (crates/server/src/rate_limiter.rs). Off by default; enable by setting METRALE_RATE_LIMIT_RPM (requests/min) and/or METRALE_RATE_LIMIT_TPM (tokens/min) > 0 (bursts via METRALE_RATE_LIMIT_BURST_RPM / METRALE_RATE_LIMIT_BURST_TPM, default = the cap). A MAX_KEYS DoS guard bounds the key table.
• Body-size limit env-configurable via METRALE_MAX_BODY_BYTES (default 32 MiB — main_modules/serve_router.rs).

Changing the model without restarting

A running server can replace its model in place. Three routes reach it, and they share one code path:

• The dashboard. met serve with no MODEL starts the listener and opens the Library, where a model and one of its recipes can be picked. Selecting one loads it and returns to the Main view.
• The Library, on a server that is already serving. Same flow; the running model is released first.
• A client request naming a different model, Ollama-style — off unless --auto-swap is passed.

For deployments where the served model is part of the contract, leave --auto-swap off (the default): no client can then change what the endpoint is running. There is no opposing --no-auto-swap; a boolean flag names only its non-default state (see Boolean flags).

Even with --auto-swap, a swap needs a request whose model resolves to a DIFFERENT model with a known recipe. A name that is absent, unrecognised, or already live is served by the current model, exactly as before the flag existed.

What a swap preserves. Stored responses and conversations, rate-limiter buckets, the API-key policy and --dump all belong to the process, not the model, and survive unchanged — including while no model is loaded, when GET /v1/conversations/{id} still answers rather than reporting the model missing.

What it cannot change. The listening socket is bound once for the process lifetime, so a recipe's host/port are ignored with a warning and the model serves on the address already bound. Hot-swap is single-node: a recipe needing more than one rank is refused rather than half-applied.

While it runs. In-flight requests finish on the model they started on. New requests that need a model get 503 model_not_loaded — retriable, and the same shape /health already reports during startup. A load that fails restores the previous model, and one this build has no kernels for is refused before anything is released.

Chat templating

Tokenization uses the HF tokenizers crate plus minijinja for chat templates. Metrale Engine ships its own template overrides for a handful of models in jinja-templates/<model_type>.jinja when the upstream template has known issues (e.g. template-forced <think> seeding). The filename is the checkpoint's config.json model_type; --disable-template-overrides ignores it.

Observability

Prometheus-style metrics are exposed on /metrics (registered in crates/server/src/metrics.rs, plus the prefix-cache, token-entropy and kernel-lookup series that api/misc_handlers.rs appends). Among them:

• metrale_requests_total, metrale_requests_active
• metrale_time_to_first_token_seconds{model}
• metrale_prompt_tokens_total, metrale_generation_tokens_total, metrale_decoded_tokens_total
• metrale_spec_decode_verify_total{k, outcome}
• metrale_tool_calls_total
• metrale_prefix_cache_hits_total, metrale_prefix_cache_misses_total, metrale_prefix_cache_hit_rate

Structured logs go to stderr; they're the primary operational signal. Metrale Engine logs a brief line per completed request (model, prompt tokens, generated tokens, TTFT, elapsed, tools used). Per-token DECODE spam is deliberately not logged — it's useless.

A safe production config (Qwen3.5-35B, agents)

serve Sehyo/Qwen3.5-35B-A3B-NVFP4 \
  --port 8888 \
  --max-seq-len 16384 \
  --max-batch-size 4 \
  --kv-cache-dtype nvfp4 \
  --gpu-memory-utilization 0.88 \
  --scheduler slai \
  --tbt-deadline-ms 100 \
  --enable-prefix-caching \
  --speculative --mtp-quantization nvfp4 \
  --auto-compact 0.85 \
  --adaptive-sampling

Claude Code uses 16k+ context for tool use; running with --max-seq-len 4096 will make agents fail mid-session. Always size up when running an agent workload.