Reproducibility

Source: Dev.to | Dec 16, 2025 (UTC)

Purpose

This document defines reproducibility guarantees in FACET-compliant systems.

Reproducibility means that an AI execution can be:

• replayed exactly
• audited after the fact
• diffed across versions
• cached safely
• reasoned about as a deterministic system

FACET treats reproducibility as a first-class property, not an emergent side-effect.

The Core Problem

Most LLM-based systems are not reproducible.

Even when developers fix:

• model version
• temperature
• prompt text

results still drift due to:

• implicit context truncation
• provider-specific tool sequencing rules
• non-deterministic serialization
• runtime retries and fallbacks
• hidden defaults in SDKs

This makes:

• debugging unreliable
• audits unverifiable
• regression testing meaningless

FACET exists specifically to eliminate these failure modes.

FACET’s Reproducibility Contract

A FACET execution is reproducible if and only if all of the following are identical:

• FACET document
• runtime inputs (@input values)
• execution mode (Pure)
• lens registry and versions
• compiler version

Under these conditions:

• execution order is fixed
• context layout is fixed
• tool schemas are fixed
• Canonical JSON output is fixed

This is a hard guarantee, not a best-effort promise.

Deterministic Execution Stack

Reproducibility emerges from the interaction of five enforced layers:

1. Typed AST — no ambiguous values
2. Reactive DAG (R-DAG) — fixed execution order
3. Token Box Model — deterministic context packing
4. Canonical JSON — stable intermediate representation
5. Provider Adapters — isolated, stateless renderers

If any layer fails to produce a valid deterministic output, execution aborts.

FACET never emits a “mostly correct” state.

Canonical JSON as the Replay Artifact

Canonical JSON is the replay key.

Given a stored Canonical JSON document:

• provider payloads can be regenerated
• execution history can be reconstructed
• outputs can be revalidated
• hashes can be recomputed

This enables:

• exact replay in incident analysis
• post-hoc audits
• cross-provider migration

Canonical JSON decouples execution truth from provider behavior.

Snapshot Testing and CI

FACET enables snapshot-based testing of AI systems.

In @test blocks:

• Canonical JSON is emitted
• JSON is hashed and stored as a golden snapshot
• future runs compare against the snapshot

Any difference indicates:

• a semantic change. or
• a regression. or
• an intentional evolution.

This allows AI behavior to be tested like compiler output.

What Reproducibility Enables

Reproducibility unlocks capabilities that are otherwise impossible:

• deterministic caching and memoization
• cost prediction and budgeting
• stable rollbacks
• compliance and audit trails
• long-term system evolution

Without reproducibility, AI systems remain operationally fragile.

What FACET Does Not Claim

FACET does not claim:

• models are deterministic
• outputs are always identical across providers
• creative sampling is eliminated

FACET claims:

• nondeterminism is explicit, bounded, and isolated
• deterministic layers remain deterministic
• probabilistic behavior cannot corrupt system state

Design Principle

Reproducibility is not about freezing intelligence.
It is about freezing contracts.

FACET enforces reproducibility by turning AI execution into a compiled, replayable process, not an ephemeral interaction.

Status

This document defines the normative reproducibility guarantees for FACET v2.0 Hypervisor Profile.

All compliant implementations MUST uphold these guarantees in Pure Mode execution.