LP-400: Blockchain-Native Datastore (Quasar-Coordinated OLAP)

Abstract

This LP specifies a blockchain-native OLAP datastore whose replicated metadata log reaches leaderless, post-quantum finality through Quasar (LP-0110), the Lux primary-network consensus engine. The datastore is a ClickHouse-derived columnar engine: it stores MergeTree parts as immutable objects on S-Chain (Hanzo S3, hanzoai/s3), is migrating all node-to-node traffic onto the ZAP wire protocol (hanzoai/zap; §6), and keeps the ZooKeeper-style KeeperStorage key-value semantics as the local replicated-KV contract the Replicated* table engines are written against. What changes — and the whole point of this proposal — is the engine that orders and finalizes that log: instead of a ZooKeeper/Keeper ensemble running Raft/ZAB, the replication-log entries are ordered and finalized by the Quasar validator set, where a value is final only when α distinct validators have signed ACCEPT over the same position and their cumulative stake strictly exceeds ⅔ of total stake (LP-0013, LP-0110 §3).

The proposal is framed as a deliberate, decomplected two-datastore comparison:

• luxfi/datastore (this LP, new) — the Lux-native distribution. Coordination is Quasar over the validator set; there is no ZooKeeper/Keeper ensemble tier. Finality is leaderless and post-quantum.
• hanzoai/datastore (the baseline) — the same ClickHouse-fork OLAP core, but it keeps the ZooKeeper/Keeper coordination ensemble and makes only its transport ZAP-native. This is the "ZK" (ZooKeeper-coordinated) control against which the Lux-native design is measured.

Both rebind the same proven integration seam — KeeperStateMachine::commit(log_idx, buf) — so the storage, query, and replication layers cannot tell which engine ordered the log. Only the ordering engine and the coordination topology differ. The real leaderless post-quantum engine is the Go luxfi/consensus (its quorum-certificate gates and triple-seal legs are cited in §4). Reaching real finality is not a thin shim: the Go finality engine (Transitive) gathers votes by querying live validators, so the coordinator must host — or call out to — a complete, networked Quasar participant, and the clean long-term resolution is a native C++ consensus participant (the consensus2 2.0 line) linked directly into the C++ datastore (no Go runtime). §4 specifies the binding options honestly; §10 states what does and does not exist yet.

Motivation

OLAP analytics clusters (ClickHouse and its derivatives) coordinate replicated tables through an external ZooKeeper/Keeper ensemble: replica membership, part assignment, the replication queue, leader election for merges, and distributed DDL all live as znodes in a ZAB/Raft-replicated key-value tree. This works, but it has three properties that are wrong for a blockchain-grade data platform:

1. A separate coordination tier with its own trust and failure model. The Keeper ensemble is a second quorum system, independent of the chain's validator set, with its own leader, its own liveness assumptions, and its own operational surface.
2. Classical, leader-based finality. Raft/ZAB finalize through a single elected leader and a simple-majority quorum with no post-quantum signature envelope. A forged or replayed leader decision is a single-point safety risk, and the quorum is not stake-weighted.
3. No cryptographic record of agreement. A Raft commit leaves a log index, not a portable certificate. Any party that wants to verify that the cluster agreed on a part-set transition must trust the Keeper, not check a signature.

Lux already operates a consensus engine — Quasar — that solves exactly the ordering-and-finality problem, leaderlessly, with a ⅔-by-stake quorum certificate and an optional post-quantum signature envelope (BLS + Pulsar + ML-DSA; LP-0110, LP-4440, LP-4450, LP-4540). The datastore's coordination log is small (a few metadata entries per second; see §8) and is exactly the kind of ordered, finality-bearing log Quasar already finalizes for the P/C/X chains. Binding the datastore's replication log to Quasar collapses the second trust tier into the chain's own validator set and upgrades coordination finality from "trust the Keeper leader" to "verify the quorum certificate."

The baseline (hanzoai/datastore) keeps the Keeper ensemble and only modernizes its transport (ZAP). Keeping it as a named control is the point: it lets the network measure the Lux-native design against a faithful ZooKeeper-coordinated implementation that shares every other layer.

Design Principles

1. Decomplect the state machine from the engine. A Keeper is two separable things: the ZooKeeper API (KeeperStateMachine over KeeperStorage — the contract Replicated* speaks) and the consensus engine that orders the write log and drives commit. This LP changes only the second. The state machine is a value; the engine is a place. We keep the value and swap the place.
2. One seam, two engines. The single join point is KeeperStateMachine::commit. Every coordination design — Raft, the Quasar PoC, or production Quasar over the validator set — meets the storage layer at that one method. No second integration path is introduced.
3. Finality is leaderless and verifiable. A part-set transition is committed only when a Quasar quorum certificate exists for it (α distinct ACCEPT signers, stake strictly > ⅔). No node, not even a proposer, finalizes alone (LP-0013).
4. Storage rides S-Chain; consensus orders only commitments. Bulk columnar data (the MergeTree parts) lives as objects on S-Chain, each pinned by a cryptographic digest carried in the manifest (§5). Consensus orders only the small manifest entries (which parts compose the committed table state), never the bytes. This is the Celestia-shaped separation of ordering from data availability.
5. Post-quantum by profile, not by fork. The same engine selects a classical (BLS) or post-quantum (Pulsar/Aurora/Polaris) certificate profile at runtime (LP-4900, LP-4910). Quantum-safety is a configuration of one engine, not a second codebase.
6. GPU is not in this picture. Consensus ordering/finalization is latency-bound branchy control flow — "Quasar has nothing to dispatch" — and the coordinator commits a handful of small entries per second, so GPU acceleration is irrelevant to both. The only place GPU applies is validator-side batch threshold-signature verification, transparently inside the Go consensus engine and invisible to the datastore (§8).

Specification

1. Scope and the two-datastore decomplection

This LP standardizes the coordination contract and storage/transport substrate shared by two distributions of one ClickHouse-derived OLAP datastore, and the Lux-native coordination binding that distinguishes luxfi/datastore.

Axis	luxfi/datastore (this LP)	hanzoai/datastore (baseline / control)
OLAP engine	ClickHouse-fork MergeTree family	ClickHouse-fork MergeTree family (same lineage)
Local KV contract	KeeperStorage znode semantics (kept)	KeeperStorage znode semantics (kept)
Coordination engine	Quasar over the validator set	ZooKeeper/Keeper ensemble (NuRaft today)
Coordination topology	No separate ensemble tier — validators coordinate	Dedicated Keeper ensemble (3/5/7 nodes)
Finality	Leaderless, ⅔-by-stake quorum certificate	Leader-based, simple-majority Raft/ZAB
Post-quantum	Yes — Pulsar/Aurora/Polaris cert profiles	No (classical Raft)
Verifiable agreement	Portable QuasarCert	Log index only (trust the Keeper)
Transport	ZAP (in progress; native TCP today)	ZAP (in progress; native TCP today)
Bulk storage	S-Chain objects (hanzoai/s3)	S-Chain objects (hanzoai/s3)
Engine binding	embedded/RPC Quasar participant; native C++ consensus2 is the target (§4)	NuRaft (contrib/NuRaft)
Liveness domain	shares fate with consensus (regression — see §Security 8)	independent of consensus

The two distributions are not two ClickHouse forks. They share one OLAP core and one coordination seam; they differ only in which engine is bound at that seam and whether a separate coordination tier exists.

2. The shared OLAP substrate

Both distributions inherit the ClickHouse-derived columnar engine maintained as a fork at hanzoai/datastore (upstream ClickHouse/ClickHouse). The substrate properties this LP depends on:

• Storage/compute separation. MergeTree parts are written as immutable objects to S-Chain; stateless compute replicas share one physical copy via zero-copy replication. A part is an immutable unit pinned by a cryptographic digest (§5); replicas reference parts, they do not own bytes.
• Replicated* engines coordinate through a KV tree. ReplicatedMergeTree (and distributed DDL) express every replicated decision — "part P is now part of the committed set", replica liveness, the merge/mutation queue — as creates, sets, and multi-ops against a KeeperStorage tree (sequential nodes, ephemeral nodes, watches). This KV API is the only coupling between the storage layer and coordination.
• OLAP-only surface. External stream/CDC/foreign-DB engines (Kafka, RabbitMQ, MySQL/PostgreSQL/MongoDB read engines, Hive, HDFS, etc.) are out of scope and disabled at build time. The kept engine set is the OLAP/object-storage shape: MergeTree family, Replicated*, Distributed, S3/ObjectStorage, ObjectStorageQueue, Memory/Log/View/MaterializedView, RocksDB, the Dictionary engine (without foreign-DB sources), and the Keeper/KV coordination itself.

3. The coordination seam (normative)

The integration point between the OLAP engine and any coordination engine is a single C++ method on the Keeper state machine:

// src/Coordination/KeeperStateMachine.h
nuraft::ptr<nuraft::buffer> commit(const uint64_t log_idx, nuraft::buffer & data) override;

commit(log_idx, buf) deserializes one ordered batch of ZooKeeper write requests, applies them to KeeperStorage (in the two-phase pre_commit/commit order), pushes responses, fires the dispatcher's commit callback, and advances the committed index that the read-after-write barrier waits on. NuRaft calls this method today; the Quasar engine calls the identical method. Nothing in the query, storage, or replication layers references nuraft:: types beyond the trivial container aliases (buffer, log_entry, snapshot, ptr), which are slated to be replaced with native types as the final migration step.

The reusable engine that occupies the NuRaft raft_server slot is QuasarKeeperConsensus (src/Coordination/QuasarKeeperConsensus.{h,cpp}). Its contract — the surface the dispatcher needs — is:

// src/Coordination/QuasarKeeperConsensus.h  (abridged)
struct AppendOutcome { bool accepted; uint64_t last_log_idx; };

class QuasarKeeperConsensus {
public:
    void startup();                                       // bring up the Quasar chain
    void shutdown();
    AppendOutcome append(const KeeperRequestsForSessions & batch); // order + commit, leaderless
    bool     isLeader()          const;                   // dispatcher routing
    bool     isLeaderAlive()     const;
    bool     isRunning()         const;
    uint64_t lastCommittedIndex() const;                  // read-after-write barrier value
private:
    struct Engine;                  // hides the libluxconsensus handle (PIMPL)
    std::unique_ptr<Engine> engine; // ← links the cgo c-archive of Go luxfi/consensus (§4; to build)
};

append(batch) assigns each request a monotonic log index, gets agreement on the order from the consensus engine, then runs the state machine's two-phase apply in that order. The Engine PIMPL is the binding point: it hides the consensus library handle so the consensus header never leaks into the rest of the server. Note: this PoC append is synchronous (correct for the single-node 1-of-1 case); the production binding MUST instead use the async putRequestBatch → future surface (§4, §Security 8) so a multi-validator finality wait never blocks the serialization path.

This seam is proven. Two opt-in programs (src/Coordination/examples/keeper_quasar_poc.cpp, keeper_quasar_engine_test.cpp) drive the real KeeperStateMachine<KeeperMemoryStorage> — the exact class ReplicatedMergeTree coordinates through — to ordered commit through this engine, asserting that the real KeeperStorage tree reflects every committed request exactly once in consensus-decided order (build-verified on a 26.6.1.1 source tree, aarch64/clang-21: 10 ordered writes, blocks_accepted == 10; engine test: 2 batches, 9 ordered commits, monotonic committed index). See §10 for the honest caveat on which engine those PoCs were bound to.

4. Quasar binding — replication log to leaderless PQ finality

Where the real engine lives. The leaderless post-quantum Quasar engine exists today only in Go, in luxfi/consensus:

• engine/chain/quorum_cert.go is the finality authority. It admits a quorum certificate only if it passes both gates — ErrQCBelowThreshold rejects fewer than α distinct ACCEPT votes (with ErrQCNotStrictlyIncreasing enforcing that voters are distinct and sorted), and ErrQCStakeBelowSupermajority rejects a cert whose signer stake does not exceed ⅔ of total stake ("count quorum reached but not stake-weighted supermajority"). It fails closed when no vote verifier is wired (ErrQCVerifierNil) or when epoch total stake is 0. The count gate and the stake gate are independent precisely because, on a PoS chain with unequal stake, "α distinct voters" is not the same as "≥⅔ of stake."
• protocol/quasar/consensus_cert.go defines the triple-seal certificate legs — LegPulsarMLDSA ∥ LegCoronaLattice ∥ LegMagnetarSLHDSA (plus LegClassical for the BLS aggregate) — the post-quantum envelope around the quorum.

Reaching finality is not a thin shim (the central design constraint). The Go component that actually reaches finality is the Transitive engine (engine/chain/engine.go): it PushQuerys the validator set, runs the re-poll/backoff loop, and drives VoteSigner/VoteVerifier/CertGossiper against a live validator-set source and a VM. Critically, AssembleQuorumCert(pos, threshold, votes) (engine/chain/quorum_cert.go) takes the validator votes as input — it does not gather them. A single process holding one key therefore cannot manufacture α distinct real validator signatures; finality is a property of the networked validator set, not of a library call. Any "thin c-archive that returns finality" framing is wrong and is retracted.

The binding is consequently one of three real options, in increasing cleanliness:

#	Binding	What it is	Posture
(a) Interim — embed	The coordinator hosts a complete networked Quasar participant inside the C++ server: a cgo-linked Transitive + ZAP transport + P-chain validator/stake source + staking signer + a VM mapping replication-log entries → blocks + cert gossip.	A full Go-runtime consensus node embedded in ClickHouse.	Heavy; real finality
(b) Interim — RPC	The coordinator calls a co-located luxd consensus RPC; luxd runs the participant out-of-process.	An extra process + IPC on the commit path.	Operationally simpler; real finality
(c) Target — the 2.0 line	A native C++ consensus2 participant linked directly into the C++ datastore — no Go runtime, no cgo boundary; it speaks ZAP and the validator-set/stake interfaces natively.	A C++ participant (design in progress, referenced as direction).	Clean resolution

This LP specifies the contract the coordinator must satisfy regardless of option (a/b/c) and names option (c), consensus2, as the strategic target. It does not claim any of the three is built (§10): (a)/(b) embed an existing Go participant; (c)'s C++ participant is planned, its design produced separately, and is cited here as direction, not as an existing API.

The dispatcher surface the binding must implement is the full contract the Keeper dispatcher needs from the engine slot — not a 3-operation toy — and it is asynchronous (mirroring the existing putRequestBatch async-result contract) so that finality latency never blocks the request-serialization path (§Security 8):

Operation	Meaning
putRequestBatch(batch) → future	Submit an ordered entry; returns immediately. Finality is delivered later via the commit callback / future, never by blocking the caller.
commit callback	Fires once per entry, in log order, when the participant finalizes height h under both quorum-cert gates → drives KeeperStateMachine::commit.
isLeader / isLeaderAlive / getLeaderID	Dispatcher routing (in leaderless mode, "leader" = current cert assembler / liveness sentinel).
lastCommittedIndex	Read-after-write barrier value (per-node monotonic; see §Security 9).
createSnapshot / applyConfigUpdate	Snapshotting and validator-set/membership changes (epoch-anchored).
recovery flags	Startup/catch-up state the dispatcher inspects before serving.

The coordinator applies a committed entry to KeeperStorage at the commit seam only when the participant's callback signals finality for that height. Finality is governed by the Quasar quorum rule (LP-0110 §3), enforced by the two gates and reproduced here as the binding invariant:

A replication-log entry at height h is COMMITTED to KeeperStorage iff the Go
engine admits a QuasarCert for h — i.e. it passes BOTH gates:
  (1) ≥ α distinct validators signed ACCEPT over (h, entry-hash, set-root)
      [else ErrQCBelowThreshold], AND
  (2) Σ STAKE(signers) strictly exceeds ⌊ 2/3 · Σ(total stake) ⌋
      [else ErrQCStakeBelowSupermajority].          (STRICT > ⅔)

For the common equal-stake case this is α = ⌊2N/3⌋ + 1 signers tolerating f = N − α ≈ N/3 Byzantine or unresponsive validators. The committee size K is the live validator set, read per epoch at a deterministic P-chain height, so every honest node computes the identical (N, K, α) and the same set-root binding (a cross-epoch stake change cannot retroactively flip a correct certificate). With the current 5-validator fleet this is K = 5, α = 4 (4-of-5, tolerating one laggard); it recomputes automatically as validators join or leave.

   luxfi/datastore — Lux-native coordination (leaderless, PQ)

   compute replica ──┐
   compute replica ──┤  ZAP   ┌──────────────────────────────────────┐
   compute replica ──┴──────▶ │  QuasarKeeperConsensus                │
                              │   append(batch)                       │
                              │     │ assign height h                 │
                              │     ▼                                 │
                              │   Engine.submit(h, entry) ───────────┐│
                              └──────────────────────────────────────┘│
                                                                      ▼
                       ┌──────────────── Quasar validator set ─────────────────┐
                       │  Photon→Wave→Focus  ·  sign ACCEPT(h, hash, set-root)  │
                       │  QuasarCert when Σstake(signers) > ⌊2/3·Σstake⌋        │
                       │  cert profile: BLS | Pulsar | Aurora | Polaris         │
                       └───────────────────────────────────────────────────────┘
                                                                      │ finalized(h)
                              ┌──────────────────────────────────────┘
                              ▼
                       KeeperStateMachine::commit(h, buf) ──▶ KeeperStorage (znodes)
                              │
                              ▼  read-after-write barrier advances
                       Replicated* sees the new committed part-set

There is no separate Keeper ensemble in this path. The validators that finalize the chain are the validators that order the datastore's coordination log; membership and "leadership" (the right to assemble the certificate, which any node may do) are derived from the Quasar validator set, not from a ZAB/Raft election.

4a. Finality-wiring obligations (normative)

The Go engine ships two routes that an embedded single-process binding can fall into to fake finality — the same forgeability class as the toy stub, through a different door. Both are forbidden for the coordination chain:

1. No ForceAccept. ForceAccept (engine/chain/consensus.go) marks a block accepted WITHOUT a quorum and is gated to single-validator engines (ErrForceAcceptRequiresSingleValidator, k == 1). The coordination chain MUST run with the live validator set (k ≥ α ≥ 4 on the current fleet) and MUST NOT call ForceAccept. A k == 1 coordination engine is non-finalizing by definition.
2. No count-only Verify. The engine exposes two verifiers: QuorumCert.Verify(verifier, epochHeight) checks distinct-vote count only, while QuorumCert.VerifyWeighted(verifier, stake, epochHeight) additionally enforces the ⅔-stake gate. The coordination chain MUST verify through VerifyWeighted with a non-nil StakeSource bound to the P-chain epoch height; a nil StakeSource (tally 0) MUST fail closed.
3. Conformance test (required, normative). A test MUST fail the build if the coordination engine is constructed with k == 1, or with a nil StakeSource, or if any path can reach KeeperStateMachine::commit without a VerifyWeighted-passing QuasarCert. This test is the line that separates luxfi/datastore from the forgeable PoC; it is not optional.

5. Storage on S-Chain — manifest/data separation and availability

Consensus orders commitments, not bytes. A MergeTree part is written to S-Chain as an immutable object. It is not content-addressed: SeaweedFS (the hanzoai/s3 lineage) keys objects by fid (volume + file id), not by a hash of the bytes; and ClickHouse's own part checksum is non-cryptographic — SipHash128 over CityHash128 file hashes (MergeTreeDataPartChecksum: uint128 = CityHash_v1_0_2::uint128, getSipHash128AsPair), which a malicious compute replica can grind a collision against. Therefore the manifest MUST carry a cryptographic digest D — SHA-256 or BLAKE3 over the object bytes — in addition to (and independent of) the S3 fid key and the ClickHouse part checksum. What reaches Quasar finality is the manifest entry — the small znode write that declares "the committed table state now includes part P (object key K, cryptographic digest D, row count, min/max block)". The bytes are made available by S-Chain; the order, finality, and integrity binding of the manifest are made authoritative by Quasar.

   bulk columnar data (large)           coordination manifest (small)
   ───────────────────────────          ─────────────────────────────
   MergeTree part bytes                  "part P (fid K, CRHF digest D) is committed"
        │                                       │
        ▼                                       ▼
   S-Chain object store                  Quasar replication log
   (hanzoai/s3, fid-keyed)               (ordered, ⅔-stake finalized)
        │     verify SHA-256/BLAKE3 D on read    │
        └──────────── a committed part-set = (manifest ∧ object matches D) ────────┘

Data availability. A committed manifest is only meaningful if the referenced objects are retrievable and match their digest. S-Chain provides availability through replication; on every read the replica recomputes the CRHF digest and rejects the object unless it equals the finalized D, so a malicious or faulty S-Chain node cannot substitute bytes (the non-cryptographic fid and part checksum do not provide this — the manifest digest does). For deployments that require erasure-coded, sample-verifiable availability of the underlying objects, the part objects MAY be stored through the Lux DA layer (lp-8501 erasure-coded DA with sampling) rather than plain replication. DA is therefore an orthogonal property of the object tier, not of the coordination engine — either datastore can use either availability mode.

6. ZAP transport (direction; partial today)

The target transport for all node-to-node traffic — compute-to-coordinator request batches, coordinator-to-validator submissions, validator vote/certificate gossip, and interserver part fetches — is the ZAP wire protocol (hanzoai/zap; node-level impl github.com/luxfi/node/network/zap/): a fixed-layout, zero-copy binary frame over TCP with TLS 1.3 and a post-quantum X25519MLKEM768 handshake (Go 1.26 default). Status (honest): in the datastore the ZAP server stub has landed but is not yet the query path — the live path is the native ClickHouse TCP protocol (port 9000). This LP marks ZAP as the direction, not a shipped guarantee. The validator-side consensus messaging (vote/cert gossip) is ZAP within luxfi/consensus; the datastore↔coordinator and interserver paths are mid-migration. The baseline hanzoai/datastore is on the same migration — ZAP is shared direction, not a differentiator. ZAP is a node-level protocol, not an LP; do not cite it as a standards-track LP.

7. Post-quantum security

Quasar finalizes under a runtime-selectable certificate profile (LP-4900, LP-4910). The datastore inherits this directly; no datastore-specific crypto is introduced.

Profile	Legs	Posture	Datastore use
BLS-only	BLS12-381 threshold	Classical fast path	Dev/testnet, or where PQ is not required
Pulsar (PQ floor)	Pulsar (threshold ML-DSA-65, FIPS 204 byte-equal)	Post-quantum threshold	Default PQ posture
Aurora	Pulsar ∥ Corona (Module-LWE threshold)	Intra-lattice diversity	Higher assurance
Polaris	Pulsar ∥ Corona ∥ Magnetar (SLH-DSA, FIPS 205)	Cross-family, maximum assurance	Critical deployments

• Pulsar (luxfi/pulsar) — 2-round threshold ML-DSA whose signatures verify under unmodified FIPS 204 ML-DSA-65 (LP-4450; NIST MPTC submission).
• Corona (luxfi/corona) — Module-LWE 2-round threshold signature with production DKG/resharing (LP-4440; ePrint 2024/1113 lineage).
• Magnetar (luxfi/magnetar) — SLH-DSA leg (LP-4540). Honest scope: the sound, production Magnetar regime is per-validator standalone (each validator holds its own FIPS 205 keypair, signs independently, consensus collects N signatures into a ValidatorAggregateCert). Threshold SLH-DSA variants are TEE-relocated or research-grade and are NOT relied on for no-leak threshold custody here.

Validator signing keys (BLS for consensus, plus the PQ leg keypairs) are managed under the Key Management System (lp-0070); the datastore does not hold, sign with, or rotate validator keys — it only consumes the finalized certificate.

8. GPU scope: a non-goal for consensus and for the coordinator

"GPU-native consensus" is a category error and is not claimed by this LP. Consensus sampling, vote tallying, and finalization are latency-bound, branchy control flow; as the Quasar GPU audit states plainly, "Quasar has nothing to dispatch — it composes Pulsar + Corona + ML-DSA + BLS" (luxfi/consensus/PQ-GPU-AUDIT.md:41). GPU acceleration lives only in the per-primitive crypto paths, never in the consensus loop. Separately, the datastore coordinator commits a handful of small entries per second (part-manifest mutations and replica liveness, not data), so it is bounded by network round trips and certificate assembly — GPU does nothing for the coordinator either.

The single place GPU applies is validator-side batch threshold-signature verification: when a validator must verify large batches of signature shares (on the order of >64 at once), the per-primitive batch-verify paths can dispatch to GPU kernels (batched ML-DSA verify, NTT, Keccak) through the Lux acceleration substrate (luxfi/accel LatticeOps/HashOps, luxfi/lattice NTT under -tags gpu, resolved by the CRYPTO_BACKEND GPU → CGo → Vanilla fallback in luxfi/crypto). This is internal to the Go consensus engine and entirely transparent to the datastore: the coordinator submits an entry and waits for finality; whether a validator verified the certificate's signatures on CPU or GPU is invisible to and undriven by the datastore. The underlying Metal/CUDA kernels are maintained in the private Lux GPU tree and are not verifiable from this workstation (§10); this LP makes no GPU performance claim on their behalf.

9. Topology comparison

  BASELINE  hanzoai/datastore (ZooKeeper-coordinated, "ZK")
  ──────────────────────────────────────────────────────────
   compute ─┐                 ┌────────────────────────────┐
   compute ─┤  ZAP            │  Keeper ENSEMBLE (3/5/7)    │
   compute ─┴───────────────▶ │  NuRaft: elect leader,      │
                              │  replicate log, majority    │
                              │  commit → KeeperStorage      │
                              └────────────────────────────┘
   • second trust tier   • leader-based   • classical majority   • no portable cert


  LUX-NATIVE  luxfi/datastore (Quasar-coordinated)
  ──────────────────────────────────────────────────────────
   compute ─┐                 ┌────────────────────────────┐
   compute ─┤  ZAP            │  QuasarKeeperConsensus      │
   compute ─┴───────────────▶ │  submit → Quasar validator  │
                              │  set → QuasarCert (⅔ stake) │
                              │  → KeeperStorage             │
                              └────────────────────────────┘
   • no second tier   • leaderless   • ⅔-by-stake cert   • portable QuasarCert   • PQ

The two share the entire left side (compute replicas, ZAP, the KeeperStorage contract, S-Chain). They differ only in the box: an ensemble running Raft vs. the validator set running Quasar.

Rationale

Why keep the KeeperStorage contract instead of a new coordination API

ReplicatedMergeTree and distributed DDL are tens of thousands of lines written against ZooKeeper semantics (sequential nodes, ephemeral nodes, watches, multi-ops, versioned compare-and-set). Rewriting them against a bespoke Quasar API would be a massive, bug-prone change to the storage layer for zero behavioral gain — the storage layer does not care how the log was ordered, only that writes are linearized and durable. Keeping KeeperStorage as the local KV contract and swapping only the engine is the minimal, decomplected change: it touches src/Coordination/, not the query/storage/replication layers. The PoC's thesis — "the state machine cannot tell the difference" — is the design principle made executable.

What is NOT the engine (two explicitly excluded bindings)

Two existing artifacts are easy to mistake for the consensus engine. Neither is, and neither is the binding:

• luxfi/consensus/pkg/c (libluxconsensus) — the C shim the proof-of-concept programs link. It is sufficient to prove the seam in single-node mode (always-leader, 1-of-1 "finality"), but it is a forgeable toy: it has a local check_decision_threshold + verify_callback, but no real validator quorum, no stake weighting, and no networked vote collection — finality is a local cumulative count. It MUST NOT be cited as working BFT/PQ finality.
• lux-private/consensus (a.k.a. luxcpp/consensus) — a CPU-only BLS verifier (Verify + AggregateThresholdSignatures). It has no ordering, commit, or log surface and is not a consensus engine; its "GPU kernels + e2e harness" description is misleading (no .cu kernels, no commit-shaped API). It is not the binding and the datastore does not depend on it. (It is also not present on this workstation, so any claim about it here would be unverifiable.)

The engine is the Go luxfi/consensus (§4): the real α-of-K + ⅔-stake gates and the triple-seal legs live there. This LP depends on the seam (which pkg/c helped validate) and on a to-be-built cgo c-archive of that Go engine for finality. Any claim that the PoC achieved real BFT or post-quantum finality is false and is explicitly disclaimed.

Why a separate luxfi/datastore repo rather than a flag in hanzoai/datastore

The two distributions exist to be compared, and a comparison needs two independently buildable, independently deployable artifacts with disjoint default topologies — one that defaults to a Keeper ensemble, one that defaults to Quasar over the validator set. Keeping them as separate repos that share the OLAP core (via the common ClickHouse-fork lineage and the one coordination seam) keeps each artifact's defaults honest while still honoring "one way to do everything" at the layer that matters: there is exactly one OLAP engine, one KV contract, one transport, one storage substrate, and one coordination seam. Only the engine bound at that seam differs.

LP numbering: why 400–499 (proposed new sub-range)

LP-0099 reserves 100–999 for Core Protocols, sub-divided as: 100–199 consensus, 200–299 validator coordination, 300–399 epoch management, 400–599 reserved, 600–799 protocol extensions, 800–999 reserved. There is no dedicated storage/datastore range anywhere in the IA. This proposal therefore opens a new sub-range and justifies it:

• The datastore is core-protocol infrastructure: a replicated state machine whose write log reaches Quasar finality. It is not a chain (no chain letter → not 5xxx–9xxx), not a crypto primitive (→ not 4xxx), and not a chain identity spec (→ not 1xxx).
• It is not consensus itself (100–199), not validator coordination (200–299), and not epoch management (300–399) — it is a protocol that is built on all three.
• The natural slot is therefore the reserved band immediately after epoch management. This LP claims 400–499 as "State Machine Replication & Storage Coordination" — protocols that replicate a deterministic state machine over Lux consensus and the storage substrates they ride — and opens it with LP-400 as the range flagship (matching house convention where round-hundred numbers anchor a range: LP-1000 P-Chain core, LP-3000 token registry, LP-4000 crypto cross-cuts, LP-5000 A-Chain core, LP-6000 B-Chain core).
• 500–599 is deliberately left reserved. Two existing Draft DA LPs (LP-8501, LP-8505) carry a dangling "(LP-500)" reference for a rollup framework; per the chain-letter IA a rollup framework belongs in the 8500–8599 L2/ZK-rollup band, not in core protocols. Claiming 400–499 avoids that ambiguity entirely.

Open governance question (must resolve before this LP can be Final). LP-0099 is a Final Meta LP; a Standards-Track LP cannot self-amend it. The 400–499 sub-range is therefore a request, not a fait accompli. Two routes, to be decided by the LP editors:

• Route A — amend LP-0099 first. Ratify the following addition to the 100–999 sub-range table as an LP-0099 amendment; only then does LP-400 occupy a valid slot.

  Sub-range	Purpose
  400–499	State Machine Replication & Storage Coordination (new — replicated datastores, coordination services over consensus)

• Route B — place it outside Core Protocols. This LP itself argues (Backwards Compatibility) that the datastore is a consumer of consensus, not a core protocol — which weakens the 100–999 claim. If the editors prefer, LP-400 should move to a consumer/infrastructure range rather than carve a new Core-Protocol sub-range.

This LP does not presume the outcome; it flags the dependency and proceeds under Route A as the working assumption, with the number subject to change.

Backwards Compatibility

• Wire/query compatibility. The native TCP query protocol, the SQL surface, the MergeTree/Replicated* engines, and the system.* tables are unchanged. Clients and BI tools see an ordinary ClickHouse-compatible OLAP database.
• KV-contract compatibility. KeeperStorage semantics (znode model, watches, multi-ops, read-after-write barrier) are preserved exactly, so Replicated* and distributed DDL work without modification.
• Operational migration. A baseline (hanzoai/datastore, ZK-coordinated) cluster migrates to Lux-native coordination by switching the engine bound at the seam; the on-disk part format and the S-Chain object layout are identical, so parts do not move. The two distributions can read the same S-Chain part objects.
• No consensus changes. This LP introduces no change to Quasar, the validator set, epoch handling, or the wire certificate format. It is a consumer of LP-0110/LP-0013/LP-0120; it adds nothing to the consensus protocol.

Security Considerations

1. Finality safety = Quasar's safety. The datastore's coordination is exactly as safe as a Quasar quorum certificate: a part-set transition cannot finalize unless the Go engine admits the cert under both gates — ErrQCBelowThreshold (≥ α distinct ACCEPT signers) and ErrQCStakeBelowSupermajority (signer stake strictly exceeds ⅔ of total). The strict > (never ≥), the distinct-voter rule (ErrQCNotStrictlyIncreasing), the fail-closed paths (ErrQCVerifierNil, TotalStake == 0), and the set-root epoch binding are inherited from engine/chain/quorum_cert.go / LP-0110 §3 and MUST NOT be weakened by the datastore. The datastore performs no quorum math of its own; it waits on the engine's finality signal and treats the certificate as authoritative.
2. No second trust tier. Removing the Keeper ensemble removes its independent leader-election and liveness assumptions. The attack surface of coordination collapses into the validator set's surface, which is already monitored, staked, and slashable.
3. Manifest/object integrity needs a CRHF (not the part checksum, not the fid). The finalized manifest MUST bind each part object by a cryptographic digest D (SHA-256 or BLAKE3), verified on every read. This is load-bearing: ClickHouse's part checksum is SipHash128-over-CityHash128 (non-cryptographic, grindable) and the SeaweedFS object key is a fid, not a content hash — neither resists substitution. With D in the finalized manifest, a malicious or faulty S-Chain node cannot serve substituted bytes without detection. An unavailable object is a liveness failure (the part cannot be read), not a safety failure (no wrong data is served).
4. Replay and ordering. Each replication-log entry is bound to a monotonic height and the validator-set-root of its epoch; a certificate gathered under one epoch's set cannot be replayed into another. Log order is assigned by a single committer per height, but — per obligation in §8 — finality is awaited asynchronously and never under a held serialization lock; duplicate or reordered application is rejected by the monotonic committed-index barrier.
5. Transport. ZAP over TLS 1.3 with staker certificates authenticates peers; the PQ handshake (X25519MLKEM768) protects the channel against harvest-now/decrypt- later. CRC32-C detects corruption (not tampering); integrity/authentication come from TLS.
6. Post-quantum posture. Under the Pulsar/Aurora/Polaris profiles the finality certificate itself is quantum-safe, so a future quantum adversary cannot forge a coordination certificate. The classical BLS profile is for environments that explicitly accept classical finality (e.g. local/dev).
7. Honest non-goal. This LP does not claim that the proof-of-concept achieved real BFT or PQ finality; the PoC's pkg/c engine is single-node and forgeable, and lux-private/luxcpp/consensus is a CPU BLS verifier, not an engine (§Rationale). Real finality arrives only when a complete networked Quasar participant — embedded, via co-located luxd RPC, or the native C++ consensus2 target — is wired at the seam (§4), none of which is built today (§10).
8. Liveness coupling is a REGRESSION vs the ZK baseline (stated honestly). Folding coordination into the validator set deletes the one liveness domain that did not share fate with consensus: in the baseline, a consensus stall does not by itself stop the Keeper ensemble. Here, a validator-set stall stalls coordination. To bound the blast radius, the binding MUST (a) use the async dispatcher surface (submit returns; finality via callback/future), (b) make finality waits cancellable with a deadline, and (c) never hold the request-serialization lock across a finality wait — otherwise a stall cascades into session timeouts and replicas marking each other dead. This coupling is the explicit price of removing the second tier; it is documented, not hidden.
9. Read-after-write is per-node monotonic, cross-node eventually-consistent. The lastCommittedIndex barrier guarantees a node sees its own committed writes; it does not make a just-committed write instantly visible on every other replica. This is the same guarantee as the ZK baseline, not a stronger one — no over-claim.
10. Distinguish the two fault bounds. BFT safety requires f_byz < N/3 Byzantine validators (the ⅔-stake cert is what enforces it). The quorum slack f = N − α is a crash/liveness bound (how many non-responsive validators finality tolerates). They coincide numerically near N/3 but are different properties; do not conflate "tolerates N−α laggards" with "safe against N−α Byzantine nodes."

Implementation

Concern	Location	Status
Coordination seam	hanzoai/datastore src/Coordination/KeeperStateMachine.h — commit(log_idx, buf)	Exists (upstream); the join point
Reusable Quasar engine	hanzoai/datastore src/Coordination/QuasarKeeperConsensus.{h,cpp}	Stage 1 merged (single-node)
Seam proof programs	hanzoai/datastore src/Coordination/examples/keeper_quasar_poc.cpp, keeper_quasar_engine_test.cpp	Build-verified 26.6.1.1 (aarch64/clang-21)
Engine binding (the deliverable)	a complete networked Quasar participant at QuasarKeeperConsensus::Engine — option (a) embed cgo Transitive+ZAP+validator/stake source+signer+VM, (b) co-located luxd RPC, or (c) native C++ consensus2 (target)	TO BUILD — none exists; a single key cannot synthesize α votes
Finality engine (real)	luxfi/consensus engine/chain/engine.go Transitive (PushQuery/re-poll); AssembleQuorumCert(pos,threshold,votes) takes votes as input	Production (Go); LP-0110
Finality gates (real)	engine/chain/quorum_cert.go — ErrQCBelowThreshold (α), ErrQCStakeBelowSupermajority (⅔ stake), VerifyWeighted vs count-only Verify; ForceAccept gated to k==1	Production (Go)
Triple-seal legs (real)	protocol/quasar/consensus_cert.go — LegPulsarMLDSA/LegCoronaLattice/LegMagnetarSLHDSA/LegClassical	Production (Go)
PQ threshold legs	luxfi/pulsar, luxfi/corona, luxfi/magnetar	LP-4450 / LP-4440 / LP-4540
GPU (validator-side only)	luxfi/accel, luxfi/lattice, luxfi/crypto; batch threshold-verify under the Go engine, transparent to the datastore	accel/lattice in Go; Metal/CUDA kernels private/unverified
Transport	ZAP — node impl github.com/luxfi/node/network/zap/, hanzoai/zap	DIRECTION; datastore live path is native TCP 9000, ZAP server is a stub
Bulk storage + integrity	hanzoai/s3 (S-Chain, fid-keyed) + manifest CRHF (SHA-256/BLAKE3)	Storage prod; CRHF binding is new (this LP)
NOT the engine (excluded)	luxfi/consensus/pkg/c (forgeable toy stub) · lux-private/luxcpp/consensus (CPU BLS verifier, no ordering/commit surface)	Do not depend on either

Implementation Status and Roadmap (honest)

• Proven. The seam is real and exercised end-to-end against the real Keeper state machine: a non-NuRaft engine orders a batch of ZooKeeper requests, the real KeeperStorage tree reflects every committed request exactly once in engine-decided order, and the committed-index barrier advances. Single-node ordering and commit are complete.
• Not yet real finality. The PoC engine underneath the seam was luxfi/consensus/pkg/c, single-node and forgeable; it proves the plumbing, not BFT/PQ finality. The real Go engine has the correct gates but reaching finality needs a networked participant (a single process cannot synthesize α votes — §4).
• The gap to close (this LP's deliverable surface), in order.
  1. Stand up a networked Quasar participant at the seam. Interim: embed a cgo Transitive participant (ZAP transport + P-chain validator/stake source + staking signer + a VM mapping log entries → blocks + cert gossip), or call a co-located luxd consensus RPC. Target: the native C++ consensus2 participant (no Go runtime; the strategic end state, design produced separately). A build-spike is empirically confirming the embed/RPC trade-off.
  2. Enforce the §4a obligations as tests. Fail the build on k == 1, on a nil StakeSource, on ForceAccept, or on any commit path not gated by a VerifyWeighted-passing QuasarCert.
  3. Async dispatcher surface (HIGH liveness). Implement the full putRequestBatch async-result contract (commit callback, snapshot, config/membership, leader id, recovery flags); never block request serialization on a finality wait; finality waits are cancellable with a deadline.
  4. cgo hardening (if option a/b). A recover() at every //export boundary (a Go panic crossing into C++ is UB); resolve Go↔ClickHouse signal-handler coexistence (SIGURG preemption, SIGPROF, SIGSEGV) — a named build-spike gate before any embed lands.
  5. Wire the cert-profile selector (BLS/Pulsar/Aurora/Polaris); membership/leadership derive from the Quasar validator set.
  6. Excise residual nuraft:: container types and remove contrib/NuRaft.
• Gate. Each step lands under the full gtest_coordination suite on a Linux build host with submodules initialized (the engine cannot be configured/built on a macOS workstation — cmake/tools.cmake and submodule init are the gates). No step is considered done until that suite is green.
• Comparison deliverable. A standing benchmark harness measures Lux-native vs. ZK-baseline coordination on identical workloads (DDL throughput, part-commit latency, finality latency under f failures, recovery time, certificate verification cost) — the empirical core of the two-datastore framing.

References

• LP-0013 — Leaderless (the no-single-proposer finality invariant)
• LP-0110 — Quasar Unified Consensus Protocol (§3 committee sizing and weighted quorum)
• LP-0111…LP-0116 — Photon, Flare, Wave, Focus, Horizon, Prism (Quasar sub-protocols)
• LP-0118 — Warp signature aggregation
• LP-0120 — Quasar mainnet defaults; LP-0121 — Blockchain go-live standard
• LP-8501 — Data Availability Layer (erasure/DAS) — the canonical DA LP
• LP-4440 — Corona (Module-LWE threshold); LP-4450 — Pulsar (threshold ML-DSA-65); LP-4540 — Magnetar (SLH-DSA)
• LP-4900 — Aurora cert profile; LP-4910 — Polaris cert profile
• LP-0070 — Key Management System; LP-0099 — LP numbering scheme (Route A amendment, §Rationale)
• ZAP transport — node-level protocol, github.com/luxfi/node/network/zap/ + hanzoai/zap (NOT an LP; lp-9027 is the unrelated DEX wire spec)
• Engine source — luxfi/consensus engine/chain/{engine.go,quorum_cert.go,consensus.go}, protocol/quasar/consensus_cert.go
• Seam source — hanzoai/datastore src/Coordination/{KeeperStateMachine.h,QuasarKeeperConsensus.{h,cpp},examples/}

Copyright

Copyright and related rights waived via CC0.