BrainOS Context Lab

The Context Lab wraps a pinned BrainOS build behind an adapter and compares it against classical ways of managing a long conversation: full context, a sliding window, lexical RAG, BrainOS alone and BrainOS with RAG. A deterministic benchmark generator produces conversations whose context rots in controlled ways, and the platform measures what each strategy recalls, how faithfully it answers and how it degrades as length grows.

Claims about memory systems are easy to make and hard to test. Without controlled baselines, a pinned dependency, a reproducible dataset and effect sizes, a token-reduction number is marketing rather than evidence.

BrainOS needed an external harness that did not modify the runtime under test. The lab exists to validate the runtime's context claims with the same discipline as an experiment: fixed seeds, hashed datasets, paired statistics and a taxonomy of how answers fail.

Baselines A–E and ablations D1–D4 share one evaluation loop; a security module and a cognitive-trace UI sit alongside.

Baselines

A full context, B sliding window, C lexical RAG, D BrainOS, E BrainOS + RAG; ablations D1–D4 switch individual BrainOS mechanisms off.

Benchmark generator

Seven categories — single-hop, multi-hop, temporal, conflict, distractor, cross-session and abstention — from an 800-token smoke tier up to a 5k–120k-token ladder, with a SHA-256-pinned dataset.

Metrics

Recall@K, evidence-in-prompt, faithfulness and degradation AUC; paired t-tests, Cohen's d / Hedges' g and sign tests; a nine-label failure taxonomy.

Security

A seven-family injection guard, a findings ledger and a credential scanner.

UI

Gradio 6 with Chat, Memory, Context, Cognitive Trace and Security tabs.

• Base package with zero mandatory dependencies; OpenAI and OpenAI-compatible providers.
• SQLite storage with session isolation and secure delete.
• The README reports 1,004 passing tests and a clean ruff run.

1. 01

   Pin the system under test

   BrainOS is consumed at a fixed upstream commit and never modified by the lab.

2. 02

   Report observations, not results

   The author labels current numbers as integration-validation observations — one seed, one length tier, an estimated token counter and no model in the loop — and declines to call them research results.

Experimental design

5 conditions · 4 ablations · 7 categories

Statistics

Paired tests with effect sizes

Degradation AUC is only defined across length tiers, so it is reported as null on single-tier runs.

*No headline numbers are repeated here by design: the repository explicitly states that current figures are not research results.

• Building an honest abstention category — the author records that abstention does not work yet.
• Keeping the harness independent of the runtime it evaluates.

• Lexical rather than semantic scoring; English-only pattern extraction.
• One seed and one length tier so far, with an estimated token counter.
• The Evaluation tab is a placeholder pending a later phase.

• An evaluation platform is only as credible as the caveats it publishes about itself.

• Repository (opens in a new tab)github.com/NiravRVaghasiya/BrainOS-Context-Lab
• Limitations (opens in a new tab)github.com/NiravRVaghasiya/BrainOS-Context-Lab/blob/main/docs/limitations.md