Chess Analysis Engine

A chess game goes in as a PGN, pasted moves or a Lichess URL. A LangGraph pipeline evaluates every position with Stockfish, labels each move, computes per-player accuracy and ACPL, finds the critical moments and writes plain-language explanations of why each key move helped or hurt. The final stage checks selected numerical and move-label claims against engine output before displaying the note.

Most chess-analysis demos stop at printing centipawns. The interesting problem is the one after that: a language model writing about chess will happily contradict the engine — praising a move the engine calls a blunder, or quoting a percentage it never reported.

The guardrail is enforced twice. The prompt forbids unsupported claims, and check_faithfulness compares selected claims in the produced text with the engine's numbers. In the author's framing, a prompt is an instruction while a check creates an enforceable boundary for the claims it covers. The headline test case is the Immortal Game: Anderssen's celebrated 18.Bd6 is a Blunder by Stockfish's numbers, and the app keeps saying so.

Deterministic stages first, generative stage last. Parser, evaluator and classifier are pure and fully testable without a model or network; retrieval and annotation can fail without taking the analysis down.

Parser

python-chess reads pasted text, a file or a Lichess URL, and splits the game into phases (the opening is the first 24 plies; the endgame starts when non-pawn material drops to 20 or below).

Evaluator

Stockfish over UCI with Threads=1 and Hash=16 for determinism, depth 18 by default, a six-ply principal variation, mate scores mapped to ±10000 and an evaluation cache keyed on position and depth. An AnalysisEngine protocol lets tests inject fakes.

Classifier

A pure function converts centipawns to win probability using the Lichess model, then labels each move by the drop in the mover's win chance: Brilliant (a rule-based sacrifice check), Best, Good (< 2), Inaccuracy (< 5), Mistake (< 10) and Blunder (≥ 10). It also computes ACPL, a volatility-weighted game accuracy and the critical moments.

Retriever

ChromaDB with two paths: a metadata lookup by ECO code for opening theory, and a vector search over a text description of derived position features for tactical motifs. Only critical moments trigger retrieval; a deterministic hashing embedder is the default.

Annotator

A system prompt, few-shot pairs and a templated user turn carrying the FEN, the move, its label, win chance before and after, the engine's best line, board-derived piece activity and retrieved theory. Providers: stub (default), OpenAI or Ollama. Every note passes through check_faithfulness.

Reporter and surfaces

One report model rendered by a FastAPI endpoint, a CLI, and Streamlit and Gradio front ends that share a single renderer so they cannot drift apart.

• A LangGraph StateGraph over a Pydantic AnalysisState. Conditional edges send parse or evaluation failures to END; retriever and annotator exceptions are logged and the run continues without coach notes.
• Move accuracy and win probability follow the published Lichess formulas; all evaluations are anchored to White and flipped exactly once into the mover's perspective.
• Docker image on python:3.11-slim with Stockfish installed from apt; CI runs ruff, mypy, pytest with the stub provider, then the evaluation suite.
• Zero-secret default: an unknown provider or missing key falls back to the stub, so the demo starts with no credentials.
• The README reports 369 tests across 19 test modules, including famous-game regressions and a test that the Immortal Game's 18.Bd6 is never praised.

PGN/URL → Parser → Evaluator → Classifier → Retriever → Annotator → Reporter          (chess)  (stockfish)  (deltas)     (RAG)       (LLM)       (JSON+UI) Annotator output → check_faithfulness → accept | discard & fall back to stub

1. 01

   Check the prose, not just the prompt

   Faithfulness is verified on the free text the model actually produced: praise after a drop of ten or more points, criticism of a Best or Brilliant move, or any percentage more than three points from the engine's figures. A failing note is discarded and replaced.

2. 02

   A fallback that cannot lie

   The stub provider is a template writer that parses the numbers back out of the rendered prompt, so by construction it cannot make a chess claim the engine does not support. Tests assert its output passes the checker for all six labels.

3. 03

   Deterministic engine settings

   Single thread, fixed hash and fixed depth keep fixtures and labels reproducible across machines and CI.

4. 04

   Describe positions, don't embed FENs

   Retrieval queries are text descriptions of derived features — king safety, open files, material — so the vector search has semantics to match rather than a board string.

5. 05

   Degrade, don't fail

   If retrieval or the model is unavailable, the user still receives evaluations, labels and metrics — just without notes.

Immortal Game, ply 35 (18.Bd6)

74.3% → 50.0% · Blunder

Depth-16 fixture: 288 centipawns lost, engine preferred Re1. The stub annotation reads “a drop of 24 points” and the checker accepts it.

Regression evaluation

20 reviewed expectations

Fourteen move labels, four player-metric bounds and two critical-moment checks, each with a written rationale.

Faithfulness evaluation

Every critical moment of two famous games

Opera and Immortal games annotated from fixtures and run through the checker, plus a detector probe of eight texts with known verdicts.

Deployment

Live on Hugging Face Spaces

Gradio front end with a depth slider and preloaded demo games, verified running.

*Test counts are as reported in the repository README. The hosted demo runs on shared CPU at a reduced search depth.

• Detecting brilliancies without multi-PV search meant settling for a rule-based sacrifice proxy — the docstring cites the upgrade path.
• Labels shift with search depth, so fixtures pin depth 16 while the hosted demo runs shallower to stay responsive.
• Shipping on the free Hugging Face tier required adding a Gradio front end alongside Streamlit and making both render through the same module.

• Brilliant detection is a heuristic proxy rather than a multi-PV analysis.
• Chess.com game URLs are not supported; Lichess URLs are.
• The evaluation cache keys on position without move counters, so fifty-move-rule positions can be misread.
• By default the annotator is the deterministic stub; a language model is only used when an API key or local Ollama is configured.

• A prompt is an instruction; a post-generation check creates an enforceable boundary for the claim types it covers.
• Put deterministic components first and generative ones last, and make the fallback path as trustworthy as the happy path.
• Famous games make excellent regression tests because the ‘right’ answer is culturally surprising and numerically unambiguous.

• Repository (opens in a new tab)github.com/NiravRVaghasiya/chess-analysis-engine
• Live demo (Hugging Face) (opens in a new tab)huggingface.co/spaces/Nirav12321/chess-analysis-engine
• Design document (opens in a new tab)github.com/NiravRVaghasiya/chess-analysis-engine/blob/main/DESIGN.md