← all challenges

// challenge brief · hack a ton 2026

Contradiction Catcher

Given a long text, find the paragraphs that don't fit, and build the framework to measure how well an AI can do it.

// Proposed by
Cult of Coders
// Industry
AI / LLM evaluation
// Difficulty
🟡 Intermediate

// the problem

"John is in his thirties. When he was 45 he used to ride a bike."

A human spots that instantly; an AI over a 50,000-token document often doesn't. Detecting internal contradictions at scale is an unsolved, genuinely useful problem, and the only way to improve at it is to be able to measure it.

// your mission

Build a system that, given a story, surfaces the paragraphs that are internally inconsistent, and scales to texts up to ~50k tokens. Deliver two things:

  • (a) A benchmark: a way to generate/curate test cases and score results, so you can see how well any approach performs.
  • (b) A harness: an agentic workflow that actually performs the task.

// how it works

  1. 1Ingest: take a long narrative text.
  2. 2Reason: the agent identifies claims and finds ones that contradict each other (age, timeline, identity, facts).
  3. 3Report: flag the offending paragraph(s) and explain the contradiction.
  4. 4Measure: the benchmark scores correctness against known-inconsistent test cases.

// what we provide

  • Example texts with seeded inconsistencies.
  • A hidden evaluation set.

// suggested approach

  • Inference: for model access, we suggest [LLMok](https://llmok.app). Use code AMBASADA26 for 50% off.
  • Build the benchmark first: synthesize stories, inject controlled inconsistencies, and you have ground truth for free.
  • For the harness: chunking plus a claim-extraction pass plus cross-checking. Use an agent that reasons over long context rather than one giant prompt.
  • Floor version: short texts, a single inconsistency type (e.g. age/timeline), exact-match scoring. Scale length and contradiction types from there.

// how you'll be judged

  • Detection accuracy on the hidden set (precision and recall, since false alarms matter).
  • How well the approach holds up as text length grows toward 50k tokens.
  • Quality and honesty of the benchmark itself.
  • Rigor of the evaluation and reporting.

// stretch goals

  • Multiple contradiction types (factual, temporal, identity, spatial).
  • Calibrated confidence per flagged paragraph.
  • Cost/latency reporting alongside accuracy.

// deliverables

  • Git repo: code + README + run instructions.
  • Demo: benchmark run + harness on the provided texts.
  • 5-minute pitch: architecture, decisions, trade-offs.