MADCAP (Multi-Agent Debate with a Critic-And-Profile loop). A toy project exploring structured multi-agent debate between LLMs; with an information bottleneck, a memoryless critic and a cross-round profiling loop.
Fair warning: this is a work-in-progress. Changes get pushed to the repo whenever something interesting occurs to me, and they might break existing functionality without notice. The latest version might not even work at all. This is a playground for experimentation, not a stable library. Treat it accordingly.
The repo is MIT-licensed and lives at: https://github.com/arepetti/madcap
The short version
I wanted to see what happens when you stop trusting a single LLM reply and instead make three of them argue it out. A confident Answerer drafts a response, a sceptical Critic stress-tests it, and an impartial Judge runs the show and writes the verdict. The user watches the whole thing unfold: the rephrased question, every round of debate, and a final answer with a confidence label.
That's MADCAP (Multi-Agent Debate with a Critic-And-Profile loop). It's a toy. It's a playground. It's not trying to be a product.
Turns out the papers flag a lot of failure modes. So the interesting part isn't "three LLMs talk to each other" (that's a weekend project). The interesting part is trying not to let them agree with each other into uselessness.
What makes this one different
Mechanically it's a Multi-Agent Debate (MAD) setup in the lineage of Du et al. [1] and Liang et al. [11], a practical cousin of the older "AI safety via debate" idea [2]. Thesis, antithesis, synthesis: the answerer proposes, the critic pushes back, the judge weighs and rules. Nothing radical there.
Three design choices give it its own flavour:
The user's question is never sent verbatim to the debate. The judge first rewrites it into a neutral, precise form (and may ask the user to clarify before doing so). Everyone debates that sanitised version. The point: defuse sycophancy [3, 4] at the door, so the answerer never sees whatever leading framing the user typed. This isn't a new idea (prompt rewriting shows up under various names in the bias literature [8, 9, 20]) but doing it before the debate starts (rather than as a post-hoc trick) is specific to this setup.
A one-way information bottleneck between answerer and critic. After every answerer reply, the judge restates it as neutral facts (stripped of rhetoric, hedging, persuasive framing) and that restatement is the only thing the critic sees. The answerer hears the critic raw; the critic only ever hears the answerer through the judge's filter. The goal: the critic argues with the claim, not the prose [8, 16].
A cross-round profile of the answerer's weaknesses. The judge quietly watches for recurring patterns in the answerer's failures and feeds them into future critics. So if the answerer keeps over-generalising from single examples, round 5's critic already knows to probe for that.
The three actors
Answerer
The one that actually tries to answer the question. It's told to be direct, concede when wrong, and never invent sources. It keeps its memory across rounds (so it can build on previous discussions), but can't rely on the critic remembering anything, because...
Critic
...the critic is rebuilt from scratch every single question. No memory. It never saw the last debate. Its entire job is to stress-test: argue the strongest opposing position, flag unsupported claims, surface hidden assumptions.
One specific trick it uses: negation-as-inversion. Take any claim X, force yourself to argue "what if not X?". Hidden assumptions are much easier to spot in their negated form ("we should use microservices" vs. "what if we didn't use microservices?"). This is a runtime version of the "consider the opposite" debiasing strategy from classic cognitive-bias research [21].
Crucially: the critic never sees the answerer's actual words. Only the judge's neutral restatement reaches it. And it says "no further objections" when genuinely done: no manufactured drama.
Why memoryless? Because the MAD literature [6, 10, 11] keeps finding the same thing: when agents share context and accumulate history, the "debate" stops being adversarial and becomes an echo chamber. Sycophantic conformity [10], degeneration of thought [11], a stronger model drifting toward a weaker peer's wrong answer, the usual suspects. Wiping the critic is the cheapest defence against all of them.
Judge
The busiest role. It's simultaneously an LLM-as-a-Judge [5] and a Meta-Judge in the sense of Ma et al. [6]: mediating, normalising, and ruling. Three phases per question:
- Phase 1: rephrase. Is the question clear enough? If yes, rewrite it neutral. If no, ask the user to clarify and loop until ready.
- Phase 2: mediate and verdict. Restate each answerer reply as neutral facts (the bottleneck). At the end, given the critic's challenges as context, deliver the final answer with a confidence label.
- Phase 3: profile note. After the debate, produce at most one observation about the answerer's tendencies. Stylistic stuff (length, formatting, tone) is rejected. Only substantive weaknesses count.
End-to-end flow for one question
- Setup. Judge and critic get rebuilt fresh. The answerer keeps its memory.
- Rephrase. The judge rewrites the question or asks clarifications until satisfied.
- Debate. Bounded loop: answerer replies → judge restates as neutral facts → critic sees only the restatement and either objects or declares done → if not done, answerer hears the critique verbatim and replies again.
- Verdict. The judge gets the critic's challenges as context and delivers the final answer + confidence label + remaining uncertainty.
- Profile update. If enabled, the judge extracts a profile note. Deduplication, stylistic filter, and surfacing threshold rules decide whether it sticks.
Memory model
What survives between rounds:
- The answerer's full conversation memory (every debate it took part in).
- The list of rephrased questions (so the critic and judge stay terminologically consistent).
- The answerer profile (active entries + pending candidates).
What dies at the start of every question:
- The critic and judge themselves. They're rebuilt so their system prompts reflect the latest state.
What each actor sees inside one round:
- The answerer sees the rephrased question + every critic reply verbatim.
- The critic sees only judge restatements. Never the answerer's words.
- The judge sees every answerer reply (to restate them) and gets the critic's challenges at verdict time.
What does not persist:
- Debate transcripts (the answerer carries them in its own memory; nobody else does).
- Clarification replies (stay inside that round).
- Restatements (shown to the user for transparency, then discarded).
- The verdict itself (lives on only as part of the answerer's conversation memory, which is how follow-up questions feel natural).
The profile
This is the only cross-round learning the system does, and the rules are intentionally conservative so one bad round can't poison everything:
- Source: Only the judge writes. Only when the critic raised a real substantive criticism.
- Stylistic filter: Anything about length, formatting, tone, emoji is rejected outright. The profile is about what the answerer reasons poorly about, not how it writes.
- Deduplication: Simple word overlap. If a new note is close enough to an existing one, it bumps the count instead of adding a duplicate.
- Surfacing threshold: A note must be observed at least twice to become visible to the critic. Single-shot observations stay hidden.
- Capacity cap: When full, the oldest un-corroborated candidate gets evicted. Confirmed entries are protected.
- Effect: Active entries show up in the critic's instructions as "observed answerer tendencies: probe these with extra care".
Why each piece is there
None of this is arbitrary. Each choice is trying to kill a specific failure mode from the literature. Here's the mapping.
The restatement bottleneck
Two ideas behind it:
The "lossy paraphrase" intuition: Tishby's information bottleneck [7] is the classic frame for "compress to just what's useful". We borrow the intuition (keep the claims, drop the rhetoric) and nothing more. The judge is a prompted paraphraser, not a learned encoder.
A mediator, not a blind reviewer: this isn't double-blind review (that hides author identity; reviewers still read original prose). This is closer to a court mediator or a Wikipedia-style neutral rewrite. LLM critics have been shown to favour or reject answers based on writing style rather than content [8, 9, 16]. Paraphrasing as debiasing is studied directly in [8]. So: every answerer-to-critic message goes through the judge's rewrite, and the critic argues with substance.
The caveat [8]: this helps but isn't magic. Authorship leaks through perplexity, word choice, structure. Treat it as useful, not bulletproof.
The memoryless critic
Echo chambers are the #1 failure mode in MAD literature [6, 10, 11]. When agents share context: sycophantic conformity, degeneration of thought, stronger models drifting toward weaker peers' wrong answers. Naive MAD can amplify biases rather than correct them. A fresh critic every round can't be slowly trained into politeness, can't build "we agreed last time" inertia, and won't echo earlier positions.
A single Meta-Judge, not a panel
Ma et al. [6] found that MAD-as-Judge amplifies the usual biases (position, verbosity, bandwagon) badly after round one, while a Meta-Judge structure holds up. One judge with procedural duties is the closest match. The verdict prompt only reaches the judge at the end, with the critic's challenges as context, not thrown into a free-for-all.
Related good news: Khan et al. [12] show adversarial debate can improve truthfulness even under a weaker judge. But that result assumes the judge only arbitrates, not also writes the surface the critic sees. So it doesn't transfer cleanly here (see Known Issues).
A twist worth noting: in most LLM-as-Judge work, the meta-judge scores a debate that already finished. Here the judge is in the loop on every turn: mediator, restater, and arbiter. Same architectural lever, more demanding job.
The profile, and why stylistic notes are banned
Instead of seeding the answerer with a fixed personality, the system learns where it actually fails and feeds that to the critic. The loop "critic probes under a rubric → judge arbitrates and updates the rubric" is a runtime cousin of Constitutional AI [17], except the rubric grows from observed behaviour instead of being written upfront. Without the stylistic filter, the profile degenerates into "this model writes long bulleted answers" (true, but useless to a critic).
Trade-offs
- The judge is a single point of failure. If it mis-restates, the critic argues with a strawman. It's the LLM telephone game, and the side-by-side display (raw reply + restatement) exists specifically so the human can catch it.
- Latency and token cost. One round costs: answer + restate + critique + verdict. On local models that's annoying. On hosted APIs it's a real bill. The extra cost has to be earned by the bottleneck and the profile: we haven't measured the gap against Self-Refine [18] yet.
Things to try next
Some directions I'm thinking about:
-
Structured restatement. Replace free-form prose restatements with a structured object (
core_claims,assumptions,omitted_variables,evidence_offered). Makes drift between turns mechanically detectable and the critic's job auditable. - Split the judge into Mediator + Arbiter. Two specialised prompts driving the same model. Relieves the worst objective conflict at zero implementation cost.
- Neuro-symbolic verification. Once the restatement is structured, have the judge emit logic fragments that an actual solver can consistency-check. LINC, FLARE, Proof of Thought, THOUGHT-LIKE-PRO [14] show the pattern is practical.
- Multiple specialised critics. A "literature critic" for citations, a "logic critic" for derivations, an "edge-case critic" for preconditions. They'd never see each other, so no echo-chamber risk.
- Calibrated confidence. Right now low/medium/high is just a self-report. Calibrate it against the track record using profile data.
- Built-in CoT baseline. A single-shot, no-critique mode in the same CLI, so the comparison against DAB is always one command away.
Known issues
These aren't bugs, they're structural limits I'm aware of and haven't solved.
The judge does three jobs with conflicting pressures
Rephrase (neutralisation), restate (information preservation), and verdict (decisive arbitration) are genuinely different objectives crammed into one prompted persona. Worse: the judge writes the surface the verdict is evaluated against. Every answerer turn reaches the verdict only through the judge's own paraphrase, which makes this structurally closer to self-evaluation than mediation. Self-preference bias [8] applies, perplexity cues leak through even when identities are hidden.
Second-order problem: if the judge is the weakest model (the realistic case on local hardware), it fails at restatement first. Weak compressors silently drop claims, and the critic critiques strawmen. Khan et al. [12] doesn't save us here because their result assumes a pure arbiter, not one that also writes the surface.
No internal detection mechanism. No recovery path.
Default models are too small for the design
The bottleneck story assumes faithful restatement. On 7–8B models (llama3.1:8b, qwen2.5:7b), abstractive paraphrase routinely drops claims and softens hedges. The defaults exist because 16 GB of RAM and 8 GB VRAM is what most tinkering machines have, not because they're ideal. Numbers from this CLI on defaults are fine for within-configuration comparisons but shouldn't be quoted against papers that evaluate on GPT-4-class models.
The profile can self-reinforce bad notes
The profile is judge-write-only. The answerer has no rebuttal channel. If the judge mis-identifies a tendency: next critic probes a phantom weakness → answerer over-defends → judge sees "confirmation" → profile hardens. The surfacing threshold helps but both observations come from the same biased judge, so they're not independent.
A weak judge makes this compound across rounds.
References
[1] Du, Y., Li, S., Torralba, A., Tenenbaum, J. B., & Mordatch, I. (2023). Improving Factuality and Reasoning in Language Models through Multiagent Debate. arXiv:2305.14325. The landmark MAD paper. Society-of-Mind framing.
[2] Irving, G., Christiano, P., & Amodei, D. (2018). AI safety via debate. arXiv:1805.00899. The earlier alignment-flavoured proposal of using debate between agents as a way to surface truth under a human judge.
[3] Sharma, M. et al. (2023). Towards Understanding Sycophancy in Language Models. arXiv:2310.13548 (ICLR 2024). Documents the tendency of RLHF-trained assistants to agree with user-stated premises, including against factual evidence. The "imitation bias" referred to throughout this document.
[4] Perez, E. et al. (2022). Discovering Language Model Behaviors with Model-Written Evaluations. arXiv:2212.09251. Earlier and broader documentation of sycophantic and persona-conditioned behaviours; complement to [3].
[5] Zheng, L. et al. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. arXiv:2306.05685 (NeurIPS 2023). The canonical LLM-as-a-Judge reference; also flags position, verbosity, and self-enhancement biases.
[6] Ma, C. et al. (2025). Judging with Many Minds: Do More Perspectives Mean Less Prejudice? On Bias Amplification and Resistance in Multi-Agent Based LLM-as-Judge. arXiv:2505.19477. The empirical claim behind the Meta-Judge choice here: MAD amplifies bias, Meta-Judge resists it.
[7] Tishby, N. & Zaslavsky, N. (2015). Deep Learning and the Information Bottleneck Principle. arXiv:1503.02406. Invoked here only as a loose metaphor for "lossy paraphrase that preserves claim-content". The principle itself is a quantitative object this design does not instantiate.
[8] Wataoka, K., Takahashi, T., & Ri, R. (2024). Self-Preference Bias in LLM-as-a-Judge. arXiv:2410.21819. Quantifies self-preference in LLM judges and links it to the perplexity (familiarity) of the evaluated text, regardless of whether the text is self-generated. The motivating finding for the voice-normalising restatement, and the caveat that stylistic normalisation is useful but partial.
[9] Li, J. et al. (2024). A Survey on LLM-as-a-Judge. arXiv:2411.15594. Broader overview of biases in LLM-as-a-Judge and mitigation strategies.
[10] Wynn, A., Satija, H., & Hadfield, G. (2025). Talk Isn't Always Cheap: Understanding Failure Modes in Multi-Agent Debate. arXiv:2509.05396. Empirical evidence that MAD can degrade accuracy: stronger models shift toward weaker peers' incorrect reasoning rather than challenging it.
[11] Liang, T. et al. (2023). Encouraging Divergent Thinking in Large Language Models Through Multi-Agent Debate. arXiv:2305.19118. The other landmark MAD paper. Explicitly diagnoses "degeneration of thought" / conformity across rounds: the failure mode the memoryless critic is built to defeat.
[12] Khan, A. et al. (2024). Debating with More Persuasive LLMs Leads to More Truthful Answers. arXiv:2402.06782. Empirical follow-up to [2]: adversarial debate can improve truthfulness even under a weaker judge, with explicit measurement of sensitivity to judge calibration.
[13] Michael, J. et al. (2023). Debate Helps Supervise Unreliable Experts. arXiv:2311.08702. Closest empirical kin to [2]; tests the judge-asymmetry assumption.
[14] Neuro-symbolic LLM reasoning: e.g. LINC: A Neurosymbolic Approach for Logical Reasoning by Combining LLMs with First-Order Logic Provers (ACL 2023), FLARE: Faithful Logic-Aided Reasoning and Exploration (arXiv:2410.11900), Proof of Thought (arXiv:2409.17270), THOUGHT-LIKE-PRO (arXiv:2407.14562). Practical templates for structured-restatement + symbolic-verification.
[15] Minsky, M. (1986). The Society of Mind. Simon & Schuster. The conceptual ancestor cited by Du et al. for "multiple minds debating one problem".
[16] Wang, P. et al. (2023). Large Language Models are not Fair Evaluators. arXiv:2305.17926. Documents position, verbosity, and self-enhancement biases in LLM-as-Judge; standard companion to [5].
[17] Bai, Y. et al. (2022). Constitutional AI: Harmlessness from AI Feedback. arXiv:2212.08073. The original "model critiques model under a rubric, then a judge rules" pattern; the critic-and-judge roles here are a runtime cousin of the CAI training loop.
[18] Madaan, A. et al. (2023). Self-Refine: Iterative Refinement with Self-Feedback. arXiv:2303.17651. The single-agent self-critique baseline any multi-agent debate design has to beat to justify its cost.
[19] Saunders, W. et al. (2022). Self-critiquing models for assisting human evaluators. arXiv:2206.05802. Earlier work training a model to find flaws in another model's output; direct ancestor of the critic role.
[20] Wei, J. et al. (2023). Simple Synthetic Data Reduces Sycophancy in Large Language Models. arXiv:2308.03958. Sycophancy is mitigable at training time, which contextualises the inference-time rephrasing defence used here.
[21] Lord, C. G., Lepper, M. R., & Preston, E. (1984). Considering the opposite: A corrective strategy for social judgment. Journal of Personality and Social Psychology, 47(6), 1231–1243. The original "consider the opposite" debiasing study; the cognitive-science parent of negation-as-inversion.
Code
The architecture is intentionally minimal: about 15 modules total, no persistence, no tests. debate.py is the entry point; it picks a provider, runs a setup wizard, builds the three actors, and hands off to a read-eval-print loop. The loop dispatches !commands (like !stats and !new) and forwards everything else to the DebatePipeline, which is the one place that knows how a question becomes a verdict.
src/
├── debate.py # entry point, wires everything
├── install.ps1 # Windows one-shot installer
├── requirements.txt
├── personas/ # one .txt per (preset, role)
└── debate_pkg/
├── chat.py # read-eval-print loop
├── pipeline.py # the per-question state machine
├── context.py # shared session state
├── setup.py # interactive setup wizard
├── personas.py # persona file loading
├── profile.py # cross-round Answerer profile
├── tokens.py # token counting for !stats
├── ui.py / colors.py # all terminal I/O
├── actors/ # Answerer, Critic, Judge
├── commands/ # !help, !new, !personas, !stats
└── providers/ # LLMProvider interface + OllamaProvider
The provider layer is the only thing that talks to the LLM backend. Swapping Ollama for OpenAI, Anthropic, or Azure is a single new file + one line changed in debate.py. The actors are intentionally dumb (all orchestration logic lives in pipeline.py) so extending the debate (structured restatements, multiple critics, etc.) means touching one module, not the whole tree.
