Agents can overfit benchmarks or succeed on formal syntax without generating meaningful new mathematics.
A possibility worth exploring ●
Formal Mathematics Benchmarking
Evaluate proof agents on source-linked research statements, known partial results, false-proof traps and machine-checkable subgoals.
The possibility
Evaluate proof agents on source-linked research statements, known partial results, false-proof traps and machine-checkable subgoals.
Research-level benchmarks can measure whether AI systems formalize, retrieve, decompose and verify mathematics rather than merely produce plausible prose.
A living benchmark can absorb newly formalized results and later test reconstruction of genuine breakthroughs.
What to keep in mind
What stands between here and there?
Benchmark scores must not be described as proof progress on the underlying open problem.
Riemann Hypothesis
RH provides a network of precise conditional and computational tasks.
Understand the question →P versus NP
P versus NP supplies formal statements and barriers, though benchmark performance is not progress on the theorem.
Understand the question →Collatz Conjecture
Collatz offers finite certificates and graded partial statements.
Understand the question →Twin Prime Conjecture
Twin primes combine an open target with well-defined bounded-gap milestones.
Understand the question →Ideas it sparks
Ways to start thinking today.
These are prompts for curiosity—not roadmaps or promises.
Formal Proof-Agent Benchmark
A graded benchmark of formal statements, known lemmas, finite certificates, dependency retrieval and adversarial false-proof traps.
Why it is thinkable now
Explore the idea →Yes, using existing Lean repositories and source-backed partial results.
Sources and uncertainty 4
Formal ConjecturesGoogle DeepMind and contributorsSource verifiedFormal Conjectures: An Open and Evolving Benchmark for Verified Discovery in MathematicsMoritz Firsching, Paul Lezeau, Salvatore Mercuri, Miklós Z. Horváth, Yaël Dillies, Calle Sönne, Eric Wieser, Fred Zhang, Thomas Hubert, Blaise Agüera y Arcas, Pushmeet KohliSource verifiedCollatz Conjecture Formalization ChallengeCommunity projectSource verifiedTwin Primes formal statementGoogle DeepMind Formal Conjectures contributorsSource verified
AI summarized and source-backed · Qualified expert review pending.