AuditOwl
Sign in Start a free audit

Deep reasoning verification for research code

Code audit for scientific manuscripts For editors and authors

AuditOwl runs a deep analysis of a manuscript and the code repository behind it, and reports discrepancies between the two as well as methodological problems in the code that could change the paper's conclusions.

Start a free audit See how it works

Beta: one free audit per day.

An example finding, as it appears in the report

Severity:high Finding §3.4 Confidence medium

The checkpoint kept for evaluation is chosen by test_success_rate, not validation loss

eval/test_utils.py:93 
92# Track best model for this planner type93if metrics.test_success_rate > best_results[planner_type]["success_rate"]:94    best_results[planner_type]["success_rate"] = metrics.test_success_rate95    best_results[planner_type]["model_type"] = model_type96    best_results[planner_type]["epoch"] = epoch
Paper · §3.4, p. 5

“We select the model checkpoint that achieves the lowest loss on the validation set for evaluation.”

The code instead keeps whichever checkpoint scores best on the test set, so the reported number is not the validation-selected result the paper describes.

Finding 1 of 5: The checkpoint kept for evaluation is chosen by test_success_rate, not validation loss

These mistakes usually survive peer review

What it checks

AuditOwl looks for code failures that quietly invalidate a scientific claim. They are often invisible in the PDF alone, so they slip through the peer review. Spot them before an awkward email arrives after publication.

  • Methodological problems

    AuditOwl detects domain specific methodological mishaps like data leakage or invalid statistical testing that often can change the conclusion of a paper.

  • Reproducibility gaps

    AuditOwl analyzes if the paper is reproducible for example by making sure that all important code or configurations are present.

  • Claim ↔ code contradictions

    The paper says multiple testing correction was used, but the authors forget to actually apply it? AuditOwl spots this.

  • Conclusion changing bugs

    Sometimes bugs are just reproduction nuisances, but sometimes they can fully invalidate a scientific claim. AuditOwl catches deep semantic bugs, before a reviewer does.

From claim to code

How it works

AuditOwl first finds the major scientific claims in the paper: figure panels, the numbers in tables, the results of statistical tests, and quantitative claims in the text.

It then traces each claim to the code that backs it. If a table reports results for two machine-learning models, AuditOwl traces the model code, the preprocessing and training code, and the evaluation code that produces the numbers in that table. Then it asks four questions.

  1. Are major parts of the code missing?

    For example, one of the models is not implemented, or the training code is missing.

  2. Is the code technically correct?

    For example, is there a bug that prevents a baseline from learning properly?

  3. Is the claim methodologically valid?

    For example, a model may have been trained on leaky features that would not be available in practice. Errors like this can often only be found in the code, not in the text.

  4. Do paper and code severely mismatch?

    For example, the paper says one model component improved the result, but a different component was used.

(Caveat)

Its empirical error rate is very low, but AuditOwl is an agentic AI tool and can make mistakes. Do not use its output unchecked to form a review decision. AuditOwl does not replace human review; it augments it.

Two readers, one audit

For editors

Run AuditOwl on papers your journal has published, or on preprints already on arXiv. Get back a numbered list of concrete, checkable claims, so reviewer time goes to the science instead of to someone else's data-loading code.

It builds your journal's reputation for rigorous, reproducible science that can be trusted.

For authors

Audit your preprint and its public repository, and fix problems before reviewers or readers find them.

Make sure your science is rigorous and reproducible, and build your reputation as a researcher who can be trusted.

AuditOwl makes your work easier to reuse, which can increase your citation count.

Start free, go deeper when it counts

Pricing

Higher tiers spend more inference compute on your work and surface progressively deeper issues.

01 / Free

Available now

Light

A fast first pass over the paper and repository.

Run a free audit

02 / Subscription

Coming soon

Standard

Deeper analysis with SOTA models.

Coming soon

03 / Subscription

Coming soon

Pro

The most thorough audit using multiple SOTA models.

Coming soon

Beta: one free audit per day.

Standard and Pro are coming soon.

Questions we get

Does this replace peer review?
No. It replaces the many hours a referee would spend reading someone else’s data-loading code for a thorough code review. It returns anchored evidence about discrepancies for a human to confirm or reject.
Can I audit an unpublished paper or a private repository?
Not during the beta. Please only use papers and code that are already public, for example on arXiv and GitHub. You can still upload public code as a zipped archive, for example from a paper’s supplementary material.
Which languages does it handle?
AuditOwl is programming language agnostic.
What does it cost?
During the beta you can run one free Light audit per day. The deeper Standard and Pro tiers are coming soon.
How long does an audit take?
It depends on the size of your project, but in our experience it’s less than half an hour.

Find problems before a referee does.

(Upload, point, read)

Upload a paper, point AuditOwl at its code, and run one free audit a day during the beta.

Start a free audit Sign in