Skip to main content
| On a pull request, Verify reads the changes, writes test scenarios for the behavior they affect, and runs them against the pull request’s code in a sandbox. It reports the results in one comment that it keeps up to date, and keeps the evidence in the pull request’s Change Stack. Before Verify can test a pull request, the repository must be added to Verify with a prepared environment, and Enable PR testing must be on. See Set up Verify.

Start a run with a comment

Post this as a new comment on the pull request:
Post the command as a new pull request comment. Verify ignores it in a review-thread reply, without starting a run or replying.
To steer the run, add instructions after verify, on the same line or on the following lines:
Instructions shape which scenarios Verify prioritizes, the coverage it emphasizes, and the behavior it asserts. On a pull request that already has saved scenarios, instructions can add scenarios or refocus saved ones, but every saved scenario still runs. To drop a scenario, delete it on the pull request’s Verify tab in Change Stack. Instructions don’t make Verify test a change that has no material effect on behavior. The command has these rules:
  • It must start a line, outside a fenced code block.
  • Each comment starts at most one run. Editing a comment that already started a run doesn’t start another.
  • You need write access to the repository in your Git provider. Otherwise Verify replies: “This command requires write access to the repository. Only users with write or admin permissions can trigger CodeRabbit to commit or create pull requests.”
  • On GitLab, Verify also checks your membership in the GitLab organization. On GitHub, if the repository’s configuration sets chat.allow_non_org_members to false, you must be a member of the GitHub organization.
  • If a run is already testing the pull request’s current commit, Verify replies “Verify is already testing the current PR head.” with a View running test link instead of starting another. To run with different instructions, wait for that run to finish or stop it, then comment again.
On GitHub Cloud, when the repository is added to Verify with a ready environment, the CodeRabbit walkthrough comment includes a CodeRabbit Verify section with the checkbox Run CodeRabbit Verify to test the changes in this pull request. Selecting it starts a run if you have write access, like the command.

Automatic runs

When Auto Verify is on for the repository, Verify considers each pull request when it is opened. With Also verify new commits also on, it considers the pull request again whenever new commits are pushed. Automatic runs start alongside CodeRabbit’s review. Automatic runs skip draft pull requests, pull requests opened by bots, closed and merged pull requests, and public repositories on GitHub. Editing a comment or the title, reopening a pull request, or marking a draft ready for review doesn’t start one either. To test any of these except a pull request closed without merging, comment @coderabbitai verify. To turn automatic runs on, see Turn on automatic runs.

Selection rules

Before an automatic run, Verify checks whether the pull request’s changes qualify. By default it includes:
  • PRs that change product behavior, including UI, APIs, validation, permissions, data, or background workflows.
  • PRs that mix functional and non-functional changes.
  • PRs where it’s unclear whether product behavior changes.
And it excludes:
  • PRs limited to documentation, test maintenance, CI, build tooling, infrastructure, or internal logging changes.
  • Refactoring and performance improvements that preserve product behavior.
To change this, write Custom selection rules on the repository’s Verify settings page, under Automatic runs > Selection rules. Your rules override conflicting defaults; leave the field empty to use the defaults. For example: “Include product changes. Exclude infrastructure-only changes.” If Verify can’t complete the check, it runs the tests anyway. A run you start with @coderabbitai verify is never filtered. When the check excludes a pull request, Verify posts a CodeRabbit Verify comment that says “Verify testing was intentionally skipped under the PR testing criteria. This is not a failed test result.” and names the commits it evaluated, followed by “No Verify tests were run.” No run is created, so you can still start one with @coderabbitai verify or, on GitHub, with the Run a verifier button in that comment.

The Verify comment

Verify posts one comment headed CodeRabbit Verify and edits it at three points:
  1. When the run starts, with the label Planning scenarios.
  2. When the scenarios are ready, with the label Scenarios ready and a list of the scenarios Verify will test.
  3. When the run ends, with the final label and a row per scenario.
Progress during testing appears only on the pull request’s Verify tab in Change Stack. The finished comment contains:
  • A link to the pull request’s Verify tab in Change Stack, labeled View progress before scenarios exist, View scenarios once they do, and Review evidence once there’s evidence.
  • A bold status line with the run label, how many scenarios are complete, and counts per result. On GitHub repositories with the comment header, the run label appears in the heading instead, such as CodeRabbit Verify ¡ All scenarios passed, and the bold line starts with the scenario counts.
  • The commit the run covers, labeled Target commit until a scenario passes or fails and Tested commit after that (Analyzed commit when scenarios were only generated), and the Total time.
  • A table with the columns Result, Scenario, and Run summary. Each row links to the scenario’s Evidence or Details, and to Watch recording when a recording is available. With more than five scenarios, passed scenarios are collapsed.
  • Footer links: View test results (on GitHub), Revise scenarios, and Turn off automatic Verify. Revise scenarios opens the pull request’s Verify tab in Change Stack, where you can edit or delete scenarios and run the tests again. See Review results in Change Stack.
The comment never contains evidence files. Its links open Change Stack, where you sign in and your repository access is checked.

Run labels

Each scenario row shows a result: Bugs, Blockers, Passed, Failed, Error, or Inconclusive. The CodeRabbit app and the Activity page use other names for some run labels and results. See Comment labels and app outcomes.

Review results in Change Stack

The pull request’s Verify tab in Change Stack shows the run’s progress, each scenario with its evidence and recording, and earlier runs under Recent Verify runs. There you can edit, revise, add, or delete scenarios, and start a new run with Run tests, which also requires write access to the repository. To act on findings, see Findings on a pull request. If the run you’re viewing tested an older commit, Change Stack says “These Verify results belong to a different commit.” and offers Run tests for latest commit.

New commits and outdated runs

When a new run starts for a newer commit, automatically or with @coderabbitai verify, Verify stops any run still testing an older commit. A run that finishes after you push shows Outdated run and a note that it doesn’t cover the current pull request head, and its results apply only to the commit it tested. To test the new commit, turn on Also verify new commits or comment @coderabbitai verify. With Also verify new commits off, an automatic run that hasn’t started testing when you push is cancelled, and the comment says “Verify was cancelled before testing started because a new commit was pushed. Automatic runs on new commits are disabled for this repository.” Comment @coderabbitai verify to test the latest commit. Later runs on the same pull request reuse its saved scenarios, including the ones you add or revise in Change Stack. If you add instructions after @coderabbitai verify, Verify can add scenarios or refocus saved ones to cover them, and every saved scenario still runs. To drop a scenario, delete it on the pull request’s Verify tab in Change Stack.

Stop a run

You can’t stop a run from the pull request. Open the pull request’s Verify tab in Change Stack from the Verify comment, select View agent run, and then select Stop in the page header. You need the Coding Agent write permission. Editing or deleting the @coderabbitai verify comment doesn’t stop the run. A stopped run ends as superseded, like a run interrupted by a new commit.

Verify doesn’t block merges

Verify posts no check run and no commit status, so a failing or unfinished Verify run never blocks a merge. Read the comment before you merge.

What’s next

Results and bugs

Read outcomes and evidence, and turn bugs into tickets or coding tasks.

User journeys

Test critical flows on a branch, on demand or on a schedule.

Troubleshooting

Fix runs that don’t start, get skipped, or end without a result.