> ## Documentation Index
> Fetch the complete documentation index at: https://docs.coderabbit.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Test pull requests with CodeRabbit Verify

> Run Verify on a pull request with a comment or automatically, read the Verify comment, and understand new pushes, stopped runs, and merge behavior.

GitHub Cloud (Verify is available on GitHub Cloud. GitHub Enterprise Server is not supported.) GitLab.com (Verify is available on GitLab.com. GitLab self-managed is not supported.) Bitbucket Cloud (Verify is available on Bitbucket Cloud. Bitbucket Data Center is not supported.) | Cloud Only (This feature is available on CodeRabbit Cloud only. It is not available on self-hosted CodeRabbit deployments.) Open Beta (Verify is in beta and free during the beta.)

On a pull request, Verify reads the changes, writes test scenarios for the behavior they affect, and runs them against the pull request's code in a sandbox. It reports the results in one comment that it keeps up to date, and keeps the evidence in the pull request's [Change Stack](https://docs.coderabbit.ai/change-stack.md).

Before Verify can test a pull request, the repository must be added to Verify with a prepared environment, and **Enable PR testing** must be on. See [Set up Verify](https://docs.coderabbit.ai/verify/setup.md).

## Start a run with a comment

Post this as a new comment on the pull request:

```text wrap theme={null}
@coderabbitai verify
```

> [!WARNING]
>
> Post the command as a new pull request comment. Verify ignores it in a review-thread reply, without starting a run or replying.

To steer the run, add instructions after `verify`, on the same line or on the following lines:

```text wrap theme={null}
@coderabbitai verify check that the login form shows an error after a wrong password
```

Instructions shape which scenarios Verify prioritizes, the coverage it emphasizes, and the behavior it asserts. On a pull request that already has saved scenarios, instructions can add scenarios or refocus saved ones, but every saved scenario still runs. To drop a scenario, delete it on the pull request's **Verify** tab in Change Stack. Instructions don't make Verify test a change that has no material effect on behavior.

The command has these rules:

* It must start a line, outside a fenced code block.
* Each comment starts at most one run. Editing a comment that already started a run doesn't start another.
* You need write access to the repository in your Git provider. Otherwise Verify replies: "This command requires write access to the repository. Only users with write or admin permissions can trigger CodeRabbit to commit or create pull requests."
* On GitLab, Verify also checks your membership in the GitLab organization. On GitHub, if the repository's configuration sets [`chat.allow_non_org_members`](https://docs.coderabbit.ai/reference/configuration#param-chat-allow-non-org-members) to `false`, you must be a member of the GitHub organization.
* If a run is already testing the pull request's current commit, Verify replies "Verify is already testing the current PR head." with a **View running test** link instead of starting another. To run with different instructions, wait for that run to finish or [stop it](#stop-a-run), then comment again.

On GitHub Cloud, when the repository is added to Verify with a ready environment, the CodeRabbit walkthrough comment includes a **CodeRabbit Verify** section with the checkbox **Run CodeRabbit Verify to test the changes in this pull request.** Selecting it starts a run if you have write access, like the command.

## Automatic runs

When **Auto Verify** is on for the repository, Verify considers each pull request when it is opened. With **Also verify new commits** also on, it considers the pull request again whenever new commits are pushed. Automatic runs start alongside CodeRabbit's review.

Automatic runs skip draft pull requests, pull requests opened by bots, closed and merged pull requests, and public repositories on GitHub. Editing a comment or the title, reopening a pull request, or marking a draft ready for review doesn't start one either. To test any of these except a pull request closed without merging, comment `@coderabbitai verify`.

To turn automatic runs on, see [Turn on automatic runs](https://docs.coderabbit.ai/verify/setup.md#turn-on-automatic-runs).

### Selection rules

Before an automatic run, Verify checks whether the pull request's changes qualify. By default it includes:

* PRs that change product behavior, including UI, APIs, validation, permissions, data, or background workflows.
* PRs that mix functional and non-functional changes.
* PRs where it's unclear whether product behavior changes.

And it excludes:

* PRs limited to documentation, test maintenance, CI, build tooling, infrastructure, or internal logging changes.
* Refactoring and performance improvements that preserve product behavior.

To change this, write **Custom selection rules** on the repository's **Verify** settings page, under **Automatic runs** > **Selection rules**. Your rules override conflicting defaults; leave the field empty to use the defaults. For example: "Include product changes. Exclude infrastructure-only changes."

If Verify can't complete the check, it runs the tests anyway. A run you start with `@coderabbitai verify` is never filtered.

When the check excludes a pull request, Verify posts a **CodeRabbit Verify** comment that says "Verify testing was intentionally skipped under the PR testing criteria. This is not a failed test result." and names the commits it evaluated, followed by "No Verify tests were run." No run is created, so you can still start one with `@coderabbitai verify` or, on GitHub, with the **Run a verifier** button in that comment.

## The Verify comment

Verify posts one comment headed **CodeRabbit Verify** and edits it at three points:

1. When the run starts, with the label **Planning scenarios**.
2. When the scenarios are ready, with the label **Scenarios ready** and a list of the scenarios Verify will test.
3. When the run ends, with the final label and a row per scenario.

Progress during testing appears only on the pull request's **Verify** tab in Change Stack. The finished comment contains:

* A link to the pull request's **Verify** tab in Change Stack, labeled **View progress** before scenarios exist, **View scenarios** once they do, and **Review evidence** once there's evidence.
* A bold status line with the run label, how many scenarios are complete, and counts per result. On GitHub repositories with the comment header, the run label appears in the heading instead, such as **CodeRabbit Verify · All scenarios passed**, and the bold line starts with the scenario counts.
* The commit the run covers, labeled **Target commit** until a scenario passes or fails and **Tested commit** after that (**Analyzed commit** when scenarios were only generated), and the **Total time**.
* A table with the columns **Result**, **Scenario**, and **Run summary**. Each row links to the scenario's **Evidence** or **Details**, and to **Watch recording** when a recording is available. With more than five scenarios, passed scenarios are collapsed.
* Footer links: **View test results** (on GitHub), **Revise scenarios**, and **Turn off automatic Verify**. **Revise scenarios** opens the pull request's **Verify** tab in Change Stack, where you can edit or delete scenarios and run the tests again. See [Review results in Change Stack](#review-results-in-change-stack).

The comment never contains evidence files. Its links open Change Stack, where you sign in and your repository access is checked.

### Run labels

| Label | Meaning |
| - | - |
| **Planning scenarios** | The run is queued, preparing the environment, or writing scenarios. |
| **Scenarios ready** | Verify wrote the scenarios, and the comment lists them. Follow testing progress in Change Stack. |
| **Scenarios generated · not run** | You generated or revised scenarios in Change Stack without running them. |
| **All scenarios passed** | Every scenario passed within the tested scope. |
| **Bugs found** | At least one scenario found a product defect. |
| **Testing blocked** | Missing prerequisites, such as credentials or test data, limited testing. |
| **Run failed** | One or more scenarios failed, but Verify couldn't classify the findings. |
| **Results inconclusive** | No bug, blocker, or error was found, but not every scenario passed, or the observations weren't enough for a reliable result. |
| **Testing error** | An execution or setup error stopped the run. The comment gives the reason. |
| **Skipped** | Verify selected no scenarios for this change. The comment says "No scenarios were selected for this change. No tests were executed." |
| **Outdated run** | A newer commit or a stop request superseded the run. |

Each scenario row shows a result: **Bugs**, **Blockers**, **Passed**, **Failed**, **Error**, or **Inconclusive**. The CodeRabbit app and the Activity page use other names for some run labels and results. See [Comment labels and app outcomes](https://docs.coderabbit.ai/verify/results.md#comment-labels-and-app-outcomes).

## Review results in Change Stack

The pull request's **Verify** tab in Change Stack shows the run's progress, each scenario with its evidence and recording, and earlier runs under **Recent Verify runs**. There you can edit, revise, add, or delete scenarios, and start a new run with **Run tests**, which also requires write access to the repository. To act on findings, see [Findings on a pull request](https://docs.coderabbit.ai/verify/results.md#findings-on-a-pull-request). If the run you're viewing tested an older commit, Change Stack says "These Verify results belong to a different commit." and offers **Run tests for latest commit**.

## New commits and outdated runs

When a new run starts for a newer commit, automatically or with `@coderabbitai verify`, Verify stops any run still testing an older commit. A run that finishes after you push shows **Outdated run** and a note that it doesn't cover the current pull request head, and its results apply only to the commit it tested. To test the new commit, turn on **Also verify new commits** or comment `@coderabbitai verify`.

With **Also verify new commits** off, an automatic run that hasn't started testing when you push is cancelled, and the comment says "Verify was cancelled before testing started because a new commit was pushed. Automatic runs on new commits are disabled for this repository." Comment `@coderabbitai verify` to test the latest commit.

Later runs on the same pull request reuse its saved scenarios, including the ones you add or revise in Change Stack. If you add instructions after `@coderabbitai verify`, Verify can add scenarios or refocus saved ones to cover them, and every saved scenario still runs. To drop a scenario, delete it on the pull request's **Verify** tab in Change Stack.

## Stop a run

You can't stop a run from the pull request. Open the pull request's **Verify** tab in Change Stack from the Verify comment, select **View agent run**, and then select **Stop** in the page header. You need the Coding Agent write permission. Editing or deleting the `@coderabbitai verify` comment doesn't stop the run. A stopped run ends as superseded, like a run interrupted by a new commit.

## Verify doesn't block merges

Verify posts no check run and no commit status, so a failing or unfinished Verify run never blocks a merge. Read the comment before you merge.

## What's next

> [card] [Results and bugs](https://docs.coderabbit.ai/verify/results.md)
>
> Read outcomes and evidence, and turn bugs into tickets or coding tasks.

> [card] [User journeys](https://docs.coderabbit.ai/verify/user-journeys.md)
>
> Test critical flows on a branch, on demand or on a schedule.

> [card] [Troubleshooting](https://docs.coderabbit.ai/verify/troubleshooting.md)
>
> Fix runs that don't start, get skipped, or end without a result.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.