Outcomes
Each run and each scenario or journey has one primary outcome. The app uses these labels:
When a run has several kinds of results, the outcome follows this order: Bugs first, then Blocked, then Blockers, then the test verdict. Other findings stay visible in the scenarioâs details.
Comment labels and app outcomes
The Verify comment on a pull request, the CodeRabbit app, and the Activity pageâs filters name some results differently. The Activity pageâs Outcomes filter matches runs that contain at least one scenario with that result.
In the commentâs scenario rows, Error corresponds to the appâs Blocked. See Run labels.
Find results
- Pull request runs: in the Verify comment on the pull request, and on the pull requestâs Verify tab in Change Stack. See Test pull requests.
- One repository: on the repositoryâs page, Pull Requests > PR runs lists pull request runs, and Journey Testing > Run history lists user journey runs.
- All repositories: the Activity page at app.coderabbit.ai/verify/activity. It isnât in the sidebar; the same dashboard also appears below the repository list when you select Verify.
Findings on a pull request
A pull request runâs findings stay with its scenarios on the pull requestâs Verify tab in Change Stack; they donât appear on the repositoryâs Bugs and Blockers pages. Select a scenario to see its result, its findings, and its evidence. From there you can:- Hand findings to a coding agent: select Fix issues with AI (or Investigate with AI) for the run, or Fix with AI (or Investigate with AI) on a scenario. The Handoff to coding agent dialog lets you choose findings and evidence, then Copy markdown or Open in Codex.
- Change the scenarios: select one scenario and choose Edit manually or Edit with a prompt, select scenarios and choose Revise or Delete, or choose Add scenario, so later runs on the pull request test what you intend.
Judging a finding
A finding is what Verify observed, checked against what it expected. Before you act on one, watch the recording and compare the steps and the expected behavior with what your change intends. Verify has no action to dismiss a finding or mark it as not a bug, and findings donât block merging. If a scenario expects behavior that your change intentionally alters, edit or delete the scenario so later runs on the pull request donât report it again.Evidence
A run can attach these kinds of evidence to a scenario, journey, or finding:- Screenshots that the tester takes as it works.
- Session recordings of the browser. When both exist, Watch shortened recording plays a version with idle stretches cut out and Watch full recording plays the whole session. The recordingâs timeline marks where assertions passed or failed and where bugs and blockers were found; select a marker to jump to it.
- Text files, such as each scenarioâs execution result and log. Text files up to 2 MiB are previewed in the app; download larger ones.
- API requests, for pull request runs, on the scenario in Change Stack: the method, path, response status and duration, and the request and response bodies.
Bugs and blockers from user journeys
This section and the next cover findings from user journey runs. For pull request runs, see Findings on a pull request. Each repository has Journey Testing > Bugs and Journey Testing > Blockers pages for findings from user journey runs. The counts next to them show open findings on the default branch across all environments. Filter the list by Branch, Environment, and Status (Open or Closed). Its columns are Finding, Journey, Type, Status, and Tickets. Open a finding to see its details:- A bug shows Description, Actual behavior, Expected behavior, the Journey, Reproduction steps, and its evidence.
- A blocker shows what blocked testing, its Impact, the Blocked tasks, Suggested remediation, and its evidence.
Copy a finding or fix it with Coding Agent
On an open finding:- Copy copies the finding as Markdown, with its repository, branch, and environment.
- Fix with Coding Agent opens a new Coding Agent task, prefilled with the finding, the repository, the tested branch, and a link back to the finding in Verify. Review the task and start it yourself; nothing runs until you do. Evidence files stay in Verify and arenât attached to the task.
Create Jira or Linear tickets
Verify creates tickets in Jira and Linear. It needs a Jira or Linear connection in your organization.1
Save the repository's ticket defaults
On the repositoryâs page, select Journey Testing > Settings and open Issue tracker settings. Choose the Issue tracker type, the Connection, the Linear team or Jira project, and the Default status. Optionally set Default assignee (optional), Default priority (optional), Linear project (optional), and Default labels. Then select Save changes.
2
Create a ticket
On a finding, select Create ticket. Verify creates the ticket with your saved defaults and confirms with its key, and the ticket appears in the Tickets column.
Automatic tickets from schedules
A schedule can create tickets for you. In the schedule editor, under Issue tracker, turn on Auto create issues and choose Bugs only or Bugs & blockers. Only new findings create issues. The repository must have saved issue tracker settings first; until it does, the schedule editor links to the issue tracker settings instead of showing Auto create issues. When a scheduled retest closes a finding, Verify moves its linked issue to Done in Jira or to the teamâs completed state in Linear, whether the issue was created automatically or by hand.Whatâs next
Troubleshooting
Fix blocked, inconclusive, and failed runs and missing evidence.
Coding Agent
Review and deliver a fix started from a Verify finding.
User journeys
Schedule journeys so that fixed bugs get retested automatically.