Before you begin
- Runtime verification must be enabled for your organization. It is available for GitHub Cloud repositories on CodeRabbit Cloud. If the tab shows Coming Soon, the feature is not available for that finding or organization.
- You need Security read access and access to the repository to view results or start an individual verification. Changing the repository’s automatic verification setting requires Security write access.
- Verification uses Agent minutes, separately from Deep Scan credits. You need available Agent minutes or enabled usage billing.
- To use Runtime Analysis, the repository needs a prepared coding environment in CodeRabbit with the dependencies and setup required to run the application. You can follow the guide in your own staging or test environment without configuring an environment in CodeRabbit.
Generate and use a reproducer guide
1
Open the finding
Go to Security > Findings, open a Deep Scan vulnerability, and select Runtime verification.
2
Generate a reproducer
Select Generate Reproducer. CodeRabbit first rechecks whether the finding is still present in the current source. If it is no longer present, no guide or runtime test is needed. If the check cannot reach a conclusion, generation stops and the tab explains why.When the finding is still present, CodeRabbit analyzes the source and generates a guide tied to the displayed Revision.
3
Review the guide
Expand Reproducer Guide to review prerequisites, steps, expected results, missing context, and limitations. A published guide describes how to test the finding; its expected results are based on source analysis and are not runtime evidence.
4
Reproduce in your own environment
Follow the guide’s steps in your internal staging or test environment. Check the displayed revision, satisfy the prerequisites, and supply any environment-specific values identified by the guide. Compare your observations with the expected results to assess whether you can reproduce the finding in your setup.You can use the guide independently of Runtime Analysis; generating it does not start a runtime test in CodeRabbit.
Run the guide in CodeRabbit
Runtime Analysis can execute the generated guide in an isolated environment within CodeRabbit and retain evidence from the run. Use it when you want CodeRabbit to run and assess the reproduction steps for you.1
Configure a coding environment
Follow Coding environments to choose toolchains, install dependencies, configure environment variables and startup instructions, and assign the repository. Wait for the environment and repository assignment to be ready.You can also select Configure environment from the finding if no prepared environment is available.
2
Run the guide
Return to the finding’s Runtime verification tab. In Runtime Analysis, select a prepared Environment and choose Run. The environment selection applies only to this run and does not change the repository’s default environment.
3
Inspect the result
Follow the run as CodeRabbit prepares the environment, executes the guide, and assesses the evidence. Expand Runtime Analysis to read the result and supporting observations. You can use Cancel run while execution is active.
Verify findings automatically
In the Run Deep Scan dialog, choose a Runtime verification level before starting the scan. When you have Security write access, the choice is also saved for the repository; otherwise, it applies to that scan only. To configure future scans, go to Security > Repositories, select the repository, and open Settings. In Runtime verification, choose the level under Verify new findings. If Turn on appears, select it to save the initial opt-in.
After a Deep Scan completes, CodeRabbit generates guides for eligible open findings that the scan newly detected or reopened at the selected severity. Findings carried over unchanged from earlier scans are not automatically verified again.
When a guide is ready and the repository has a prepared default environment, CodeRabbit starts Runtime Analysis automatically. Without a prepared default environment, the guide can still be generated. You can follow it in your own staging or test environment, or select a prepared environment in CodeRabbit and run it manually.
Open a finding’s Runtime verification tab to follow its progress and inspect the result. A completed Deep Scan does not mean its runtime tests have finished.
Interpret runtime results
Check Application ready and Target exercised alongside the conclusion. They show whether CodeRabbit established that the application was ready and that the relevant code path was exercised. A setup failure or an untested path is not evidence that a vulnerability is absent.
Under Assessment and Evidence, inspect the completed steps, retained output, limitations, and counter-evidence. You can download retained text evidence for closer inspection. The result also identifies the environment, tested commit, and Agent minutes used.
Results apply to the published guide and tested commit. Runtime Analysis does not change the finding’s state. Use the evidence to decide whether to resolve or dismiss the finding, or start Fix with AI.
Usage and billing
Runtime verification uses Agent minutes rather than the credits used to scan the repository. Automatic verification is billed as Agent minutes, up to $6 per finding.- For a scan you start, automatic verification is billed to you.
- For scheduled and coverage-triggered scans, it is billed to the person who last saved the runtime verification setting in repository settings. Saving that setting makes you the payer for those runs.
- If the payer has no available Agent minutes and usage billing is off, automatic verification is skipped. The Deep Scan can still produce findings.
Troubleshoot verification
What’s next
Coding environments
Configure an environment in CodeRabbit to run a guide with Runtime Analysis.
Triage and remediate findings
Use the evidence to resolve, dismiss, or fix a finding.
Steer Deep Scans
Give future scans the repository context and security instructions they need.