Skip to main content

ā€‹
How Custom Checks Work

Custom checks run in a secure, read-only environment against your PR. The agent has access to:
  • Changed files, code snippets, and relevant git history
  • PR title, description, linked issues, and review discussion
  • Pattern and code search tools (ast-grep, ripgrep)
  • Sandboxed shell commands to inspect the repo
  • Web lookups for public documentation
  • Connected MCP tools for internal systems
Process:
1

Analyze

CodeRabbit interprets your instructions against the PR context
2

Verify

The agent gathers evidence using available tools to substantiate findings
3

Decide

Emits Passed, Failed, or Inconclusive with reasoning

ā€‹
Limitations

Custom checks run in a sandboxed environment. Instructions that rely on unavailable capabilities will return Inconclusive or produce unreliable results.
Custom checks cannot:
  • Run your test suite ā€” dependencies are not installed in the sandbox
  • Access node_modules, dist, or build artifacts ā€” build steps are not executed
  • Execute arbitrary repository code ā€” security restriction
  • Post inline comments on specific lines ā€” results appear in the summary table only
  • Check PR approval status or reviewer assignments ā€” not available to the agent
  • Modify the CodeRabbit review ā€” use Path Instructions instead

ā€‹
Writing Effective Instructions

Think of instructions as guidance for a smart teammate who needs explicit criteria, not subjective judgment.

Be specific and actionable

Avoid vague instructions like ā€œCheck for security issuesā€

Define clear pass/fail criteria

Say exactly what causes failure

One concern per check

Donā€™t bundle security, testing, and documentation into one instruction

Include examples for complex rules

Show what passes and what fails
Avoid these anti-patterns:
Anti-patternExampleWhy it fails
Vague instructionsā€Verify best practicesā€No concrete pass/fail criteria
Unavailable informationā€Ensure PR is approved by @security-teamā€Agent cannot check approval status
Speculationā€Assess if there are obvious performance optimizationsā€No definitive criteria; relies on subjective judgment
Review modificationsā€Keep review comments conciseā€Use Path Instructions instead

ā€‹
Examples

Custom checks can enforce a wide range of guardrails tailored to your teamā€™s needs. Here are a few examples to get you started. 
Fail if any log statement may include passwords, API keys, tokens,
SSNs, or payment card data. Check print(), console.log, logger.*,
System.out.println, and fmt.Print variants.

Scan for hardcoded credentials matching: sk_live_*, AKIA[A-Z0-9]{16},
ghp_[a-zA-Z0-9]{36}, or variables named *_SECRET, *_KEY, *_PASSWORD.
Exclude test files and .example files.

For files in db/migrations/: verify both up() and down() methods exist.
Flag migrations dropping tables without restoration in down().

All breaking changes to public APIs, CLI flags, environment variables,
or database schemas must be documented in the "Breaking Change" section
of the PR description and in CHANGELOG.md.

Disallow creation of new Java source files. Java may be modified,
but new code should be written in Kotlin.

Pass: PR modifies existing .java files or adds no new .java files.
Fail: A new *.java file is added to source control.