Routing Rules

Overview

Routing Rules provide dynamic, expression-based control over request routing. They execute before governance provider selection and can override it, allowing you to make sophisticated routing decisions based on request context, headers, parameters, capacity metrics, and organizational hierarchy.

Unlike governance routing (which uses static provider weights), routing rules use CEL expressions (Common Expression Language) to evaluate conditions at runtime and make routing decisions dynamically.

Scope and Precedence

You attach each rule to a scope - Virtual Key, Team, Customer, or Global - and a priority. When a request arrives, rules are checked most-specific scope first and the first matching rule wins (no later rules run). Use this to decide which rule takes effect when several could match:

Virtual Key scope   (most specific - checked first)
    ↓
Team scope
    ↓
Customer scope
    ↓
Global scope        (applies to all - checked last)

Within a single scope, rules with a lower priority number are checked first (priority 0 before 10). If no rule matches, the request’s provider/model is left unchanged and normal governance/load-balancing routing applies.

Practical guidance:

Put narrow, high-priority overrides (premium tier, a specific team) at the Virtual Key or Team scope.
Put organization-wide defaults (capacity failover, data-residency) at the Global scope.
Use priority numbers with gaps (0, 10, 20) so you can insert rules later without renumbering.

CEL Expression Guide

Available Variables

Routing rules evaluate CEL expressions with these available variables:

Request Context

model          // Requested model (string)
provider       // Current provider (string)
request_type   // Request type (chat_completion, embedding, batch, image_generation, moderation, transcription, translation)

Headers & Parameters

headers["header-name"]     // Request header (case-insensitive key lookup)
params["param-name"]        // Query parameter

Header Examples:

headers["x-tier"] == "premium"
headers["x-api-version"] == "v2"
headers["user-agent"].contains("mobile")

Organization Context

virtual_key_id             // VK ID (string, empty if no VK)
virtual_key_name           // VK name (string)
team_id                    // Team ID (string, empty if not in team)
team_name                  // Team name (string)
customer_id                // Customer ID (string)
customer_name              // Customer name (string)

Organization Examples:

team_name == "ml-research"
customer_id == "acme-corp"
virtual_key_name.startsWith("prod-")

Capacity Metrics (as percentages: 0-100)

budget_used      // Budget usage percentage for provider/model (0.0 to 100.0)
tokens_used      // Token rate limit usage percentage (0.0 to 100.0)
request          // Request rate limit usage percentage (0.0 to 100.0)

Capacity Examples:

budget_used > 80           // Route to fallback when 80%+ of budget used
tokens_used < 50           // Route to fast provider when below 50% token limit
request > 90               // Switch providers when request limit near max

CEL Operators & Functions

Comparison Operators

==    // Equal
!=    // Not equal
>     // Greater than
<     // Less than
>=    // Greater or equal
<=    // Less or equal

Logical Operators

&&    // AND
||    // OR
!     // NOT

String Functions

.startsWith("prefix")      // Check string prefix
.endsWith("suffix")        // Check string suffix
.contains("substring")     // Check substring
.matches("regex")          // Regex match

Collections

"value" in ["item1", "item2", "item3"]    // Check membership

Expression Examples

Simple Conditions

// Route based on header value
headers["x-tier"] == "premium"

// Route based on team
team_name == "research"

// Route based on model
model == "gpt-4o"

// Route based on request type
request_type == "embedding"

// Route to fallback when budget high
budget_used > 80

Complex Conditions (Multiple Criteria)

// Premium tier research team
headers["x-tier"] == "premium" && team_name == "research"

// High capacity or premium
budget_used > 90 || headers["x-priority"] == "high"

// Specific team and model
team_name == "ml-ops" && model.startsWith("claude-")

// Region-based with capacity check
headers["x-region"] == "us-east" && tokens_used < 75

// Route embeddings to cheaper provider
request_type == "embedding" && budget_used > 50

Pattern Matching

// Match models starting with prefix
model.startsWith("gpt-4")

// Match custom headers
headers["x-environment"] in ["staging", "testing"]

// Email domain matching
headers["x-user-email"].contains("@company.com")

// Regex patterns
headers["x-app-version"].matches("[0-9]+\\.[0-9]+\\.[0-9]+")

Validation & Error Handling

Invalid CEL syntax → Rule logs warning, skipped, evaluation continues
Missing header/parameter → Expression returns false (graceful no-match)
Type mismatches → Logged as warning, rule skipped
Empty expression → Rule always matches (use true/false for explicit behavior)

Configuration

Manage routing rules from the dashboard:

Open Routing Rules in the dashboard.

From the list you can:
- See all rules with scope, priority, and enabled status
- Filter by scope or scope ID
- Create, edit, and delete rules
- View rule expressions and targets
- Enable or disable rules without deleting them
- Drag to reorder priority
Click Create rule (or edit an existing rule) to open the rule sheet and fill in the fields:
- Name (required): Unique rule identifier
- Description (optional): Internal notes
- Enabled: Toggle rule on/off
- CEL Expression: Visual or manual expression builder
- Targets (required): One or more weighted routing targets - each has Provider (optional), Model (optional), API Key (optional, requires Provider to be set), and Weight (%). Weights must sum to 1. When multiple targets are defined, one is selected probabilistically at request time.
- Fallbacks (optional): Array of fallback providers
- Scope: Where rule applies (global, customer, team, virtual_key)
- Scope ID: Required if scope is not global
- Priority: Lower = evaluated first (default: 0)
Save the rule. It takes effect immediately for matching requests.

Visual CEL Builder

The dashboard includes a visual query builder for CEL expressions:

Condition Builder: Select field, operator, value
Logical Operators: Combine conditions with AND/OR
Manual Mode: Switch to edit CEL directly
Validation: Real-time syntax validation
Conversion: Auto-converts visual rules to CEL

Real-World Use Cases

When to use Routing Rules:

Dynamic routing based on request headers or parameters
Capacity-based routing (route to fallback when budget/rate limit is high)
Organization-based routing (different rules for different teams/customers)
A/B testing or canary deployments
Conditional provider override based on complex logic

Use Case 1: Tier-Based Routing

Route requests based on customer tier using headers:

{
  "name": "Premium Tier Fast Track",
  "cel_expression": "headers[\"x-tier\"] == \"premium\"",
  "targets": [
    { "provider": "openai", "model": "gpt-4o", "weight": 1 }
  ],
  "fallbacks": ["azure/gpt-4o"],
  "scope": "global",
  "priority": 10
}

Use Case 2: Capacity-Based Failover

Route to cheaper provider when budget is exhausted:

{
  "name": "Budget Exhaustion Fallback",
  "cel_expression": "budget_used > 90",
  "targets": [
    { "provider": "groq", "model": "llama-2-70b", "weight": 1 }
  ],
  "fallbacks": [],
  "scope": "global",
  "priority": 5
}

Use Case 3: Team-Specific Routing

Route team-specific requests to their preferred provider:

{
  "name": "ML Team Anthropic Preference",
  "cel_expression": "team_name == \"ml-research\"",
  "targets": [
    { "provider": "anthropic", "model": "claude-3-opus-20240229", "weight": 1 }
  ],
  "fallbacks": ["bedrock/claude-3-opus"],
  "scope": "team",
  "scope_id": "team-ml-research-uuid",
  "priority": 0
}

Use Case 4: Complex Multi-Condition Routing

Combine multiple criteria for sophisticated routing:

{
  "name": "Production Premium Route",
  "cel_expression": "headers[\"x-environment\"] == \"production\" && headers[\"x-priority\"] == \"high\" && tokens_used < 75",
  "targets": [
    { "provider": "openai", "model": "gpt-4o", "weight": 1 }
  ],
  "fallbacks": ["azure/gpt-4o"],
  "scope": "global",
  "priority": 5
}

Use Case 5: Probabilistic A/B Testing

Split traffic across providers or models by weight for canary deployments or cost optimization:

{
  "name": "Split Traffic OpenAI vs Groq",
  "cel_expression": "true",
  "targets": [
    { "provider": "openai", "model": "gpt-4o",        "weight": 0.7 },
    { "provider": "groq",   "model": "llama-3.1-70b", "weight": 0.3 }
  ],
  "scope": "global",
  "priority": 15
}

Each request matching this rule has a 70% chance of going to OpenAI and a 30% chance of going to Groq. Weights must always sum to 1.

Use Case 6: Regional Routing

Route based on region headers:

{
  "name": "EU Data Residency",
  "cel_expression": "headers[\"x-region\"] == \"eu\"",
  "targets": [
    { "provider": "azure", "model": "gpt-4o", "weight": 1 }
  ],
  "fallbacks": [],
  "scope": "global",
  "priority": 0
}

How Rules Combine With Governance & Load Balancing

A matching routing rule takes precedence over a Virtual Key’s governance provider_configs. Use this to predict where a request goes:

A rule matches → its selected target sets the provider/model/fallbacks (and pins the key if you set key_id); the Virtual Key’s provider_configs are ignored. Adaptive load balancing still picks the best key unless you pinned one.
No rule matches → governance routing selects the provider/model by weighted random, then adaptive load balancing picks the best key.

Example: governance is configured 70% Azure / 30% OpenAI, and you also have the rule budget_used > 85 → groq. A request arriving at 90% budget usage routes to Groq (the rule wins); the governance split is ignored for that request.

Fallbacks

Add a fallbacks array to a rule to define the failover order if the chosen provider fails:

{
  "provider": "openai",
  "fallbacks": ["azure/gpt-4o", "groq/gpt-3.5-turbo"]
}

If OpenAI fails, the request retries Azure, then Groq.

Tuning Priority and Scope

Routing-rule evaluation is fast, but a few habits keep your rule set predictable and maintainable:

Order rules by likelihood: give frequently matching rules a lower priority number so they’re checked first.
Use the narrowest scope that fits: prefer Team or Virtual Key scope for targeted overrides; reserve Global scope for organization-wide defaults.
Prefer exact comparisons: use == and in [...] over .matches() regex where you can - they’re easier to reason about and less error-prone.
Keep expressions simple: split complex conditions into separate rules when it improves clarity.
Leave gaps in priority numbers (0, 10, 20) so you can insert rules later without renumbering.

Best Practices

Rule Naming

✅ Good names:

“Premium Tier Fast Track”
“Budget Exhaustion Fallback”
“ML Team Anthropic Route”
“Production High Priority Route”

❌ Bad names:

“Rule 1”
“Fix”
“Temp”
“TODO”

CEL Expression Safety

✅ Safe patterns:

headers["x-tier"] == "premium"                    // Exact match
headers["x-region"] in ["us", "eu", "asia"]      // Membership
team_name.startsWith("prod-")                    // Prefix check
budget_used > 80                                 // Numeric comparison

❌ Risky patterns:

headers["x-tier"].matches(".*premium.*")         // Complex regex
headers["x-config"].contains("json")             // Fragile
model.length() > 5 && ...                        // Undocumented behavior

Scope Management

✅ Good scope design:

Global rules for organization-wide policies
Customer scope for compliance (EU, data residency)
Team scope for team preferences
Virtual Key scope for specific integrations

❌ Avoid:

Too many virtual key-level rules (maintenance nightmare)
Conflicting rules across scopes
Rules that duplicate governance routing

Testing & Validation

✅ Validate before deployment:

Test CEL expression with expected headers
Verify provider/model exist in your setup
Check fallbacks are valid providers
Confirm scope_id matches actual entity
Test with from_memory=true to verify in-memory state

❌ Don’t:

Deploy rules without testing
Use nonexistent providers
Create circular fallback chains

Monitoring

✅ Track rule usage:

Monitor routing decisions by scope on the dashboard
Alert on unexpected provider selection patterns
Review priority order occasionally

❌ Don’t forget:

Disabling unused rules (instead of deleting)
Updating documentation when rules change
Testing failover chains

Troubleshooting

Rule Not Matching

Symptom: Rule expression is correct but doesn’t match

Diagnosis:

Check if the rule is enabled in the dashboard
Verify scope matches (check the Virtual Key’s team/customer hierarchy)
Check rule priority vs other rules in scope (lower priority evaluates first)
Verify variable values: confirm the headers/parameters your expression references are actually present on the request

Solutions:

Open the rule in the dashboard and confirm it is enabled, its scope applies, and its expression references variables that are present on the request.
Remember header keys are matched case-insensitively but should be written lowercase in CEL.

Expression Compilation Error

Symptom: “Failed to compile rule: invalid CEL syntax”

Common causes:

Unclosed quotes: headers["x-tier (missing closing quote)
Invalid operators: headers["x"] ?? (not standard CEL)
String escaping: headers["x-\type"] (incorrect escape)

Solutions:

Use the visual CEL builder to avoid syntax errors
Test expressions incrementally
Check CEL operator documentation above
Wrap complex expressions in parentheses: (A && B) || (C && D)

Wrong Provider Selected

Symptom: Request routed to unexpected provider

Diagnosis:

Multiple rules matching? (first-match-wins means earlier rules take precedence)
Governance routing already determined provider? (check scope hierarchy)
Load balancing changed key? (rule sets provider, LB sets key)

Solutions:

Lower the priority number of the rule you want to win
Verify scope precedence (VirtualKey > Team > Customer > Global)
Check whether another rule with a lower priority matches first
Review the rules listed in the dashboard and confirm enabled state, scope, and priority

Header/Parameter Not Found

Symptom: “no such key” error in CEL evaluation

This is normal! DeepIntShield treats missing headers as non-matches:

headers["x-optional"] == "value"  # Returns false if header missing

If you need to check if header exists:

headers["x-optional"] != ""  # True only if present and non-empty

Verifying Active Rules

To confirm which rules are live and in what order they will be evaluated, review the routing rules list in the dashboard and check each rule’s enabled state, scope, and priority.

Cross-check against your request: confirm the matching rule is enabled, its scope applies to the request’s Virtual Key/team/customer, and no higher-precedence (lower priority number, or more specific scope) rule matches first.

Additional Resources

Provider Routing

Understand how routing rules fit into the complete routing pipeline

Open →

Virtual Keys

Configure Virtual Keys that scope routing rules

Open →

Governance

Learn about the governance layer (applied after routing rules determine provider selection when no rule matches)

Open →

CEL Language Spec

Complete CEL expression language documentation

Open →