Speculative dispatch

Speculative dispatch fires the provider call in parallel with input-guard evaluation. On the allow path (>95% of typical traffic), the user sees max(guards, model) latency instead of guards + model.

For most workloads this is the single biggest latency win available.

Default behavior

Off (opt-in). This is a safety tradeoff - see below. Enable per-deployment when you’re comfortable with the semantics.

# Plugin config (preferred, live-reloadable):
{ "speculative_input_guards": true }

# Or env var:
DEEPINTSHIELD_GUARD_SPECULATIVE_INPUT_GUARDS=true

What changes

With this enabled, the provider call and input-guard checks run at the same time. The response is only released once the guard verdict is in.

• On the allow path: total latency is max(guards, model) instead of guards + model.
• On the deny path: the model’s response is discarded and the user sees the standard guardrail_blocked error. You pay for one wasted provider call - that’s the safety vs. latency tradeoff.

The two safety semantics

Read this before enabling

Speculative dispatch changes one behavior compared to strict sequential mode:

1. Deny verdicts - the model has already responded by the time the deny lands. DeepintShield discards the response and returns guardrail_blocked. The wasted provider call is the cost of the latency win.
2. Input redact verdicts - the model has already seen the unredacted input by the time the redact lands. Serving the model’s response would defeat the redactor, so DeepintShield promotes input-redact verdicts to block in speculative mode. If you rely heavily on input redaction (e.g. PII scrubbing on prompts), leave speculative dispatch off.

When to enable

• Latency-sensitive chat and copilot UIs where a 200–800ms reduction is visible.
• Workloads with low deny rates (typical: <5%) so the wasted-provider-call cost is small.
• Workloads that don’t rely on input redaction for safety.

When to leave it off

• Audit-heavy workloads where every guarded request must complete before the provider call even starts (some regulated environments).
• Workloads with high deny rates (>20%) where the wasted-call cost is real.
• Streaming responses - speculative dispatch is automatically disabled for streaming, since there’s no clean way to discard tokens after the first one hits the wire.