OpenTelemetry (OTel)

Overview

The OTel plugin enables seamless integration with OpenTelemetry Protocol (OTLP) collectors, allowing you to send LLM traces to your existing observability infrastructure. Connect DeepIntShield to platforms like Grafana Cloud, Datadog, New Relic, Honeycomb, or self-hosted collectors.

All traces follow OpenTelemetry semantic conventions, making it easy to correlate LLM operations with your broader application telemetry.

Supported Trace Formats

The plugin supports multiple trace formats to match your observability platform:

Format	Description	Use Case	Status
`genai_extension`	OpenTelemetry GenAI semantic conventions	Recommended - Standard OTel format with rich LLM metadata	✅ Released
`vercel`	Vercel AI SDK format	For Vercel AI SDK compatibility	🔄 Coming soon
`open_inference`	Arize OpenInference format	For Arize Phoenix and OpenInference tools	🔄 Coming soon

Configuration

Required Fields

Field	Type	Required	Description
`service_name`	`string`	❌ No	Service name to be used for tracing, defaults to `deepintshield`
`collector_url`	`string`	✅ Yes	OTLP collector endpoint URL
`trace_type`	`string`	✅ Yes	One of: `genai_extension`, `vercel`, `open_inference`
`protocol`	`string`	✅ Yes	Transport protocol: `http` or `grpc`
`headers`	`object`	❌ No	Custom headers for authentication (supports `env.VAR_NAME`)
`tls_ca_cert`	`string`	❌ No	File path to client CA certificate for TLS. Optional. Works with both gRPC and HTTP protocol

Environment Variable Substitution

Headers support environment variable substitution using the env. prefix:

{
  "headers": {
    "Authorization": "env.OTEL_API_KEY",
    "X-Custom-Header": "env.CUSTOM_VALUE"
  }
}

Resource Attributes

The plugin supports the standard OTEL_RESOURCE_ATTRIBUTES environment variable. Any attributes defined in this variable will be automatically attached to every span emitted by the plugin.

export OTEL_RESOURCE_ATTRIBUTES="deployment.environment=production,service.version=1.2.3,team.name=platform"

These attributes appear as resource-level metadata on all traces:

{
  "resource": {
    "attributes": {
      "service.name": "deepintshield",
      "deployment.environment": "production",
      "service.version": "1.2.3",
      "team.name": "platform"
    }
  }
}

This is useful for:

Environment identification - Distinguish between production, staging, and development traces
Service versioning - Track which version of your service generated the trace
Team attribution - Tag traces with team ownership for filtering and alerting
Custom metadata - Add any key-value pairs relevant to your observability needs

Setup

Otel UI setup

Open the dashboard and go to Settings > Plugins.
Add the OTel plugin and enable it.
Set the Collector URL, Trace Type (genai_extension), and Protocol (http or grpc).
Add any authentication Headers your collector requires.
Click Save. New requests start streaming traces to your collector immediately.

If you need to connect to an OTEL collector that requires TLS, set the CA certificate (tls_ca_cert) field so the plugin trusts your collector’s certificate.

Quick Start with Docker

Get started quickly with a complete observability stack using the included Docker Compose configuration:

services:
  otel-collector:
    image: otel/opentelemetry-collector-contrib:latest
    container_name: otel-collector
    command: ["--config=/etc/otelcol/config.yaml"]
    configs:
      - source: otel-collector-config
        target: /etc/otelcol/config.yaml
    ports:
      - "4317:4317"   # OTLP gRPC
      - "4318:4318"   # OTLP HTTP
      - "8888:8888"   # Collector /metrics
      - "9464:9464"   # Prometheus scrape endpoint
      - "13133:13133" # Health check
      - "1777:1777"   # pprof
      - "55679:55679" # zpages
    restart: unless-stopped
    depends_on:
      - tempo

  tempo:
    image: grafana/tempo:latest
    container_name: tempo
    command: [ "-config.file=/etc/tempo.yaml" ]
    configs:
      - source: tempo-config
        target: /etc/tempo.yaml
    ports:
      - "3200:3200"   # tempo HTTP API
    expose:
      - "4317"        # OTLP gRPC (internal)
    volumes:
      - tempo-data:/var/tempo
    restart: unless-stopped

  prometheus:
    image: prom/prometheus:latest
    container_name: prometheus
    depends_on:
      - otel-collector
    command:
      - "--config.file=/etc/prometheus/prometheus.yml"
      - "--storage.tsdb.path=/prometheus"
      - "--web.console.libraries=/usr/share/prometheus/console_libraries"
      - "--web.console.templates=/usr/share/prometheus/consoles"
      - "--web.enable-remote-write-receiver"
    ports:
      - "9090:9090"
    volumes:
      - prometheus-data:/prometheus
    configs:
      - source: prometheus-config
        target: /etc/prometheus/prometheus.yml
    restart: unless-stopped

  grafana:
    image: grafana/grafana:latest
    container_name: grafana
    depends_on:
      - prometheus
      - tempo
    environment:
      GF_SECURITY_ADMIN_USER: admin
      GF_SECURITY_ADMIN_PASSWORD: admin
      GF_AUTH_ANONYMOUS_ENABLED: "true"
      GF_AUTH_ANONYMOUS_ORG_ROLE: Viewer
      GF_PLUGINS_ALLOW_LOADING_UNSIGNED_PLUGINS: "grafana-pyroscope-app,grafana-exploretraces-app,grafana-metricsdrilldown-app"
      GF_PLUGINS_ENABLE_ALPHA: "true"
      GF_INSTALL_PLUGINS: ""
    ports:
      - "4000:3000"
    volumes:
      - grafana-data:/var/lib/grafana
    configs:
      - source: grafana-datasources
        target: /etc/grafana/provisioning/datasources/datasources.yml
    restart: unless-stopped

configs:
  otel-collector-config:
    content: |
      receivers:
        otlp:
          protocols:
            grpc:
              endpoint: 0.0.0.0:4317
            http:
              endpoint: 0.0.0.0:4318

      processors:
        batch:

      exporters:
        prometheus:
          endpoint: 0.0.0.0:9464
          namespace: otel
          const_labels:
            source: otelcol

        otlp/tempo:
          endpoint: tempo:4317
          tls:
            insecure: true

        debug:
          verbosity: detailed

      extensions:
        health_check:
          endpoint: 0.0.0.0:13133
        pprof:
          endpoint: 0.0.0.0:1777
        zpages:
          endpoint: 0.0.0.0:55679

      service:
        extensions: [health_check, pprof, zpages]
        telemetry:
          logs:
            level: debug
          metrics:
            level: detailed
        pipelines:
          traces:
            receivers: [otlp]
            processors: [batch]
            exporters: [debug, otlp/tempo]
          metrics:
            receivers: [otlp]
            processors: [batch]
            exporters: [debug, prometheus]
          logs:
            receivers: [otlp]
            processors: [batch]
            exporters: [debug]

  tempo-config:
    content: |
      server:
        http_listen_port: 3200
        log_level: info

      distributor:
        receivers:
          otlp:
            protocols:
              grpc:
                endpoint: 0.0.0.0:4317

      ingester:
        max_block_duration: 5m
        trace_idle_period: 10s

      compactor:
        compaction:
          block_retention: 1h

      storage:
        trace:
          backend: local
          wal:
            path: /var/tempo/wal
          local:
            path: /var/tempo/blocks

      metrics_generator:
        registry:
          external_labels:
            source: tempo
        storage:
          path: /var/tempo/generator/wal
          remote_write:
            - url: http://prometheus:9090/api/v1/write

  prometheus-config:
    content: |
      global:
        scrape_interval: 15s
      scrape_configs:
        - job_name: "otelcol-internal"
          static_configs:
            - targets: ["otel-collector:8888"]
        - job_name: "otelcol-exporter"
          static_configs:
            - targets: ["otel-collector:9464"]
        - job_name: "tempo"
          static_configs:
            - targets: ["tempo:3200"]

  grafana-datasources:
    content: |
      apiVersion: 1
      datasources:
        - name: Prometheus
          uid: prometheus
          type: prometheus
          access: proxy
          orgId: 1
          url: http://prometheus:9090
          isDefault: true
          editable: true
        - name: Tempo
          uid: tempo
          type: tempo
          access: proxy
          orgId: 1
          url: http://tempo:3200
          editable: true
          jsonData:
            tracesToMetrics:
              datasourceUid: prometheus
            nodeGraph:
              enabled: true

volumes:
  prometheus-data:
  grafana-data:
  tempo-data:

This launches:

OTel Collector - Receives traces on ports 4317 (gRPC) and 4318 (HTTP)
Tempo - Distributed tracing backend
Prometheus - Metrics collection
Grafana - Visualization dashboard

Access Grafana at http://<your-grafana-host>:3000 (default credentials: admin/admin)

Popular Platform Integrations

In the OTel plugin settings, set:

Field	Value
Collector URL	`https://otlp-gateway-prod-us-central-0.grafana.net/otlp`
Trace Type	`genai_extension`
Protocol	`http`
Headers	`Authorization`: `env.GRAFANA_CLOUD_API_KEY`

Set the environment variable:

export GRAFANA_CLOUD_API_KEY="Basic <your-base64-encoded-token>"

In the OTel plugin settings, set:

Field	Value
Collector URL	`https://trace.agent.datadoghq.com`
Trace Type	`genai_extension`
Protocol	`http`
Headers	`DD-API-KEY`: `env.DATADOG_API_KEY`

Set the environment variable:

export DATADOG_API_KEY="your-datadog-api-key"

In the OTel plugin settings, set:

Field	Value
Collector URL	`https://otlp.nr-data.net:4318`
Trace Type	`genai_extension`
Protocol	`http`
Headers	`api-key`: `env.NEW_RELIC_LICENSE_KEY`

Set the environment variable:

export NEW_RELIC_LICENSE_KEY="your-license-key"

In the OTel plugin settings, set:

Field	Value
Collector URL	`https://api.honeycomb.io`
Trace Type	`genai_extension`
Protocol	`http`
Headers	`x-honeycomb-team`: `env.HONEYCOMB_API_KEY`, `x-honeycomb-dataset`: `deepintshield-traces`

Set the environment variable:

export HONEYCOMB_API_KEY="your-api-key"

Langfuse is an open-source LLM observability platform that accepts OpenTelemetry traces via its OTLP endpoint.

Configure the OTel plugin with the following settings:

Field	Value
Collector URL	`https://cloud.langfuse.com/api/public/otel` (EU) or `https://us.cloud.langfuse.com/api/public/otel` (US)
Trace Type	`genai_extension`
Protocol	`http` (required - Langfuse does not support gRPC)
Headers	`Authorization`: `env.LANGFUSE_AUTH`

Set up the environment variable with your Langfuse API keys:

# Generate base64 auth string from your Langfuse API keys
export LANGFUSE_AUTH="Basic $(echo -n 'pk-lf-xxx:sk-lf-xxx' | base64)"

Replace pk-lf-xxx and sk-lf-xxx with your Langfuse public and secret keys from your project settings.

See the Langfuse OpenTelemetry documentation for more details.

Use the included Docker Compose stack or point to your own collector. In the OTel plugin settings, set:

Field	Value
Collector URL	`http://your-collector:4318`
Trace Type	`genai_extension`
Protocol	`http`

Captured Data

Each trace includes comprehensive LLM operation metadata following OpenTelemetry semantic conventions:

Span Attributes

Span Name: Based on request type (gen_ai.chat, gen_ai.text, gen_ai.embedding, etc.)
Service Info: service.name=deepintshield, service.version
Provider & Model: gen_ai.provider.name, gen_ai.request.model

Request Parameters

Temperature, max_tokens, top_p, stop sequences
Presence/frequency penalties
Tool configurations and parallel tool calls
Custom parameters via ExtraParams

Input/Output Data

Complete chat history with role-based messages
Prompt text for completions
Response content with role attribution
Tool calls and results

Performance Metrics

Token usage (prompt, completion, total)
Cost calculations in dollars
Latency and timing (start/end timestamps)
Error details with status codes

Example Span

{
  "name": "gen_ai.chat",
  "attributes": {
    "gen_ai.provider.name": "openai",
    "gen_ai.request.model": "gpt-4",
    "gen_ai.request.temperature": 0.7,
    "gen_ai.request.max_tokens": 1000,
    "gen_ai.usage.prompt_tokens": 45,
    "gen_ai.usage.completion_tokens": 128,
    "gen_ai.usage.total_tokens": 173,
    "gen_ai.usage.cost": 0.0052
  }
}

Supported Request Types

The OTel plugin captures all DeepIntShield request types:

Chat Completion (streaming and non-streaming) → gen_ai.chat
Text Completion (streaming and non-streaming) → gen_ai.text
Embeddings → gen_ai.embedding
Speech Generation (streaming and non-streaming) → gen_ai.speech
Transcription (streaming and non-streaming) → gen_ai.transcription
Responses API → gen_ai.responses

Protocol Support

HTTP (OTLP/HTTP)

Uses HTTP/1.1 or HTTP/2 with JSON or Protobuf encoding:

{
  "collector_url": "http://<your-otel-collector>:4318",
  "protocol": "http"
}

Default port: 4318

gRPC (OTLP/gRPC)

Uses gRPC with Protobuf encoding for lower latency:

{
  "collector_url": "<your-otel-collector>:4317",
  "protocol": "grpc"
}

Default port: 4317

Metrics Push (Cluster Mode)

The OTel plugin supports push-based metrics export via OTLP, which is essential for multi-node cluster deployments. Instead of relying on Prometheus scraping each node’s /metrics endpoint (which can miss nodes behind a load balancer), all nodes actively push metrics to a central OTEL Collector.

Configuration

Field	Type	Required	Description
`metrics_enabled`	`boolean`	❌ No	Enable push-based metrics export (default: `false`)
`metrics_endpoint`	`string`	✅ Yes (if enabled)	OTLP metrics endpoint URL
`metrics_push_interval`	`integer`	❌ No	Push interval in seconds (default: `15`, range: 1-300)

In the OTel plugin settings, set:

Field	Value
Collector URL	`http://otel-collector:4318/v1/traces`
Trace Type	`genai_extension`
Protocol	`http`
Metrics Enabled	`true`
Metrics Endpoint	`http://otel-collector:4318/v1/metrics`
Metrics Push Interval	`15`

In the OTel plugin settings, set:

Field	Value
Collector URL	`otel-collector:4317`
Trace Type	`genai_extension`
Protocol	`grpc`
Metrics Enabled	`true`
Metrics Endpoint	`otel-collector:4317`
Metrics Push Interval	`15`

Pushed Metrics

These are the same Prometheus-style metrics from the telemetry plugin, pushed via OTLP protocol to a central collector:

Metric	Type	Description
`deepintshield_upstream_requests_total`	Counter	Total requests to upstream providers
`deepintshield_success_requests_total`	Counter	Successful upstream requests
`deepintshield_error_requests_total`	Counter	Error requests with reason labels
`deepintshield_input_tokens_total`	Counter	Total input tokens
`deepintshield_output_tokens_total`	Counter	Total output tokens
`deepintshield_cache_hits_total`	Counter	Cache hits
`deepintshield_cost_total`	Counter	Total cost in USD
`deepintshield_upstream_latency_seconds`	Histogram	Upstream request latency
`deepintshield_stream_first_token_latency_seconds`	Histogram	Time to first token
`deepintshield_stream_inter_token_latency_seconds`	Histogram	Inter-token latency
`http_requests_total`	Counter	Total HTTP requests
`http_request_duration_seconds`	Histogram	HTTP request duration

OTEL Collector Configuration

Configure your OTEL Collector to receive OTLP metrics and export to your preferred backend (Datadog, Prometheus, etc.):

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:
    timeout: 10s
    send_batch_size: 1000

exporters:
  # For Datadog
  datadog:
    api:
      key: ${DD_API_KEY}

  # Or for Prometheus remote write
  prometheusremotewrite:
    endpoint: "http://prometheus:9090/api/v1/write"

service:
  pipelines:
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [datadog]  # or prometheusremotewrite

Why Push vs Pull?

Aspect	Pull (`/metrics` scrape)	Push (OTEL metrics)
Load balancer	May miss nodes	All nodes push
Service discovery	Required	Not required
Scraper configuration	Per-node endpoints	Single collector
Cluster aggregation	Query-side `sum()`	Collector handles it

For single-node deployments, pull-based /metrics scraping works well. For multi-node clusters, push-based metrics ensures all nodes are captured.

Good to Know

No latency impact: Tracing runs off the request path and adds no measurable latency to your LLM calls. You can leave it enabled in production.
Streaming requests: Streaming responses produce a single complete span when the stream finishes, with accurate token counts and costs - no extra configuration needed.
Keep credentials out of plain text: In the Headers field, reference secrets with the env. prefix (for example Authorization: env.OTEL_API_KEY) so they are read from the environment at runtime instead of being stored verbatim.

When to Use

OTel Plugin

Choose the OTel plugin when you:

Have existing OpenTelemetry infrastructure
Need to correlate LLM traces with application traces
Require compliance with enterprise observability standards
Want vendor flexibility (switch backends without code changes)
Need multi-service distributed tracing

vs. Built-in Observability

Use Built-in Observability for:

Local development and testing
Simple self-hosted deployments
No external dependencies
Direct database access to logs

Troubleshooting

Connection Issues

Verify collector is reachable:

# Test HTTP endpoint
curl -v http://<your-otel-collector>:4318/v1/traces

# Test gRPC endpoint (requires grpcurl)
grpcurl -plaintext <your-otel-collector>:4317 list

Missing Traces

Check DeepIntShield logs for emission errors:

# Enable debug logging
deepintshield-http --log-level debug

Authentication Failures

Verify environment variables are set:

echo $OTEL_API_KEY

Next Steps

Built-in Observability - Local logging for development
Telemetry - Prometheus metrics and dashboards