Log Exports

Log Exports

DeepIntShield Enterprise provides comprehensive log export capabilities, allowing you to automatically export request logs, traces, and telemetry data to various storage systems and data lakes on configurable schedules.

Log exports are managed entirely from the DeepIntShield Workspace under Logs → Exports. Each export is created as a job with a destination, a data configuration, and an optional schedule.

Overview

The log export system enables:

• Scheduled Exports: Daily, weekly, or monthly automated exports
• Multiple Destinations: Object stores, data warehouses, and data lakes
• Format Flexibility: JSON, CSV, and Parquet, with optional compression
• Filtering & Transformation: Export specific data subsets with custom transformations
• Compliance: Meet data retention and audit requirements

Creating an Export Job

From Logs → Exports in the DeepIntShield dashboard:

1. Click New Export and give the job a descriptive Name.
2. Choose a Destination and fill in its connection fields (see Supported Export Destinations).
3. Configure the Data to export - the format, compression, which datasets to include, and any filters or transformations.
4. (Optional) Set a Schedule to run the export automatically, or leave it off to run on demand.
5. Save the job, then use Run now to export immediately. Completed exports appear in the Exports list, where you can download the produced artifacts.

The sections below describe the destination, data, schedule, and transformation options you configure on this page.

Supported Export Destinations

Each destination is configured with a type and a set of credentials and location fields entered in the export form.

Object Storage

Amazon S3

Field	Example
Bucket	deepintshield-logs
Region	us-west-2
Prefix	logs/{year}/{month}/{day}/
Access Key ID	${AWS_ACCESS_KEY_ID}
Secret Access Key	${AWS_SECRET_ACCESS_KEY}

Google Cloud Storage

Field	Example
Bucket	deepintshield-logs
Prefix	logs/{year}/{month}/{day}/
Service Account Key	${GCP_SERVICE_ACCOUNT_KEY}

Azure Blob Storage

Field	Example
Container	deepintshield-logs
Account Name	${AZURE_ACCOUNT_NAME}
Account Key	${AZURE_ACCOUNT_KEY}
Prefix	logs/{year}/{month}/{day}/

Data Warehouses & Lakes

Snowflake

Field	Example
Account	your-account.snowflakecomputing.com
Database	DEEPINTSHIELD_LOGS
Schema	PUBLIC
Table	request_logs
Warehouse	COMPUTE_WH
Username	${SNOWFLAKE_USERNAME}
Password	${SNOWFLAKE_PASSWORD}

Amazon Redshift

Field	Example
Host	deepintshield-cluster.xxxxxx.us-west-2.redshift.amazonaws.com:5439
Database	deepintshield_logs
Schema	public
Table	request_logs
Username	${REDSHIFT_USERNAME}
Password	${REDSHIFT_PASSWORD}

Google BigQuery

Field	Example
Project ID	your-project-id
Dataset	deepintshield_logs
Table	request_logs
Service Account Key	${GCP_SERVICE_ACCOUNT_KEY}

Export Schedules

Enable a schedule on the export to run it automatically. Set the frequency, time, timezone, and (for weekly/monthly) the day. Leave the schedule off to run the export on demand.

Frequency	Settings
Daily	Time (e.g. 02:00) and timezone (e.g. UTC).
Weekly	Day of week (e.g. sunday), time, and timezone.
Monthly	Day of month (e.g. 1), time, and timezone.

Data Formats

Choose the output format for the export and, optionally, compression (e.g. gzip).

JSON Format

{
  "timestamp": "2024-01-15T10:30:00Z",
  "request_id": "req_123456789",
  "customer_id": "cust_abc123",
  "provider": "openai",
  "model": "gpt-4",
  "endpoint": "/v1/chat/completions",
  "method": "POST",
  "status_code": 200,
  "latency_ms": 1250,
  "input_tokens": 100,
  "output_tokens": 150,
  "cost_usd": 0.0045
}

CSV Format

timestamp,request_id,customer_id,provider,model,endpoint,method,status_code,latency_ms,input_tokens,output_tokens,cost_usd
2024-01-15T10:30:00Z,req_123456789,cust_abc123,openai,gpt-4,/v1/chat/completions,POST,200,1250,100,150,0.0045

Parquet Schema

message log_record {
  required int64 timestamp;
  required binary request_id (UTF8);
  required binary customer_id (UTF8);
  required binary provider (UTF8);
  required binary model (UTF8);
  required binary endpoint (UTF8);
  required binary method (UTF8);
  required int32 status_code;
  required int32 latency_ms;
  optional int32 input_tokens;
  optional int32 output_tokens;
  optional double cost_usd;
}

Data Filtering & Transformation

Narrow the exported data and reshape it directly in the export’s Data configuration.

Filtering Options

Restrict the export to a subset of logs by date range, provider, model, status code, customer, latency bounds, or error state:

{
  "filters": {
    "date_range": {
      "start": "2024-01-01T00:00:00Z",
      "end": "2024-01-31T23:59:59Z"
    },
    "providers": ["openai", "anthropic", "azure"],
    "models": ["gpt-4", "claude-3-sonnet"],
    "status_codes": [200, 201, 400, 401, 403, 404, 500],
    "customers": ["cust_123", "cust_456"],
    "min_latency_ms": 100,
    "max_latency_ms": 10000,
    "has_errors": true
  }
}

Transformation Options

Aggregate, anonymize, or enrich records before they leave DeepIntShield:

{
  "transformations": [
    {
      "type": "aggregate",
      "group_by": ["provider", "model", "date"],
      "metrics": [
        "count",
        "avg_latency",
        "p95_latency",
        "total_tokens",
        "total_cost",
        "error_rate"
      ]
    },
    {
      "type": "anonymize",
      "fields": ["customer_id", "request_id"],
      "method": "hash"
    },
    {
      "type": "enrich",
      "add_fields": {
        "export_timestamp": "${EXPORT_TIMESTAMP}",
        "export_version": "${EXPORT_VERSION}"
      }
    }
  ]
}