Skip to main content
Version: latest

GenAI Distributed Tracing

Envoy AI Gateway's router joins and records distributed traces when supplied with an OpenTelemetry collector endpoint.

This guide provides an overview of the spans recorded by the AI Gateway and how export them to your choice of OpenTelemetry collector.

Overview​

Envoy AI Gateway's router joins and records distributed traces when supplied with an OpenTelemetry collector endpoint.

Requests to the OpenAI Chat Completions and Embeddings endpoints are recorded as Spans which include typical timing and request details. In addition, there are GenAI attributes representing the LLM or Embeddings call including full request and response details, defined by OpenInference semantic conventions.

OpenInference attributes default to include full request and response data for both chat completions and embeddings. This can be toggled with configuration, but when enabled allows systems like Arize Phoenix to perform evaluations of production requests captured in OpenTelemetry spans.

For chat completions, this includes traditional LLM metrics such as correctness and hallucination detection. For embeddings, it enables agentic RAG evaluations focused on retrieval and semantic analysis.

Trying it out​

Before you begin, you'll need to complete the basic setup from the Basic Usage guide, which includes installing Envoy Gateway and AI Gateway.

Install Phoenix for LLM observability​

# Install Phoenix using PostgreSQL storage.
helm install phoenix oci://registry-1.docker.io/arizephoenix/phoenix-helm \
  --namespace envoy-ai-gateway-system \
  --set auth.enableAuth=false \
  --set server.port=6006

Configure AI Gateway with OpenTelemetry​

Upgrade your AI Gateway installation with OpenTelemetry configuration:

helm upgrade ai-eg oci://docker.io/envoyproxy/ai-gateway-helm \
  --version v0.0.0-latest \
  --namespace envoy-ai-gateway-system \
  --set "extProc.extraEnvVars[0].name=OTEL_EXPORTER_OTLP_ENDPOINT" \
  --set "extProc.extraEnvVars[0].value=http://phoenix-svc:6006" \
  --set "extProc.extraEnvVars[1].name=OTEL_METRICS_EXPORTER" \
  --set "extProc.extraEnvVars[1].value=none"
# OTEL_SERVICE_NAME defaults to "ai-gateway" if not set
# OTEL_METRICS_EXPORTER=none because Phoenix only supports traces, not metrics

Wait for the gateway pod to be ready:

kubectl wait --for=condition=Ready -n envoy-gateway-system \
  pods -l gateway.envoyproxy.io/owning-gateway-name=envoy-ai-gateway-basic

Generate traces​

Make requests to your AI Gateway to generate traces. Follow the instructions from Testing the Gateway in the Basic Usage guide to:

  1. Set up port forwarding (if needed)
  2. Test the chat completions endpoint

Each request will generate traces that are sent to Phoenix.

Check Phoenix is receiving traces​

kubectl logs -n envoy-ai-gateway-system deployment/phoenix | grep "POST /v1/traces"

Access Phoenix UI​

Port-forward to access the Phoenix dashboard:

kubectl port-forward -n envoy-ai-gateway-system svc/phoenix 6006:6006

Then open http://localhost:6006 in your browser to explore the traces.

Privacy Configuration​

Control sensitive data in traces by adding OpenInference configuration to Helm values when you reconfigure the AI Gateway. There is similar config for embeddings:

For example, if you are using a values.yaml file instead of command line arguments, you can add the following to control redaction:

extProc:
  extraEnvVars:
    # Base OTEL configuration...
    # Hide sensitive data (all default to false)
    - name: OPENINFERENCE_HIDE_INPUTS
      value: "true"  # Hide input messages to the LLM
    - name: OPENINFERENCE_HIDE_OUTPUTS
      value: "true"  # Hide output messages from the LLM
    # Reduce volume for embeddings (all default to false)
    - name: OPENINFERENCE_HIDE_EMBEDDINGS_TEXT
      value: "true"  # Hide embeddings input
    - name: OPENINFERENCE_HIDE_EMBEDDINGS_VECTORS
      value: "true"  # Hide embeddings output

Note: Hiding inputs/outputs prevents human or LLM-as-a-Judge evaluation of your LLM requests, such as done with the Phoenix Evals library.

Cleanup​

To remove Phoenix and disable tracing:

# Uninstall Phoenix
helm uninstall phoenix -n envoy-ai-gateway-system

# Disable tracing in AI Gateway
helm upgrade ai-eg oci://docker.io/envoyproxy/ai-gateway-helm \
  --version v0.0.0-latest \
  --namespace envoy-ai-gateway-system \
  --reuse-values \
  --unset extProc.extraEnvVars

See Also​