Category: Kubernetes

The Problem

Hybrid cloud is the future, but monitoring remains stuck in the past. Many organizations embrace hybrid infrastructure, yet struggle with fragmented observability tools. Why? Because monitoring providers still operate in silos.

One of the primary reasons hybrid monitoring isn’t as prevalent is the challenge of instrumentation. Many cloud providers offer their own monitoring solutions. Instrumentation libraries are often incompatible with one another, making cross-platform integration difficult.

The good news? With OpenTelemetry, Grafana, and Datadog, hybrid monitoring is becoming easier and more flexible. 🚀

The Solution

One promising development is the rise of open-source, vendor-neutral instrumentation frameworks like OpenTelemetry.

In essence, open standards are reducing incompatibility issues and allowing “a vendor-agnostic approach to get data from the sources you need to the observability service of your choice.”​

A step toward solving the hybrid cloud monitoring challenge came when Grafana introduced the otelcol.receiver.datadog component.

Now, with otelcol.receiver.datadog, Grafana users can ingest and process Datadog telemetry directly within OpenTelemetry pipelines, unlocking several advantages:

1. Expanding Grafana’s Reach to Datadog Customers
2. Seamless Integration with OpenTelemetry Pipelines
3. Avoiding Vendor Lock-in While Retaining Datadog’s Strengths
4. Cost Optimization by Centralizing Hybrid Monitoring

How it works together?

Requirements

1. Grafana – for web UI
2. Grafana Alloy – to receive, process and export telemetry data
3. Grafana Tempo – to collect and visualize traces from Datadog instrumented apps
4. Datadog agent – with enabled APM feature
5. Instrumented application with Datadog trace library

Quick check list:

1. you have running: Grafana, Alloy and Tempo services
2. You have running Datadog agents
3. You have instrumented applications with Datadog trace library
4. You add Tempo as Datasource in Grafana

Grafana Alloy configuration

The core of the solution is to set up Datadog receiver in Alloy config:

receivers:
  datadog:
    endpoint: "0.0.0.0:9126"
    output:
      traces: [otelcol.exporter.otlp.tempo.input]

# I am skipping extra steps which you might want to use to pre-proces data

otelcol:
  exporter:
    otlp:
      tempo:
        client:
          endpoint: "https://tempo-distributor.example.com:443"
          tls:
            insecure: false
            insecure_skip_verify: true

To avoid ports conflict with Datadog agent we choose 9126 as a port for Alloy Datadog receiver.

Running Alloy as Daemonset with hostNetwork access allow agent to be present on each Node.

alloy:
  extraPorts:
    - name: "datadog"
      port: 9126
      targetPort: 9126
      protocol: "TCP"

controller:
  hostNetwork: true
  hostPID: true

service:
  internalTrafficPolicy: "Local"

Application set up

Application has to be configured to send APM traces to Node IP:1926.

The Node IP can be extracted form Kubernetes meta information and passed as environment variable:

env:
- name: NODE_IP
  valueFrom:
    fieldRef:
      fieldPath: status.hostIP

Datadog Agent configuration

With Datadog Agent we have two options:

1. Send traces to Alloy Datadog receiver as addition to Datadog host
2. Send traces only to Alloy Datadog receiver

With option 1 we assume you are using both Datadog and Grafana solutions in hybrid mode. In that case Datadog agent has to have following configuration:

agents:
  containers:
    agent:
      env:
        - name: DD_ADDITIONAL_ENDPOINTS
          value: '{"http://alloy-service:9126": ["datadog-receiver"]}'
        - name: DD_APM_ADDITIONAL_ENDPOINTS
          value: '{"http://alloy-service:9126": ["datadog-receiver"]}'
    traceAgent:
      enabled: true
      env:
        - name: DD_ADDITIONAL_ENDPOINTS
          value: '{"http://alloy-service:9126": ["datadog-receiver"]}'
        - name: DD_APM_ADDITIONAL_ENDPOINTS
          value: '{"http://alloy-service:9126": ["datadog-receiver"]}'

For option 2 where traces must go only to Alloy Datadog receiver following configuration might work:

datadog:
  apm:
    socketEnabled: true                    # Use the Unix Domain Socket (default). Can be true even if port is enabled.
    portEnabled: true                      # Enable TCP port 8126 for traces.
    useLocalService: false 

  env:
    - name: DD_APM_DD_URL
      value: "http://alloy-service:9126"  # URL for OTel collector’s Datadog trace receiver
    - name: DD_APM_NON_LOCAL_TRAFFIC
      value: "true" 

After all done and agents restarted you can navigate to Explore page in Grafana, select Tempo as datasource and get recent Datadog traces.

Conclusions

1. Kudos to Grafana and Datadog – Their collaboration on the otelcol.receiver.datadog makes transitioning between monitoring platforms smoother than ever, reducing friction for hybrid observability.
2. Hybrid Monitoring is the New Normal – Applications no longer rely on a single monitoring provider throughout their lifespan. As infrastructure evolves, businesses will inevitably switch or integrate multiple observability tools.
3. Stay Agile with Open Standards – Using OpenTelemetry ensures flexibility, allowing teams to adapt their monitoring stack without vendor lock-in, keeping observability seamless across hybrid and multi-cloud environments.

By embracing open standards, organizations can future-proof their monitoring strategies while ensuring complete visibility across their hybrid infrastructure. 🚀