Collecting Datadog APM traces using Grafana Alloy and Tempo

The Problem

Hybrid cloud is the future, but monitoring remains stuck in the past. Many organizations embrace hybrid infrastructure, yet struggle with fragmented observability tools. Why? Because monitoring providers still operate in silos.

One of the primary reasons hybrid monitoring isn’t as prevalent is the challenge of instrumentation. Many cloud providers offer their own monitoring solutions. Instrumentation libraries are often incompatible with one another, making cross-platform integration difficult.

The good news? With OpenTelemetry, Grafana, and Datadog, hybrid monitoring is becoming easier and more flexible. 🚀

The Solution

One promising development is the rise of open-source, vendor-neutral instrumentation frameworks like OpenTelemetry.

In essence, open standards are reducing incompatibility issues and allowing “a vendor-agnostic approach to get data from the sources you need to the observability service of your choice.”

A step toward solving the hybrid cloud monitoring challenge came when Grafana introduced the otelcol.receiver.datadog component.

Now, with otelcol.receiver.datadog, Grafana users can ingest and process Datadog telemetry directly within OpenTelemetry pipelines, unlocking several advantages:

  1. Expanding Grafana’s Reach to Datadog Customers
  2. Seamless Integration with OpenTelemetry Pipelines
  3. Avoiding Vendor Lock-in While Retaining Datadog’s Strengths
  4. Cost Optimization by Centralizing Hybrid Monitoring

How it works together?

Requirements

  1. Grafana – for web UI
  2. Grafana Alloy – to receive, process and export telemetry data
  3. Grafana Tempo – to collect and visualize traces from Datadog instrumented apps
  4. Datadog agent – with enabled APM feature
  5. Instrumented application with Datadog trace library

Quick check list:

  1. you have running: Grafana, Alloy and Tempo services
  2. You have running Datadog agents
  3. You have instrumented applications with Datadog trace library
  4. You add Tempo as Datasource in Grafana

Grafana Alloy configuration

The core of the solution is to set up Datadog receiver in Alloy config:

receivers:
datadog:
endpoint: "0.0.0.0:9126"
output:
traces: [otelcol.exporter.otlp.tempo.input]

# I am skipping extra steps which you might want to use to pre-proces data

otelcol:
exporter:
otlp:
tempo:
client:
endpoint: "https://tempo-distributor.example.com:443"
tls:
insecure: false
insecure_skip_verify: true

To avoid ports conflict with Datadog agent we choose 9126 as a port for Alloy Datadog receiver.

Running Alloy as Daemonset with hostNetwork access allow agent to be present on each Node.

alloy:
extraPorts:
- name: "datadog"
port: 9126
targetPort: 9126
protocol: "TCP"

controller:
hostNetwork: true
hostPID: true

service:
internalTrafficPolicy: "Local"

Application set up

Application has to be configured to send APM traces to Node IP:1926.

The Node IP can be extracted form Kubernetes meta information and passed as environment variable:

env:
- name: NODE_IP
valueFrom:
fieldRef:
fieldPath: status.hostIP

Datadog Agent configuration

With Datadog Agent we have two options:

  1. Send traces to Alloy Datadog receiver as addition to Datadog host
  2. Send traces only to Alloy Datadog receiver

With option 1 we assume you are using both Datadog and Grafana solutions in hybrid mode. In that case Datadog agent has to have following configuration:

agents:
containers:
agent:
env:
- name: DD_ADDITIONAL_ENDPOINTS
value: '{"http://alloy-service:9126": ["datadog-receiver"]}'
- name: DD_APM_ADDITIONAL_ENDPOINTS
value: '{"http://alloy-service:9126": ["datadog-receiver"]}'
traceAgent:
enabled: true
env:
- name: DD_ADDITIONAL_ENDPOINTS
value: '{"http://alloy-service:9126": ["datadog-receiver"]}'
- name: DD_APM_ADDITIONAL_ENDPOINTS
value: '{"http://alloy-service:9126": ["datadog-receiver"]}'

For option 2 where traces must go only to Alloy Datadog receiver following configuration might work:

datadog:
apm:
socketEnabled: true # Use the Unix Domain Socket (default). Can be true even if port is enabled.
portEnabled: true # Enable TCP port 8126 for traces.
useLocalService: false

env:
- name: DD_APM_DD_URL
value: "http://alloy-service:9126" # URL for OTel collector’s Datadog trace receiver
- name: DD_APM_NON_LOCAL_TRAFFIC
value: "true"

After all done and agents restarted you can navigate to Explore page in Grafana, select Tempo as datasource and get recent Datadog traces.

Conclusions

  1. Kudos to Grafana and Datadog – Their collaboration on the otelcol.receiver.datadog makes transitioning between monitoring platforms smoother than ever, reducing friction for hybrid observability.
  2. Hybrid Monitoring is the New Normal – Applications no longer rely on a single monitoring provider throughout their lifespan. As infrastructure evolves, businesses will inevitably switch or integrate multiple observability tools.
  3. Stay Agile with Open Standards – Using OpenTelemetry ensures flexibility, allowing teams to adapt their monitoring stack without vendor lock-in, keeping observability seamless across hybrid and multi-cloud environments.

By embracing open standards, organizations can future-proof their monitoring strategies while ensuring complete visibility across their hybrid infrastructure. 🚀

Grafana a new look

As much as l like managed monitoring solutions it’s hard not to be excited about setting up your own platform based on Prometheus Grafana stack and save some bucks for the company.

After 5 years of using proprietary solutions I am again on the way to get into Grafana world.

Here I want to wrap up what I learnt so far about Grafana.

Big ecosystem of tools

The biggest change for me is ecosystem of projects which supports almost all imaginable monitoring needs.

  • For Logs: Loki and Promtail
  • For Traces: Tempo
  • For Profiling: Peryscope
  • For open telemetry collecting: Alloy
  • For eBPF: Beyla
  • For Synthetics: Prometheus BlackBox and K6 API
  • and many more

Automated Deployment process

A lot was done in the deployment process where using Helm charts you can get it up and running in a minutes.

All that comes with scalability and high availability in mind.

Though you would need to connect and customize each component, it’s still can be just a Helm configuration change.

Huge ecosystem of Components to simplify instrumentation

It’s clear your code has to be instrumented to be observable. Grafana and their Open Source community developed a number of tools to make it easy and simple:

  1. Compatible receivers for big providers like Datadog
  2. New discovery methods like eBPF
  3. No code way of instrumentation like sidecar container

And a lot more see the Alloy components section to get started.

New fresh UI

Grafana Logs and Metrics get new look.

You can see multiple logs and metrics on the same page.

Summary

It’s cool, it’s fresh and it’s a lot of fun to use!