Datadog, a leading monitoring and analytics platform, helps teams implement monitoring for various aspects of their organization whether it is cloud infrastructure, application performance monitoring, or general incident management. Leveraging infrastructure as code (IaC) practices with tools like Terraform or OpenTofu makes the management of Datadog easier to scale.
The Datadog Terraform Provider seamlessly integrates Datadog's monitoring capabilities into your infrastructure provisioning process, allowing you to define and manage monitoring resources alongside your infrastructure code. In this blog, we'll explore how to leverage the Datadog Terraform Provider to streamline your monitoring setup as well as the integrations that Scalr has with Datadog to assist in monitoring your Terraform operations.
Before diving into the implementation details, let's briefly highlight the benefits of using the Datadog Terraform Provider:
To start using the Datadog Terraform or OpenTofu Provider, follow these steps:
Start by configuring the Datadog Terraform provider in your Terraform code. Open your Terraform configuration file (commonly named main.tf) and add the following block:
terraform {
required_providers {
datadog = {
source = "datadog/datadog"
}
}
}
provider "datadog" {
api_key = "your-datadog-api-key"
app_key = "your-datadog-app-key"
}
Replace "your-datadog-api-key" with your Datadog API key and "your-datadog-app-key" with the application keys you generated earlier. The latest version of the official Datadog Terraform documentation can be found in the registry here.
Here's a simple example demonstrating how to create a basic monitor using the Datadog Terraform Provider:
resource "datadog_monitor" "example" {
name = "Scalr-Example"
type = "metric alert"
message = "Alert! Please check the monitor"
query = "sum(last_1h):sum:system.load.5{host:host0} > 2.0""
monitor_thresholds {
warning = 2
critical = 4
}
}
In this example, we define a monitor that triggers when the 5-minute load average on a host named "host0" exceeds 2.0.
Here's a simple example demonstrating how to set downtime in Datadog using the Datadog Terraform Provider:
resource "datadog_downtime" "example" {
scope = ["*"]
start = 1363105000
end = 1363234500
recurrence {
type = "days"
period = 1
}
}
In this example, we’ll use the datadog_monitor datasource to pull details about the monitor that was created in a previous step:
data "datadog_monitor" "example" {
name_filter = "Scalr-Example"
}
This will return details about the monitor such as the message that the monitor has set as well as the thresholds.
Once you have created your Terraform or Tofu code for the Datadog resources or datasources, you can then execute it with the following commands:
terraform init
terraform plan
terraform apply
tofu init
tofu plan
tofu apply
Terraform or Tofu will initialize the Datadog provider and apply the changes to your Datadog environment. Upon a successful Terraform apply, the state file will be created.
Utilize Terraform variables to make your configurations more dynamic. Instead of hardcoding values, use variables to create reusable and flexible scripts.
provider "datadog" {
api_key = "var.api-key"
app_key = "var.app-key"
}
These variables can be defined as environment variables to make it more dynamic and avoid having secrets in the code.
For example, you can specify the following before running the Terraform init to specific the Datadog API key:
export TF_VAR_api-key=<your-api-key>
Consider using remote state management to store your Terraform state files securely. Services like Scalr or AWS S3 can be configured as remote backends to store state files. Here is an example of connecting to Scalr:
terraform {
backend "remote" {
hostname = "<account-name>.scalr.io"
organization = "<scalr-environment-name>"
workspaces {
name = "<workspace-name>"
}
}
}
Find out more about remote state management here.
This guide covered the basic steps to configure the Terraform Datadog provider. As you explore further, consider exploring additional Datadog resources supported by the Datadog provider, such as roles, logs, and various integrations with clouds like AWS. The library.tf documentation for the Terraform Datadog provider is a valuable resource for in-depth information and examples.
By integrating Datadog into your Terraform workflows, you're not just managing infrastructure – you're managing monitoring with the efficiency and scalability that infrastructure as code brings.
Scalr, a Terraform Automation and Collaboration platform, provides a best-in-class integration with Datadog. Scalr is featured in the Datadog catalog, making it easier than ever to integrate the two products. Below, we'll explore the seamless integration process and benefits of integrating Scalr with Datadog.
The Scalr to Datadog integration for events can stream event details for Terraform and OpenTofu runs executed in Scalr. Datadog users can build reports based on the source of the run, whether it was from the Terraform CLI, a VCS provider like Github, or manually executed through the UI. Users can also track the result of the run, the execution time, and much more.
The events are sent through the Datadog API integration, see the official documentation here.
Scalr will send metrics to Datadog for in-depth analysis and reporting such as queued runs, queue state, the number of environments, and workspace count. These metrics are visualized in their out-of-the-box dashboard to help correlate deployments with other infrastructure changes and to track trends within your pipeline. The metrics functionality is an agent-based integration, which means you must use the Datadog agent. See how to install the Datadog agent and enable the Scalr integration here.
Scalr sends all of its audit logs to Datadog for further analysis. Audit logs allow you to get insights into all actions taken, who performed the action, how it was done, and more. The audit log feature can use the same Datadog connection that is used for events or a new one can be created.
For example, you may want to know how and when a Terraform run was discarded in the Scalr pipeline. Scalr will send the following data to Datadog:
{
"id": "AgAAAY2x_1pxzx914QAAAAAAAAAYAAAAAEFZMnhfMXRKQUFDY1RCNUtZb3FoTlFBQQAAACQ12345656E4ZGIxZmYtNWE3MS00MTU3LWJlMTctODhmMTZhYTU5Nzhl",
"content": {
"timestamp": "2024-02-16T12:58:38.577Z",
"tags": [
"scalr-workspace-name:demo-ws",
"scalr-environment-name:cs-m",
"scalr-workspace:ws-v0o370ouv5kbjmk9h",
"scalr-user-email:[email protected]",
"scalr-action:discard-run",
"scalr-environment:org-sscctbkdgkdr123",
"source:scalr",
"datadog.submission_auth:private_api_key"
],
"host": "docs.scalr.io",
"service": "audit-log",
"attributes": {
"actor": {
"id": "user-stp8qjepev3a123",
"access-token": {
"type": "session",
"token": "...B6wifA"
},
"type": "user",
"email": "[email protected]"
},
"request": {
"ip-address": "69.206.111.123",
"action": "discard-run",
"id": "a3a473d725c730382558c25d52bd1234",
"source": "ui",
"user-agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36"
},
"hostname": "docs.scalr.io",
"service": "audit-log",
"outcome": {
"result": "SUCCESS",
"status-code": 202
},
"target": {
"display-name": "run-v0o370pcff0e53123",
"context": {
"environment": {
"display-name": "CS-M",
"id": "org-sscctbkdgkdrqg0"
},
"workspace": {
"display-name": "demo-ws",
"id": "ws-v0o370ouv5kbjmk9h"
},
"account": {
"display-name": "docs",
"id": "acc-sscctbisjkl3123"
}
},
"id": "run-v0o370pcff0e53",
"type": "runs"
},
"timestamp": "2024-02-16T12:58:38.577846"
}
}
}
Datadog provides many capabilities to take your Terraform or OpenTofu operations to the next level. Whether you are using the Datadog provider to manage Datadog itself or using the Scalr integration with Datadog to monitor your Terraform and Tofu operations, Datadog is at the forefront of helping users scale.