Documentation Grafana Cloud Monitor infrastructure Cloud Provider Observability AWS observability CloudWatch metrics Configure CloudWatch metrics scrape

Configure CloudWatch metrics scrape jobs

You can configure CloudWatch metrics scrape jobs with Terraform or using a combination of the Cloud Provider UI, and CloudFormation or the AWS IAM console. The connection and configuration process for CloudWatch metrics scrape jobs includes the following tasks:

• Navigate to Cloud provider AWS configuration
• Create an AWS IAM role
• Connect to AWS account
• Create a scrape job

Navigate to Cloud provider AWS configuration

1. Open your Grafana Cloud portal.
2. Expand Observability > Cloud Provider in the main menu of your Grafana Cloud stack.
3. Select AWS > Add your AWS services if you haven’t added any AWS services yet. If you have added a service, click the Configuration tab.

Create an AWS IAM role

To configure CloudWatch metrics scrape jobs, you must create an AWS IAM role to grant Grafana Cloud Access to read your CloudWatch metrics and associated resource metadata. You can choose to configure the AWS role either automatically or manually in the AWS Management Console. With automatic configuration, you can use either CloudFormation or Terraform. Choose the one that fits with your setup.

Automatically with CloudFormation

1. From the Configuration tab, click the AWS accounts tile.
2. At the Accounts page, click Add new account to open the Create new account page. For Create a new AWS role, Automatically > Use CloudFormation should be selected.
3. Click Launch stack.
4. Follow the steps to create the IAM role in AWS CloudFormation.
5. Return to the AWS accounts Create new account page in Grafana when you have finished in AWS.

Automatically with Terraform

Create an AWS IAM role so that Grafana can then assume a role that has access only to your CloudWatch data, with no need to share access and secret keys.

The input variables for the IAM role are:

• external_id: The username / instance ID for your Grafana Cloud Prometheus. AWS uses an external ID to provide an extra layer of security when giving Grafana access to pull your CloudWatch metrics into Grafana Cloud. Click Details in the Prometheus card of the Grafana Cloud Portal or use the grafana_cloud_stack data source to find this information.
• iam_role_name: A customizable name of the IAM role used by Grafana for the CloudWatch integration. The default value is GrafanaCloudWatchIntegration.

The output variable is role_arn, which is the AWS Identity and Access Management (IAM) role Amazon Resource Name (ARN) you need to use when you create the scrape job.

To create a new AWS role in Grafana Cloud using the AWS Command Line Interface (CLI):

1. Configure authentication for the AWS Terraform provider.

2. Copy the following code snippet into your Terraform file:

   terraform Copy

   terraform {
     required_providers {
       aws = {
         source  = "hashicorp/aws"
         version = "~> 3.0"
       }
     }
   }
   
   locals {
     grafana_account_id = "008923505280"
   }
   
   variable "external_id" {
     type        = string
     description = "This is your Grafana Cloud identifier and is used for security purposes."
     validation {
       condition     = length(var.external_id) > 0
       error_message = "ExternalID is required."
     }
   }
   
   variable "iam_role_name" {
     type        = string
     default     = "GrafanaLabsCloudWatchIntegration"
     description = "Customize the name of the IAM role used by Grafana for the CloudWatch integration."
   }
   
   data "aws_iam_policy_document" "trust_grafana" {
     statement {
       effect = "Allow"
       principals {
         type        = "AWS"
         identifiers = ["arn:aws:iam::${local.grafana_account_id}:root"]
       }
       actions = ["sts:AssumeRole"]
       condition {
         test     = "StringEquals"
         variable = "sts:ExternalId"
         values   = [var.external_id]
       }
     }
   }
   
   resource "aws_iam_role" "grafana_labs_cloudwatch_integration" {
     name        = var.iam_role_name
     description = "Role used by Grafana CloudWatch integration."
     # Allow Grafana Labs' AWS account to assume this role.
     assume_role_policy = data.aws_iam_policy_document.trust_grafana.json
   }
   
   resource "aws_iam_role_policy" "grafana_labs_cloudwatch_integration" {
     name = "GrafanaLabsCloudWatchIntegrationPolicy"
     role = aws_iam_role.grafana_labs_cloudwatch_integration.id
     # This policy allows the role to discover metrics via tags and export them.
     policy = jsonencode({
       Version = "2012-10-17"
       Statement = [
         {
           Effect = "Allow"
           Action = [
             "tag:GetResources",
             "cloudwatch:GetMetricData",
             "cloudwatch:ListMetrics",
             "apigateway:GET",
             "aps:ListWorkspaces",
             "autoscaling:DescribeAutoScalingGroups",
             "dms:DescribeReplicationInstances",
             "dms:DescribeReplicationTasks",
             "ec2:DescribeTransitGatewayAttachments",
             "ec2:DescribeSpotFleetRequests",
             "shield:ListProtections",
             "storagegateway:ListGateways",
             "storagegateway:ListTagsForResource"
           ]
           Resource = "*"
         }
       ]
     })
   }
   
   // Allow some time for IAM (global) changes to propagate
   resource "time_sleep" "wait_10_seconds" {
     depends_on = [
       aws_iam_role.grafana_labs_cloudwatch_integration,
       aws_iam_role_policy.grafana_labs_cloudwatch_integration
     ]
     create_duration = "10s"
   }
   
   output "role_arn" {
     depends_on = [
       time_sleep.wait_10_seconds
     ]
     value       = aws_iam_role.grafana_labs_cloudwatch_integration.arn
     description = "The ARN for the role created, copy this into Grafana Cloud installation."
   }

3. Run the terraform apply command, and either set variables directly in the CLI or create a tfvars file.

   • To set the variables directly in the CLI, use the following example command: terraform apply -var="external_id=<your external ID>" -var="iam_role_name=GrafanaCloudWatchIntegration"

   • To create a tfvars file (terraform.tfvars), add the following text to the Terraform file:

     terraform Copy

     external_id="<your external ID>"
     iam_role_name="GrafanaCloudWatchIntegration"

     Then run the following command:

     bash Copy

     terraform apply

4. After the Terraform apply command has finished creating the IAM Role, it outputs your role_arn, as in the following example:

   bash Copy

   role_arn = "arn:aws:iam::<yourAWSAccountID>:role/<iam_role_name>"

Manually with the AWS IAM console

Creating the AWS IAM role manually is a lengthy, tedious, error-prone process. We recommend using CloudFormation or Terraform to configure your IAM role to enable scraping CloudWatch metrics.

1. From the Configuration tab, click the AWS accounts tile.
2. At the AWS accounts page, click Add new account to open the Create new account page.
3. For Create a new AWS role, select Manually > Open AWS IAM Console.

Continue the following steps in the AWS IAM Console in a separate browser tab. Leave the Grafana Cloud browser tab open to find information needed for subsequent steps.

1. Click Roles > Create role in the AWS IAM Console.
2. Select AWS Account > Another AWS account.
3. For Account ID, enter the Grafana AWS Account ID found on the Create new account configuration page in Grafana Cloud.
4. Select Require external ID, and enter the Grafana Cloud/External ID found on the Create new account page in Grafana Cloud.
5. Click Next > Next.
6. Enter a Role name for this role and click Create role > View role.
7. Click Add Permissions > Create inline policy and select JSON on the policy editor toggle.
8. Delete the existing JSON and replace it with the JSON policy configuration found on the Create new account page in Grafana Cloud and click Next.
9. Enter a Policy name and click Create policy. Copy the ARN for your new role to use when you connect to AWS account in Grafana Cloud.

Connect to AWS account

After you have created the AWS IAM role, you need to connect your AWS account with your Grafana Cloud instance so Grafana Cloud can scrape metrics. You can connect your AWS account with Grafana Cloud either in the UI or using Terraform.

Connect to AWS account in UI

If you aren’t using Terraform to configure your scrape job, you can connect your AWS account in the UI using the following steps:

1. Enter the name of your account on the AWS Accounts Create new account page in Grafana Cloud, for the Account name(optional) box. Give your account a unique name that contains only alphanumeric characters, dashes, and underscores.
2. In the ARN box, paste the ARN you copied from the AWS IAM role you created.
3. Select the regions where you have services you want to monitor from the AWS Regions drop-down menu.
4. Click Add account to ensure the connection is working and to save your new account.
5. Click Create a scrape job to continue creating your configuration. The Create new scrape job configuration page opens with the preselected account.

Connect to AWS account using Terraform

If you are using Terraform to configure your CloudWatch metrics scrape job, you need to connect to AWS account by completing the tasks:

• Create a Grafana Cloud access policy
• Obtain the regional Cloud Provider API endpoint
• Set up the Grafana Cloud Terraform Provider

Create a Grafana Cloud access policy

To create an access policy for your organization in the Grafana Cloud portal, refer to the Create an access policy for a stack steps.

Add scopes to the Grafana Cloud access policy

In step 6 of the Create an access policy for a stack steps, use the Add scopes drop-down to add the following specific scopes to the access policy for Grafana to have the access it needs to integrate with your cloud provider:

• integration-management:read
• integration-management:write
• stacks:read

Create a Grafana Cloud access policy token

On your Grafana Cloud access policy, click Add token to generate a token to authenticate the provider with the Cloud Provider API. Give your token an appropriate name and select an Expiration date. We recommend you select a specific expiration date and do not set the Expiration date to No expiry, as this can create a security vulnerability.

Obtain the regional Cloud Provider API endpoint

You need the URL for the Cloud Provider API to communicate. To complete these steps, you need to have cURL and jq installed.

1. Copy and use the following script to return a list of all the Grafana Cloud stacks you own, along with their respective cloud provider API hostnames:

   bash Copy

    curl -sH "Authorization: Bearer <Access Token from previous step>" "http://grafana.com/api/instances" | \
    jq '[.items[]|{stackName: .slug, clusterName:.clusterSlug, cloudProviderAPIURL: "http://cloud-provider-api-\(.clusterSlug).grafana.net"}]'

2. Select the hostname for the stack you want to manage. In the following example, the correct hostname for the herokublogpost stack is http://cloud-provider-api-prod-us-central-0.grafana.net.

   json Copy

   [
     {
       "stackName": "herokublogpost",
       "clusterName": "prod-us-central-0",
       "cloudProviderAPIURL": "http://cloud-provider-api-prod-us-central-0.grafana.net"
     }
   ]

Set up the Grafana Terraform Provider

Set up the Grafana Terraform provider with either of these methods:

• Terraform commands
• Environmental variables

Configure the Grafana Terraform Provider

1. Include the provider as a dependency in your Terraform configuration, as in the following example:

   terraform Copy

    terraform {
      required_providers {
        grafana = {
          source  = "grafana/grafana"
          version = ">= 3.13.1" # minimum required version that includes Cloud Provider support
        }
      }
    }

2. Configure AWS support for the Grafana Terraform provider with the following snippet, which uses the access token and cloud provider API URL obtained in the previous steps.

   Copy

    provider "grafana" {
      // ...
      cloud_provider_url = <Cloud Provider API URL from previous step>
      cloud_provider_access_token = <Access Token from previous step>
    }

Environment variables

When running Terraform commands, set the cloud provider URL and access token in an empty Grafana provider block with environment variables, such as GRAFANA_CLOUD_PROVIDER_ACCESS_TOKEN and GRAFANA_CLOUD_PROVIDER_URL.

Copy

 provider "grafana" {}

Create a scrape job

After you have added an AWS account to Grafana Cloud, you can create a CloudWatch metrics scrape job to scrape metrics from that AWS account. You can create a CloudWatch metrics scrape job with Grafana Cloud either in the UI or using Terraform.

Create a scrape job in the UI

To get to the Create new scrape job configuration page, either click Create a scrape job after connecting your AWS account, or follow the steps to Navigate to Cloud provider AWS configuration, and click CloudWatch metrics scrape > Add new scrape job.

1. Select an AWS account. Either use the dropdown or click add a new one to add a different AWS account.
2. Enter a name for your scrape job. Give your scrape job a unique name that contains only alphanumeric characters, dashes, and underscores.
3. Include your AWS resource tags is selected by default. For more information, refer to Query tag data. Ensure your tags adhere to AWS best practices, such as not containing personally identifiable information or other confidential or sensitive information.

   Note

   Including tags increases the total number of active series, which can impact your Grafana Cloud costs.

4. Optionally, add static labels for easier filtering and grouping. These labels are added to all metrics exported by this scrape job.
5. Choose the services you want to scrape. You can search in the search box or browse in the list of services.
6. Optionally, enter a custom namespace in the Namespace name box and click Add.
7. Click Edit Metrics next to the service or namespace if you want to customize the metrics that are collected for that service or namespace. A default set of metrics is included for each service. For custom namespaces, you must enter the metrics.
   1. Select or deselect metrics in the Edit dialog box.
   2. For each metric selected, choose the statistics you want to include. You can also choose statistics to apply to all metrics you have selected. Refer to AWS documentation to determine which statistics are possible or best for each metric.
   3. Select the scrape interval.
   4. Add AWS tags you want to include. To pull metrics from Amazon CloudWatch into Grafana Cloud, you must have tagged resources in AWS. Optionally, you can narrow the metrics you pull by adding the tag you have placed on a metric in AWS, using the exact AWS tag format.
   5. Click Save service settings.
8. Click Create scrape job to begin collecting metrics or click Export as terraform to export the scrape settings in Terraform format.

Create a scrape job using Terraform

Grafana Terraform provider resources

You can define the following resources and data sources with the Grafana Terraform provider:

Scrape job example Terraform

The following example is for obtaining Amazon Elastic Compute Cloud (EC2) and Amazon Relational Database Service (RDS) metrics. Replace the values with your values.

terraform Copy

variable "access_token" {
  type      = string
  sensitive = true
}

variable "cloud_provider_url" {
  type        = string
  description = "The Grafana Cloud Provider API URL from the previous step"
}

variable "role_arn" {
  type        = string
  description = "The AWS role ARN output from the previous step"
}

variable "stack_slug" {
  type        = string
  description = "The Grafana Cloud Stack name"
}

terraform {
  required_providers {
    grafana = {
      source  = "grafana/grafana"
      version = ">= 3.13.1"
    }
  }
}

provider "grafana" {
  # this token is used for calling the grafana.com API, and easily fetching the Grafana instance Stack ID
  cloud_access_policy_token   = var.access_token
  cloud_provider_url          = var.cloud_provider_url
  cloud_provider_access_token = var.access_token
}

data "grafana_cloud_stack" "thestack" {
  slug     = var.stack_slug
}

resource "grafana_cloud_provider_aws_account" "myaccount" {
  stack_id = data.grafana_cloud_stack.thestack.id
  role_arn = var.role_arn
  regions = [
    "us-east-1",
    "us-east-2",
  ]
}

resource "grafana_cloud_provider_aws_cloudwatch_scrape_job" "myaccount-ec2" {
  stack_id                = data.grafana_cloud_stack.thestack.id
  name                    = "tf-managed-scrape-job"
  aws_account_resource_id = grafana_cloud_provider_aws_account.myaccount.resource_id

  service {
    name = "AWS/EC2"

    metric {
      name       = "CPUUtilization"
      statistics = ["Average"]
    }

    metric {
      name       = "StatusCheckFailed"
      statistics = ["Maximum"]
    }

    scrape_interval_seconds = 300
    tags_to_add_to_metrics  = ["eks:cluster-name"]
  }

  service {
    name = "AWS/RDS"

    metric {
      name       = "CPUUtilization"
      statistics = ["Average", "Maximum"]
    }

    scrape_interval_seconds = 300
    tags_to_add_to_metrics  = ["name"]
  }

  custom_namespace {
    name = "MyApp"

    metric {
      name       = "MyMetric"
      statistics = ["Maximum", "Sum"]
    }

    scrape_interval_seconds = 300
  }
}

Export Terraform for existing scrape jobs

If you already have a scrape job in Cloud provider observability, you can export the Terraform configuration for that scrape job.

To export the Terraform configuration for existing scrape jobs in Grafana Cloud:

1. Open your Grafana Cloud portal.
2. Expand Observability > Cloud Provider in the main menu of your Grafana Cloud stack.
3. Click the Configuration tab.
4. Click CloudWatch metrics scrape.
5. Select the checkbox next to the scrape job you want the Terraform configuration for.
6. Click Export as terraform.
7. Click Copy to clipboard and save the Terraform in a file.

Export Terraform for new scrape jobs

If you don’t already have a scrape job, but you want a quicker way to collect a default set of metrics, you can use the Cloud provider observability UI to select the services with multiple configurable default metrics and export the configuration as Terraform.

To use the Cloud provider observability UI to configure your scrape job:

1. Open your Grafana Cloud portal.
2. Expand Observability > Cloud Provider in the main menu of your Grafana Cloud stack.
3. Click the Configuration tab.
4. Click CloudWatch metrics scrape.
5. Click Add new scrape job.

Explore your AWS service data

On the Cloud provider AWS Configuration tab, click Install dashboards and alerts to install the following prebuilt dashboards and alerts:

• Prebuilt dashboards
• Preconfigured alerts

Add, edit, or delete a scrape job

To add a scrape job, on the Your scrape jobs page, click Add new scrape job.

To edit a scrape job:

1. Click the Configuration tab, then click the CloudWatch metrics scrape tile.
2. At the Your scrape jobs page, open the edit view by one of these methods:
   • Click the name of the scrape job.
   • Click the three-dot icon next to the scrape job, and select Edit.
3. In the Edit scrape job view, make your changes.
4. Click Save scrape job.

To delete a scrape job, at the Your scrape jobs page, you can either:

• Click the three-dot menu icon next to the scrape job, select Delete, then click Delete to confirm.
• Click the name of the scrape job to open the Edit scrape job page, click Delete scrape job, then click Delete to confirm.

Enable account_alias label in collected metrics

Amazon CloudWatch Metrics supports pulling the AWS Account alias, as an additional label, account_alias, into all collected metrics. If the configured IAM role doesn’t have enough permissions to fetch the account alias, it isn’t added.

To enable the collection and addition of the account_alias label in all collected metrics, add the iam:ListAccountAliases permission to the IAM Policy used by Grafana.

You can check if the account_alias label is present by running the following query:

promql Copy

group({__name__=~"aws.+"}) by (account_alias)