Axiom
Overview

Send data from Amazon CloudWatch to Axiom

Axiom CloudWatch Forwarder includes templates for the following CloudFormation stacks:

  • Forwarder creates a Lambda function that forwards logs from Amazon CloudWatch to Axiom.
  • Subscriber runs once to create subscription filters on Forwarder for Amazon CloudWatch log groups specified by a combination of names, prefix, and regular expression filters.
  • Listener creates a Lambda function that listens for new log groups and creates subscription filters for them on Forwarder. This way, you don’t have to create subscription filters manually for new log groups.
  • Unsubscriber runs once to remove subscription filters on Forwarder for Amazon CloudWatch log groups specified by a combination of names, prefix, and regular expression filters.

To determine the best method to send data from different AWS services, see Send data from AWS to Axiom.

The Axiom CloudWatch Forwarder is an open-source project and welcomes your contributions. For more information, see the GitHub repository.

Prerequisites#

Installation#

To install the Axiom CloudWatch Forwarder, choose one of the following:

Install with Cloudformation stacks#

  1. Launch the Forwarder stack template on AWS. Copy the Forwarder Lambda ARN because it’s referenced in the Subscriber stack.
  2. Launch the Subscriber stack template on AWS.
  3. Launch the Listener stack template on AWS.

Install with Terraform module#

1
Step

Create a new Forwarder module in your Terraform file in the following way:

module "forwarder" {
  source           = "axiomhq/axiom-cloudwatch-forwarder/aws//modules/forwarder"
  axiom_dataset    = "DATASET_NAME"
  axiom_token      = "API_TOKEN"
  prefix           = "axiom-cloudwatch-forwarder"
}

Info: Replace API_TOKEN with the Axiom API token you have generated. For added security, store the API token in an environment variable.

Replace DATASET_NAME with the name of the Axiom dataset where you send your data.

Alternatively, create a dataset with the Axiom Terraform provider.

2
Step

Create a new Subscriber module in your Terraform file in the following way:

module "subscriber" {
  source               = "axiomhq/axiom-cloudwatch-forwarder/aws//modules/subscriber"
  prefix               = "axiom-cloudwatch-forwarder"
  forwarder_lambda_arn = module.forwarder.lambda_arn
  log_groups_prefix    = "/aws/lambda/"
}
3
Step

Create a new Listener module in your Terraform file in the following way:

module "listener" {
  source               = "axiomhq/axiom-cloudwatch-forwarder/aws//modules/listener"
  prefix               = "axiom-cloudwatch-forwarder"
  forwarder_lambda_arn = module.forwarder.lambda_arn
  log_groups_prefix    = "/aws/lambda/"
}
4
Step

In your terminal, go to the folder of your main Terraform file, and then run terraform init.

5
Step

Run terraform plan to check the changes, and then run terraform apply.

Filter Amazon CloudWatch log groups#

The Subscriber and Unsubscriber stacks allow you to filter the log groups by a combination of names, prefix, and regular expression filters. If no filters are specified, the stacks subscribe to or unsubscribe from all log groups. You can also whitelist a specific set of log groups using filters in the CloudFormation stack parameters. The log group names, prefix, and regular expression filters included are additive, meaning the union of all provided inputs is matched.

Example#

For example, you have the following list of log groups:

/aws/lambda/function-foo
/aws/lambda/function-bar
/aws/eks/cluster/cluster-1
/aws/rds/instance-baz
  • To subscribe to the Lambda log groups exclusively, use a prefix filter with the value of /aws/lambda.
  • To subscribe to EKS and RDS log groups, use a list of names with the value of /aws/eks/cluster/cluster-1,/aws/rds/instance-baz.
  • To subscribe to the EKS log group and all Lambda log groups, use a combination of prefix and names list.
  • To use the regular expression filter, write a regular expression to match the log group names. For example, \/aws\/lambda\/.* matches all Lambda log groups.
  • To subscribe to all log groups, leave the filters empty.

Listener architecture#

The optional Listener stack does the following:

  • Creates an Amazon S3 bucket for AWS CloudTrail.
  • Creates a trail to capture the creation of new log groups.
  • Creates an event rule to pass those creation events to an Amazon EventBridge event bus.
  • Sends an event via EventBridge to a Lambda function when a new log group is created.
  • Creates a subscription filter for each new log group.

Remove subscription filters#

To remove subscription filters for one or more log groups, launch the Unsubscriber stack template on AWS.

The log group filtering works the same way as the Subscriber stack. You can filter the log groups by a combination of names, prefix, and regular expression filters.

Alternatively, to turn off log forwarding to Axiom, create a new Unsubscriber module in your Terraform file in the following way:

module "unsubscriber" {
  source           = "axiomhq/axiom-cloudwatch-forwarder/aws//modules/unsubscriber"
  prefix           = "axiom-cloudwatch-forwarder"
  forwarder_lambda_arn = module.forwarder.lambda_arn
  log_groups_prefix    = "/aws/lambda/"
}

Updated

Was this page helpful?