Axiom
Overview

Send data from Fluentd to Axiom

With Fluentd, you can collect logs from multiple sources and ship it instantly into Axiom

Prerequisites#

Installation#

Visit the Fluentd download page to install Fluentd on your system.

Configuration#

Fluentd lifecycle consist of five different components which are:

  • Setup: Configure your fluent.conf file.
  • Inputs: Define your input listeners.
  • Filters: Create a rule to allow or disallow an event.
  • Matches: Send output to Axiom when input data match and pair specific data from your data input within your configuration.
  • Labels: Groups filters and simplifies tag handling.

When setting up Fluentd, the configuration file .conf is used to connect its components.

Configuring Fluentd using the HTTP output plugin#

The example below shows a Fluentd configuration that sends data to Axiom using the HTTP output plugin:

<source>
  @type forward
  port 24224
</source>
 
<match *.**>
  @type http
 
  endpoint https://AXIOM_DOMAIN/v1/ingest/DATASET_NAME
  # Authorization Bearer should be an ingest token
  headers {"Authorization": "Bearer API_TOKEN"}
  json_array false
  open_timeout 3
 
  <format>
    @type json
  </format>
 
  <buffer>
    flush_interval 5s
  </buffer>
</match>

Replace AXIOM_DOMAIN with the base domain of your edge deployment. For more information, see Edge deployments.

Replace API_TOKEN with the Axiom API token you have generated. For added security, store the API token in an environment variable.

Replace DATASET_NAME with the name of the Axiom dataset where you send your data.

Configuring Fluentd using the OpenSearch output plugin#

The example below shows a Fluentd configuration that sends data to Axiom using the OpenSearch plugin:

<source>
  @type tail
  @id input_tail
  <parse>
    @type apache2
  </parse>
  path /var/log/*.log
  tag td.logs
</source>
 
<match **>
  @type opensearch
  @id out_os
  @log_level info
  include_tag_key true
  include_timestamp true
  host "#{ENV['FLUENT_OPENSEARCH_HOST']  || 'AXIOM_DOMAIN'}"
  port "#{ENV['FLUENT_OPENSEARCH_PORT'] || '443'}"
  path "#{ENV['FLUENT_OPENSEARCH_PATH']|| '/v1/ingest/DATASET_NAME/elastic'}"
  scheme "#{ENV['FLUENT_OPENSEARCH_SCHEME'] || 'https'}"
  ssl_verify "#{ENV['FLUENT_OPENSEARCH_SSL_VERIFY'] || 'true'}"
  ssl_version "#{ENV['FLUENT_OPENSEARCH_SSL_VERSION'] || 'TLSv1_2'}"
  user "#{ENV['FLUENT_OPENSEARCH_USER'] || 'axiom'}"
  password "#{ENV['FLUENT_OPENSEARCH_PASSWORD'] || 'xaat-xxxxxxxxxx-xxxxxxxxx-xxxxxxx'}"
  index_name "#{ENV['FLUENT_OPENSEARCH_INDEX_NAME'] || 'fluentd'}"
</match>

Replace AXIOM_DOMAIN with the base domain of your edge deployment. For more information, see Edge deployments.

Replace DATASET_NAME with the name of the Axiom dataset where you send your data.

Configure buffer interval with filter patterns#

The example below shows a Fluentd configuration to hold logs in memory with specific flush intervals, size limits, and how to exclude specific logs based on patterns.

# Collect common system logs
<source>
  @type tail
  @id system_logs
  <parse>
    @type none
  </parse>
  path /var/log/*.log
  pos_file /var/log/fluentd/system.log.pos
  read_from_head true
  tag system.logs
</source>
 
# Collect Apache2 logs (if they’re located in /var/log/apache2/)
<source>
  @type tail
  @id apache_logs
  <parse>
    @type apache2
  </parse>
  path /var/log/apache2/*.log
  pos_file /var/log/fluentd/apache2.log.pos
  read_from_head true
  tag apache.logs
</source>
 
# Filter to exclude certain patterns (optional)
<filter **>
  @type grep
  <exclude>
    key message
    pattern /exclude_this_pattern/
  </exclude>
</filter>
 
# Send logs to Axiom
<match **>
  @type opensearch
  @id out_os
  @log_level info
  include_tag_key true
  include_timestamp true
  host "#{ENV['FLUENT_OPENSEARCH_HOST']  || 'AXIOM_DOMAIN'}"
  port "#{ENV['FLUENT_OPENSEARCH_PORT'] || '443'}"
  path "#{ENV['FLUENT_OPENSEARCH_PATH']|| '/v1/ingest/DATASET_NAME/elastic'}"
  scheme "#{ENV['FLUENT_OPENSEARCH_SCHEME'] || 'https'}"
  ssl_verify "#{ENV['FLUENT_OPENSEARCH_SSL_VERIFY'] || 'true'}"
  ssl_version "#{ENV['FLUENT_OPENSEARCH_SSL_VERSION'] || 'TLSv1_2'}"
  user "#{ENV['FLUENT_OPENSEARCH_USER'] || 'axiom'}"
  password "#{ENV['FLUENT_OPENSEARCH_PASSWORD'] || 'xaat-xxxxxxxxxx-xxxxxxxxx-xxxxxxx'}"
  index_name "#{ENV['FLUENT_OPENSEARCH_INDEX_NAME'] || 'fluentd'}"
 
  <buffer>
    @type memory
    flush_mode interval
    flush_interval 10s
    chunk_limit_size 5M
    retry_max_interval 30
    retry_forever true
  </buffer>
</match>

Replace AXIOM_DOMAIN with the base domain of your edge deployment. For more information, see Edge deployments.

Replace DATASET_NAME with the name of the Axiom dataset where you send your data.

Collect and send PHP logs to Axiom#

The example below shows a Fluentd configuration that sends PHP data to Axiom.

# Collect PHP logs
<source>
  @type tail
  @id php_logs
  <parse>
    @type multiline
    format_firstline /^\[\d+-\w+-\d+ \d+:\d+:\d+\]/
    format1 /^\[(?<time>\d+-\w+-\d+ \d+:\d+:\d+)\] (?<message>.*)/
  </parse>
  path /var/log/php*.log
  pos_file /var/log/fluentd/php.log.pos
  read_from_head true
  tag php.logs
</source>
 
# Send PHP logs to Axiom
<match php.logs>
  @type opensearch
  @id out_os
  @log_level info
  include_tag_key true
  include_timestamp true
  host "#{ENV['FLUENT_OPENSEARCH_HOST']  || 'AXIOM_DOMAIN'}"
  port "#{ENV['FLUENT_OPENSEARCH_PORT'] || '443'}"
  path "#{ENV['FLUENT_OPENSEARCH_PATH']|| '/v1/ingest/DATASET_NAME/elastic'}"
  scheme "#{ENV['FLUENT_OPENSEARCH_SCHEME'] || 'https'}"
  ssl_verify "#{ENV['FLUENT_OPENSEARCH_SSL_VERIFY'] || 'true'}"
  ssl_version "#{ENV['FLUENT_OPENSEARCH_SSL_VERSION'] || 'TLSv1_2'}"
  user "#{ENV['FLUENT_OPENSEARCH_USER'] || 'axiom'}"
  password "#{ENV['FLUENT_OPENSEARCH_PASSWORD'] || 'xaat-xxxxxxxxxx-xxxxxxxxx-xxxxxxx'}"
  index_name "#{ENV['FLUENT_OPENSEARCH_INDEX_NAME'] || 'php-logs'}"
 
  <buffer>
    @type memory
    flush_mode interval
    flush_interval 10s
    chunk_limit_size 5M
    retry_max_interval 30
    retry_forever true
  </buffer>
</match>

Replace AXIOM_DOMAIN with the base domain of your edge deployment. For more information, see Edge deployments.

Replace DATASET_NAME with the name of the Axiom dataset where you send your data.

Collect and send Scala logs to Axiom#

The example below shows a Fluentd configuration that sends Scala data to Axiom

# Collect Scala logs
<source>
  @type tail
  @id scala_logs
  <parse>
    @type multiline
    format_firstline /^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d{3}/
    format1 /^(?<time>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d{3}) \[(?<thread>.*)\] (?<level>\w+) (?<class>[\w\.$]+) - (?<message>.*)/
  </parse>
  path /var/log/scala-app.log
  pos_file /var/log/fluentd/scala.log.pos
  read_from_head true
  tag scala.logs
</source>
 
# Send Scala logs using HTTP plugin to Axiom
<match scala.logs>
  @type http
  endpoint "#{ENV['FLUENT_HTTP_ENDPOINT'] || 'https://AXIOM_DOMAIN/v1/ingest/DATASET_NAME'}"
  headers {"Authorization": "Bearer #{ENV['FLUENT_HTTP_TOKEN'] || '<your-token>'}"}
  <format>
    @type json
  </format>
 
  <buffer>
    @type memory
    flush_mode interval
    flush_interval 10s
    chunk_limit_size 5M
    retry_max_interval 30
    retry_forever true
  </buffer>
</match>

Replace AXIOM_DOMAIN with the base domain of your edge deployment. For more information, see Edge deployments.

Replace DATASET_NAME with the name of the Axiom dataset where you send your data.

Send virtual machine logs to Axiom using the HTTP output plugin#

The example below shows a Fluentd configuration that sends data from your virtual machine to Axiom using the apache source type.

<source>
  @type tail
  @id input_tail
  <parse>
    @type apache2
  </parse>
  path /var/log/**/*.log
  pos_file /var/log/fluentd/fluentd.log.pos
  tag vm.logs
  read_from_head true
</source>
 
<filter vm.logs>
  @type record_transformer
  <record>
    hostname "#{Socket.gethostname}"
    service "vm_service"
  </record>
</filter>
 
<match vm.logs>
  @type http
  @id out_http_axiom
  @log_level info
  endpoint "#{ENV['AXIOM_URL'] || 'https://AXIOM_DOMAIN'}"
  path "/v1/ingest/DATASET_NAME"
  ssl_verify "#{ENV['AXIOM_SSL_VERIFY'] || 'true'}"
  <headers>
    Authorization "Bearer API_TOKEN"
    Content-Type "application/json"
  </headers>
  <format>
    @type json
  </format>
  <buffer>
    @type memory
    flush_mode interval
    flush_interval 5s
    chunk_limit_size 5MB
    retry_forever true
  </buffer>
</match>

Replace AXIOM_DOMAIN with the base domain of your edge deployment. For more information, see Edge deployments.

Replace API_TOKEN with the Axiom API token you have generated. For added security, store the API token in an environment variable.

Replace DATASET_NAME with the name of the Axiom dataset where you send your data.

The example below shows a Fluentd configuration that sends data from your virtual machine to Axiom using the nginx source type.

<source>
  @type tail
  @id input_tail
  <parse>
    @type nginx
  </parse>
  path /var/log/nginx/access.log, /var/log/nginx/error.log
  pos_file /var/log/fluentd/nginx.log.pos
  tag nginx.logs
  read_from_head true
</source>
 
<filter nginx.logs>
  @type record_transformer
  <record>
    hostname "#{Socket.gethostname}"
    service "nginx"
  </record>
</filter>
 
<match nginx.logs>
  @type http
  @id out_http_axiom
  @log_level info
  endpoint "#{ENV['AXIOM_URL'] || 'https://AXIOM_DOMAIN'}"
  path "/v1/ingest/DATASET_NAME"
  ssl_verify "#{ENV['AXIOM_SSL_VERIFY'] || 'true'}"
  <headers>
    Authorization "Bearer API_TOKEN"
    Content-Type "application/json"
  </headers>
  <format>
    @type json
  </format>
  <buffer>
    @type memory
    flush_mode interval
    flush_interval 5s
    chunk_limit_size 5MB
    retry_forever true
  </buffer>
</match>

Replace AXIOM_DOMAIN with the base domain of your edge deployment. For more information, see Edge deployments.

Replace API_TOKEN with the Axiom API token you have generated. For added security, store the API token in an environment variable.

Replace DATASET_NAME with the name of the Axiom dataset where you send your data.

Updated

Was this page helpful?