Axiom
Overview

Send data from Kubernetes Cluster to Axiom

Prerequisites#

Send Kubernetes Cluster logs to Axiom using Filebeat#

Ingest logs from your Kubernetes cluster into Axiom using Filebeat.

The following is an example of a DaemonSet configuration to ingest your data logs into Axiom.

Configuration#

apiVersion: v1
kind: ServiceAccount
metadata:
  name: filebeat
  namespace: kube-system
  labels:
    k8s-app: filebeat
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: filebeat
  labels:
    k8s-app: filebeat
rules:
  - apiGroups: [''] # "" indicates the core API group
    resources:
      - namespaces
      - pods
      - nodes
    verbs:
      - get
      - watch
      - list
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: filebeat
subjects:
  - kind: ServiceAccount
    name: filebeat
    namespace: kube-system
roleRef:
  kind: ClusterRole
  name: filebeat
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: v1
data:
  filebeat.yml: |-
    filebeat.autodiscover:
      providers:
        - type: kubernetes
          node: ${NODE_NAME}
          hints.enabled: true
          hints.default_config:
            type: container
            paths:
              - /var/log/containers/*${data.kubernetes.container.id}.log
    allow_older_versions: true
    processors:
      - add_cloud_metadata:
    output.elasticsearch:
      hosts: ['${AXIOM_HOST}/v1/ingest/${AXIOM_DATASET}/elastic']
      api_key: 'axiom:${AXIOM_TOKEN}'
    setup.ilm.enabled: false
kind: ConfigMap
metadata:
  annotations: {}
  labels:
    k8s-app: filebeat
  name: filebeat-config
  namespace: kube-system
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  labels:
    k8s-app: filebeat
  name: filebeat
  namespace: kube-system
spec:
  selector:
    matchLabels:
      k8s-app: filebeat
  template:
    metadata:
      annotations: {}
      labels:
        k8s-app: filebeat
    spec:
      containers:
        - args:
            - -c
            - /etc/filebeat.yml
            - -e
          env:
            - name: AXIOM_HOST
              value: AXIOM_DOMAIN
            - name: AXIOM_DATASET
              value: DATASET_NAME
            - name: AXIOM_TOKEN
              value: API_TOKEN
            - name: NODE_NAME
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: spec.nodeName
          image: docker.elastic.co/beats/filebeat-oss:8.11.1
          imagePullPolicy: IfNotPresent
          name: filebeat
          resources:
            limits:
              memory: 200Mi
            requests:
              cpu: 100m
              memory: 100Mi
          securityContext:
            runAsUser: 0
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          volumeMounts:
            - mountPath: /etc/filebeat.yml
              name: config
              readOnly: true
              subPath: filebeat.yml
            - mountPath: /usr/share/filebeat/data
              name: data
            - mountPath: /var/lib/docker/containers
              name: varlibdockercontainers
              readOnly: true
            - mountPath: /var/log
              name: varlog
              readOnly: true
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      serviceAccount: filebeat
      serviceAccountName: filebeat
      terminationGracePeriodSeconds: 30
      volumes:
        - configMap:
            defaultMode: 416
            name: filebeat-config
          name: config
        - hostPath:
            path: /var/lib/docker/containers
            type: ''
          name: varlibdockercontainers
        - hostPath:
            path: /var/log
            type: ''
          name: varlog
        - hostPath:
            path: /var/lib/filebeat-data
            type: ''
          name: data
  updateStrategy:
    rollingUpdate:
      maxUnavailable: 1
    type: RollingUpdate

Replace AXIOM_DOMAIN with the base domain of your edge deployment. For more information, see Edge deployments.

Replace API_TOKEN with the Axiom API token you have generated. For added security, store the API token in an environment variable.

Replace DATASET_NAME with the name of the Axiom dataset where you send your data.

After editing your values, apply the changes to your cluster using kubectl apply -f daemonset.yaml

Send Kubernetes Cluster logs to Axiom using Vector#

Collect logs from your Kubernetes cluster and send them directly to Axiom using the Vector daemonset.

Configuration#

apiVersion: v1
kind: ServiceAccount
metadata:
  name: vector
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: vector
rules:
  - apiGroups: [""]
    resources:
      - pods
      - nodes
      - namespaces
    verbs:
      - get
      - list
      - watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: vector
subjects:
  - kind: ServiceAccount
    name: vector
    namespace: kube-system
roleRef:
  kind: ClusterRole
  name: vector
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: vector-config
  namespace: kube-system
data:
  vector.yml: |-
    sources:
      kubernetes_logs:
        type: kubernetes_logs
        self_node_name: ${VECTOR_SELF_NODE_NAME}
    sinks:
      axiom:
        type: axiom
        inputs:
          - kubernetes_logs
        compression: zstd
        dataset: ${AXIOM_DATASET}
        token: ${AXIOM_TOKEN}
        healthcheck:
          enabled: true
          log_level: debug
        logging:
          level: debug
        log_level: debug
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: vector
  namespace: kube-system
spec:
  selector:
    matchLabels:
      name: vector
  template:
    metadata:
      labels:
        name: vector
    spec:
      serviceAccountName: vector
      containers:
        - name: vector
          image: timberio/vector:0.37.0-debian
          args:
            - --config-dir
            - /etc/vector/
          env:
            - name: AXIOM_HOST
              value: AXIOM_DOMAIN
            - name: AXIOM_DATASET
              value: DATASET_NAME
            - name: AXIOM_TOKEN
              value: API_TOKEN
            - name: VECTOR_SELF_NODE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
          volumeMounts:
            - name: config
              mountPath: /etc/vector/vector.yml
              subPath: vector-config.yml
            - name: data-dir
              mountPath: /var/lib/vector
            - name: var-log
              mountPath: /var/log
              readOnly: true
            - name: var-lib
              mountPath: /var/lib
              readOnly: true
          resources:
            limits:
              memory: 500Mi
            requests:
              cpu: 200m
              memory: 100Mi
          securityContext:
            runAsUser: 0
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
      volumes:
        - name: config
          configMap:
            name: vector-config
            items:
              - key: vector.yml
                path: vector-config.yml
        - name: data-dir
          hostPath:
            path: /var/lib/vector
            type: DirectoryOrCreate
        - name: var-log
          hostPath:
            path: /var/log
        - name: var-lib
          hostPath:
            path: /var/lib
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
  updateStrategy:
    rollingUpdate:
      maxUnavailable: 1
    type: RollingUpdate

Replace AXIOM_DOMAIN with the base domain of your edge deployment. For more information, see Edge deployments.

Replace API_TOKEN with the Axiom API token you have generated. For added security, store the API token in an environment variable.

Replace DATASET_NAME with the name of the Axiom dataset where you send your data.

After editing your values, apply the changes to your cluster using kubectl apply -f daemonset.yaml

Send Kubernetes Cluster logs to Axiom using Fluent Bit#

Collect logs from your Kubernetes cluster and send them directly to Axiom using Fluent Bit.

Configuration#

apiVersion: v1
kind: ServiceAccount
metadata:
  name: fluent-bit
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: fluent-bit
rules:
  - apiGroups: [""]
    resources:
      - pods
      - nodes
      - namespaces
    verbs:
      - get
      - list
      - watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: fluent-bit
subjects:
  - kind: ServiceAccount
    name: fluent-bit
    namespace: kube-system
roleRef:
  kind: ClusterRole
  name: fluent-bit
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: fluent-bit-config
  namespace: kube-system
data:
  fluent-bit.conf: |-
    [SERVICE]
        Flush          1
        Log_Level      debug
        Daemon         off
        Parsers_File   parsers.conf
        HTTP_Server    On
        HTTP_Listen    0.0.0.0
        HTTP_Port      2020
 
    [INPUT]
        Name               tail
        Tag                kube.*
        Path               /var/log/containers/*.log
        Parser             docker
        DB                 /var/log/flb_kube.db
        Mem_Buf_Limit      7MB
        Skip_Long_Lines    On
        Refresh_Interval   10
 
    [FILTER]
        Name                  kubernetes
        Match                 kube.*
        Kube_URL              https://kubernetes.default.svc:443
        Kube_CA_File          /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        Kube_Token_File       /var/run/secrets/kubernetes.io/serviceaccount/token
        Kube_Tag_Prefix       kube.var.log.containers.
        Merge_Log             On
        Merge_Log_Key         log_processed
        K8S-Logging.Parser    On
        K8S-Logging.Exclude   Off
 
    [OUTPUT]
        Name            http
        Match           *
        Host            ${AXIOM_HOST}
        Port            443
        URI             /v1/ingest/${AXIOM_DATASET}
        Header          Authorization Bearer ${AXIOM_TOKEN}
        Format          json
        Json_date_key   time
        Json_date_format iso8601
        Retry_Limit     False
        Compress        gzip
        tls             On
        tls.verify      Off
 
  parsers.conf: |-
    [PARSER]
        Name        docker
        Format      json
        Time_Key    time
        Time_Format %Y-%m-%dT%H:%M:%S.%L
        Time_Keep   On
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluent-bit
  namespace: kube-system
spec:
  selector:
    matchLabels:
      name: fluent-bit
  template:
    metadata:
      labels:
        name: fluent-bit
    spec:
      serviceAccountName: fluent-bit
      containers:
        - name: fluent-bit
          image: fluent/fluent-bit:1.9.9
          env:
            - name: AXIOM_HOST
              value: AXIOM_DOMAIN
            - name: AXIOM_DATASET
              value: DATASET_NAME
            - name: AXIOM_TOKEN
              value: API_TOKEN
          volumeMounts:
            - name: config
              mountPath: /fluent-bit/etc/fluent-bit.conf
              subPath: fluent-bit.conf
            - name: config
              mountPath: /fluent-bit/etc/parsers.conf
              subPath: parsers.conf
            - name: varlog
              mountPath: /var/log
            - name: varlibdockercontainers
              mountPath: /var/lib/docker/containers
              readOnly: true
      volumes:
        - name: config
          configMap:
            name: fluent-bit-config
        - name: varlog
          hostPath:
            path: /var/log
        - name: varlibdockercontainers
          hostPath:
            path: /var/lib/docker/containers
      terminationGracePeriodSeconds: 10

Replace AXIOM_DOMAIN with the base domain of your edge deployment. For more information, see Edge deployments.

Replace API_TOKEN with the Axiom API token you have generated. For added security, store the API token in an environment variable.

Replace DATASET_NAME with the name of the Axiom dataset where you send your data.

After editing your values, apply the changes to your cluster using kubectl apply -f daemonset.yaml

Extract top-level fields from log body#

To extract top-level fields from the log body, configure the filelog receiver with operators to parse container log formats and JSON-structured log bodies.

Configure the following settings in your OpenTelemetry Collector configuration:

Setting Description
Receiver Use filelog receiver targeting /var/log/pods/*/*/*.log
Container operator Add the container operator to parse container log formats (Docker, CRI-O, Containerd) and extract the inner log message as the body
JSON parser operator Add the json_parser operator to parse JSON-structured log bodies into structured fields
Transform processor Optional: Use the transform processor to promote parsed JSON fields to top-level attributes instead of keeping them in the body

Example configuration:

receivers:
  filelog/k8s-logs:
    include: [/var/log/pods/*/*/*.log]
    start_at: end
    operators:
      - type: container
      - type: json_parser
        if: body matches '^{.*}$'

For complete configuration details, including optional timestamp and severity extraction, see the OpenTelemetry Collector documentation.

Updated

Was this page helpful?