Manage datasets

What datasets are#

Axiom’s datastore is tuned for the efficient collection, storage, and analysis of timestamped event data. An individual piece of data is an event, and a dataset is a collection of related events. Datasets contain incoming event data.

Best practices for organizing datasets#

Use datasets to organize your data ready for querying based on the event schema. Common ways to separate include environment, signal type, and service.

Separate by environment#

If you work with data sourced from different environments, separate them into different datasets. For example, use one dataset for events from production and another dataset for events from your development environment.

You might be tempted to use a single environment attribute instead, but this risks causing confusion when results show up side-by-side in query results. Although some organizations choose to collect events from all environments in one dataset, they’ll often rely on applying an environment filter to all queries, which becomes a chore and is error-prone for newcomers.

Separate by signal type#

If you work with distributed applications, consider splitting your data into different datasets. For example:

• A dataset with traces for all services
• A dataset with app logs for all services
• A dataset with frontend web vitals
• A dataset with infrastructure logs
• A dataset with security logs
• A dataset with CI logs

If you look for a specific event in a distributed system, you are likely to know its signal type but not the related service. By splitting data into different datasets using the approach above, you can find data easily.

Separate by service#

Another common practice is to separate datasets by service. This approach allows for easier access control management.

For example, you might separate engineering services with datasets like kubernetes, billing, or vpn, or include events from your wider company collectors like product-analytics, security-logs, or marketing-attribution.

This separation enables teams to focus on their relevant data and simplifies querying within a specific domain. It also works well with Axiom’s role-based access control feature as you can restrict access to sensitive datasets to those who need it.

Avoid the “kitchen sink”#

While it might seem convenient to send all events to a single dataset, this “kitchen sink” approach is generally not advisable for several reasons:

• Field count explosion: As you add more event types to a single dataset, the number of fields grows rapidly. This can make it harder to understand the structure of your data and impacts query performance.
• Query inefficiency: With a large, mixed dataset, queries often require multiple filters to isolate the relevant data. This is tedious, but without those filters, queries take longer to execute since they scan through more irrelevant data.
• Schema conflicts: Different event types may have conflicting field names or data types, leading to unnecessary type coercion at query time.
• Access management: With all data in one dataset, it becomes challenging to provide granular access controls. You might end up giving users access to more data than they need.

Don’t create multiple Axiom organizations to separate your data. For example, don’t use a different Axiom organization for each deployment. If you’re on the Axiom Cloud (Personal) plan, this might go against Axiom’s fair use policy. Instead, separate data by creating a different dataset for each deployment within the same Axiom organization.

Access to datasets#

The datasets that individual users have access to determine the following:

• The data they see in dashboards. If a user has access to a dashboard but only to some of the datasets referenced in the dashboard’s elements, the user only sees data from the datasets they have access to.
• The monitors they see. A user only sees the monitors that reference the datasets that the user has access to. If a user has access to the monitors of an organization but only to some of the datasets referenced in the monitors, the user only sees the monitors that reference the datasets they have access to. If a monitor joins several datasets, a user can only see the monitor if the user has access to all of the datasets.

Limits on ingested data#

For more information on limits and requirements imposed by Axiom, see Limits.

Create dataset#

To create a dataset using Console, follow these steps:

1. Click Settings > Datasets and views.
2. Click New dataset.
3. Name the dataset and add an optional description.
4. In Kind, select one of the following:
   • Select Events if you plan to use the dataset for logs or traces data.
   • Select Metrics if you plan to send OTel metrics data to the dataset.
5. In Data retention, select for how long to store your data in this dataset. For more information, see Specify data retention period.
6. In Edge deployment, select the edge deployment you want to use for this dataset. You can select from the list of edge deployments you have added to the organization. For more information, see Edge deployments and Add edge deployments.
7. Click Save dataset.

To create a dataset using the Axiom API, send a POST request to the datasets endpoint.

Dataset names are 1 to 128 characters in length. They only contain ASCII alphanumeric characters and the hyphen (-) character.

Import data#

You can import data to your dataset in one of the following formats:

• Newline delimited JSON (NDJSON)
• Arrays of JSON objects
• CSV

To import data to a dataset, follow these steps:

1. Click Settings > Datasets and views.
2. In the list, find the dataset where you want to import data, and then click Import on the right.
3. Optional: Specify the timestamp field. This is only necessary if your data contains a timestamp field and it’s different from _time.
4. Upload the file, and then click Import.

Trim dataset#

Trimming permanently deletes data stored in blocks that are older than a date you specify. Note that this action doesn’t delete data that’s older than the specified date but shares a block with newer data.

This can be useful if your dataset takes up too much storage space and you want to reduce its size to ensure you stay within the allowed limits. If your dataset contains too many fields, delete the fields you don’t need instead.

Trimming is an asynchronous operation. When you submit a trim request, Axiom queues the deletion and processes it in the background. The trimmed data may take several hours to be fully removed from your dataset. During this time:

• The trim operation appears as successful in the audit log.
• Query results may still include data that's queued for deletion.
• The storage usage displayed in your settings may not immediately reflect the reduction.

To trim a dataset, follow these steps:

1. Click Settings > Datasets and views.
2. In the list, find the dataset that you want to trim, and then click Trim dataset on the right.
3. Specify the date before which you want to delete blocks of data.
4. Enter the name of the dataset, and then click Trim.

Vacuum fields#

The data schema of your dataset is defined on read. Axiom continuously creates and updates the data structures during the data ingestion process. At the same time, Axiom only retains data for the retention period you specify. This means that the data schema can contain fields that you ingested into the dataset in the past, but these fields are no longer present in the data currently associated with the dataset. This can be an issue if the number of fields in the dataset exceeds the allowed limits.

In this case, vacuuming fields in a dataset can help you reduce the number of fields associated with a dataset and stay within the allowed limits. Vacuuming fields resets the number of fields associated with a dataset to the fields that occur in events within your retention period. Technically, it wipes the data schema and rebuilds it from the data you currently have in the dataset, which is partly defined by the retention period. For example, you have ingested 500 fields over the last year and 50 fields in the last 95 days, which is your retention period. In this case, before vacuuming, your data schema contains 500 fields. After vacuuming, the dataset only contains 50 fields.

Vacuuming fields doesn’t delete any events from your dataset. To delete events, trim the dataset. Vacuuming only removes fields that no longer occur in any event within your retention period. If you accidentally ingested events with fields you didn’t want to send to Axiom, and these events are within your retention period, vacuuming alone doesn’t solve your problem. In this case, delete the unintended fields instead. Deleting fields also vacuums the dataset.

You can vacuum the fields of a dataset whose schema is locked. In this case, Axiom vacuums the fields and then regenerates the locked schema so that it matches the vacuumed fields. For more information, see Lock dataset schema.

To vacuum fields, follow these steps:

1. Click Settings > Datasets and views.
2. In the list, find the dataset where you want to vacuum fields, and then click Vacuum fields on the right.
3. Select the checkbox, and then click Vacuum.

Delete fields#

Deleting fields removes specific fields from the data schema of an event dataset. This is useful if you accidentally ingested fields you don’t want in Axiom, or if the number of fields in the dataset approaches the allowed limits and you want to remove the fields you don’t need.

Deleting fields differs from the other ways of reducing the number of fields:

• Vacuuming fields only removes fields that no longer occur in any event within your retention period. Deleting fields removes the fields you select, even if events within your retention period contain them. Deleting fields also vacuums the dataset, so fields that vacuuming would remove are removed as well.
• Trimming a dataset deletes events. Deleting fields doesn’t delete any events. All other fields of the events remain queryable.

Deleting fields works in the following way:

• Axiom removes the deleted fields from the schema, the list of fields in the Datasets tab, and query autocomplete. Field descriptions, units, and hidden settings for the deleted fields are removed as well.
• The values of the deleted fields stay in storage but you can’t access them anymore. Deleting fields doesn’t reduce the storage usage of your dataset.
• As the last step of the deletion, Axiom vacuums the fields of the dataset. This rebuilds the data schema from the events currently in the dataset, so fields that no longer occur in any event within your retention period disappear together with the fields you selected. For more information, see Vacuum fields.
• After the deletion completes, saved queries, dashboards, monitors, and virtual fields that reference a deleted field return an error. Review these before you delete fields.
• Deleting fields doesn’t prevent Axiom from adding the fields again. If you ingest events that contain a deleted field after the deletion, Axiom adds the field to the schema again and you can query the newly ingested values. The values you ingested before the deletion remain inaccessible. To stop a field from coming back, fix the data source before you delete the field, or lock the schema after the deletion completes. If Axiom keeps receiving the field while the deletion runs, the field might remain in the dataset after the deletion finishes.
• Field deletion is an asynchronous operation. When you submit a request, Axiom queues the deletion and processes it in the background. The deletion can take several hours to complete, depending on the size of the dataset. During this time, the fields still appear in the Fields panel, and queries might return the values of the deleted fields for some events but not for others. The request appears as deleteDatasetFields in the audit log.

To delete fields, follow these steps:

1. Go to the Datasets tab.
2. In the list, select the dataset from which you want to delete fields.
3. In the Fields panel, click More > Delete fields.
4. Enter a pattern that matches the names of the fields you want to delete, and then click Find fields. Axiom lists all fields whose name contains the pattern. The pattern is case-insensitive. Leave the pattern empty to list all fields.
5. If you entered a pattern, Axiom selects all matching fields that you can delete. If you left the pattern empty, Axiom doesn’t select any fields. Select the fields you want to delete and clear the ones you want to keep. Use Check all and Uncheck all to change the selection of all matching fields. Fields you can’t delete appear greyed out with an explanation.
6. Click Delete N fields, where N is the number of fields you selected.
7. Review the list of fields, and then click Delete N fields to confirm.

Axiom displays a message that the deletion started. The deleted fields disappear from the Fields panel when the deletion completes.

To delete fields using the Axiom API, send a POST request to the delete fields endpoint with the list of field names in the request body. To delete a single field, send a DELETE request to the delete field endpoint. Both endpoints return the status code 202 when the deletion is queued, and the status code 429 with a Retry-After header if a field deletion started on the dataset in the last 5 minutes.

The API doesn’t match field names by pattern. Specify each field name exactly as it appears in the schema of the dataset. To get the field names, use the list fields endpoint. If any field in the request doesn’t exist in the dataset, is a system field, or is a map field, Axiom rejects the whole request with the status code 400 and doesn’t delete any fields. If the field in a DELETE request doesn’t exist in the dataset, the delete field endpoint returns the status code 404.

Lock dataset schema#

By default, the data schema of a dataset grows together with your data. When you ingest an event with a field that isn’t yet part of the data schema, Axiom adds the new field to the schema. Similarly, when you ingest an event where the value of an existing field has a new type, Axiom extends the field’s type to support both the existing and the new type.

Locking the schema of a dataset freezes its set of fields and the type of each field. While the schema is locked, Axiom discards the following during the data ingestion process:

• Fields that aren’t part of the locked schema.
• Field values whose type doesn’t match the type of the field in the locked schema and can’t be promoted to that type. For example, if the locked schema defines a field as a number and you ingest an event where the field contains a string, Axiom drops the field’s value from that event. Axiom keeps values that it can promote to the locked type. For example, if the locked type of a field is float, Axiom accepts integer values in that field. Null values never violate a locked schema.

Axiom still ingests the events themselves and only drops the fields that don’t match the locked schema. When Axiom drops a field, the response to the ingest request contains a warning message that identifies the field and the reason. For example:

"messages": [
  {
    "priority": "warn",
    "code": "schema_rules_drop:debug",
    "count": 1,
    "msg": "field \"debug\" dropped: not in declared schema"
  },
  {
    "priority": "warn",
    "code": "schema_rules_drop:status",
    "count": 1,
    "msg": "field \"status\" dropped: type rule violated (allowed integer, saw string)"
  }
]

Axiom calculates ingest usage based on the data as it arrives. Fields that Axiom discards because the schema is locked still count towards your ingest usage.

Locking the schema can be useful if you want to protect a dataset with a well-defined schema from unexpected fields, ensure that fields have consistent types, or keep the number of fields within the allowed limits. To reduce the number of fields already associated with a dataset, delete fields or vacuum its fields.

Locking and unlocking the schema take effect immediately and don’t change the data stored in the dataset. When you unlock the schema, Axiom restores the default behavior and adds new fields to the schema during the data ingestion process. Unlocking doesn’t restore fields or values that Axiom discarded while the schema was locked.

To lock the schema of a dataset, follow these steps:

1. Go to the Datasets tab.
2. In the list, select the dataset whose schema you want to lock.
3. In the Fields panel, click More > Lock schema.

Axiom displays a Schema locked message. While the schema is locked, the title of the Fields panel is Fields (Locked).

To unlock the schema of a dataset, follow these steps:

1. Go to the Datasets tab.
2. In the list, select the dataset whose schema you want to unlock.
3. In the Fields panel, click More > Unlock schema.

Axiom displays a Schema unlocked message.

Share datasets#

You can share your datasets with other Axiom organizations. The receiving organization:

• can query the shared dataset.
• can create other Axiom resources that rely on query access such as dashboards and monitors.
• can’t ingest data into the shared dataset.
• can‘t modify the shared dataset.

No ingest usage associated with the shared dataset accrues to the receiving organization. Query usage associated with the shared dataset accrues to the organization running the query.

To share a dataset with another Axiom organization:

1. Ensure you have the necessary privileges to share datasets. By default, only users with the Owner role can share datasets.
2. Click Settings > Datasets and views.
3. In the list, find the dataset that you want to share, and then click Share dataset on the right.
4. In the Sharing links section, click + to create a new sharing link.
5. Copy the URL and share it with the receiving user in the organization with which you want to share the dataset. For example, https://app.axiom.co/s/dataset/{sharing-token}.
6. Ask the receiving user to open the sharing link. When opening the link, the receiving user sees the name of the dataset and the email address of the Axiom user that created the sharing link. They click Add dataset to confirm that they want to receive the shared dataset.

Delete sharing link#

Organizations can gain access to the dataset with an active sharing link. To deactivate the sharing link, delete the sharing link. Deleting a sharing link means that organizations that don’t have access to the dataset can’t use the sharing link to join the dataset in the future. Deleting a sharing link doesn’t affect the access of organizations that already have access to the shared dataset.

To delete a sharing link:

1. Click Settings > Datasets and views.
2. In the list, find the dataset, and then click Share dataset on the right.
3. To the right of the sharing link, click Delete.
4. Click Delete sharing link.

Remove access to shared dataset#

If your organization has previously shared a dataset with a receiving organization, and you want to remove the receiving organization’s access to the dataset, follow these steps:

1. Click Settings > Datasets and views.
2. In the list, find the dataset, and then click Share dataset on the right.
3. In the list, find the organization whose access you want to remove, and then click Remove.
4. Click Remove access.

Remove shared dataset#

If your organization has previously received access to a dataset from a sending organization, and you want to remove the shared dataset from your organization, follow these steps:

1. Ensure you have Delete permissions for the shared dataset.
2. Click Settings > Datasets and views.
3. In the list, click the shared dataset that you want to remove, and then click Remove dataset.
4. Enter the name of the dataset, and then click Remove.

Specify data retention period#

The data retention period determines how long Axiom stores your data. By default, the data retention period is the same for all datasets. You can configure custom retention periods for individual datasets. As a result, Axiom automatically trims data after the specified time period instead of the default period. For example, this can be useful if your dataset contains sensitive event data that you don’t want to retain for a long time.

The retention period of a dataset is set to Default unless you select a different value. Datasets set to Default follow your organization’s default dataset retention period. Changing the default therefore also changes the retention period of these datasets, and deletes data in them if you lower the default. When you select any other retention period for a dataset, the dataset uses that custom retention period and becomes independent of the default.

The custom retention period you can set for an individual dataset depends on your plan:

• On the Axiom Cloud plan, you can set any retention period, including a period that’s longer than your organization’s default, or Forever.
• On other plans, the custom retention period must be shorter than the default data retention period. For example, if you’re on the Personal plan, the default data retention period is 30 days and you can only specify a shorter period.

For more information, see Pricing-based limits and Set default dataset retention period.

To change the data retention period for a dataset, follow these steps:

1. Click Settings > Datasets and views.
2. In the list, find the dataset for which you want to change the retention period, and then click Edit dataset retention on the right.
3. Select a retention period, or select Custom to enter your own retention period in days. To make the dataset follow your organization’s default dataset retention period, select Default.
4. Click Submit.

Delete dataset#

To delete a dataset, follow these steps:

1. Click Settings > Datasets and views.
2. In the list, click the dataset that you want to delete, and then click Delete dataset.
3. Enter the name of the dataset, and then click Delete.

Manage apps and endpoints#

Apps allow you to enrich your Axiom organization with dedicated apps. For more information, see Introduction to apps.

Endpoints allow you to easily integrate Axiom into your existing data flow using tools and libraries that you already know. With endpoints, you can build and configure your existing tooling to send data to Axiom so you can start monitoring your logs immediately.

Edit app#

1. Click Settings > Apps.
2. Find the app in the list, and then click More > Edit.

Disconnect app#

1. Click Settings > Apps.
2. Find the app in the list, and then click More > Disconnect.

Create endpoint#

1. Click Settings > Endpoints.
2. Click New endpoint.
3. Click the type of endpoint you want to create.
4. Name the endpoint.
5. Select the dataset where you want to send data.
6. Copy the URL displayed for the newly created endpoint. This is the target URL where you send the data.

Delete endpoint#

1. Click Settings > Endpoints.
2. Find the endpoint in the list, and then click Delete endpoint on the right.