> ## Documentation Index
> Fetch the complete documentation index at: https://docs.chronosphere.io/llms.txt
> Use this file to discover all available pages before exploring further.
# Normalize your log data
> Learn how to use field mapping in Chronosphere Observability Platform to normalize your log data at ingest time.
During the ingestion phase, after parsing your data, Chronosphere Observability
Platform applies your preferred field normalization rules to populate predefined keys
such as `service`, `severity`, and `timestamp`.
```mermaid actions={false} theme={null}
flowchart LR
Send["Send data"]
subgraph OP["Observability Platform"]
Parse[Parse data]
Normalize[Normalize data]
Control[Control data]
Format[Format data]
Store[Store data]
end
Send --> Parse --> Normalize --> Control --> Format --> Store
click Parse "/control/shaping/shape-logs/parse-logs"
click Control "/control/shaping/shape-logs/control-logs"
click Format "/control/shaping/shape-logs/lookup-logs/"
classDef stages fill:#2fbf71,stroke-width:2px,stroke:#2fbf71,color:#FFFFFF;
class Normalize stages
```
This mapping provides consistency in your data, and optimizes querying in Logs
Explorer. After applying these mappings, Observability Platform applies
[control rules](/control/shaping/shape-logs/control-logs), and then stores your
log data.
In addition to these mappings, you can apply custom field mappings to normalize log
data, such as normalizing values from several fields that all relate to different
environments. See [Examples](#examples) for examples of how you can use custom field
mappings.
Custom field mappings are applied after log data is ingested, but before
Observability Platform stores it.
To indicate nesting within a source field, use a dot (`.`). For example, if your logs
are in a format like `{"resource": {"labels": {"project_id": "..."}}}`, then enter
`resource.labels.project_id`.
If your logs already use dot notation, such as `{"resource.labels.project_id": "..."}`,
enclose the selector in double quotes and brackets like
`["resource.labels.project_id"]`. See
[target nested or flat fields with dot notation](#target-nested-or-flat-fields-with-dot-notation)
for additional examples.
## View custom field mappings
Select from the following methods to view custom field mappings.
To view custom field mappings:
1. In the navigation menu, click ** Go to Admin**
and then select
** Control > Logs Ingest**.
All field mappings display in the **Field normalization** section, including both
predefined and custom field mappings.
2. Choose which custom field mappings to view:
* To view an individual custom field mapping, click it in the list to open the
configuration in the **Edit field mapping** drawer.
* To view the code definition for all configured custom field mappings, click the
**Code config** tab.
To use [Chronoctl](/tooling/chronoctl) to return all custom field mappings, use this
command:
```shell theme={null}
chronoctl log-ingest-config read
```
[Use the Code Config tool](/tooling/gitops#use-the-code-config-tool) in Observability
Platform to view a custom field mappings's Terraform representation.
To complete this action with the Chronosphere API, use the
[`ReadLogIngestConfig`](/tooling/api-info/definition/operations/ReadLogIngestConfig)
endpoint.
Because the Chronosphere API requires authentication, include an API token with your
`curl` request, as shown in the following example. For more details, see
[Create an API token](/tooling/api-info#create-an-api-token).
```shell /"TOKEN"/ /INSTANCE/ /METHOD/ /ENDPOINT_PATH/ theme={null}
export CHRONOSPHERE_API_TOKEN="TOKEN"
export CHRONOSPHERE_DOMAIN="INSTANCE.chronosphere.io"
curl -H "API-Token: ${CHRONOSPHERE_API_TOKEN}" \
-X METHOD "https://${CHRONOSPHERE_DOMAIN}/ENDPOINT_PATH"
```
Replace the following:
* *`TOKEN`*: Your API token.
* *`INSTANCE`*: The subdomain name for your organization's Observability Platform instance.
* *`METHOD`*: The HTTP method to use with the request, such as `GET` or `POST`.
* *`ENDPOINT_PATH`*: The specific endpoint you want to access.
## Create custom field mappings
Use one of the following methods to create custom field mappings.
Define the custom field mapping definition in Observability Platform, and then
[use the Code Config tool](/tooling/gitops#use-the-code-config-tool) to apply the definition.
1. In the navigation menu, click ** Go to Admin**
and then select
** Control > Logs Ingest**.
2. In the **Field normalization** section, click **Create**.
3. In the **Custom field name** field, enter the custom field name to map data to,
such as `environment`, `region`, or `userID`.
4. In the **Mapped fields** section, in the **Source field name** field, enter a
source field name from your log data to map to the custom field you're creating.
Fields are matched in priority order.
To add more fields, click **Add field mapping** and then enter another source
field.
Use a dot (`.`) to indicate nesting within a field. For example, if your logs are
in a format like `{"resource": {"labels": {"project_id": "..."}}}`, then enter
`resource.labels.project_id`.
If your logs already use dot notation, such as `{"resource.labels.project_id"}`,
enclose the selector in double quotes and brackets like
`["resource.labels.project_id"]`.
5. Optional: In the **Default value** field, enter a default value as the mapping if
the custom field contains no values.
6. Optional: Select either **Regex** or **Value map** as the normalization mode.
* **Regex**: Enter a regular expression pattern to extract and normalize values,
such as for removing extraneous data. For example, use the following regular
expression to remove prefixes and suffixes from any environments including
`production` in the name:
```shell theme={null}
^.*?(production).*?$
```
* **Value map**: Enter key:value pairs to map additional metadata within your
custom field. For example, map `"warn"` to `"Warning"` and `"err"` to `"Error"`
for different severity types.
To add more value mappings, click **Add value mapping** and then define the
value mapping.
7. Click **Save** to save the custom field mapping definition.
8. Click the **Code config** tab and
[use the Code Config tool](/tooling/gitops#use-the-code-config-tool) to apply your
changes.
To use [Chronoctl](/tooling/chronoctl) to create custom field mappings, use this
command:
```shell theme={null}
chronoctl log-ingest-config create
```
1. Run the following command to generate a sample configuration you can use as a
template:
```shell theme={null}
chronoctl log-ingest-config scaffold
```
In the template, `kind: LogIngestConfig` defines an individual log ingest
configuration.
2. Use the `custom_field_normalization` object to define the plain text parser to run
on your log data.
3. With a completed definition, submit it with:
```shell /FILE_NAME/ theme={null}
chronoctl log-ingest-config create -f FILE_NAME
```
Replace *`FILE_NAME`* with the name of the YAML definition file you want to use.
When you run `terraform plan` to generate an execution plan, Chronosphere automatically
tests configurations that include notification policies by submitting them as dry runs.
For details, see the
[Terraform provider](/tooling/infrastructure/terraform#validate-plans-with-dry-runs)
documentation.
To create a custom field mapping with [Terraform](/tooling/infrastructure/terraform):
1. Create a Terraform resource with the `chronosphere_log_ingest_config` resource
type, followed by a resource name.
2. Use the `custom_field_normalization` object to define the custom field mapping to
run on your log data.
3. Add the definition to a Terraform file.
4. Run this command to create the resource:
```shell theme={null}
terraform apply
```
To complete this action with the Chronosphere API, use the
[`CreateLogIngestConfig`](/tooling/api-info/definition/operations/CreateLogIngestConfig)
endpoint.
Use the `"custom_field_normalization"` object to define the custom field mapping to
run on your log data.
Because the Chronosphere API requires authentication, include an API token with your
`curl` request, as shown in the following example. For more details, see
[Create an API token](/tooling/api-info#create-an-api-token).
```shell /"TOKEN"/ /INSTANCE/ /METHOD/ /ENDPOINT_PATH/ theme={null}
export CHRONOSPHERE_API_TOKEN="TOKEN"
export CHRONOSPHERE_DOMAIN="INSTANCE.chronosphere.io"
curl -H "API-Token: ${CHRONOSPHERE_API_TOKEN}" \
-X METHOD "https://${CHRONOSPHERE_DOMAIN}/ENDPOINT_PATH"
```
Replace the following:
* *`TOKEN`*: Your API token.
* *`INSTANCE`*: The subdomain name for your organization's Observability Platform instance.
* *`METHOD`*: The HTTP method to use with the request, such as `GET` or `POST`.
* *`ENDPOINT_PATH`*: The specific endpoint you want to access.
## Edit custom field mappings
When creating or editing a custom field mapping, you can
[use the Code Config tool](/tooling/gitops#use-the-code-config-tool)
to view code representations of a field mapping. The displayed code also responds to
changes you make in the **Visual editor** tab.
Select from the following methods to edit custom field mappings. After selecting a
field mapping to edit, you can change the field mappings order. Fields are matched in
priority order, so an order of `1` has higher precedence over an order of `6`.
1. In the navigation menu, click ** Go to Admin**
and then select
** Control > Logs Ingest**.
2. In the **Field normalization** section, click the custom field mapping you want to
edit.
3. In the **Edit field mapping** drawer, change your custom field mapping.
4. To reorder field mappings, drag and drop them to change the order.
5. Click **Save**. Observability Platform saves changes to your custom field mapping.
6. Click the **Code config** tab and
[use the Code Config tool](/tooling/gitops#use-the-code-config-tool) to apply your
changes.
To update custom field mappings with [Chronoctl](/tooling/chronoctl), use this
command:
```shell theme={null}
chronoctl log-ingest-config update
```
1. Update the custom field mapping definition file.
2. Run the following command to submit the changes:
```shell /FILE_NAME/ theme={null}
chronoctl log-ingest-config update -f FILE_NAME.yaml
```
Replace *`FILE_NAME`* with the name of the YAML definition file you want to use.
Users can modify Terraform-managed resources only by using Terraform.
[Learn more](/tooling/infrastructure/terraform#prevent-changes-to-managed-resources).
To edit custom field mappings using [Terraform](/tooling/infrastructure/terraform):
1. Create or edit a Terraform file that updates the resource's existing properties.
2. Run this command to apply the changes:
```shell theme={null}
terraform apply
```
You can also [use the Code Config tool](/tooling/gitops#use-the-code-config-tool) to
view the custom field mapping's Terraform representation.
To complete this action with the Chronosphere API, use the
[`UpdateLogIngestConfig`](/tooling/api-info/definition/operations/UpdateLogIngestConfig)
endpoint.
Because the Chronosphere API requires authentication, include an API token with your
`curl` request, as shown in the following example. For more details, see
[Create an API token](/tooling/api-info#create-an-api-token).
```shell /"TOKEN"/ /INSTANCE/ /METHOD/ /ENDPOINT_PATH/ theme={null}
export CHRONOSPHERE_API_TOKEN="TOKEN"
export CHRONOSPHERE_DOMAIN="INSTANCE.chronosphere.io"
curl -H "API-Token: ${CHRONOSPHERE_API_TOKEN}" \
-X METHOD "https://${CHRONOSPHERE_DOMAIN}/ENDPOINT_PATH"
```
Replace the following:
* *`TOKEN`*: Your API token.
* *`INSTANCE`*: The subdomain name for your organization's Observability Platform instance.
* *`METHOD`*: The HTTP method to use with the request, such as `GET` or `POST`.
* *`ENDPOINT_PATH`*: The specific endpoint you want to access.
## Delete custom field mappings
Select from one of the following methods to delete custom field mappings.
1. In the navigation menu, click ** Go to Admin**
and then select
** Control > Logs Ingest**.
2. In the **Edit field mapping** drawer, in the row of the field mapping you want to
delete, click the **** trash can icon.
3. Click **Save** to save your changes.
4. Click the **Code config** tab and
[use the Code Config tool](/tooling/gitops#use-the-code-config-tool) to apply your
changes.
Observability Platform deletes your custom field mapping.
To delete a custom field mapping with [Chronoctl](/tooling/chronoctl), use this
command:
```shell theme={null}
chronoctl log-ingest-config delete
```
Users can modify Terraform-managed resources only by using Terraform.
[Learn more](/tooling/infrastructure/terraform#prevent-changes-to-managed-resources).
To delete a resource that's managed by [Terraform](/tooling/infrastructure/terraform):
1. Edit your Terraform configuration file to remove the pre-existing resource
definition.
2. Run this command to remove the resource from Observability Platform:
```shell theme={null}
terraform apply
```
To complete this action with the Chronosphere API, use the
[`DeleteLogIngestConfig`](/tooling/api-info/definition/operations/DeleteLogIngestConfig)
endpoint.
Because the Chronosphere API requires authentication, include an API token with your
`curl` request, as shown in the following example. For more details, see
[Create an API token](/tooling/api-info#create-an-api-token).
```shell /"TOKEN"/ /INSTANCE/ /METHOD/ /ENDPOINT_PATH/ theme={null}
export CHRONOSPHERE_API_TOKEN="TOKEN"
export CHRONOSPHERE_DOMAIN="INSTANCE.chronosphere.io"
curl -H "API-Token: ${CHRONOSPHERE_API_TOKEN}" \
-X METHOD "https://${CHRONOSPHERE_DOMAIN}/ENDPOINT_PATH"
```
Replace the following:
* *`TOKEN`*: Your API token.
* *`INSTANCE`*: The subdomain name for your organization's Observability Platform instance.
* *`METHOD`*: The HTTP method to use with the request, such as `GET` or `POST`.
* *`ENDPOINT_PATH`*: The specific endpoint you want to access.
## Target nested or flat fields with dot notation
The `source` selectors for custom field mappings support dot notation to read values
from fields inside nested structures. How Observability Platform interprets a selector
depends on whether you enclose the segments in brackets in double quotes:
* An unbracketed dot splits the selector into path segments, which traverses nested
fields. A source of `metadata.code` reads a `code` field nested inside a `metadata`
object.
* A bracket-quoted segment such as `["metadata.code"]` treats the quoted string as a
single, literal field name that contains a dot. This reads a flat field named
`metadata.code` instead of a nested structure.
Consider a custom field mapping that populates a `status_code` target from a source
field.
To read from a nested field, set the source selector to `metadata.code`:
```json theme={null}
{
"target": "status_code",
"normalization": {
"source": [
{ "selector": "metadata.code" }
]
}
}
```
This selector reads the value from the nested `code` field:
```json theme={null}
{
"metadata": {
"code": "500"
}
}
```
To read from a single flat field whose name contains a dot, enclose the source
selector in double quotes and brackets:
```json theme={null}
{
"target": "status_code",
"normalization": {
"source": [
{ "selector": "[\"metadata.code\"]" }
]
}
}
```
This selector reads the value from one literal field name:
```json theme={null}
{
"metadata.code": "500"
}
```
## Examples
The following examples show common `custom_field_normalization` patterns. Each object
is an element of the `custom_field_normalization` array.
### Standardize and enrich data
Normalize values from several fields that all relate to different environments, such
as `environment` and `region`. You can then add optional mappings for individual
environments, such as mapping `dev` to `development`.
```json theme={null}
{
"target": "environment",
"normalization": {
"source": [
{
"selector": "env"
},
{
"selector": "region"
}
],
"default_value": "production",
"value_map": {
"dev": "development",
"prod": "production",
"stg": "staging"
}
}
}
```
### Consolidate values from multiple fields
Map a single `region` field from whichever source field is present, checking `region`
first and then `datacenter`. If neither contains a value, apply a default value.
```json theme={null}
{
"target": "region",
"normalization": {
"source": [
{ "selector": "region" },
{ "selector": "datacenter" }
],
"default_value": "us-east-1"
}
}
```
### Standardize values with a value map
Normalize inconsistent HTTP status codes into a smaller set of status classes, and
label anything unmatched as `unknown`.
```json theme={null}
{
"target": "status_class",
"normalization": {
"source": [
{ "selector": "http.status_code" }
],
"value_map": {
"200": "success",
"201": "success",
"400": "client_error",
"404": "client_error",
"500": "server_error",
"503": "server_error"
},
"default_value": "unknown"
}
}
```
### Extract a value with a regular expression
Use `sanitize_patterns` to extract a substring from a larger field. Each pattern must
contain exactly one capturing group, and Observability Platform uses that group as the
result. This example extracts an uppercase log level from a free-form `message` field.
```json theme={null}
{
"target": "log_level",
"normalization": {
"source": [
{ "selector": "message" }
],
"sanitize_patterns": ["^.*level=([A-Z]+).*$"],
"default_value": "INFO"
}
}
```
### Map a value from a nested field
Use dot notation to read a value from a nested structure, such as `user.id` inside
`{"user": {"id": "..."}}`. Sources are still matched in priority order.
```json theme={null}
{
"target": "userID",
"normalization": {
"source": [
{ "selector": "user.id" },
{ "selector": "actor.userId" }
]
}
}
```
### Map a value from a flat field that contains dots
If a source field name already contains dots, bracket-quote the selector so
Observability Platform treats it as one literal field name instead of a nested path.
```json theme={null}
{
"target": "team",
"normalization": {
"source": [
{ "selector": "[\"kubernetes.labels.team\"]" }
],
"default_value": "unassigned"
}
}
```