> ## Documentation Index > Fetch the complete documentation index at: https://docs.chronosphere.io/llms.txt > Use this file to discover all available pages before exploring further. # Monitor data model > Understand the YAML data model for monitors, including how teams, collections, monitors, and notification policies relate. Monitors use a custom definition in YAML format. When [adding or editing](/investigate/alerts/monitors) a monitor in Chronosphere Observability Platform, click the [Code Config tool](/tooling/gitops#use-the-code-config-tool) to view code representations of a monitor for Terraform, Chronoctl, and the Chronosphere API. Whichever method you use to create a monitor, the monitor structure uses the following data model. A full list of values is available [in the API](/tooling/api-info/definition/operations/CreateMonitor). ```mermaid actions={false} theme={null} flowchart TD Team["Team"] ==> CollectionsStack Team ==> PoliciesStack CollectionsStack ==> MonitorsStack CollectionsStack ==> PoliciesStack MonitorsStack ==> PoliciesStack %% Stacked effect for Collections subgraph CollectionsStack["Collections"] direction LR subgraph Collections["Contain team resources"] C1["Collection A"] C2["Collection B"] C3["Collection C"] end end subgraph MonitorsStack["Monitors"] direction LR subgraph Monitors["When to trigger alerts"] M1["Monitor 1"] M2["Monitor 2"] M3["Monitor 3"] end end subgraph PoliciesStack["Notification Policies"] direction LR subgraph Policies["How to route alerts"] P1["Policy X"] P2["Policy Y"] P3["Policy Z"] end end %% Styles classDef green fill:#2fbf71,stroke-width:2px,stroke:#2fbf71,color:#FFFFFF,font-weight:bold; class Team,CollectionsStack,MonitorsStack,PoliciesStack green ``` ## Collection A [collection](/administer/collections) is an organization of resources associated with a [team](/administer/accounts-teams/teams). All monitors must belong to exactly one collection. Each team can own many [collections](/administer/collections), which are a group of resources like monitors and dashboards. The identified team owns the systems that the associated dashboards and monitors track. Each collection can have either a default [notification policy](/investigate/alerts/notifications/policies) for all monitors, or you can set notification policies explicitly for each monitor. These policies apply a function to time series data that determines whether an alert triggers. ## Name A unique descriptive name for the monitor. You can change a monitor's name after it's created. Monitor names are static strings, so they can't include label variables such as `$labels.LABEL_NAME`. ## Slug A unique identifier that you can't change after creating the monitor. Observability Platform generates a slug when you create a monitor. Slugs have a maximum length of 254 alphanumeric characters. ## Notification policy The notification policy field determines which [notification policy](/investigate/alerts/notifications/policies) to use at a particular alert severity. You define the policies in a collection. When you create a collection with Chronoctl or Terraform, you can reference a notification policy in any collection. ## Notification template You can optionally customize the title and description of notifications when this monitor's alerts trigger or resolve. Set a `notification_template` object on the monitor with `title` and `description` template strings. Observability Platform evaluates those strings at notification time using alert context, with Go `text/template` syntax. Configure the field in the monitor UI, or through the Chronosphere API, Chronoctl, or Terraform. If a monitor sets a `notification_template`, it overrides any template information in the notifier to which the notification is sent. For variables, functions, and examples, see [Notification templates](/investigate/alerts/monitors/notification-templates). For the exact field shapes, see the monitor resource on [`CreateMonitor`](/tooling/api-info/definition/operations/CreateMonitor). In Chronoctl YAML, `notification_template` exists in the `spec` section of the monitor. The same field names apply in Terraform and in the API request body. ```yaml theme={null} spec: notification_template: title: "[{{.Severity}}] {{.Labels.service}} threshold exceeded" description: "{{.Labels.env}} crossed {{.ThresholdOp}} {{.Threshold}}" ``` ## Labels Labels are user-defined key-value pairs associated with a monitor. Labels identify, categorize, and route alerting time series to specific notifiers through notification policies. Labels don't affect monitor performance. ## Annotations Annotations are notes you can [add to a monitor](/investigate/alerts/monitors/monitor-annotations), such as runbooks, summaries, and descriptions. URL-valued annotations render as [templated links](/investigate/alerts/monitors/annotation-links) on monitor and SLO pages. Annotations help on-call engineers troubleshoot by surfacing links to related resources or displaying a summary that encapsulates the meaning of the overall query without having to scan the entire Series Legend. Annotation keys are validated against the same regular expression as [Prometheus metric names and labels](https://prometheus.io/docs/concepts/data_model/#metric-names-and-labels). Annotation values are arbitrary strings. For example, you can create a summary for a monitor that uses labels from the monitor query. Consider the following monitor query that matches on labels for `kube_pod_status`: ```text theme={null} kube_pod_status{cluster="staging",namespace="test",pod=~"staging-[0-9a-z]+-[0-9a-z]+"} ``` You can create an annotation that uses variables to substitute values from this monitor query. For example: ```text theme={null} Cluster: {{$labels.cluster}} | Namespace: {{$labels.namespace}} | Pod: {{$labels.pod}} is stale after 10m | Value: {{$value}} ``` When you save your monitor, the annotation replaces variables with values from the monitor query and renders as: ```text theme={null} Cluster: staging | Namespace: test | Pod: staging-37s799fcsj-ekg12 is stale after 10m | Value: 1 ``` For a list of available variables, see [Use annotations with monitors](/investigate/alerts/monitors/monitor-annotations). You can also create annotations for monitors that link to trace metrics, which lets you provide direct links for your on-call engineers to help diagnose issues. For more information, refer to [linking to tracing data](/investigate/querying/create-links). Annotation values support Markdown formatting, including bold text (`**text**`), inline code (`` `code` ``), named links (`[label](url)`), and plain HTTP URLs. Annotation values can also span multiple lines. Newlines separate paragraphs in the rendered output. ## Notifiers [Notifiers](/investigate/alerts/notifications/notifiers) can reference monitor labels by using the `{{ .CommonLabels.LABEL }}` variable, replacing *`LABEL`* with the monitor label name. Notifiers can reference monitor annotations by using the `{{ .CommonAnnotations. }}` variable, replacing `` with the monitor annotation name. Building on the previous example, you can create a Slack notifier that references the `summary` annotation: ```yaml Chronoctl example icon="square-terminal" theme={null} api_version: v1/config kind: Notifier spec: name: slack-summary-notifier slug: slack-notifier slack: # ... text: "Summary: {{ .CommonAnnotations.summary }}" # ... ``` ## Query A Prometheus or Graphite query, and a check interval of 15, 30, or 60 seconds that defines how often to check the latest data against configured conditions. When querying monitors, recent data is typically the most relevant. For this reason, Chronosphere recommends the `topk` function instead of the `head_max` function. The `topk` function takes the top `K` data points at query runtime and returns a filtered list of data points. The `head_max` function takes the maximum values for each time series across the entire query time range, selects the top `K` time series based on the maximum value, and then returns the filtered list of time series. ## Signals [Signals](/investigate/alerts/notifications/signals) are an optional group of notifications that use a unique combination of labels. When you create a monitor and enable the signals option for notifications, you can add the labels from the monitor's time series to the group. The following query parameters are available for signals: | Parameter | Configurable? | Default | | -------------------------------------------------------------------------------- | ------------- | ---------------- | | `group_wait` | No | 10 seconds | | `group_interval` | No | 5 minutes | | [`repeat_interval`](/investigate/alerts/notifications/policies#repeat-intervals) | Yes | `3600` (seconds) | | [`disable_repeat`](/investigate/alerts/notifications/policies#repeat-intervals) | Yes | `false` | ## Conditions Checks run against every time series resulting from the query. You define a condition and sustain period and can assign the resulting alert a severity (warning or critical). If a time series triggers that condition for the sustain period, Observability Platform generates an alert. Evaluating a sustain period of greater than 0 seconds requires at least two checks, defined in the query check interval, before triggering an alert. For example, if the sustain is five seconds, but the check interval is 30 seconds, the alert isn't triggered for at least 30 seconds, the time it takes for two checks to occur. A sustain period of zero seconds causes the alert to trigger after only one check, but might lead to noisy alerts. ### Missing data conditions A monitor can have different conditions that indicate missing data. For more information, see the [CreateMonitor](/tooling/api-info/definition/operations/CreateMonitor) endpoint. A `NOT_EXISTS` condition triggers only when the entire monitor query returns no time series. Use a `SIGNAL_NOT_EXISTS` condition when a missing data alert should use the same [signal grouping](/investigate/alerts/notifications/signals) as the rest of the monitor. `SIGNAL_NOT_EXISTS` triggers when a specific signal no longer has matching series (for example after series for one label value drop out while others remain). That behavior differs from `NOT_EXISTS`, which only considers whether the full query result is empty. Monitors can have the following signals to group time series: * Per monitor (no `signal_grouping`, or equivalent): same as `NOT_EXISTS`. The alert triggers when every series from the query is missing. * Per signal (`label_names`): the alert triggers when every series in a signal is missing (all series that share that signal's label set). * Per series (`signal_per_series`): the alert triggers when any individual series disappears. For example, a **Prometheus** monitor runs `sum by (cluster) (error_rate())` with signal grouping on `cluster`, so each cluster is its own signal. 1. While `cluster="dev"` and `cluster="prod"` both return series, neither operator triggers a missing data alert from that condition alone. 2. If series for `prod` disappear but `dev` still returns data, `SIGNAL_NOT_EXISTS` can alert for the `prod` signal. `NOT_EXISTS` doesn't trigger, because the query still returns results. 3. If `dev` then disappears, the query returns no time series. `SIGNAL_NOT_EXISTS` can alert for the `dev` signal as well, and `NOT_EXISTS` triggers because the entire query is empty. `SIGNAL_NOT_EXISTS` alerts can resolve without user intervention after the missing data condition clears. `SIGNAL_NOT_EXISTS` works for **Prometheus** monitors, not for **Graphite** queries. The **Sustain** field is required and must be between five minutes and 24 hours. Omit `value` (or set it to zero), the same as for `EXISTS` and `NOT_EXISTS`. Use `SIGNAL_NOT_EXISTS` only on default conditions, not in [condition overrides](#condition-overrides). Resolve thresholds (`resolve_value`) aren't supported for this operator. In Chronoctl YAML and Terraform, set `op` to `NOT_EXISTS` or `SIGNAL_NOT_EXISTS` on a condition under `series_conditions`, with the sustain fields your format expects and with `value` omitted or zero. The [Chronoctl](#chronoctl-example-prometheus) and [Terraform](#terraform-example-prometheus) examples show the full monitor definition. In the Chronoctl YAML, use the inline comments before the sample threshold conditions when replacing or adding a missing data condition. In Terraform, use the same `series_conditions` layout and the comments on `op` and `value` in the example resource as a guide. ### Resolve threshold You can set a resolve threshold on a condition so an alert clears at a different value than where it triggers. Resolve thresholds reduce flapping when a metric repeatedly triggers and resolves while crossing a single threshold, instead of staying in a steady passing or alerting range. If you omit a resolve threshold, the alert resolves using the same threshold and comparison as the trigger. Resolve thresholds are valid only when **Alert when value** uses one of these comparisons: greater than `>`, greater than or equal to `>=`, less than `<`, or less than or equal to `<=`. In Chronoctl YAML, Terraform, and the API, those same comparisons are often written as `GT` (greater than), `GEQ` (greater than or equal to), `LT` (less than), and `LEQ` (less than or equal to). The value of the resolve threshold must be opposite the value set for the alert: * For greater than `>` or greater than or equal to `>=`, the resolve threshold must be less than or equal to the trigger threshold. For example, trigger when the value is greater than `10` and set the resolve threshold to `5` so the alert clears when the value is at most `5`. * For less than `<` or less than or equal to `<=`, the resolve threshold must be greater than or equal to the trigger threshold. For example, trigger when the value is less than `10` and set the resolve threshold to `10` so the alert clears when the value is at least `10`. A resolve threshold is separate from the [resolve window](#resolve-window), which is a time-based setting for how long the query must stay clear before the alert resolves. How you configure `resolve_value` depends on how you create the monitor: * Observability Platform: In the **Conditions** section, set the optional resolve threshold for each numeric comparison (see [Create a monitor](/investigate/alerts/monitors#create-a-monitor)). * Terraform: Optional `resolve_value` block (`enabled` and `value`) inside each `condition` in `series_conditions`. * Chronoctl or the [`CreateMonitor`](/tooling/api-info/definition/operations/CreateMonitor) API: Optional `resolve_value` object with `enabled` and `value` on each condition. See the API definition for the condition schema. ### Condition overrides Defines [overrides](/investigate/alerts/monitors#override-a-monitor-alert) for specific conditions regarding the selected monitor. Use the `EXACT_MATCHER_TYPE` matcher to override exact key-value pairs, or the `REGEXP_MATCHER_TYPE` matcher to match with regular expressions. These are the only supported matcher types for condition overrides. ### Resolve window When you have an alert that's repeatedly resolving and continuing to trigger, define a resolve window. Within the resolve window, Observability Platform suppresses additional triggers for the same alert until the alert hasn't triggered for the defined duration. * Alerts configured with the `sustain` field trigger if the PromQL expression for the alert returns a populated value for the defined duration. * A triggered alert with a defined resolve window resolves after the PromQL expression returns an empty value for the defined duration. Use resolve windows with conditions or condition overrides. The resolve window defaults to `0s`. How you define the resolve window differs depending on how you [create the monitor](/investigate/alerts/monitors#create-a-monitor): * Terraform: Define the resolve window with the `resolve_sustain` field in the `series_conditions` object. * Chronoctl or the [CreateMonitor API endpoint](/tooling/api-info/definition/operations/CreateMonitor): Define the resolve window with the `resolve_sustain_secs` field in the `series_conditions` object. * Observability Platform: In the **Conditions** pane, define the resolve window with the **Resolve when clear for** field. The duration defined in the resolve window definition behaves like the Prometheus [`keep_firing_for`](https://prometheus.io/docs/prometheus/latest/configuration/recording_rules/#rule) field. Changing a monitor's configuration might result in resolve windows not being respected. This behavior can cause an alert to be triggered again, which would enter a new resolve window. When a monitor's resolve configuration is modified during an active alert it can take up to the maximum of whichever is higher, the previous resolve sustain setting or the current resolve sustain setting, to resolve the alert. ## Schedule By default, Observability Platform monitors are constantly active. Limit a monitor's activity to specific time ranges within a weekly schedule. A schedule defines weekly time ranges when Observability Platform runs the monitor's queries and evaluates its conditions. The monitor is inactive outside those ranges. Define a schedule when you [create](/investigate/alerts/monitors#create-monitors) or [edit](/investigate/alerts/monitors#edit-a-monitor) a monitor. Avoid ending a schedule's time range at 23:59. Using this value can inadvertently stop a monitor from executing during the last minute of the day. To run the monitor through the end of the day, use `24:00`. To span days, end the first day at `24:00` and begin the next day at `00:00`. A monitor that's actively sending alerts resolves after entering an inactive period in a defined schedule. When a monitor re-enters an active state, the monitor must meet defined conditions to trigger a new alert. ## Configuration examples The following examples show complete monitor definitions for each tool and query type. Each example groups series into signals based on the `source` and `service_environment` label keys, and includes a `schedule` that runs the monitor on Mondays from 7:00 to 10:10 and 15:00 to 22:30, and Thursdays from 21:15 through the end of the day. All times are in UTC. If you define `label_names` in the `signal_grouping` section, enter a label name that differs in name and case from the label you enter in the `labels` section. For example, if you enter `environment` as a key in the `labels` section, you might use `Environments` in the `label_names` section. The following YAML configuration defines a Prometheus monitor named `Disk Getting Full`. The `series_conditions` trigger a warning notification when the value exceeds 30 for more than 300 seconds, and a critical notification when it exceeds 60 for more than 300 seconds. ```yaml expandable Chronoctl example icon="square-terminal" theme={null} api_version: v1/config kind: Monitor spec: # Required name of the monitor. Can be modified after the monitor is created. name: Disk Getting Full # PromQL query. If set, you can't set graphite_query. prometheus_query: max(disk:last{measurement="used_percent"}) by (source, service_environment, region) # Annotations are visible in notifications generated by this monitor. # You can template annotations with labels from notifications. annotations: key_1: "{{ $labels.job }}" # Slug of the collection the monitor belongs to. collection_slug: loadgen # Optional setting for configuring how often alerts are evaluated. # Defaults to 60 seconds. interval_secs: 60 # Labels are visible in notifications generated by this monitor, # and can be used to route alerts with notification overrides. labels: key_1: kubernetes_cluster # Optional notification policy used to route alerts generated by the monitor. notification_policy_slug: custom-notification-policy schedule: # The timezone of the time ranges. timezone: UTC weekly_schedule: monday: active: ONLY_DURING_RANGES # The time ranges that the monitor is active on this day. Required if # active is set to ONLY_DURING_RANGES. ranges: - # End time in the in format ":", such as "15:30". end_hh_mm: "15:00" # Start time in the in format ":", such as "15:30". start_hh_mm: "10:10" tuesday: active: NEVER wednesday: active: NEVER thursday: active: ONLY_DURING_RANGES # The time ranges that the monitor is active on this day. Required if ranges: # active is set to ONLY_DURING_RANGES. - # End time in the in format ":", such as "15:30". end_hh_mm: "24:00" # Start time in the in format ":", such as "15:30". start_hh_mm: "21:15" friday: active: NEVER saturday: active: NEVER sunday: active: NEVER # Conditions evaluated against each queried series to determine the severity of each series. series_conditions: defaults: critical: # List of conditions to evaluate against a series. # Only one condition must match to assign a severity to a signal. conditions: # Missing data: `NOT_EXISTS` (entire query empty) or `SIGNAL_NOT_EXISTS` # (signal-aware; sustain must be from 5m to 24h; not for Graphite). - op: GT # How long the op operation needs to evaluate for the condition # to evaluate to true. sustain_secs: 300 # The value to compare to the metric value using the op operation. value: 60 # Optional. resolve at a different threshold (GT/GEQ/LT/LEQ only). # resolve_value: # enabled: true # value: 50 # How long the operation needs to evaluate false to resolve resolve_sustain_secs: 60 warn: # List of conditions to evaluate against a series. # Only one condition must match to assign a severity to a signal. conditions: - op: GT # How long the op operation needs to evaluate for the condition # to evaluate to true. sustain_secs: 300 # The value to compare to the metric value using the op operation. value: 30 # Optional. resolve at a different threshold (GT/GEQ/LT/LEQ only). # resolve_value: # enabled: true # value: 20 # How long the operation needs to evaluate false to resolve resolve_sustain_secs: 60 # Defines how the set of series from the query are split into signals. signal_grouping: label_names: - source - service_environment # If true, each series will have its own signal and label_names can't be set. signal_per_series: false ``` The following YAML configuration defines a logs monitor named `Kubernetes errors in production us-west`. The `series_conditions` trigger a warning notification when the error count exceeds 30 for more than 300 seconds, and a critical notification when it exceeds 60 for more than 300 seconds. ```yaml expandable Chronoctl example icon="square-terminal" theme={null} api_version: v1/config kind: Monitor spec: # Required name of the monitor. Can be modified after the monitor is created. name: Kubernetes errors in production us-west # Logging query to return data for. logging_query: severity = "ERROR" AND kubernetes.cluster_name = "production-us-west" | make-series by service # Annotations are visible in notifications generated by this monitor. # You can template annotations with labels from notifications. annotations: key_1: "{{ $labels.job }}" # Slug of the collection the monitor belongs to. collection_slug: production-team # Optional setting for configuring how often alerts are evaluated. # Defaults to 60 seconds. interval_secs: 60 # Labels are visible in notifications generated by this monitor, # and can be used to route alerts with notification overrides. labels: key_1: kubernetes_cluster # Optional notification policy used to route alerts generated by the monitor. notification_policy_slug: custom-notification-policy schedule: # The timezone of the time ranges. timezone: UTC weekly_schedule: monday: active: ONLY_DURING_RANGES # The time ranges that the monitor is active on this day. Required if # active is set to ONLY_DURING_RANGES. ranges: - # End time in the in format ":", such as "15:30". end_hh_mm: "15:00" # Start time in the in format ":", such as "15:30". start_hh_mm: "10:10" tuesday: active: NEVER wednesday: active: NEVER thursday: active: ONLY_DURING_RANGES # The time ranges that the monitor is active on this day. Required if ranges: # active is set to ONLY_DURING_RANGES. - # End time in the in format ":", such as "15:30". end_hh_mm: "24:00" # Start time in the in format ":", such as "15:30". start_hh_mm: "21:15" friday: active: NEVER saturday: active: NEVER sunday: active: NEVER # Conditions evaluated against each queried series to determine the severity of each series. series_conditions: defaults: critical: # List of conditions to evaluate against a series. # Only one condition must match to assign a severity to a signal. conditions: # Missing data: `NOT_EXISTS` (entire query empty) or `SIGNAL_NOT_EXISTS` # (signal-aware; sustain must be from 5m to 24h; not for Graphite). - op: GT # How long the op operation needs to evaluate for the condition # to evaluate to true. sustain_secs: 300 # The value to compare to the metric value using the op operation. value: 60 # Optional. resolve at a different threshold (GT/GEQ/LT/LEQ only). # resolve_value: # enabled: true # value: 50 # How long the operation needs to evaluate false to resolve resolve_sustain_secs: 60 warn: # List of conditions to evaluate against a series. # Only one condition must match to assign a severity to a signal. conditions: - op: GT # How long the op operation needs to evaluate for the condition # to evaluate to true. sustain_secs: 300 # The value to compare to the metric value using the op operation. value: 30 # Optional. resolve at a different threshold (GT/GEQ/LT/LEQ only). # resolve_value: # enabled: true # value: 20 # How long the operation needs to evaluate false to resolve resolve_sustain_secs: 60 # Defines how the set of series from the query are split into signals. signal_grouping: label_names: - source - service_environment # If true, each series will have its own signal and label_names can't be set. signal_per_series: false ``` The following Terraform resource creates a Prometheus monitor that Terraform refers to by `infra`, with a human-readable name of `Infra Example monitor`. ```terraform expandable Terraform example icon="square-terminal" theme={null} resource "chronosphere_monitor" "infra" { name = "Infra Example monitor" # Reference to the collection the alert belongs to. collection_id = chronosphere_collection.infra.id # Override the notification policy. # By default, uses the policy from the collection_id. notification_policy_id = chronosphere_collection.infra_testing.id # Arbitrary set of labels to assign to the alert. labels = { "priority" = "sev-1" } # Arbitrary set of annotations to include in alert notifications. annotations = { "runbook" = "http://default-runbook" } # Interval at which to evaluate the monitor, for example 15s, 30s, or 60s. # Defaults to 60s. interval = "30s" query { # PromQL query to evaluate for the alert. # Alternatively, you can use graphite_expr instead. prometheus_expr = "sum (rate(grpc_server_handled_total{grpc_code!=\"OK\"}[1m])) by (app, grpc_service, grpc_method)" } # The remaining examples are optional signals specifying how to group the # series returned from the query. # No signal_grouping clause = Per monitor # signal_grouping with label_names set = Per signal for labels set # signal_grouping with signal_per_series set to true = Per series signal_grouping { # Set of labels names used to split series into signals. # Each unique combination of labels results in its own signal. label_names = ["app", "grpc_service"] # As an alternative to label_names, signal_per_series creates an alert for # every resulting series from the query. # signal_per_series = true } # Container for the conditions determining the severity of each series from the query. # The highest severity series of a signal determines that signal's severity. series_conditions { # Condition assigning a warn threshold for series above a certain threshold. condition { # Severity of the condition, which can be "warn" or "critical". severity = "warn" # Value to compare against each series from the query result. # For EXISTS, NOT_EXISTS, or SIGNAL_NOT_EXISTS, value must be zero or omitted. value = 5.0 # Operator to use when comparing the query result versus the threshold. # Valid values include GT, LT, LEQ, GEQ, EQ, NEQ, EXISTS, NOT_EXISTS, # SIGNAL_NOT_EXISTS. op = "GT" # Amount of time the query needs to fail the condition check before # an alert is triggered. Must be an integer. Accepts one of s (seconds), m # (minutes), or h (hours) as units. Optional. sustain = "240s" # Amount of time the query needs to no longer trigger before resolving. Must be # an integer. Accepts one of s (seconds), m (minutes), or h (hours) as units. resolve_sustain = "60s" } condition { severity = "critical" value = 10.0 op = "GT" sustain = "120s" resolve_sustain = "60s" # Optional. resolve at a different threshold (GT/GEQ/LT/LEQ only). resolve_value { enabled = true value = 5.0 } } # Multiple optional overrides can be defined for different sets of conditions # to series with matching labels. override { # One or more matchers for labels on a series. label_matcher { # Name of the label name = "app" # How to match the label, which can be "EXACT_MATCHER_TYPE" or # "REGEXP_MATCHER_TYPE". type = "EXACT_MATCHER_TYPE" # Value of the label. value = "dbmon" } condition { severity = "critical" value = 1.0 op = "GT" sustain = "60s" } } } # If you define a schedule, Observability Platform evaluates the monitor only during # the specified time ranges. The monitor is inactive during all unspecified # time ranges. # If you define an empty schedule, Observability Platform never evaluates the monitor. schedule { # Valid values: Any IANA timezone string timezone = "UTC" range { # Time range for the monitor schedule. Valid values for day can be full # day names, such as "Sunday" or "Monday". # Valid time values must be specified in the range of 00:00 to 24:00. day = "Monday" start = "07:00" end = "10:10" } range { day = "Monday" start = "15:00" end = "22:30" } range { day = "Thursday" start = "21:15" end = "24:00" } } } ``` The following Terraform resource creates a logs monitor that Terraform refers to by `k8s_production`, with a human-readable name of `Kubernetes errors in production us-west`. ```terraform expandable Terraform example icon="square-terminal" theme={null} resource "chronosphere_monitor" "k8s_production" { name = "Kubernetes errors in production us-west" # Reference to the collection the alert belongs to. collection_id = chronosphere_collection.k8s_production.id # Override the notification policy. # By default, uses the policy from the collection_id. notification_policy_id = chronosphere_collection.k8s_testing.id # Arbitrary set of labels to assign to the alert. labels = { "priority" = "sev-1" } # Arbitrary set of annotations to include in alert notifications. annotations = { "runbook" = "http://default-runbook" } # Interval at which to evaluate the monitor, for example 15s, 30s, or 60s. # Defaults to 60s. interval = "30s" query { # Logging query to evaluate. logging_query = "severity='ERROR' AND kubernetes.cluster_name='production-us-west' | make-series by service" } # The remaining examples are optional signals specifying how to group the # series returned from the query. # No signal_grouping clause = Per monitor # signal_grouping with label_names set = Per signal for labels set # signal_grouping with signal_per_series set to true = Per series signal_grouping { # Set of labels names used to split series into signals. # Each unique combination of labels results in its own signal. label_names = ["app", "grpc_service"] # As an alternative to label_names, signal_per_series creates an alert for # every resulting series from the query. # signal_per_series = true } # Container for the conditions determining the severity of each series from the query. # The highest severity series of a signal determines that signal's severity. series_conditions { # Condition assigning a warn threshold for series above a certain threshold. condition { # Severity of the condition, which can be "warn" or "critical". severity = "warn" # Value to compare against each series from the query result. # For EXISTS, NOT_EXISTS, or SIGNAL_NOT_EXISTS, value must be zero or omitted. value = 5.0 # Operator to use when comparing the query result versus the threshold. # Valid values include GT, LT, LEQ, GEQ, EQ, NEQ, EXISTS, NOT_EXISTS, # SIGNAL_NOT_EXISTS. op = "GT" # Amount of time the query needs to fail the condition check before # an alert is triggered. Must be an integer. Accepts one of s (seconds), m # (minutes), or h (hours) as units. Optional. sustain = "240s" # Amount of time the query needs to no longer trigger before resolving. Must be # an integer. Accepts one of s (seconds), m (minutes), or h (hours) as units. resolve_sustain = "60s" } condition { severity = "critical" value = 10.0 op = "GT" sustain = "120s" resolve_sustain = "60s" # Optional. resolve at a different threshold (GT/GEQ/LT/LEQ only). resolve_value { enabled = true value = 5.0 } } # Multiple optional overrides can be defined for different sets of conditions # to series with matching labels. override { # One or more matchers for labels on a series. label_matcher { # Name of the label name = "app" # How to match the label, which can be "EXACT_MATCHER_TYPE" or # "REGEXP_MATCHER_TYPE". type = "EXACT_MATCHER_TYPE" # Value of the label. value = "dbmon" } condition { severity = "critical" value = 1.0 op = "GT" sustain = "60s" } } } # If you define a schedule, Observability Platform evaluates the monitor only during # the specified time ranges. The monitor is inactive during all unspecified # time ranges. # If you define an empty schedule, Observability Platform never evaluates the monitor. schedule { # Valid values: Any IANA timezone string timezone = "UTC" range { # Time range for the monitor schedule. Valid values for day can be full # day names, such as "Sunday" or "Monday". # Valid time values must be specified in the range of 00:00 to 24:00. day = "Monday" start = "07:00" end = "10:10" } range { day = "Monday" start = "15:00" end = "22:30" } range { day = "Thursday" start = "21:15" end = "24:00" } } } ```