Configuration options

Config file

By default karma will try to read configuration file named karma.yaml from current directory. Configuration file uses YAML format and it needs to have .yaml extension. Custom filename and directory can be passed via command line flags or environment variables:

Example with flags:

karma --config.file docs/example.yaml

Example with environment variables:

CONFIG_FILE="docs/example.yaml"

Alertmanagers

alertmanager section allows setting Alertmanager servers that should be queried for alerts. You can configure one or more Alertmanager servers, alerts with identical label set will be deduplicated and labeled with each Alertmanager server they were observed at. This allows using karma to collect alerts from a pair of Alertmanager instances running in HA mode. Syntax:

alertmanager:
  interval: duration
  servers:
    - name: string
      uri: string
      external_uri: string
      timeout: duration
      proxy: bool
      tls:
        ca: string
        cert: string
        key: string
        insecureSkipVerify: bool
      headers:
        any: string

Note: there are multiple supported combination of URI settings which result in a slightly different behavior. Settings that control it are:

Breakdown of all combination of settings:

  1. Only uri is set:

    uri: http://localhost:123
    

    Karma would use those URIs for:

    Backend Silence management Silence links
    http://localhost:123 http://localhost:123 http://localhost:123
  2. Proxy mode is enabled:

    uri: http://localhost:123
    proxy: true
    

    Karma would use those URIs for:

    Backend Silence management Silence links
    http://localhost:123 Karma internal URI http://localhost:123
  3. external_uri is set, but proxy mode is disabled:

    uri: http://localhost:123
    external_uri: http://example.com
    

    Karma would use those URIs for:

    Backend Silence management Silence links
    http://localhost:123 http://example.com http://example.com
  4. Proxy mode is enabled and external_uri is set:

    uri: http://localhost:123
    proxy: true
    external_uri: http://example.com
    

    Karma would use those URIs for:

    Backend Silence management Silence links
    http://localhost:123 Karma internal URI http://example.com

Example with two production Alertmanager instances running in HA mode and a staging instance that is also proxied and requires a custom auth header:

alertmanager:
  interval: 1m
  servers:
    - name: production1
      uri: https://alertmanager1.prod.example.com
      timeout: 20s
      proxy: false
    - name: production2
      uri: https://alertmanager2.prod.example.com
      timeout: 20s
      proxy: false
    - name: staging
      uri: https://alertmanager.staging.example.com
      timeout: 30s
      proxy: true
      tls:
        ca: /etc/ssl/staging-ca.crt
      headers:
        X-Auth-Token: aValidToken
    - name: protected
      uri: https://alertmanager-auth.prod.example.com
      timeout: 20s
      tls:
        cert: /etc/ssl/client.pem
        key: /etc/ssl/client.key
    - name: self-signed
      uri: https://test.example.com
      tls:
        insecureSkipVerify: true

Defaults:

alertmanager:
  interval: 1m
  servers: []

There is no default for alertmanager.servers and it’s a required option for setting multiple Alertmanager servers. For cases where only a single server needs to be configured without a config file see Simplified Configuration.

Alert acknowledgement

Prometheus Alertmanager allows alerts to be in 3 states:

A silence rule can be used to mark an alert as acknowledged and being worked on. To simplify creating of such silences karma provides a one click button that will create a silence matching alert group it was clicked for. alertAcknowledgement allows to enable this feature and customize it’s configuration. Syntax:

alertAcknowledgement:
  enabled: bool
  duration: duration
  author: string
  commentPrefix: string

Defaults:

alertAcknowledgement:
  enabled: false
  duration: 15m0s
  author: karma
  commentPrefix: ACK!

A common problem is setting a correct duration for the silence. If set for too short it can expire before the issue is resolved, and will require re-silencing all the alerts. If set for too long it mask the same problem reoccurring in the future. This requires user to expire the silence once the issue is resolved.

kthxbye is a tiny daemon that can help with managing short lived acknowledged silences. It will continuously extend short lived acknowledgement silences if there are alerts firing against those silences, which means that the user doesn’t need to worry about setting proper duration for such silences. To use it run an instance of kthxbye with every alertmanager instance or cluster and configure it to use the same comment prefix as commentPrefix. With this setup when user clicks to acknowledge an alert karma will create a short lived silence and kthxbye will keep that silence in Alertmanager until there are no alerts matching it, meaning that the issue was resolved.

Annotations

annotations section allows configuring how alert annotation are displayed in the UI. Syntax:

annotations:
  default:
    hidden: bool
  hidden: list of strings
  visible: list of strings
  keep: list of strings
  strip: list of strings

The difference between hidden/visible and keep/strip is that hidden annotations are still accessible, but they are shown in the UI collapsed by default (only name is visible, value is shown after clicking), while stripped annotations are removed entirely and never presented to the user.

Example where all annotations except summary are hidden by default. If there are additional annotation keys user will need to click on the + icon to see them.

annotations:
  default:
    hidden: true
  hidden: []
  visible:
    - summary
  keep: []
  strip:
    - help
    - verylong

Example where all annotations except details are visible by default. If details annotation is present on any alert user will need to click on the + icon to see it. Additionally secret annotation is stripped and never shown in the UI.

annotations:
  default:
    hidden: false
  hidden:
    - details
  visible: []
  keep: []
  strip:
    - secret

Defaults:

annotations:
  default:
    hidden: false
  hidden: []
  visible: []

Filters

filters section allows configuring default set of filters used in the UI.

Syntax:

filters:
  default: list of strings

Example:

filters:
  default:
    - "@state=active"
    - severity=critical

Defaults:

filters:
  default: []

Grid

grid section allows customizing how alert grid is rendered in the UI. Sorting configuration can be overridden by each user via UI settings. Syntax:

grid:
  sorting:
    order: string
    reverse: bool
    label: string
    customValues:
      labels: dict

Defaults:

grid:
  sorting:
    order: startsAt
    reverse: true
    label: alertname
    customValues:
      labels: {}

Example with sorting using severity label and value mappings for it:

grid:
  sorting:
    order: label
    reverse: false
    label: severity
    customValues:
      labels:
        severity:
          critical: 1
          warning: 2
          info: 3

Karma

karma section allows configuring miscellaneous internal options.

Syntax:

karma:
  name: string

Defaults:

karma:
  name: karma

Labels

labels section allows configuring how alert labels will be rendered in the UI. All labels will be parsed when collecting alerts from Alertmanager API and used when deduplicating alerts, but some labels aren’t useful to users and so can be removed from the UI, this is controlled by keep and strip options. colors section allows configuring which labels should have colors applied to label background in the UI. Colors can help visually identify alerts with shared labels, for example coloring hostname label will allow to quickly spot all alerts for the same host. Syntax:

labels:
  color:
    static: list of strings
    unique: list of strings
    custom:
      foo:
        - value: string
          value_re: string
          color: string
  keep: list of strings
  strip: list of strings

Example with static color for the job label (every job label will have the same color regardless of the value) and unique color for the @receiver label (every @receiver label will have color unique for each value).

labels:
  color:
    static:
      - job
    unique:
      - "@receiver"

Example where task_id label is ignored by karma:

labels:
  keep: []
  strip:
    - task_id

Example where all but instance and alertname labels are allowed:

labels:
  keep:
    - alertname
    - instance
  strip: []

Example where severity label will have a red color for critical, yellow for warning and blue for info:

labels:
  color:
    custom:
      "@alertmanager":
        - value: prod
          color: "#e6e"
      severity:
        - value: info
          color: "#87c4e0"
        - value: warning
          color: "#ffae42"
        - value: critical
          color: "#ff220c"

Example with a regex value, info, warning and critical will get colors as below, but any value not matching those 3 values will use the color from .*:

labels:
  color:
    custom:
      severity:
        - value: info
          color: "#87c4e0"
        - value: warning
          color: "#ffae42"
        - value: critical
          color: "#ff220c"
        - value_re: ".*"
          color: "#736598"

Note: be sure to set fallback values at the end of the list, so they’re only evaluated if there’s no exact value match

Defaults:

labels:
  color:
    static: []
    unique: []
    custom: {}
  keep: []
  strip: []

Listen

listen section allows configuring karma web server behavior. Syntax:

listen:
  address: string
  port: integer
  prefix: string

Example where karma would listen for HTTP requests on http://1.2.3.4:80/karma/

listen:
  address: 1.2.3.4
  port: 80
  prefix: /karma/

Defaults:

listen:
  address: "0.0.0.0"
  port: 8080
  prefix: /

Log

log section allows configuring logging subsystem. Syntax:

log:
  config: bool
  level: string
  format: string

Defaults:

log:
  config: true
  level: info
  format: text

Silences

silences section allows specifying to configure silence post post-processing. Syntax:

silences:
  comments:
    linkDetect:
      rules: list of link detection rules

Example where a string DEVOPS-123 inside a comment would be rendered as a link to a JIRA ticket https://jira.example.com/browse/DEVOPS-123.

silences:
  comments:
    linkDetect:
      rules:
        - regex: "(DEVOPS-[0-9]+)"
          uriTemplate: https://jira.example.com/browse/$1

Defaults:

jira: []

Receivers

receivers section allows configuring how alerts from different receivers are handled by karma. If alerts are routed to multiple receivers they can be duplicated in the UI, each instance will have different value for @receiver. Syntax:

receivers:
  keep: list of strings
  strip: list of strings

Example where alerts that are routed to the alertmanage2es receiver are ignored by karma.

receivers:
  strip:
    - alertmanage2es

Defaults:

receivers:
  strip: []

Sentry

sentry section allows configuring Sentry integration. See Sentry documentation for details. Syntax:

sentry:
  private: string
  public: string

Example:

sentry:
  private: https://<key>:<secret>@sentry.io/<project>
  public: https://<key>:<secret>@sentry.io/<project>

Silence form

silenceForm section allows customizing silence form behavior. author:populate_from_header subsection allows to configure fetching of author name used on the silence form from the request header. It can be used with setups where karma is deployed behind authentication proxy that adds some extra headers with username for all requests received by karma.

Syntax:

silenceForm:
  author:
    populate_from_header:
      header: string
      value_re: string
  strip:
    labels: list of strings

Example where job label won’t be auto populated onto the silence form and where the X-Auth header with value User foobar will set the default silence author to foobar.

silenceForm:
  author:
    populate_from_header:
      header: X-Auth
      value_re: ^User (.+)$
  strip:
    labels:
      - job

UI defaults

ui section allows configuring default values for UI settings controled via the configuration modal. Those defaults can be overwritten by use via UI controls.

Syntax:

ui:
  refresh: duration
  hideFiltersWhenIdle: bool
  colorTitlebar: bool
  theme: string
  minimalGroupWidth: integer
  alertsPerGroup: integer
  collapseGroups: string

Defaults:

ui:
  refresh: 30s
  hideFiltersWhenIdle: true
  colorTitlebar: false
  theme: "auto"
  minimalGroupWidth: 420
  alertsPerGroup: 5
  collapseGroups: collapsedOnMobile

Customizing karma

In order to keep the core code simple karma doesn’t support any way of extending provided functionality. There is however possibility to inject custom CSS & JavaScript code, which can be used to either override built in CSS styles or integrate with extra services, for example using error handlers other than Sentry.

custom:
  css: string
  js: string

Example:

custom:
  css: /theme/custom.css
  js: /assets/custom.js

Use at your own risk and be aware that used CSS class names might change without warning. This feature is provided as is without any guarantees.

Command line flags

Config file options are mapped to command line flags, so alertmanager:interval config file key is accessible as --alertmanager.interval flag, run karma --help to see a full list. Exceptions for passing flags:

There’s no support for configuring multiple Alertmanager servers using flags, but it’s possible to configure a single Alertmanager instance this way, see the Simplified Configuration section.

Environment variables

Environment variables are mapped in a similar way as command line flags, alertmanager:interval is accessible as ALERTMANAGER_INTERVAL env. Exceptions for passing flags:

There’s no support for configuring multiple alertmanager servers using environment variables, but it’s possible to configure a single Alertmanager instance this way, see the Simplified Configuration section.

Simplified Configuration

To configure multiple Alertmanager instances karma requires a config file, but for a single Alertmanager instance cases it’s possible to configure all Alertmanager server options that are set for alertmanager.servers config section using only flags or environment variables.

Alertmanager URI

To set the uri key from alertmanager.servers map ALERTMANAGER_URI env or --alertmanager.uri flag can be used. Examples:

ALERTMANAGER_URI=https://alertmanager.example.com karma
karma --alertmanager.uri https://alertmanager.example.com

Alertmanager external URI

To set the external_uri key from alertmanager.servers map ALERTMANAGER_EXTERNAL_URI env or --alertmanager.external_uri flag can be used. Examples:

ALERTMANAGER_EXTERNAL_URI=https://alertmanager.example.com karma
karma --alertmanager.external_uri https://alertmanager.example.com

Alertmanager name

To set the name key from alertmanager.servers map ALERTMANAGER_NAME env or --alertmanager.name flag can be used. Examples:

ALERTMANAGER_NAME=single karma
karma --alertmanager.name single

Alertmanager timeout

To set the timeout key from alertmanager.servers map ALERTMANAGER_TIMEOUT env or --alertmanager.timeout flag can be used. Examples:

ALERTMANAGER_TIMEOUT=10s karma
karma --alertmanager.timeout 10s

Alertmanager request proxy

To set the proxy key from alertmanager.servers map ALERTMANAGER_PROXY env or --alertmanager.proxy flag can be used. Examples:

ALERTMANAGER_PROXY=true karma
karma --alertmanager.proxy