Skip to main content

Steps to install Loki stack Loki and Grafana on a rke2 cluster

· 2 min read
Hreniuc Cristian-Alexandru

Here we will document how to install the Loki stack: Loki + Grafana to display our logs in grafana.

Preparations​

Add the grafana-contabo.domain.com or grafana-hetzner.domain.com to the IP of the cluester in cloudflare, we will need it, because we will generate a certificate for it using let's encrypt.

Install​

We will install it via Rancher using the helm chart.

First we will need to add the repository to Rancher. We should add this: https://grafana.github.io/helm-charts to Apps & Marketplace > Repositories > Create.

Go to Apps & Marketplace > Charts and search for loki-stack, install it using the following:

  • name - loki-stack
  • values - Use the yml from below. You will have to replace contabo with hetzner, if you install this on hetzner.
loki:
# https://github.com/grafana/helm-charts/blob/main/charts/loki/values.yaml
  enabled: true
  persistence:
    enabled: true
    storageClassName: nfs-master1-storage # https://github.com/grafana/helm-charts/blob/main/charts/loki/templates/statefulset.yaml#L145
    size: 15Gi
  # extraArgs:
  #   log.level: debug
grafana:
  enabled: true
  sidecar:
    datasources:
      enabled: true # https://github.com/grafana/loki/blob/88feda41a02f9c544d7476f61e296373e83fbe72/production/helm/loki-stack/templates/datasources.yaml#L1
  persistence:
    enabled: true # We should set it to true
    storageClassName: nfs-master1-storage
    size: 1Gi
  ingress:
    enabled: true
    annotations:
      kubernetes.io/ingress.class: nginx
      cert-manager.io/cluster-issuer: letsencrypt-staging
    hosts:
      - grafana-contabo.domain.com
    tls:
      - hosts:
        - grafana-contabo.domain.com
        secretName: grafana-contabo.domain.com-cert # Autogenerated

Note: Don't forget to change the dns based on the cluster.

To get the admin password for grafana, you should run:

kubectl get secret --namespace default loki-stack-grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo

The loki endpoint that should be used everywhere is http://loki-stack:3100, the DNS is the same as the name set when we installed the helm.

Import the grafana dashboard for Loki logs​

Go to https://grafana-contabo.domain.com/dashboard/import or https://grafana-hetzner.domain.com/dashboard/import and paste the json found here(removed). Add it to favorites.

The dashboard should be accessible here: https://grafana-contabo.domain.com/d/domain-com-loki/logs-for-domain-com-backend or https://grafana-hetzner.domain.com/d/domain-com-loki/logs-for-domain-com-backend.

Optional features​

If we want all logs from all containers to be sent to Grafana, we should enable the promtail component from the loki stack.

If we ever want logstash, to centralize the logs and store them, we can activate it.