Skip to main content

4 posts tagged with "grafana"

View All Tags

· 2 min read
Hreniuc Cristian-Alexandru

Here we will document how to install the Loki stack: Loki + Grafana to display our logs in grafana.


Add the or to the IP of the cluester in cloudflare, we will need it, because we will generate a certificate for it using let's encrypt.


We will install it via Rancher using the helm chart.

First we will need to add the repository to Rancher. We should add this: to Apps & Marketplace > Repositories > Create.

Go to Apps & Marketplace > Charts and search for loki-stack, install it using the following:

  • name - loki-stack
  • values - Use the yml from below. You will have to replace contabo with hetzner, if you install this on hetzner.
enabled: true
enabled: true
storageClassName: nfs-master1-storage #
size: 15Gi
# extraArgs:
# log.level: debug
enabled: true
enabled: true #
enabled: true # We should set it to true
storageClassName: nfs-master1-storage
size: 1Gi
enabled: true
annotations: nginx letsencrypt-staging
- hosts:
secretName: # Autogenerated

Note: Don't forget to change the dns based on the cluster.

To get the admin password for grafana, you should run:

kubectl get secret --namespace default loki-stack-grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo

The loki endpoint that should be used everywhere is http://loki-stack:3100, the DNS is the same as the name set when we installed the helm.

Import the grafana dashboard for Loki logs

Go to or and paste the json found here(removed). Add it to favorites.

The dashboard should be accessible here: or

Optional features

If we want all logs from all containers to be sent to Grafana, we should enable the promtail component from the loki stack.

If we ever want logstash, to centralize the logs and store them, we can activate it.

· 12 min read
Hreniuc Cristian-Alexandru

This document describes all steps that we need to make when we decide to start the production cluster from Hetzner. This contains:

  • server installation
  • database
  • frontend apps
  • backend apps
  • ssl
  • grafana + loki

1 Install servers

We buy the servers from the clould web interface. For each server we need to do the following steps when buying:

  • Add it to the brandName-net-01 private network(Used to access the nfs storage) In the future, maybe start the cluster on this network.

  • Add it to the brandName-firewall-01 firewall

  • Add it to the brandName-01 placement group(this way they won't end up on the same phisical server, so if one fails the others are still up)

  • Add the public IP to the brandName-firewall-01 fireawall, we have two rules that allow traffic between those servers. This is due to tha fact that we couldn't make it(rke2 cluster, here's smt similar) work on the private addresses.

· One min read
Hreniuc Cristian-Alexandru

First, download the prometheus collector from their webpage: or use docker.

Next you need to add an new user in the database:



I used % for host of the user, to allow logging in from everywhere. I did this because I encountered some errors when I used localhost: ERRO[0026] Error pinging mysqld: dial tcp :0: connect: connection refused source="exporter.go:119"

export DATA_SOURCE_NAME='mysqld_exporter:password@(' && \
nohup ./mysqld_exporter --no-collect.info_schema.tables --no-collect.slave_status > nohup.out 2>&1 &

Use this dashboard from grafana:

· 2 min read
Hreniuc Cristian-Alexandru


To run a container with prometheus, we need two things:

  • prometheus.yml file
  • Docker volume

The prometheus.yml file should contain something like this:

# my global config
scrape_interval: 5s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
evaluation_interval: 5s # Evaluate rules every 15 seconds. The default is every 1 minute.

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
# The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
- job_name: 'prometheus'
- targets: ['localhost:9090']

- job_name: 'docker'
- targets: ['localhost:9999']

- job_name: 'other_collector'
scrape_interval: 1s
- targets: ['']

To create a docker volume, run:

docker create volume prometheus_data

This way, your collected data won\'t be lost on restarts.

docker run -d --restart=always --name=prometheus --network=host -v /path/to/prometheus.yml:/etc/prometheus/prometheus.yml -v prometheus_data:/prometheus prom/prometheus

You should open the browser and check if everything works, by default it is 9090 port.


For grafana we will also use a docker volume:

docker create volume grafana_data

This way we will save all data from grafana(panels, users, etc...).

To run the grafana container, simply run:

docker run -d --name=grafana --restart=always --network=host -e GF_SERVER_ROOT_URL= -e GF_SECURITY_ADMIN_PASSWORD=secret -v grafana_data:/var/lib/grafana grafana/grafana

You can check it on port 3000.

Now you should add the Prometheus server as a data source. You can do this by going to Configuration -> Datasources -> Select Prometheus