14 posts tagged with "kubernetes"

This doc describes how to create a backup on-demand for RKE2 cluster. We already have set up a recurring backup, from the rke2 engine. But if we ever want to backup this, before an update or something, we should follow these steps.

Source

Atm only the snapshot with config file works, there is a bug in their server:

• https://github.com/rancher/rke2/issues/2103
• https://github.com/rancher/rke2/issues/2121

rke2 etcd-snapshot --config /etc/rancher/rke2/config2.yaml --debug

s3: true
s3-access-key: keyId
s3-bucket: domain_com-contabo-rke
s3-endpoint: s3.eu-central-003.backblazeb2.com
s3-region: eu-central-003
s3-secret-key: applicationKey
snapshot-compress: true

This will create a snapshot and upload it to S3(BackBlaze).

Not working atm​

rke2 etcd-snapshot \
  --snapshot-compress \
  --s3 \
  --s3-endpoint "s3.eu-central-003.backblazeb2.com" \
  --s3-bucket "domain_com-contabo-rke" \
  --s3-access-key "keiId" \
  --s3-secret-key "applicationKey"

Install vitess client locally first:

wget https://github.com/vitessio/vitess/releases/download/v14.0.0/vitess_14.0.0-9665c18_amd64.deb

sudo dpkg -i vitess_14.0.0-9665c18_amd64.deb

Start the port forwarding

Note: Make sure you have the KUBECONFIG env set when running pf.sh

cd vitess
bash pf.sh &

alias vtctlclient="vtctlclient -server localhost:15999 -alsologtostderr"
alias mysql="mysql -h 127.0.0.1 -P 15306 -u domain-com_admin"


Pass: `domain-com_admin_`

Connnect to the db to test if it works

mysql -pdomain-com_admin_

Create the backup - only the schema

mysqldump -d -h 127.0.0.1 -P 15306 -u domain-com_admin -pdomain-com_admin_ domain-com > domain-com-dev-schema.sql

Create the backup - the complete db

mysqldump -h 127.0.0.1 -P 15306 -u domain-com_admin -pdomain-com_admin_ domain-com > domain-com-dev.sql

Import the db locally

!!Make sure you use another bash terminal, nnot the one you added the aliases!!

# Create the database
mysql -u root -proot
create database domain_com_dev;
quit

# Import it
mysql -u root -proot domain_com_dev < domain-com-dev.sql

If you encounnter errors, you might have to run these commannds:

sed -i 's/utf8mb4/utf8/g' domain-com-dev.sql
sed -i 's/utf8_0900_ai_ci/utf8_general_ci/g' domain-com-dev.sql

And retry import.

Here we will document how to install the Loki stack: Loki + Grafana to display our logs in grafana.

Preparations​

Add the grafana-contabo.domain.com or grafana-hetzner.domain.com to the IP of the cluester in cloudflare, we will need it, because we will generate a certificate for it using let's encrypt.

Install​

We will install it via Rancher using the helm chart.

First we will need to add the repository to Rancher. We should add this: https://grafana.github.io/helm-charts to Apps & Marketplace > Repositories > Create.

Go to Apps & Marketplace > Charts and search for loki-stack, install it using the following:

• name - loki-stack
• values - Use the yml from below. You will have to replace contabo with hetzner, if you install this on hetzner.

loki:
# https://github.com/grafana/helm-charts/blob/main/charts/loki/values.yaml
  enabled: true
  persistence:
    enabled: true
    storageClassName: nfs-master1-storage # https://github.com/grafana/helm-charts/blob/main/charts/loki/templates/statefulset.yaml#L145
    size: 15Gi
  # extraArgs:
  #   log.level: debug
grafana:
  enabled: true
  sidecar:
    datasources:
      enabled: true # https://github.com/grafana/loki/blob/88feda41a02f9c544d7476f61e296373e83fbe72/production/helm/loki-stack/templates/datasources.yaml#L1
  persistence:
    enabled: true # We should set it to true
    storageClassName: nfs-master1-storage
    size: 1Gi
  ingress:
    enabled: true
    annotations:
      kubernetes.io/ingress.class: nginx
      cert-manager.io/cluster-issuer: letsencrypt-staging
    hosts:
      - grafana-contabo.domain.com
    tls:
      - hosts:
        - grafana-contabo.domain.com
        secretName: grafana-contabo.domain.com-cert # Autogenerated

Note: Don't forget to change the dns based on the cluster.

To get the admin password for grafana, you should run:

kubectl get secret --namespace default loki-stack-grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo

The loki endpoint that should be used everywhere is http://loki-stack:3100, the DNS is the same as the name set when we installed the helm.

Import the grafana dashboard for Loki logs​

Go to https://grafana-contabo.domain.com/dashboard/import or https://grafana-hetzner.domain.com/dashboard/import and paste the json found here(removed). Add it to favorites.

The dashboard should be accessible here: https://grafana-contabo.domain.com/d/domain-com-loki/logs-for-domain-com-backend or https://grafana-hetzner.domain.com/d/domain-com-loki/logs-for-domain-com-backend.

Optional features​

If we want all logs from all containers to be sent to Grafana, we should enable the promtail component from the loki stack.

If we ever want logstash, to centralize the logs and store them, we can activate it.

This document describes all steps that we need to make when we decide to start the production cluster from Hetzner. This contains:

• server installation
• database
• frontend apps
• backend apps
• ssl
• grafana + loki

1 Install servers​

We buy the servers from the clould web interface. For each server we need to do the following steps when buying:

• Add it to the brandName-net-01 private network(Used to access the nfs storage) In the future, maybe start the cluster on this network.

• Add it to the brandName-firewall-01 firewall

• Add it to the brandName-01 placement group(this way they won't end up on the same phisical server, so if one fails the others are still up)

• Add the public IP to the brandName-firewall-01 fireawall, we have two rules that allow traffic between those servers. This is due to tha fact that we couldn't make it(rke2 cluster, here's smt similar) work on the private addresses.