Skip to main content

Tempo

Grafana Tempo lets you scale tracing as far as possible with minimal operational cost and less complexity than ever before.

For local development purposes one should the monolithic mode. This is a single pod that runs Tempo and doesn't scale. For live clusters use the microservices mode with kind TempoStack. More info found here: https://grafana.com/docs/tempo/latest/setup/operator/monolithic/

Object Storage​

Tempo is able to use multiple different storage providers, but in our case we're standardizing on S3 / S3-compatible API's.

Tempo needs 1 Object Storage container. The given Tempo config relies on a container named havenplus-tempo. However, you can rename it according to your needs and change the reference to the bucket when creating the required secret (see steps below).

info

Keep in mind that the bucket names must be globally unique within the context of an IaaS-provider.

When running the stack on your local environment, a Minio instance is automatically deployed and this bucket is automically created as well.

For instructions on how to create these with the relevant IaaS provider, please consult the provider-specific instructions.

S3 Credentials Secret​

ℹ When deploying the stack in your local environment using the Local overlay, you can skip the following step.

Use the S3 credentials provided by the concerning IaaS-provider and use these to create a secret.yaml:

apiVersion: v1
kind: Secret
metadata:
  name: s3-credentials
stringData:
  endpoint: <see specific Provider documentation>
  bucket: tempo
  access_key_id: <access-key-id>
  access_key_secret: <secret-access-key>
type: Opaque

Next, use kubeseal to generate a sealed-secret based on the secret above:

kubeseal -f secret.yaml -w sealed.yaml  --controller-namespace sealed-secrets -n tempo

An entry for sealed.yaml has already been preconfigured in kustomization.yaml, you just need to uncomment it. Furthermore, ensure that sealed.yaml is placed in the same folder as the kustomization.yaml.

Don't forget to remove the secret.yaml and definitely do NOT commit the actual secret.yaml to Git! The TempoStack custom resource refers to the secret.

Data Retention

By default, Tempo is configured to retain data for 3 days. You can adjust this to your organisation's needs via spec.retention in the concerning Tempo custom resource manifest. Set it to e.g. 7d to retain 7 days worth of trace data.