Thanos Additional Stores

Thanos provides an HA and stable view of the metrics coming from your multiple Prometheus backends

Thanos extends the capabilities of Prometheus. High availability:

  • Thanos queries data from both replicas, deduplicating samples/alerts, and filling the gaps if necessary.

Scalability (horizontal):

  • vs. federating time series from other Prometheus into global Prometheus instance.
  • vs. multiple Prometheus data sources in Grafana dashboards

Global aggregation:

  • Query across multiple Prometheus instances

If you have additional Sauron instances or your own non-Sauron Thanos instances, this page explains how to add them to a single Sauron Thanos metrics view.

Using this feature, it is possible to create federated, global views that span across multi-regional metrics. These federated views can be custom tailored to meet your teams needs.

Add an additional store

In this example we will add a Sauron named dev-west to a Sauron named dev-global.

  • We will use the API server endpoint from both of these Saurons to complete this task.
  • From the dev-global API server add the GRPC dev-west Thanos endpoint and select a name.
PUT /v1/thanos/additionalstores


{
  "cert": "base64encodedClientCert",
  "key": "base64encodedClientKey",
  "name": "west-coast",  
  "urls": [
    "thanos-grpc.dev.west.jrosinsk.oracledx.com:443"  
  ]
}
  • Screenshot showing this information being added to the API server endpoint for the dev-global Sauron

Example 1

  • Obtain the base64 encoded additional store client cert & key, required to allow the dev-global store to access the metrics in the dev-west store

  • Use the dev-west Sauron API server endpoint to retrieve the required cert & key values.

  • Make sure thanos-grpc is entered instead of just thanos in the urls list. Thanos store is communicated through the grpc port.

GET /v1/thanos/servercerts
  • Screenshot highlighting the copy and paste of the base64 encoded server key from the Thanos dev-west endpoint.

Example 2

  • Next, copy and paste the base64 encoded cert value

  • Take a moment to verify the values you have entered look OK. Be careful of the trailing = char.

  • If everything looks good, go ahead and Execute the command

  • Screenshot shows an example of executing the put command to add the dev-west metrics store to the dev-global metrics view.

Example 3

  • Verify a successful Server Response.
    202 Additional store west-coast has been created.
Verify the new Thanos Store is setup correctly

Retrieve the new thanos store endpoint from the console

Example 4

WARNING: If Global SSO is enabled for this Sauron, it can take 10-15 minutes to register the new Thanos Store endpoint before it is accessible.

  • Click the new thanos-store link.
  • From the new popup page, click the Stores tab.
  • The new thanos store that you entered in the urls field should be listed.
  • If you don't see the expected endpoint, this means your sauron couldn't reach your new thanos store. Most of the time, it is because the key and/or cert are incorrect.

Example 5

View the metrics

You should begin to see data immediately in the dev-global Thanos metrics view

Here is the screenshot showing data from dev-west visible in the dev-global metrics view. Example 6

Setup multiple thanos stores which have same cert and key.

If you have the same cert and key for your Thanos stores, you can combine them as shown below. 

  ...
  "name": "dev-regions",
  "urls": [
    "thanos-grpc.dev.west.jrosinsk.oracledx.com:443",
    "thanos-grpc.dev.midwest.jrosinsk.oracledx.com:443"
  ]
}
Share certificate for all urls in additional store

Although additional stores support multiple urls, there is only support for a single certificate. Certificates can be shared using the GET/PUT '/v1/thanos/servercerts' endpoints.

GET /v1/thanos/servercerts

Fetch the Thanos server-side certificate and private key (in base64 format), that will be used to authenticate traffic to the Thanos GRPC endpoint.

PUT /v1/thanos/servercerts

Set the Thanos server-side certificate and private key (in base64 format) that will be used to authenticate traffic to the Thanos GRPC endpoint.

Parameters

Server-side certificate and key 

{  
"cert": "string",  
"key": "string"  
}