Restore OpenSearch Indices

title: "Restoring Opensearch indices from OCI ObjectStore" description: "This guide provides instructions on restoring older indices from the OCI ObjectStore to an OpenSearch cluster in Sauron. It covers checking the cluster status, selecting and verifying snapshots, restoring and renaming indices, and verifying the restoration process. The guide also emphasizes the importance of capacity, performance, and backup considerations when restoring a large number of indices, suggesting the creation of a temporary "read-only" Sauron in such cases."

It's possible to restore older indices (logs) from the OCI ObjectStore to your sauron's OpenSearch. There are two common scenarios:

Option 1: You want to restore just a few indices.

This is generally a self-service operation. You'll need to log into your OpenSearch Dashboards endpoint at: https://kibana.handu-phx.handu.developers.oracledx.com and go to Dev Tools.

Reference: https://www.elastic.co/guide/en/elasticsearch/reference/current/snapshot-restore.html

1) Check the status of your OpenSearch cluster

GET _cat/indices?v&s=index:asc

Make sure all indices are green before proceeding. If any indices are other than green, please contact #sauron-support before proceeding.

2) Get a list of all snapshots in the OCI ObjectStore:

GET _cat/snapshots/object-store-repo?h=id&s=id:desc

This may take a moment, but returns all OpenSearch backups currently in the ObjectStore for your Sauron.

3) Select a snapshot and verify it has the logs you're interested in

From the list returned in Step 2, you'll need to determine which snapshot to use. Let's suppose we want to restore example_index_2021.08_06 and example_index_2021.08_07. Logs can continue trickle in after an official "end-of-day", so it's wise to select a snapshot that's a day or two after the index dates you actually want. So, for in this example, we might choose the snapshot 20210809-192148 to make sure we get all the logs for the above indices.

Next, we want to make sure that chosen snapshot has those logs. Run the following command to get a list of all the indices included in that snapshot.

GET _snapshot/object-store-repo/20210809-192148?pretty

Check through the list to make sure the logs you're interested in appear.

4) Restore the indices

Okay, we're ready to restore, with one final note: we also need to rename these indices. If we don't rename them, they'll be identified as old logs by the daily backup system and removed. We recommend concatonating the numbers together and adding a -restore suffix.

POST _snapshot/object-store-repo/20210809-192148/_restore
{
  "indices": "example_index_2021.08_06",
  "index_settings": { "index.number_of_replicas": 1 },
  "rename_pattern": "example_index_2021.08_06",
  "rename_replacement": "example_index_20210806-restore"
}

POST _snapshot/object-store-repo/20210809-192148/_restore
{
  "indices": "example_index_2021.08_07",
  "index_settings": { "index.number_of_replicas": 1 },
  "rename_pattern": "example_index_2021.08_07",
  "rename_replacement": "example_index_20210807-restore"
}

Upon running the above command, the selected indices will be restored and renamed.

5) Verify the indices have been restored.

Depending on the size of the indices, it may take a few minutes to an hour to fully restore. You can run the following command to check the state of the indices. You should see: example_index_20210806-restored and example_index_20210807-restored. Again, it's a good idea to wait for all indices to become green before proceeding.

GET _cat/indices?v&s=index:asc

6) Declare victory

7) When you've finished ... delete the restored indices

When you've finished, please delete these restored indices. Be VERY careful here to make sure you are deleting the restored indices only.

DELETE example_index_20210806-restored
DELETE example_index_20210807-restored

Troubleshooting

If you run into any trouble, please contact #sauron-support.

Option 2: You want to restore a large number of indices.

An example would be: "I want to restore all daily indices from Aug 7-21, 2021." In this case, please contact #sauron-support for assistance. Restoring large numbers of indices to your current Sauron can cause a number of problems:

Capacity: Your environment must be able to hold your current data, plus all restored data.
Performance: When a lot of data is being restored, it may degrade performance of your OpenSearch cluster.
Backups: Logs are routinely backed up on a daily basis... but these logs are already backed up. There's no need to create duplicates. (The extra data will also make the daily backups take longer.)

To solve these problems, we usually create a new "read-only" sauron for you and restore the data there. The "read-only" sauron is temporary and will be retired when you've finished using it.