CockroachDB upgrades

This page provides information on how to upgrade your CockroachDB version deployed using the tools from this repository.

CockroachDB must be upgraded on all DSS instances of the pool one after the other. The rollout of the upgrades on the whole CRDB cluster must be carefully performed in sequence to keep the majority of nodes healthy during that period and prevent downtime. For a Pooled deployment, one of the DSS Instance must take the role of the upgrade "Leader" and coordinate the upgrade with other "Followers" DSS instances. In general a CockroachDB upgrade consists of: 1. Upgrade preparation: Verify that the cluster is in a nominal state ready for upgrade. 1. Decide how the upgrade will be finalized (for major upgrades only): Like CockroachDB, we recommend disabling auto-finalization. 1. Perform the rolling upgrade: This step should be performed first by the Leader and as quickly as possible by the Followers one after the other. Note that during this period, the performance of the cluster may be impacted since, as documented by CockroachDB, "a query that is sent to an upgraded node can be distributed only among other upgraded nodes. Data accesses that would otherwise be local may become remote, and the performance of these queries can suffer." 1. Roll back the upgrade (optional): Like the rolling upgrade, this step should be carefully coordinated with all DSS instances to guarantee the minimum number of healthy nodes to keep the cluster available. 1. Finish the upgrade: This step should be accomplished by the Leader.

The following sections provide links to the CockroachDB migration documentation depending on your deployment type, which can be different by DSS instance.

Important notes:

• Further work is required to test and evaluate the availability of the DSS during migrations.
• We recommend to review carefully the instructions provided by CockroachDB and to rehearse all migrations on a test environment before applying them to production.

Terraform deployment

If a DSS instance has been deployed with terraform, first upgrade the cluster using Helm or Tanka. Then, update the variable crdb_image_tag in your terraform.tfvars to align your configuration with the new state of the cluster.

Helm deployment

If you deployed the DSS using the Helm chart and the instructions provided in this repository, follow the instructions provided by CockroachDB Cluster Upgrade with Helm (See specific links below). Note that the CockroachDB documentation suggests to edit the values using helm upgrade ... --set commands. You will need to use the root key cockroachdb since the cockroachdb Helm chart is a dependency of the dss chart. For instance, setting the image tag and partition using the command line would look like this:

helm upgrade [RELEASE_NAME] [PATH_TO_DSS_HELM] --set cockroachdb.image.tag=v24.1.3 --reuse-values

helm upgrade [RELEASE_NAME] [PATH_TO_DSS_HELM] --set cockroachdb.statefulset.updateStrategy.rollingUpdate.partition=0 --reuse-values

Alternatively, you can update helm_values.yml in your deployment and set the new image tag and rollout partition like this:

cockroachdb:
  image:
    # ...
    tag: # version
  statefulset:
    updateStrategy:
      rollingUpdate:
        partition: 0

New values can then be applied using helm upgrade [RELEASE_NAME] [PATH_TO_DSS_HELM] -f [helm_values.yml]. We recommend the second approach to keep your helm values in sync with the cluster state.

21.2.7 to 24.1.3

CockroachDB requires to upgrade one minor version at a time, therefore the following migrations have to be performed:

1. 21.2.7 to 22.1: see CockroachDB Cluster upgrade for Helm.
2. 22.1 to 22.2: see CockroachDB Cluster upgrade for Helm.
3. 22.2 to 23.1: see CockroachDB Cluster upgrade for Helm.
4. 23.1 to 23.2: see CockroachDB Cluster upgrade for Helm.
5. 23.2 to 24.1.3: see CockroachDB Cluster upgrade for Helm.

Tanka deployment

For deployments using Tanka configuration, since no instructions are provided for Tanka specifically, we recommend to follow the manual steps documented by CockroachDB: Cluster Upgrade with Manual configs. (See specific links below) To apply the changes to your cluster, follow the manual steps and reflect the new values in the Leader and Followers Tanka configurations, namely the new image version (see VAR_CRDB_DOCKER_IMAGE_NAME in TFVARS.gen.md) to ensure the new configuration is aligned with the cluster state.

21.2.7 to 24.1.3

CockroachDB requires to upgrade one minor version at a time, therefore the following migrations have to be performed:

1. 21.2.7 to 22.1: see CockroachDB Cluster upgrade with Manual configs.
2. 22.1 to 22.2: see CockroachDB Cluster upgrade with Manual configs.
3. 22.2 to 23.1: see CockroachDB Cluster upgrade with Manual configs.
4. 23.1 to 23.2: see CockroachDB Cluster upgrade with Manual configs.
5. 23.2 to 24.1.3: see CockroachDB Cluster upgrade with Manual configs.