Skip to content

DSS Pooling using CockroachDB

This document describes the specifics of using CockroachDB as the data store to accomplish pooling.

How CockroachDB meets pooling requirements

"Each CockroachDB node is addressable by every other CockroachDB node"

Every CRDB node must have its own externally-accessible hostname (e.g., 1.db.dss-prod.example.com), or its own hostname:port combination (e.g., db.dss-prod.example.com:26258). The port on which CRDB communicates must be open (default CRDB port is 26257).

This requirement may be verified by conducting a standard TLS diagnostic (like this one) on the hostname:port for each CRDB node (e.g., 0.db.dss.example.com:26257). The "Trust" characteristic will not pass because the certificate is issued by "Cockroach CA" which is not a generally-trusted root CA, but we explicitly enable trust by manually exchanging the trusted CA public keys in ca.crt (see "Each CockroachDB node accepts the certificates of every other node" below). However, all other checks should generally pass.

"Each CockroachDB node is discoverable"

When a CRDB node is brought online, it must know how to connect to the existing network of nodes. This is accomplished by providing an explicit list of nodes to contact. Each node contacted will provide a list of nodes it is connected to in the network ("gossip"), so not every node must be present in the explicit list. The explicit list should contain, at a minimum, all nodes within the DSS instance, and at least one node from each other DSS instance. Standard practice should generally include two nodes from each other DSS instance in case one node is down for maintenance.

"Each CockroachDB node accepts the certificates of every other node"

CockroachDB uses TLS to secure connections, and TLS includes a mechanism to ensure the identity of the server being contacted. This mechanism requires a trusted root Certificate Authority (CA) to sign a certificate containing the public key of a particular server, so a client connecting to that server can verify that the certificate (containing the public key) is endorsed by the root CA as being genuine. CRDB certificates require a claim that standard web CAs will not sign, so instead each USS acts as their own root CA. When USS 1 is presented with certificates signed by USS 2's CA, USS 1 must know that it can trust that certificate. We accomplish this by exchanging all USSs' CA public keys out-of-band in ca.crt, and specifying that certificates signed by any of the public keys in ca.crt should be accepted when considering the validity of certificates presented to establish a TLS connection between nodes.

The private CA key make-certs.py generates is stored in ca_key_dir, and its corresponding public key is stored in ca_certs_dir (ca.crt). The private CA key is used to generate all node certificates and client certificates. Once a pool is established, a USS avoids regenerating this CA keypair, and use the existing ones by default. This default behavior can be modified by setting --overwrite-ca-cert flag to true. If a USS generates a new CA keypair, the new public key must be added to the pool's combined ca.crt, and all USSs in the pool must adopt the new combined ca.crt before any nodes using certificates generated by the new CA private key will be accepted by the pool.

"The CockroachDB cluster is initialized"

A CRDB cluster of databases is like the Ship of Theseus: it is composed of many nodes which may all be replaced, one by one, so that a given CRDB cluster eventually contains none of its original nodes. Unlike the Ship of Theseus, however, a cluster is clearly identified by its cluster ID (e.g., b2537de3-166f-42c4-aae1-742e094b8349) -- if the cluster ID is the same, it is the same cluster (and vice versa). Once for the entire lifetime of the CRDB cluster, it is created and given its cluster ID with the cockroach init command. If this command is run on a node that is part of a cluster that has already been initialized, it will fail. The Kubernetes configuration in this repository uses the VAR_SHOULD_INIT parameter to control whether this command is executed during deployment.

Additional requirements

These requirements must be met by every DSS instance joining an InterUSS-compatible pool. The deployment instructions produce a system that complies with all these requirements, so this section may be ignored if following those instructions.

  • All CockroachDB nodes must be run in secure mode, by supplying the --certs-dir and --ca-key flags.
    • Do not specify --insecure
  • The ordering of the --locality flag keys must be the same across all CockroachDB nodes in the cluster.
  • All DSS instances in the same cluster must point their ntpd at the same NTP Servers. CockroachDB recommends using Google's Public NTP when running in a multi-cloud environment.

Joining an existing pool with new instance

A USS wishing to join an existing pool (of perhaps just one instance following normal CRDB pooling) should follow the deployment instructions. They will be joining an existing cluster, and they will need to request the ca.crt that the pool is currently using (any one member of the pool may provide it). The joining USS will also need a list of node addresses to which they should connect; ideally this will include at least 2 nodes from each existing DSS instance.

As soon as the joining USS has created a new, combined ca.crt by joining the existing ca.crt provided by the pool, the joining USS must provide that new ca.crt to every existing DSS instance in the pool, and all nodes of all instances must adopt the new, combined ca.crt before the joining USS can bring their DSS instance online (with tk apply).

Once all participants in the existing pool have confirmed that the new ca.crt has been adopted by all of their nodes, the joining USS brings their system online with tk apply. The pool should then be re-verified for functionality by running the prober test on each DSS instance, and the interoperability test scenario on the full pool (including the newly-added instance).

Finally, the joining USS should provide its node addresses to all other participants in the pool, and each other participant should add those addresses to the list of CRDB nodes their CRDB nodes will attempt to contact upon restart.