Skip to content

DSS Pooling using YugabyteDB

This document describes the specifics of using YugabyteDB as the data store to accomplish pooling.

How Yugabyte meets pooling requirements

"Each Yugabyte node is addressable by every other Yugabyte node"

Every Yugabyte node must have its own externally-accessible hostname (e.g., 1.tserver.db.dss-prod.example.com), or its own hostname:port combination (e.g., db.dss-prod.example.com:26258).

There are two type of nodes in a Yugabyte cluster: Master and TServer. Both ones must be accessible. The ports on which Yugabyte communicates must be open to others participants:

  • Master: gRPC: 7100
  • TServer: gRPC: 9100
  • Master: Admin UI: 7000
  • TServer: Admin UI: 9000
  • TServer: ycql: 9042
  • TServer: ysql: 5433
  • TServer: metrics: 13000
  • TServer: metrics: 12000

The ports in bold are mandatory. The others ones are needed for management UI, the UI won't work correctly if any of those port is not reachable by other nodes, except on the master node.

Info

The Helm charts and the Tanka files only expose mandatory ports as they are the only ones secure. If usage of the UI is needed in a pool with multiple participants, you must find a way to open those ports in a way secure enough for your deployments. Most of those non-mandatory ports do not offer authentication nor encryption (or confidentiality). A secure method is required, such as an Istio mesh or a local private network.

This requirement may be verified by conducting a standard TLS diagnostic (like this one) on the hostname:port for each TServer node (e.g., 0.tserver.db.dss.example.com:7100). The "Trust" characteristic will not pass because the certificate is issued by a custom CA which is not a generally-trusted root CA, but we explicitly enable trust by manually exchanging the trusted CA public keys in ca.crt (see "Each Yugabyte node accepts the certificates of every other node" below). However, all other checks should generally pass.

Danger

It's recommended to restrict access to all ports and only allow IPs of others participants. However, guides and deployment tooling haven't been adapted yet.

"Each Yugabyte node is discoverable"

When a Yugabyte node is brought online, it must know how to connect to the existing network of nodes. This is accomplished by providing an explicit list of nodes to contact. Each node contacted will provide a list of nodes it is connected to in the network ("gossip"), so not every node must be present in the explicit list, but it's recommended to do so. The explicit list shall contain, at a minimum, all known nodes when creating the DSS instance and shall be updated regularly. Yugabyte nodes have some difficulites to locate primary nodes if they don't have the full list of known nodes.

"Each Yugabyte node accepts the certificates of every other node"

Yugabyte uses TLS to secure connections, and TLS includes a mechanism to ensure the identity of the server being contacted. This mechanism requires a trusted root Certificate Authority (CA) to sign a certificate containing the public key of a particular server, so a client connecting to that server can verify that the certificate (containing the public key) is endorsed by the root CA as being genuine. Yugabyte certificates require a claim that standard web CAs will not sign, so instead each USS acts as their own root CA. When USS 1 is presented with certificates signed by USS 2's CA, USS 1 must know that it can trust that certificate. We accomplish this by exchanging all USSs' CA public keys out-of-band in ca.crt, and specifying that certificates signed by any of the public keys in ca.crt should be accepted when considering the validity of certificates presented to establish a TLS connection between nodes.

The private CA key dss-certs.py generates is stored in the ca folder. The private CA key is used to generate all node certificates and client certificates. Once a pool is established, a USS avoids regenerating this CA keypair, and use the existing ones by default. If a USS generates a new CA keypair, the new public key must be added to the pool's combined ca.crt, and all USSs in the pool must adopt the new combined ca.crt before any nodes using certificates generated by the new CA private key will be accepted by the pool.

"The Yugabyte cluster is initialized"

A Yugabyte cluster of databases is like the Ship of Theseus: it is composed of many nodes which may all be replaced, one by one, so that a given Yugabyte cluster eventually contains none of its original nodes. Unlike the Ship of Theseus, however, a cluster is clearly identified by its cluster ID (e.g., b2537de3-166f-42c4-aae1-742e094b8349) -- if the cluster ID is the same, it is the same cluster (and vice versa). Once for the entire lifetime of the Yugabyte cluster, it is created automatically during initialization of the first set of Yugabyte nodes, if all initial nodes see each others in a uninitialized state.

Additional requirements

These requirements must be met by every DSS instance joining an InterUSS-compatible pool. The deployment instructions produce a system that complies with all these requirements, so this section may be ignored if following those instructions.

  • All Yugabyte nodes must be run in secure mode.
    • use_node_to_node_encryption enabled
    • use_client_to_server_encryption enabled
    • node_to_node_encryption_use_client_certificates enabled
    • allow_insecure_connections disabled
  • All DSS instances in the same cluster must point their ntpd at the same NTP Servers.

Placement

It's important to maintain a good placement strategy, ensuring data availability in case of failures.

We do recommend a minimum of 3 participants and one copy in each participants.

You may use the modify_placement_info command to set placement settings. Example:

  • yb-admin -certs_dir_name /opt/certs/yugabyte/ -client_node_name=`hostname -f` -master_addresses yb-master-0.yb-masters.default.svc.cluster.local:7100,yb-master-1.yb-masters.default.svc.cluster.local:7100,yb-master-2.yb-masters.default.svc.cluster.local:7100 modify_placement_info dss.uss-1,dss.uss-2,dss.uss-3 3
  • yb-admin -certs_dir_name /opt/certs/yugabyte/ -client_node_name=`hostname -f` -master_addresses yb-master-0.yb-masters.default.svc.cluster.local:7100,yb-master-1.yb-masters.default.svc.cluster.local:7100,yb-master-2.yb-masters.default.svc.cluster.local:7100 modify_placement_info dss.uss-1,dss.uss-2,dss.uss-3,dss.uss-4,dss.uss-5 5

You may however use a different strategy depending on your availability needs, e.g. you may want to avoid common datacenter between the same DSS instance. To do so, define a strategy in your pool and edit placement information as needed. More information is available in Yugabyte documentation.