Let's validate that all three Elasticsearch nodes has been successfully added to the Elasticsearch cluster once more. We can do this by sending a GET request to /_cluster/state?pretty and checking the output.

But since we want to keep the database service internal, we haven't exposed it to an external-reachable URL, so the only way to validate this is to SSH into one of the VPS and query Elasticsearch using its private IP.

However, kubectl provides a more convenient alternative. kubectl has a port-forward command, which forwards requests going into a port on localhost to a port on one of the Pods. We can use this feature to send requests from our local machine to each Elasticsearch instance.

Let's suppose that we have three Pods running Elasticsearch:

$ kubectl get pods
NAME              READY   STATUS    RESTARTS   AGE
elasticsearch-0   1/1     Running   0          34m
elasticsearch-1   1/1     Running   0          34m
elasticsearch-2   1/1     Running   0          34m

We can set up port forward on elasticsearch-0 by running the following:

$ kubectl port-forward elasticsearch-0 9200:9200
Forwarding from 127.0.0.1:9200 -> 9200
Forwarding from [::1]:9200 -> 9200

Now, on a separate terminal, send a GET request to http://localhost:9200/_cluster/state?pretty:

$ curl http://localhost:9200/_cluster/state?pretty
{
  "cluster_name" : "docker-cluster",
  "state_uuid" : "rTHLkSYrQIu5E6rcGJZpCA",
  "master_node" : "TcYdL65VSb-W1ZzXPfB8aA",
  "nodes" : {
    "ns1ZaCTCS9ywDSntHz94vg" : {
      "name" : "ns1ZaCT",
      "ephemeral_id" : "PqwcVrldTOyKSfQ-ZfhoUQ",
      "transport_address" : "10.244.24.2:9300",
      "attributes" : { }
    },
    "94Q-t8Y8SJiXnwVzsGcdyA" : {
      "name" : "94Q-t8Y",
      "ephemeral_id" : "n-7ew1dKSL2LLKzA-chhUA",
      "transport_address" : "10.244.18.3:9300",
      "attributes" : { }
    },
    "TcYdL65VSb-W1ZzXPfB8aA" : {
      "name" : "TcYdL65",
      "ephemeral_id" : "pcghJOnTSgmB8xMh4DKSHA",
      "transport_address" : "10.244.75.3:9300",
      "attributes" : { }
    }
  },
  "metadata" : {
    "cluster_uuid" : "ZF1t_X_XT0q5SPANvzE4Nw",
    ...
  },
  ...
}

As you can see, the node field contains three objects, representing each of our Elasticsearch instances. They are all part of the cluster, with a cluster_uuid value of ZF1t_X_XT0q5SPANvzE4Nw. Try port forwarding to the other Pods, and confirm that the cluster_uuid for those nodes are the same.

If everything worked, we have now successfully deployed the same Elasticsearch service on DigitalOcean!