By default, Elasticsearch runs inside the Docker container as the user elasticsearch, with both a UID and GID of 1000. Therefore, we must ensure that the data directory (/usr/share/elasticsearch/data) and all its content is going to be owned by this the elasticsearch user so that Elasticsearch can write to them.

When Kubernetes bind-mounted the PersistentVolume to our /usr/share/elasticsearch/data, it was done using the root user. This means that the /usr/share/elasticsearch/data directory is no longer owned by the elasticsearch user.

Therefore, to complete our deployment of Elasticsearch, we need to use an Init Container to fix our permissions. This can be done by running chown -R 1000:1000 /usr/share/elasticsearch/data on the node as root.

Add the following entry to the initContainers array inside stateful-set.yaml:

- name: fix-volume-permission
  image: busybox
  command:
  - sh
  - -c
  - chown -R 1000:1000 /usr/share/elasticsearch/data
  securityContext:
    privileged: true
  volumeMounts:
  - name: data
    mountPath: /usr/share/elasticsearch/data

This basically mounts the PersistentVolume and updates its owner before the app Container starts initializing, so that the correct permissions would already be set when the app container executes. To summarize, your final elasticsearch/service.yaml should look like this:

apiVersion: v1
kind: Service
metadata:
  name: elasticsearch
  labels:
    app: elasticsearch
spec:
  selector:
    app: elasticsearch
  clusterIP: None
  ports:
  - port: 9200
    name: rest
  - port: 9300
    name: transport

And your final elasticsearch/stateful-set.yaml should look like this:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: elasticsearch
  labels:
    app: elasticsearch
spec:
  replicas: 3
  serviceName: elasticsearch
  selector:
    matchLabels:
      app: elasticsearch
  template:
    metadata:
      labels:
        app: elasticsearch
    spec:
      initContainers:
      - name: increase-max-map-count
        image: busybox
        command:
        - sysctl
        - -w
        - vm.max_map_count=262144
        securityContext:
          privileged: true
      - name: increase-file-descriptor-limit
        image: busybox
        command:
        - sh
        - -c
        - ulimit -n 65536
        securityContext:
          privileged: true
      containers:
        - name: elasticsearch
          image: docker.elastic.co/elasticsearch/elasticsearch-oss:6.3.2
          ports:
          - containerPort: 9200
            name: http
          - containerPort: 9300
            name: tcp
          env:
            - name: discovery.zen.ping.unicast.hosts
              value: "elasticsearch-0.elasticsearch.default.svc.cluster.local,elasticsearch-1.elasticsearch.default.svc.cluster.local,elasticsearch-2.elasticsearch.default.svc.cluster.local"
      volumeMounts:
        - name: data
          mountPath: /usr/share/elasticsearch/data
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 2Gi
      storageClassName: do-block-storage

Delete your existing Services, StatefulSets, and Pods and try deploying them from scratch:

$ kubectl apply -f ./manifests/elasticsearch/service.yaml
service "elasticsearch" created
$ kubectl apply -f ./manifests/elasticsearch/stateful-set.yaml
statefulset.apps "elasticsearch" created