How To Use Kubernetes Secrets for Storing Sensitive Config Data

Because your keys and secrets should be just that—secret

Published in

Better Programming

8 min readDec 19, 2019

Hello and welcome 👋👋 We continue the Kubernetes in a Nutshell journey. In one of the previous blogs, we saw how to configure Kubernetes apps using the ConfigMapobject. In this post, we will explore Kubernetes Secretsand how they can be used to store sensitive configuration data that needs to be handled securely (e.g., database credentials, API keys, etc.).

As usual, this is going to be example-driven, and you will learn:

how to create Secrets (CLI, yaml etc.) and
various ways of using them in your apps (env variables, volumes, etc.)

The code (and YAML) is available on GitHub.

This piece has been divided into two logical sections: ways to create Secrets and techniques to use Secrets in your applications.

Prerequisites

To go through the examples in this post, all you need is a Minikube cluster and kubectl CLI tool to access the cluster.

Install Minikube as a single-node Kubernetes cluster in a virtual machine on your computer. On a Mac, you can simply:

curl -Lo minikube https://storage.googleapis.com/minikube/releases/latest/minikube-darwin-amd64 \
  && chmod +x minikubesudo mv minikube /usr/local/bin

Install kubectl to interact with the Minikube cluster. On a Mac, you can:

curl -LO https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/darwin/amd64/kubectlchmod +x ./kubectlsudo mv ./kubectl /usr/local/bin/kubectl

Creating Secrets

Let’s look at techniques using which you can create a Secret.

Use the `data` section

It’s possible to create a Secret along with the configuration data stored as key-value pairs in the datasection of the definition.

apiVersion: v1
kind: Secret
metadata:
  name: service-apikey
data:
  apikey: Zm9vYmFy

The Secret contains key-value data representing the sensitive info, with apikey being the key and value is a base64 encoded string.

To create this Secret in Kubernetes:

kubectl apply -f https://raw.githubusercontent.com/abhirockzz/kubernetes-in-a-nutshell/master/secrets/secret-data.yaml

To keep things simple, the YAML file is being referenced directly from the GitHub repo, but you can also download the file to your local machine and use it in the same way.

To confirm that the Secret has been created:

kubectl get secret/service-apikey -o yaml

You will get a YAML response similar to:

apiVersion: v1
data:
  apikey: Zm9vYmFy
kind: Secret
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","data":{"apikey":"Zm9vYmFy"},"kind":"Secret","metadata":{"annotations":{},"name":"service-apikey","namespace":"default"}}
  creationTimestamp: "2019-12-17T11:11:27Z"
  name: service-apikey
  namespace: default
  resourceVersion: "113009"
  selfLink: /api/v1/namespaces/default/secrets/service-apikey
  uid: 671b547c-3316-4916-b6dc-be2b551b974e
type: Opaque

Fetching the Secret details using kubectl get does not disclose its contents.

Notice that apikey: Zm9vYmFy was what we had provided in the YAML manifest. You can check the plain text form by decoding it:

echo 'Zm9vYmFy' | base64 --decode//foobar

Use the stringData section

The data attribute used in the above example is used to save base64 encoded information. If you want to store plain text data securely, you can use the stringDatasection. Here is an example:

apiVersion: v1
kind: Secret
metadata:
  name: plaintext-secret
stringData:
  foo: bar
  mac: cheese

The values for foo and mac are being passed as plain text. Create this Secret and confirm:

kubectl apply -f https://raw.githubusercontent.com/abhirockzz/kubernetes-in-a-nutshell/master/secrets/secret-plaintext.yamlkubectl get secret/plaintext-secret -o yaml

Here is the data portion of the YAML response. The actual data is stored in base64 encoded format.

data:
  foo: YmFy

If you decode the data, you can confirm that it matches the original plain text input (bar) we had provided

echo 'YmFy' | base64 --decode//bar

Note that thedata section does not accept plain text attribute. Trying to do so will result in an error similar to this: illegal base64 data at input byte 8

File contents

You can provide the contents of an entire file as input to the stringData section as well. Here is what this might look like:

apiVersion: v1
kind: Secret
metadata:
  name: secret-in-a-file
stringData:
  app-config.yaml: |-
    hello: world
    john: doe

Create this `Secret`

kubectl apply -f https://raw.githubusercontent.com/abhirockzz/kubernetes-in-a-nutshell/master/secrets/secret-file.yaml

This resulting Secret will contain a key named app-config.yaml and its contents (value) will be the base64 encoding of the provided data.

As usual, you confirm this in Kubernetes as well as decode the contents.

kubectl get secret/secret-in-a-file -o yaml
echo '<"data" content in yaml response> | base64 --decode

Note: When you use this technique, your application is responsible for parsing out the data that represents the Secret configuration. In this case, it happens to be newline-separated key-value pairs, but it could be anything else.

Using `kubectl`

You can use the kubectl create secret command to create Secret objects

Using `--from-literal`

You can use plain text data to create Secret using the CLI (this will be stored in base64 encoded format in Kubernetes):

kubectl create secret generic redis-credentials --from-literal=user=poweruser --from-literal=password='f0ob@r'

Using `--from-file`

kubectl create secret generic topsecret --from-file=api_keys.txt

This will create a secret (topsecret) with

a key with the same name of the file i.e. api_keys.txt in this case
and, value as the contents of the file

From files in a directory

You can simply point to a directory and all the files within will be used to create the Secret.

kubectl create secret generic topsecrets --from-file=/home/credentials/

You will end up with

multiple keys which will the same as the individual file name
the value will be the contents of the respective file

Using Secrets

For Secrets to be useful, we need to ensure that they are available to our applications (i.e., Pods). Let's explore the ways in which we can do this

Environment variables

You can consume the Secret data as environment variables in a Pod (just like ConfigMap). Here is an example:

apiVersion: v1
kind: Pod
metadata:
  name: pod1
spec:
  containers:
    - name: nginx
      image: nginx
      env:
        - name: API_KEY
          valueFrom:
            secretKeyRef:
              name: service-apikey
              key: apikey

We use the key apikey from Secret service-apikey and make sure its value is available as an environment variable API_KEY inside the Pod.

Create the Pod (assuming you have the Secret created from the example before) and confirm.

kubectl apply -f https://raw.githubusercontent.com/abhirockzz/kubernetes-in-a-nutshell/master/secrets/pod-secret-env.yamlkubectl get pods -w

Wait for the Pod to transition to Running state. Then, confirm that the environment variable has been injected into the Pod.

kubectl exec pod1 -- env | grep API_KEY

You should get this response — API_KEY=foobar.

Instead of referring to individual entries in a Secret, you can use envFrom to conveniently use all entries as environment variables in a Pod. This is how you might use this:

apiVersion: v1
kind: Pod
metadata:
  name: pod2
spec:
  containers:
    - name: nginx
      image: nginx
      envFrom:
        - secretRef:
            name: plaintext-secret

We are referring to the plaintext-secret Secretusing envFrom.secretRef. To create this Pod:

kubectl apply -f https://raw.githubusercontent.com/abhirockzz/kubernetes-in-a-nutshell/master/secrets/pod-secret-envFrom.yamlkubectl get pods -w

Wait for the Pod to transition to Running state and then confirm the presence of environment variables.

kubectl exec pod2 -- env | grep foo
//foo=barkubectl exec pod2 -- env | grep mac
//mac=cheese

This confirms that both foo and mac were added as environment variables into the Pod along with their decoded values i.e. bar and cheese respectively

Volumes

You can mount Secrets as Volume within a Pod. For example:

apiVersion: v1
kind: Pod
metadata:
  name: pod3
spec:
  containers:
    - name: nginx
      image: nginx
      volumeMounts:
        - name: apikey-config-volume
          mountPath: /secret
          readOnly: true
  volumes:
    - name: apikey-config-volume
      secret:
        secretName: service-apikey

The apikey-config-volume volume refers to the service-apikey Secret. To create this Pod:

kubectl apply -f https://raw.githubusercontent.com/abhirockzz/kubernetes-in-a-nutshell/master/secrets/pod-secret-volume.yamlkubectl get pods -w

Wait for the Pod to transition to Running state. Then execute the following command:

kubectl exec pod3 -- cat /secret/apikey
//foobar

This confirms that the key apikey in secret service-apikey was mounted as a file (with name apikey) in the /secret directory (as specific in Pod). The content of the file is nothing but the secret value i.e. foobar in this case.

Using `imagePullSecrets`

There is a way to use Secrets such that your application Pod can use it to authenticate and pull Docker images from private Docker registries.

There are actually three types of Secrets:

generic - used to store key-value pairs, as we have seen in the examples so far
tls - store public.private key pair info as Secrets
docker-registry - credentials for authenticating to a Docker registry.

The way this technique is used is very simple:

Use docker-registry Secret type to store private Docker registry credentials in Kubernetes.
And then, imagePullSecrets (in a Pod) to reference the Secret containing the Docker registry credentials.

An example always helps:

apiVersion: v1
kind: Pod
metadata:
  name: pod4
spec:
  containers:
    - name: privateapp
      image: abhirockzz/test-private-repo:latest
      command: ["/bin/sh"]
      args: ["-c", "while true; do date; sleep 5;done"]
  imagePullSecrets:
    - name: docker-repo-secret

See how imagePullSecrets.name refers to a Secret called docker-repo-secret. Let's create it.

But before that, please ensure that you have a private Docker registry. I used DockerHub, but you can choose any other.

Start by creating a Secret (with the name docker-repo-secret) which contain your Docker credentials using kubectl create secret docker-registrycommand.

kubectl create secret docker-registry docker-repo-secret --docker-server=DOCKER_REG_SERVER --docker-username=DOCKER_REG_USERNAME --docker-password=DOCKER_REG_PASSWORD --docker-email=DOCKER_REG_EMAIL

For Docker Hub:

kubectl create secret docker-registry docker-repo-secret --docker-server=https://index.docker.io/v1/ --docker-username=foobarbaz --docker-password=t0ps3cr3t --docker-email=foobarbaz@gmail.comkubectl get secret/docker-repo-secret -o yaml

https://index.docker.io/v1/ is the Docker Hub registry server

To test things out, we will use a busybox image and tagit

docker pull busybox
docker tag busybox [DOCKER_REG]/[DOCKER_PRIVATE_REPO]:[IMAGE_TAG]e.g. 
docker tag busybox abhirockzz/test-private-repo:latest

… and push it

docker push [DOCKER_REG]/[DOCKER_PRIVATE_REPO]:[IMAGE_TAG]e.g. 
docker push abhirockzz/test-private-repo:latest

Once the private repo is ready, you can create the Podwhich will pull the image from the private repo using the registry credentials supplied to it via the Secret

kubectl apply -f https://raw.githubusercontent.com/abhirockzz/kubernetes-in-a-nutshell/master/secrets/pod-secret-docker.yamlkubectl get pods -w

Wait for the Pod to move to Running status.

If you see an ErrImagePull error, it indicates that there might be problem authenticating to the Docker registry. To get details, use: kubectl describe pod/pod4

To confirm that the pod is working fine: kubectl logs -f pod4

Since the busybox image does not really do anything by itself, we execute: while true; do date; sleep 5;done (as provided in the Pod spec). As a result, you should see the logs (printed every 5 secs).

Tue Dec 17 14:17:34 UTC 2019
Tue Dec 17 14:17:39 UTC 2019
Tue Dec 17 14:17:44 UTC 2019
Tue Dec 17 14:18:49 UTC 2019

All good! What that means is that the Pod was able to pull down your image from a private Docker repo using the Docker credentials which were injected into the Pod using imagePullSecrets, which itself referenced a Secret

Good to Know

Here is a (non-exhaustive) list of things that you should bear in mind when using Secrets:

Secret has to be created before any Pod that wants to use it.
Secrets are applicable within a namespace i.e. they can only be used by Pods in the same namespace
The Pod will not start if there is a reference to a nonexistent key in a Secret (using secretKeyRef)
1MiB is the size limit for individual Secrets

That’s it for this edition of the Kubernetes in a Nutshell series. Stay tuned for more!

If you are interested in learning Kubernetes and Containers using Azure, simply create a free account and get going. A good starting point is to use the quickstarts, tutorials, and code samples in the documentation to familiarize yourself with the service. I also highly recommend checking out the 50-day Kubernetes Learning Path. Advanced users might want to refer to Kubernetes best practices or watch some of the videos for demos, top features, and technical sessions.

I really hope you enjoyed and learned something from this article. 🙌