Kubernetes

What is Kubernetes?

According to kubernetes.com, Kubernetes is…

an open-source system for automating deployment, scaling, and management of containerized applications.

I would define it as: Kubernetes is a program for creating and managing a computer cluster.

A cluster is a distributed system in which multiple computers work together to behave somewhat like one, bigger computer. There is a long history of cluster computing (see: beowulf clusters). While a distributed system is fundamentally different from a single computer, cluster management software (e.g. Kubernetes) helps to bridge that gap.

These days, practically all applications are distributed systems. Dealing with the inherent problems of a distributed system is often left to the application layer. Kubernetes is designed to directly address some (but not all) of the fundamental problems of working in a distributed system, like: how do you tell when a node is dead? And what should be done about that when it happens?

Kubernetes provides heavy abstractions for dealing with clusters. Some of the problems that need to be addressed are:

  • Networking
    • What does it mean for a machine to be "in" the cluster vs "out" of the cluster?
    • How should inter-process communication work when processes might be on different machines?
    • What happens when a machine is no longer responsive?
  • Process Scheduling
    • When you run a process on the cluster, which specific machine should the process actually run on?
    • What if certain processes can only run on certain machines?
  • Data
    • How can processes share data (on disk) when they might running on different machines?
    • Should every process have access to the whole host's filesystem? (Probably not.) How can we make process-local disk space?
  • Security
    • Should the cluster have some kind of RBAC (authn/authz) separate from that of each individual machine?

At first, I thought Kubernetes was mostly about process scheduling, but that is actually a very small part of Kubernetes. The most crucial part of Kubernetes is probably the networking constructs it provides: how an internal network is set up and how communication can happen between processes, regardless of the nodes on which they run.

Operators

An operator is just a controller that monitors a CRD instead of a resource defined by Kubernetes.

A controller monitors a set of resources (resources are just entries in the etcd database, representing some kind of "desired state"), and reconciles the database contents with the "actual state" of the system. Whenever the controller sees differences between the desired and actual states, it takes steps to reduce that difference.

Describing a Cluster

Version

kubectl version --short

Master node address

Display address of master node 

kubectl cluster-info

View all available resources

kubectl api-resources

Completely Delete a Cluster

kubectl delete --all all
kubectl delete secret --all
kubectl delete serviceaccount --all
kubectl delete pvc --all
kubectl delete ingress --all
kubectl delete crds --all

Kubeconfig, Context and Namespace

Set namespace for the current context 

kubectl config set-context \
        $(kubectl config current-context) \
        --namespace default

Cluster ID

Clusters don't really have an identifier. Best idea I have seen is to use the kube-system namespace UID. For a discussion, see: Cluster ID API. 

kubectl get ns kube-system \
        -o jsonpath='{.metadata.uid}'

Affinity, Taints, and Tolerations

Kubernetes supports a way to indicate that specific nodes have a certain quality. Pods must opt-in to using those nodes.

That is, some nodes can say "hey, I'm a little bit different", and then some pods can say "I tolerate that difference".

An example is Azure's serverless pods. You can have physical nodes and virtual (i.e. serverless) nodes. By default, your pods will get scheduled to your physical nodes, but you could mark some pods as "tolerating" the Azure serverless nodes.

Affinity

A Pod can specify an affinity for certain nodes. Can either be a preference or a requirement.

Taints

A Node can have a taint that will not allow Pods to be scheduled to it unless the Pod can tolerate the taint.

Tolerations

A Pod can specify a toleration which allows it (but does not require it) to be scheduled to nodes with a matching taint.

ConfigMaps

kubectl get cm aws-auth \
        -n kube-system \
        --output=json \
    | jq -r '.data'

Deployments

Scaling a deployment

kubectl scale deployment <deployment-name> --replicas=1

Nodes

Master Node IP

Note you can also get the master node ip with kubectl cluster-info. 

kubectl get node -lnode-role.kubernetes.io/master \
        -o jsonpath='{ $.items[*].status.addresses[?(@.type=="ExternalIP")].address }'

Worker Nodes

kubectl get nodes \
        -l "kubernetes.io/role = node"

Worker Node IPs

Private IP

Without jq: 

kubectl get nodes -l "kubernetes.io/role = node" \
        -o jsonpath='{range .items[*]}{.metadata.annotations.flannel\.alpha\.coreos\.com/public-ip}
{end}'

With jq: 

kubectl get nodes \
        -l "kubernetes.io/role = node" \
        -o json \
    | jq -r '.items[]
             .metadata
             .annotations["flannel.alpha.coreos.com/public-ip"]'

Public IPs

kubectl get nodes -o json \
        -l "kubernetes.io/role = node" \
    | jq -r '.items[].status.addresses[]
             | select(.type == "ExternalIP")
             | .address'

All Pods on a Node

kubectl describe node $NodeName

kubectl get pods --all-namespaces \
        --field-selector spec.nodeName=$NodeName

SSH to a Pod's Node

node=$(kubectl get pod \
               -o=custom-columns=NODE:.spec.nodeName $applianceConfigPod \
           | sed -n 2p)
ip=$(kubectl get node $node \
             -o jsonpath='{ $.status.addresses[?(@.type=="ExternalIP")].address }')

Now, ssh user@nodeIp

Pods

List Containers in a Pod

kubectl get pod $pod -o json | jq -r '.spec.containers[].name'

What Node is this Pod on?

kubectl get pod \
        -o custom-columns=NODE:.spec.nodeName \
        $pod \
    | sed -n 2p

Secrets

Create Docker Registry Secret

kubectl create secret docker-registry $SecretName \
        --docker-server=$Server \
        --docker-username=$User \
        --docker-password=$Password

Decode Docker Registry Secret

TODO: also decode =.auths["registry.ironnet.cloud"].auth 

kubectl get secret $SecretName -o json \
    | jq -r '.data[".dockerconfigjson"]' \
    | base64 --decode | jq

Update a Secret

kubectl create secret docker-registry $SecretName \
        --docker-server=$Server \
        --docker-username=$User \
        --docker-password=$Password \
        --dry-run=client -o yaml | kubectl apply -f -

Services

Get Services of Type

Get all services of type NodePort/LoadBalancer/ClusterIP: 

kubectl get services --all-namespaces -o json \
    | jq '.items[]
          | select(.spec.type == "LoadBalancer")
          | .metadata.name'

Service Accounts

Get the default service account: 

kubectl get secret -o json \
    | jq '.items[]
          | select(.metadata.annotations["kubernetes.io/service-account.name"] == "default")'

StorageClass

Get the default StorageClass: 

kubectl get sc \
        -o json \
    | jq '.items[].metadata
          | select(.annotations."storageclass.kubernetes.io/is-default-class" == "true")
          | .name'

kubectl get sc \
        -o json \
    | jq -c '.items[].metadata
             | { "name": .name,
                 "default?": .annotations."storageclass.kubernetes.io/is-default-class" }'

Running Commands in Cluster

Create a temporary pod

Start a shell in a temporary pod in the namespace you want to test: 

kubectl run -it --rm --restart=Never alpine --image=alpine sh
apk add curl

Exec in an existing pod

If the pod has multiple containers, use -c $containerName as well. 

kubectl exec $podName -- <COMMAND>

Port forwarding with kubectl

kubectl port-forward $podName 8000:8000 -n $namespace

Using the API directily

https://kubernetes.io/docs/tasks/administer-cluster/access-cluster-api/#without-kubectl-proxy 

APISERVER=$(kubectl config view \
                    --minify \
                    -o jsonpath='{.clusters[0].cluster.server}')

SA=$(kubectl get serviceaccount default -o jsonpath='{.secrets[0].name}')
TOKEN=$(kubectl get secret $SA -o jsonpath='{.data.token}' | base64 --decode )
curl $APISERVER/api --header "Authorization: Bearer $TOKEN" --insecure

Debugging

Startig a busybox pod

kubectl run -i -t busybox --image=busybox --restart=Never

Get a shell on that pod 

kubectl exec -it <name-of-pod> -n my-ns sh

etcd

Pod to help with connecting 

curl -LO git.io/etcdclient.yaml

Which gets: https://gist.githubusercontent.com/mauilion/2bab4b00eb7a0ab4fca7023ae251e8ee/raw/etcdclient.yaml

Or, just ssh to the master, and download etcdctl on the master node.

Try looking at a secret at /registry/secrets/default/mysecret

Autoscaling

There are two kinds of autoscaling in Kubernetes:

  1. Scaling the number of Pods in Deployments/StatefulSets
  2. Scaling the number of Nodes in the Cluster

Scaling Pods

Horizontal Pod Autoscaler

Kubernetes has a built-in autoscaling mechanism called the Horizontal Pod Autoscaler. An HPA can be created and configured using standard Kubernetes objects: 

kubectl get hpa

Also, HPAs can be created with the special kubectl autoscale command: 

kubectl autoscale rs foo --min=2 --max=5 --cpu-percent=80

The HPA supports scaling based on metrics exposed through the Kubernetes Metrics API (i.e. what kubectl top uses). I think this means you need a metrics-server pod running in your cluster.