Kubernetes Architecture

A Kubernetes cluster consists of some master components and some worker components. In a single node Kubernetes cluster, master and worker components may run on the same node. In a production Kubernetes cluster you typically have one or more master nodes and many worker nodes.

Kubernetes API server is the primary interface to an end user - either using kubectl or client-go or an application using the Kubernetes API.

Kubernetes API server manages versioned resources. You can add new resource types to a Kubernetes API server. Pods, Deployments, Services are examples on built in resources or Kinds. In this tutorial we will be working with custom resources like Experiments, Suggestions and Trials. Resources are grouped together into API groups and are versioned.

You can explore the API groups and resources available on your Kubernetes cluster as follows.

kubectl api-versions

You can see the available Kubernetes resources(Kinds) as follows.

kubectl api-resources

Pod

Specification for a Kubernetes resource can be done via yaml file. Kubernetes manages pods instead of containers. A pod can contain one or more containers. Containers in a pod share resources and common local network. As we will see during Katib section of the tutorial, Katib injects a metrics container to the model training pod. Here is a yaml file to run mnist example as a pod. The max_steps is set to 1 to speed-up running the mnist example.

apiVersion: v1
kind: Pod
metadata:
  name: mnistpod
spec:
  containers:
  - name: mnist
    image: gcr.io/kubeflow-ci/tf-mnist-with-summaries:1.0
    command:
    - "python"
    - "/var/tf_mnist/mnist_with_summaries.py"
    - "--max_steps=1"        
    - "--batch_size=2"        
  restartPolicy: Never

cd $HOME
git clone https://github.com/tfworldkatib/tutorial.git
cd $HOME/tutorial/examples
kubectl apply -f mnistpod.yaml

Check that the Pod mnistpod has started.

kubectl get pods

Check the logs of the Pod mnistpod

kubectl logs -f mnistpod 

Notice the Accuracy output from the mnistpod. This will be used by Katib to find accuracy results from a given hyperparameter set.

Delete the Pod mnistpod

kubectl delete -f mnistpod.yaml

CRD (Custom Resource Definition)

Kubernetes supports various types of extension capabilities at all layers, starting with the API server, scheduler, controllers all the way to the kubelet. One of the common patterns used to add new resources and capabilities to the Kubernetes API server is called the Operator Pattern. This consists of creating a custom resource(Kind) and a controller that manages this custom resource. Kubeflow and Katib use this extensively. This allows Kubeflow and Katib to be integrated with Kubernetes. You can manage and interact with Kubeflow and Katib components just as you interact with any other Kubernetes component!

You can see the available Kubernetes custom resource definitions as follows.

kubectl get crds