Initializing the Kubernetes cluster

Kubernetes and pre-requisites (every node)

Install Kubernetes on Ubuntu 18.04. Assuming version 1.14.3 is pulled, check how to fix version. On new systems, copy over the install script from the master node.

> cd init
> ./install_kubernetes.sh

Reconfigure docker runtime. Edit /etc/docker/daemon.json as follows:

{
  "exec-opts": ["native.cgroupdriver=systemd"],
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m"
  },
  "storage-driver": "overlay2"
}

On nodes with an nVidia GPU, add the following:

  "runtimes": {
    "nvidia": {
      "path": "nvidia-container-runtime",
      "runtimeArgs": []
    }
  }

Restart docker daemon:

> mkdir -p /etc/systemd/system/docker.service.d
> systemctl daemon-reload
> systemctl restart docker

Make sure swap is off

> sudo swapoff -a

Check /etc/fstab if swap is still configured there, delete if this is the case.

Spin up the master node

Use kubeadm with vanilla defaults to initialize the control plane.

> sudo systemctl enable docker.service
> sudo kubeadm init

If this fails at any point, use kubeadm reset after problems have been fixed before trying to re-initialize.

• Post-init steps to setup admin user on this account

> cd init
> ./finalize_master.sh

Update kubelet configuration for master node

Edit /etc/kubernetes/manifests/kube-controller-manager.yaml:

spec:
  containers:
  - command:
    # add these two
    - --allocate-node-cidrs=true
    - --cluster-cidr=10.244.0.0/16

Copy certs/ca.crt (certificate for ccu.uni-konstanz.de) to /usr/share/ca-certificates/ca-dex.pem.

Edit /etc/kubernetes/manifests/kube-apiserver.yaml:

spec:
  containers:
  - command:
    # add these five
    - --oidc-issuer-url=https://ccu.uni-konstanz.de:32000/dex
    - --oidc-client-id=loginapp
    - --oidc-ca-file=/usr/share/ca-certificates/ca-dex.pem
    - --oidc-username-claim=name
    - --oidc-groups-claim=groups

Daemonsets on Master node

Flannel daemonset (pod network for communication)

> cd init
> ./start_pod_network.sh

nVidia daemonset

> cd init
> ./deploy_nvidia_device_plugin.sh

The daemonset should be active on any node with an nVidia GPU.

Authentication systems

DEX with LDAP

TODO: outdated, switched to containerized DEX. Check what still needs to be done.

Set up according to this tutorial with customized install scripts in kubernetes/init/dex/

1. Create secrets for TLS connections, use certs for ccu.uni-konstanz.de
   1. Modify ca-cm.yml to contain correct ca.
   2. Run upload_ccu_tls.sh
2. Spin up login application service.
   1. Modify loginapp-cm.yml: server config
   2. Modify loginapp-ing-srv.yml: service data, mapping of ports to outside world
   3. Modify loginapp-deploy.yml: ID secret for TLS
   4. Run start-login-service.sh
3. Spin up dex
   1. Modify dex-cm.yml: server data and LDAP configuration
   2. Modify dex-ing-srv.yml: service data, mapping of ports to outside world
   3. Modify dex-deploy.yml: ID secret for TLS
   4. Run start-dex-service.sh