Editing Tutorials:Monitoring with Tensorboard on the GPU cluster (section)

=== Option 1: Using compute node global Tensorboard instance ===

As stated earlier, each compute node has its own instance of Tensorboard running. This instance will automatically display all Tensorboard summary files contained in persistent volumes with <code>storageClassName: local-tensorboard</code>.

First, find out the compute node your pod was allocated to.

<syntaxhighlight lang="python">
> kubectl get pods | grep your-username
NAME                                  READY   STATUS      RESTARTS   AGE
your-username-tf-mnist-tb-pvc-mqt9m   1/1     Running     0          3m4s

> kubectl describe pod your-username-tf-mnist-tb-pvc-mqt9m | grep Node
Node:               glasya/134.34.226.30
</syntaxhighlight>

Your pod is running on [[Cluster:Compute nodes|Glasya]], IP 134.34.226.30. You can now point your browser to 134.34.226.30:6116 to access the Tensorboard instance for the node. Note that it lists the logs for all currently mounted PVs. To find out which directory your PV corresponds to, you need to check which PV your PVC was bound to, and inspect its data:

<syntaxhighlight lang="python">
> kubectl get pvc | grep your-username
your-username-tf-mnist-tb-pvc   Bound    local-pv-d07aa16c   25Gi       RWO            local-tensorboard   19m

> kubectl describe pv local-pv-d07aa16c | grep Path
Path:  /mnt/tensorboard/glasya-pv-tb-25gb-2
</syntaxhighlight>

This means that your logs will be the ones prefixed by "glasya-pv-tb-25gb-2" in the Tensorboard instance.