Editing Tutorials:Monitoring with Tensorboard on the GPU cluster (section)

== Logging to Tensorboard from within your container ==

In your job file, make sure both PVC are mounted to the container. We use "/mnt/tensorboard" as the mount point for the tensorboard log directory.

<syntaxhighlight lang="yaml">
...
       containers:
       - name: your-username-tf-mnist-tb
        volumeMounts:
        - mountPath: "/tmp/data"
          name: pvc-mnist
        - mountPath: "/mnt/tensorboard"
          name: pvc-mnist-tb
...
      volumes:
        - name: pvc-mnist
          persistentVolumeClaim:
            claimName: test-user-tf-mnist-pvc
        - name: pvc-mnist-tb
          persistentVolumeClaim:
            claimName: test-user-tf-mnist-tb-pvc
...
</syntaxhighlight>

We will not cover the details of Tensorboard logging here, see the example code in "application/src/tf_mnist.py" for some initial ideas. Make sure to provide the correct log directory when creating the writer instance for the logs. I suggest to create a new subdirectory for each run of the program and hold the PVC, so that you can compare different runs, like this:

<syntaxhighlight lang="python">
from datetime import datetime

tb_base_directory = "/mnt/tensorboard/"
now = datetime.now()
subdir = now.strftime("%Y%m%d-%H%M%S")
tb_out_directory = tb_base_directory + subdir
writer = tf.summary.FileWriter( tb_out_directory, sess.graph )
</syntaxhighlight>

Otherwise, please refer to some of the excellent online tutorials on Tensorboard, e.g.
[https://itnext.io/how-to-use-tensorboard-5d82f8654496 this here].