Pod Monitoring with Grafana Dashboards

Grafana is essential for real-time monitoring of pod metrics in Kubernetes, allowing users to create dynamic dashboards for tracking and analyzing cluster performance with ease. Elevate monitoring with Grafana's intuitive interface and customization options.

First, Prometheus and Grafana must be installed in the Kubernetes cluster. You can see my previous article here for detailed steps on setting that up.

The next step is connecting to Grafana.  There are 2 steps needed to achieve that:

1. Port forward the Grafana service from the Kubernetes cluster to the desktop with the following command:
   1. kubectl port-forward svc/kube-prometheus-stack-grafana 3000:80 -n metrics
2. Open a browser and connect on http://localhost:3000/

The username to log in is admin and the password can be found by running the following command and decoding the results in base64 (or a cursory Google search typically yields the default value):

kubectl get secret kube-prometheus-stack-grafana -o jsonpath="{.data.admin-password}" -n metrics

Once logged in to Grafana, select the Dashboards link in the sidebar.  A list of pre-configured dashboards can be reviewed for samples of what can be done in Grafana.  Click the New button then select New Dashboard to start with a fresh dashboard.  From here, panels can be added, click Add visualization then select Prometheus as the data source to get started.  As a sample of monitoring pod memory usage, configure as follows:

1. Under Metric, select container_memory_usage_bytes
2. Under Label filters, select pod = <select the name of the pod from the dropdown>
3. Click Run queries and validate data is coming through as expected
4. Optionally update the Panel options on the right side
5. Click Apply to save the changes

Pod memory should now be coming through in the panel.  Additional panels can be added and configured by selecting Add -> Visualization.  Panels can also be resized and moved by dragging the bottom right of the panel or dragging the top of the panel.

Additionally, it's possible to configure inputs for dynamically choosing which pod metrics to display.  To do that:

1. Select the Dashboard Settings gear above the panels
2. Click the Variables tab
3. Click Add variable
4. Numerous types can be added here but for this example, select:
   
   1. Variable type: Text box
   2. Name: podname
   3. Label: Pod Name
   4. Click Apply then close
5. Click edit on the visualization created earlier
6. Above the metric, click Code next to Builder to swap the Metric to code mode
7. Update the metric so it is (podname is the variable name configured above, for additional parameters, unique names should be used):
   1. container_memory_usage_bytes{pod="$podname"}
8. Click Apply
9. Enter the name of the pod to display metrics for in the Pod Name text field above the panels then click enter

Pod metrics for the entered pod should now come through in the graph!  Numerous other metrics can also be displayed and it's even possible to set up multiple panels to display pod data side by side.  Experiment around with the panels to see what's possible!

Finally, when not using persistent storage with the Prometheus Stack app, dashboards will not be saved when the pod is restarted.  We recommend saving the dashboard configuration by:

1. Select the Dashboard Settings gear above the panel
2. Select the JSON Model tab and copy out the entire JSON structure

Then if the pod is restarted or if importing into another environment:

1. Navigate to Dashboards in the Grafana app
2. Select New -> Import
3. Copy the JSON model saved previously into the Import via the dashboard JSON model

It's also possible to use the metrics here for Horizontal Pod Auto-scaling, stay tuned to this space for more information on that!