Monitor your Portworx cluster

Prerequisites

Portworx version 2.11 or later
Portworx Operator version 1.9.0 or later

Monitoring function

You can use the following technologies to monitor your Portworx cluster:

Prometheus to collect data
Alertmanager to provide notifications
Grafana to visualize your data

Verify monitoring using Prometheus

You can monitor your Portworx cluster using Prometheus. Portworx deploys Prometheus by default, but you can verify the deployment:

Verify that Prometheus pods are running by entering the following kubectl get pods command in the namespace where you deployed Portworx. For example:

kubectl -n kube-system get pods -A | grep -i prometheus

kube-system   prometheus-px-prometheus-0                              2/2     Running            0                59m
kube-system   px-prometheus-operator-59b98b5897-9nwfv                 1/1     Running            0                60m

Verify that the Prometheus px-prometheus and prometheus operated services exist by entering the following command:

kubectl -n kube-system  get service | grep -i prometheus

prometheus-operated         ClusterIP   None             <none>        9090/TCP                       63m
px-prometheus               ClusterIP   10.99.61.133     <none>        9090/TCP                       63m

Set up Alertmanager

Prometheus Alertmanager handles alerts sent from the Prometheus server based on rules you set. If any Prometheus rule is triggered, Alertmanager sends a corresponding notification to the specified receivers. You can configure these receivers using an Alertmanager config file. Perform the following steps to configure and enable Alertmanager:

Create a valid Alertmanager configuration file and name it alertmanager.yaml. The following is a sample for Alertmanager, and the settings used in your environment may be different:

global:
  # The smarthost and SMTP sender used for mail notifications.
  smtp_smarthost: 'smtp.gmail.com:587'
  smtp_from: 'abc@test.com'
  smtp_auth_username: "abc@test.com"
  smtp_auth_password: 'xyxsy'
route:
  group_by: [Alertname]
  # Send all notifications to me.
  receiver: email-me
receivers:
- name: email-me
  email_configs:
  - to: abc@test.com
    from: abc@test.com
    smarthost: smtp.gmail.com:587
    auth_username: "abc@test.com"
    auth_identity: "abc@test.com"
    auth_password: "abc@test.com"

Create a secret called alertmanager-portworx in the same namespace as your StorageCluster object:

kubectl -n kube-system create secret generic alertmanager-portworx --from-file=alertmanager.yaml=alertmanager.yaml

Edit your StorageCluster object to enable Alertmanager:

kubectl -n kube-system edit stc <px-cluster-name>

apiVersion: core.libopenstorage.org/v1
kind: StorageCluster
metadata:
  name: portworx
  namespace: kube-system
monitoring:
    prometheus:
      enabled: true
      exportMetrics: true
      alertManager:
        enabled: true

Verify that the Alertmanager pods are running using the following command:

kubectl -n kube-system get pods | grep -i alertmanager

alertmanager-portworx-0                    2/2     Running   0              4m9s
alertmanager-portworx-1                    2/2     Running   0              4m9s
alertmanager-portworx-2                    2/2     Running   0              4m9s

NOTE: To view the complete list of out-of-the-box default rules, see step 7 below.

Access the Alertmanager UI

To access the Alertmanager UI and view the Alertmanager Status and alerts, you need to set up port forwarding and browse to the specified port. In this example, port forwarding is provided for ease of access to the Alertmanager service from your local machine using the port 9093.

Set up port forwarding:

kubectl -n kube-system port-forward service/alertmanager-portworx --address=<masternodeIP> 9093:9093

Access Prometheus UI by browsing to http://<masternodeIP>:9093/#/status

NOTE: PX-Central on-premises includes Grafana and Portworx dashboards natively, which you can use to monitor your Portworx cluster. Refer to the PX-Central documentation for further details.

Access the Prometheus UI

To access the Prometheus UI to view Status, Graph and default Alerts, you also need to set up port forwarding and browse to the specified port. In this example, Port forwarding is provided for ease of access to the Prometheus UI service from your local machine using the port 9090.

Set up port forwarding:

kubectl -n kube-system port-forward service/px-prometheus 9090:9090

Prometheus Status

Access the Prometheus UI by browsing to http://localhost:9090/alerts.

View provided Prometheus rules

To view the complete list of out-of-the-box default rules used for event notifications, perform the following steps.

Get the Prometheus rules:

kubectl -n kube-system get prometheusrules

NAME       AGE
portworx   46d

Save the Prometheus rules to a YAML file:

kubectl -n kube-system get prometheusrules portworx -o yaml > prometheusrules.yaml

View the contents of the file:
```
cat prometheusrules.yaml
```

Configure Grafana

You can connect to Prometheus using Grafana to visualize your data. Grafana is a multi-platform open source analytics and interactive visualization web application. It provides charts, graphs, and alerts.

Enter the following commands to download the Grafana dashboard and datasource configuration files:

curl -O https://docs.portworx.com/samples/k8s/pxc/grafana-dashboard-config.yaml

% Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                Dload  Upload   Total   Spent    Left  Speed
100   211  100   211    0     0    596      0 --:--:-- --:--:-- --:--:--   596

curl -O https://docs.portworx.com/samples/k8s/pxc/grafana-datasource.yaml

% Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                Dload  Upload   Total   Spent    Left  Speed
100  1625  100  1625    0     0   4456      0 --:--:-- --:--:-- --:--:--  4464

Create a configmap for the dashboard and data source:

kubectl -n kube-system create configmap grafana-dashboard-config --from-file=grafana-dashboard-config.yaml

kubectl -n kube-system create configmap grafana-source-config --from-file=grafana-datasource.yaml

Download and install Grafana dashboards using the following commands:

curl "https://docs.portworx.com/samples/k8s/pxc/portworx-cluster-dashboard.json" -o portworx-cluster-dashboard.json && \
curl "https://docs.portworx.com/samples/k8s/pxc/portworx-node-dashboard.json" -o portworx-node-dashboard.json && \
curl "https://docs.portworx.com/samples/k8s/pxc/portworx-volume-dashboard.json" -o portworx-volume-dashboard.json && \
curl "https://docs.portworx.com/samples/k8s/pxc/portworx-performance-dashboard.json" -o portworx-performance-dashboard.json && \
curl "https://docs.portworx.com/samples/k8s/pxc/portworx-etcd-dashboard.json" -o portworx-etcd-dashboard.json

kubectl -n kube-system create configmap grafana-dashboards \
--from-file=portworx-cluster-dashboard.json \
--from-file=portworx-performance-dashboard.json \
--from-file=portworx-node-dashboard.json \
--from-file=portworx-volume-dashboard.json \
--from-file=portworx-etcd-dashboard.json

Enter the following command to download and install the Grafana YAML file:
```
kubectl apply -f https://docs.portworx.com/samples/k8s/pxc/grafana.yaml
```

Verify if the Grafana pod is running using the following command:

kubectl -n kube-system get pods | grep -i grafana

grafana-7d789d5cf9-bklf2                   1/1     Running   0              3m12s

Access Grafana by setting up port forwarding and browsing to the specified port. In this example, port forwarding is provided for ease of access to the Grafana service from your local machine using the port 3000:
```
kubectl -n kube-system port-forward service/grafana 3000:3000
```
Navigate to Grafana by browsing to http://localhost:3000.
Enter the default credentials to log in.
- login: admin
- password: admin

Activate your Enterprise license