Integrate Prometheus with OSM

Prometheus and OSM Integration

To familiarize yourself on how OSM works with Prometheus, try installing a new mesh with sample applications to see which metrics are collected.

  1. Install OSM with its own Prometheus instance:

    $ osm install --set osm.deployPrometheus=true,osm.enablePermissiveTrafficPolicy=true
OSM installed successfully in namespace [osm-system] with mesh name [osm]

  2. Create a namespace for sample workloads:

    $ kubectl create namespace metrics-demo
namespace/metrics-demo created

  3. Make the new OSM monitor the new namespace:

    $ osm namespace add metrics-demo
Namespace [metrics-demo] successfully added to mesh [osm]

  4. Configure OSM’s Prometheus to scrape metrics from the new namespace:

    $ osm metrics enable --namespace metrics-demo
Metrics successfully enabled in namespace [metrics-demo]

  5. Install sample applications:

    $ kubectl apply -f https://raw.githubusercontent.com/openservicemesh/osm-docs/release-v1.1/manifests/samples/curl/curl.yaml -n metrics-demo
serviceaccount/curl created
deployment.apps/curl created
$ kubectl apply -f https://raw.githubusercontent.com/openservicemesh/osm-docs/release-v1.1/manifests/samples/httpbin/httpbin.yaml -n metrics-demo
serviceaccount/httpbin created
service/httpbin created
deployment.apps/httpbin created

    Ensure the new Pods are Running and all containers are ready:

    $ kubectl get pods -n metrics-demo
NAME                       READY   STATUS    RESTARTS   AGE
curl-54ccc6954c-q8s89      2/2     Running   0          95s
httpbin-8484bfdd46-vq98x   2/2     Running   0          72s

  6. Generate traffic:

    The following command makes the curl Pod make about 1 request per second to the httpbin Pod forever:

    $ kubectl exec -n metrics-demo -ti "$(kubectl get pod -n metrics-demo -l app=curl -o jsonpath='{.items[0].metadata.name}')" -c curl -- sh -c 'while :; do curl -i httpbin.metrics-demo:14001/status/200; sleep 1; done'
HTTP/1.1 200 OK
server: envoy
date: Tue, 23 Mar 2021 17:27:44 GMT
content-type: text/html; charset=utf-8
access-control-allow-origin: *
access-control-allow-credentials: true
content-length: 0
x-envoy-upstream-service-time: 1

HTTP/1.1 200 OK
server: envoy
date: Tue, 23 Mar 2021 17:27:45 GMT
content-type: text/html; charset=utf-8
access-control-allow-origin: *
access-control-allow-credentials: true
content-length: 0
x-envoy-upstream-service-time: 2

...

  7. View metrics in Prometheus:

    Forward the Prometheus port:

    $ kubectl port-forward -n osm-system $(kubectl get pods -n osm-system -l app=osm-prometheus -o jsonpath='{.items[0].metadata.name}') 7070
Forwarding from 127.0.0.1:7070 -> 7070
Forwarding from [::1]:7070 -> 7070

    Navigate to http://localhost:7070 in a web browser to view the Prometheus UI. The following query shows how many requests per second are being made from the curl pod to the httpbin pod, which should be about 1:

    irate(envoy_cluster_upstream_rq_xx{source_service="curl", envoy_cluster_name="metrics-demo/httpbin"}[30s])

    Feel free to explore the other metrics available from within the Prometheus UI.

  8. Cleanup

    Once you are done with the demo resources, clean them up by first deleting the application namespace:

    $ kubectl delete ns metrics-demo
namespace "metrics-demo" deleted

    Then, uninstall OSM:

    $ osm uninstall mesh
Uninstall OSM [mesh name: osm] ? [y/n]: y
OSM [mesh name: osm] uninstalled

    To remove OSM’s cluster wide resources after uninstallation, run the following command. See the uninstall guide for more context and information.

    $ osm uninstall mesh --delete-cluster-wide-resources