Istio — Customizing metrics

Istio — Service Mesh gives users a lot of flexibility in deploying and managing applications on Kubernetes. It provides observability, tracing and monitoring features for the applications.

Istio extends the envoy filter support using EnvoyFilter CRD. This filter capability gives developers more flexibility in adding any custom plugins related to monitoring, logging or tracing to the application.

In this blog, we will look at how to customize Istio’s metrics. We would be adding another field request_url_pathto the existing requests_total metric. This would help us differentiate how many requests are being hit to which URL path.

To test the use case we will be using the following setup.

  • Ubuntu 20.04 OS
  • Kubernetes v1.21.5
  • Docker 20.10.8
  • Istio 1.11.0

We deployed a single node Kubernetes cluster with calico CNI.

Below are the steps to download Istio.

# Download istio
$ curl -L https://istio.io/downloadIstio | ISTIO_VERSION=1.11.0 TARGET_ARCH=x86_64 sh -

Before installing Istio, we have to edit the demo profile to add the request_url_path attribute to the mesh Configuration. Below are the changes required to manifests/profiles/demo.yaml configuration file. Same config file is available here.

# Move to istio folder
$ cd istio-1.11.0
# Edit the manifests/profiles/demo.yaml file with the below changes.
spec:
meshConfig:
defaultConfig:
extraStatTags:
- request_url_path

After making the required changes in the demo profile, install Istio using istioctl command.

# Install or Configuring Istio
$ bin/istioctl install -f manifests/profiles/demo.yaml -y
✔ Istio core installed
✔ Istiod installed
✔ Ingress gateways installed
✔ Egress gateways installed
✔ Installation complete
Thank you for installing Istio 1.11. Please take a few minutes to tell us about your install/upgrade experience! https://forms.gle/kWULBRjUv7hHci7T6

Check to see if the pods are in running state

$ kubectl get pods -n istio-system
NAME READY STATUS RESTARTS AGE
istio-system istio-egressgateway-6cb476c7d4-mrlbv 1/1 Running 0 19s
istio-system istio-ingressgateway-68fcc69f84-xhjc9 1/1 Running 0 19s
istio-system istiod-79b65d448f-799gz 1/1 Running 0 30s

Patch ingress service to Node port for easy access

$ kubectl patch svc -n istio-system istio-ingressgateway --type='json' -p '[{"op":"replace","path":"/spec/type","value":"NodePort"}]'
service/istio-ingressgateway patched

To test the use case we will deploy a sample golang application. Source code is available here. This is a basic golang application with 2 URL support. One is the root URL and the other is a login URL. Before deploying, the default namespace is labelled for Istio’s side car injection.

# Deploying the application
$ kubectl label namespace default istio-injection=enabled
namespace/default labeled
$ kubectl apply -f https://raw.githubusercontent.com/SirishaGopigiri/Istio-WASM-plugin/main/url_path_metrics/deployment.yaml
deployment.apps/appdeploy created
service/appservice created
# Creating gateway and virtual service to access the application
$ kubectl apply -f https://raw.githubusercontent.com/SirishaGopigiri/Istio-WASM-plugin/main/url_path_metrics/gateway.yaml
gateway.networking.istio.io/appdeploy-gateway created
$ kubectl apply -f https://raw.githubusercontent.com/SirishaGopigiri/Istio-WASM-plugin/main/url_path_metrics/virtualservice.yaml
virtualservice.networking.istio.io/appdeploy created

Check the pod status.

$ kubectl get pods
NAME READY STATUS RESTARTS AGE
appdeploy-54d5889d75-n2kzv 1/1 Running 0 49s
$ kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
appservice ClusterIP 10.98.171.214 <none> 5000/TCP 61s
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 7m44s

Check to see if service is accessible.

$ kubectl get svc -n istio-system
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
istio-egressgateway ClusterIP 10.102.128.113 <none> 80/TCP,443/TCP 2m10s
istio-ingressgateway NodePort 10.97.132.174 <none> 15021:32509/TCP,80:31084/TCP,443:32498/TCP,31400:32284/TCP,15443:31180/TCP 2m9s
istiod ClusterIP 10.105.76.208 <none> 15010/TCP,15012/TCP,443/TCP,15014/TCP 2m21s
$ curl -X GET http://127.0.0.1:31084{"message":"hello world!!"}$ curl -H "username:john" -X GET http://127.0.0.1:31084/login
{"message":"hello john!! Login Successful!!"}

Before customizing metrics, we will observe the requests_total metric captured by Istio.

$ kubectl exec "$(kubectl get pod -l app=golang -o jsonpath='{.items[0].metadata.name}')" -c istio-proxy -- curl -sS 'localhost:15000/stats/prometheus'| grep istio_requests# TYPE istio_requests_total counteristio_requests_total{response_code="200",reporter="destination",source_workload="istio-ingressgateway",source_workload_namespace="istio-system",source_principal="spiffe://cluster.local/ns/istio-system/sa/istio-ingressgateway-service-account",source_app="istio-ingressgateway",source_version="unknown",source_cluster="Kubernetes",destination_workload="appdeploy",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/default",destination_app="golang",destination_version="unknown",destination_service="appservice.default.svc.cluster.local",destination_service_name="appservice",destination_service_namespace="default",destination_cluster="Kubernetes",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="istio-ingressgateway",destination_canonical_service="golang",source_canonical_revision="latest",destination_canonical_revision="latest"} 2

We can observe that the count is currently 2 since we have tested the service once with both URLs ‘/’ and ‘/login’.

Below is the Envoy filter where we will capture the URL based on which we will populate the requests_total metric.

# Envoy Filter to capture url_path as attributeapiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
name: istio-url-filter
spec:
workloadSelector:
labels:
app: golang
configPatches:
- applyTo: HTTP_FILTER
match:
context: SIDECAR_INBOUND
proxy:
proxyVersion: '1\.11.*'
listener:
filterChain:
filter:
name: "envoy.http_connection_manager"
subFilter:
name: "istio.stats"
patch:
operation: INSERT_BEFORE
value:
name: istio.attributegen
typed_config:
"@type": type.googleapis.com/udpa.type.v1.TypedStruct
type_url: type.googleapis.com/envoy.extensions.filters.http.wasm.v3.Wasm
value:
config:
configuration:
"@type": type.googleapis.com/google.protobuf.StringValue
value: |
{
"attributes": [
{
"output_attribute": "istio_url_path",
"match": [
{
"value": "login",
"condition": "request.url_path == '/login'"
},
{
"value": "root",
"condition": "request.url_path == '/'"
}
]
}

]
}
vm_config:
runtime: envoy.wasm.runtime.null
code:
local: { inline_string: "envoy.wasm.attributegen" }

Few fields to observe

  • Workload selector specifies on which application the filter should be applied to
  • INSERT_BEFORE — This filter is being applied before the stats filter so that the attribute is added.
  • Output_attribute — This key is later referenced to populate the requests_total metric. Based on the URL login the value is populated.

Now we will apply this filter to the cluster to read the attribute.

$ kubectl -n istio-system apply -f https://raw.githubusercontent.com/SirishaGopigiri/Istio-WASM-plugin/main/url_path_metrics/url_filter.yaml
envoyfilter.networking.istio.io/istio-url-filter created

Next step is to update the stats filter in Istio to add this attribute to requests_total metric. Retrieve the existing filter and update it.

$ kubectl -n istio-system get envoyfilter stats-filter-1.11 -o yaml > stats-filter-1.11.yaml

Edit the inbound configuration to add the output attribute. Complete yaml file is available here.

       value: |
{
"debug": "false",
"stat_prefix": "istio",
"disable_host_header_fallback": true,
"metrics": [
{
"dimensions": {
"destination_cluster": "node.metadata['CLUSTER_ID']",
"source_cluster": "downstream_peer.cluster_id"
}
},
{
"name": "requests_total",
"dimensions": {
"request_url_path": "istio_url_path"
}
}
]
}

Once edited, apply the changes and wait for some time for the envoy to load, approximately 30–60sec.

$ kubectl apply -f stats-filter-1.11.yaml
envoyfilter.networking.istio.io/stats-filter-1.11 configured

To validate the configuration, we will now hit the same url and check the metrics in istio-proxy pod to see if the new attribute is captured or not.

$ curl -X GET http://127.0.0.1:31084
{"message":"hello world!!"}
$ curl -H "username:john" -X GET http://127.0.0.1:31084/login
{"message":"hello john!! Login Successful!!"}
## Checking the metrics
$ kubectl exec "$(kubectl get pod -l app=golang -o jsonpath='{.items[0].metadata.name}')" -c istio-proxy -- curl -sS 'localhost:15000/stats/prometheus'| grep requests
istio_requests_total{response_code="200",reporter="destination",source_workload="istio-ingressgateway",source_workload_namespace="istio-system",source_principal="spiffe://cluster.local/ns/istio-system/sa/istio-ingressgateway-service-account",source_app="istio-ingressgateway",source_version="unknown",source_cluster="Kubernetes",destination_workload="appdeploy",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/default",destination_app="golang",destination_version="unknown",destination_service="appservice.default.svc.cluster.local",destination_service_name="appservice",destination_service_namespace="default",destination_cluster="Kubernetes",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="istio-ingressgateway",destination_canonical_service="golang",source_canonical_revision="latest",destination_canonical_revision="latest"} 2istio_requests_total{response_code="200",reporter="destination",source_workload="istio-ingressgateway",source_workload_namespace="istio-system",source_principal="spiffe://cluster.local/ns/istio-system/sa/istio-ingressgateway-service-account",source_app="istio-ingressgateway",source_version="unknown",source_cluster="Kubernetes",destination_workload="appdeploy",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/default",destination_app="golang",destination_version="unknown",destination_service="appservice.default.svc.cluster.local",destination_service_name="appservice",destination_service_namespace="default",destination_cluster="Kubernetes",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="istio-ingressgateway",destination_canonical_service="golang",source_canonical_revision="latest",destination_canonical_revision="latest",request_url_path="login"} 1istio_requests_total{response_code="200",reporter="destination",source_workload="istio-ingressgateway",source_workload_namespace="istio-system",source_principal="spiffe://cluster.local/ns/istio-system/sa/istio-ingressgateway-service-account",source_app="istio-ingressgateway",source_version="unknown",source_cluster="Kubernetes",destination_workload="appdeploy",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/default",destination_app="golang",destination_version="unknown",destination_service="appservice.default.svc.cluster.local",destination_service_name="appservice",destination_service_namespace="default",destination_cluster="Kubernetes",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="istio-ingressgateway",destination_canonical_service="golang",source_canonical_revision="latest",destination_canonical_revision="latest",request_url_path="root"} 1

This confirms that a new attribute is added to the existing metrics, with the appropriate values for request_url_path. Also note that the count is 1, since we sent the curl requests to ‘/’ and ‘/login’ once.

In this blog, we walked through how to customize metrics and adding our own attributes to the existing metrics. These attributes can be used in the next steps while computing statistics or taking any action of interest based on the statistics.

  1. https://istio.io/latest/docs/tasks/observability/metrics/customize-metrics/
  2. https://istio.io/latest/docs/reference/config/proxy_extensions/attributegen/
  3. https://github.com/gin-gonic/gin

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Sirishagopigiri

Engineer by profession. Chef by passion (applicable only for some dishes :-P). Trying to become a blogger.