Ingesting custom metrics from applications

5 minutes 5 Questions

Custom metrics in Google Cloud allow you to monitor application-specific data points that are not covered by built-in metrics. This capability is essential for gaining deeper insights into your application's performance and behavior. To ingest custom metrics from applications, you primarily use Cloud Monitoring (formerly Stackdriver Monitoring). The process involves several key steps. First, you need to instrument your application code to collect the metrics you want to track. This could include business-specific measurements like order counts, user actions, or processing times. Google provides client libraries for popular programming languages including Python, Java, Go, and Node.js. These libraries simplify the process of sending metric data to Cloud Monitoring. You create metric descriptors that define the structure of your custom metrics, including the metric type, labels, and value type. When writing metrics, you create time series data points that include timestamps and values. The Monitoring API accepts these data points and stores them for analysis. For containerized applications running on Google Kubernetes Engine, you can use the OpenTelemetry framework or the Prometheus adapter to export custom metrics. This approach provides flexibility in how metrics are collected and transmitted. Once ingested, custom metrics appear in the Cloud Monitoring console alongside standard metrics. You can create dashboards to visualize this data, set up alerting policies to receive notifications when thresholds are breached, and use the data for capacity planning. Best practices include using meaningful metric names with appropriate prefixes, adding relevant labels for filtering and grouping, and avoiding excessive cardinality in label values. Rate limiting and batching of metric writes help optimize costs and performance. Custom metrics are billed based on the volume of data ingested, so understanding your monitoring requirements helps manage expenses effectively while maintaining operational visibility.

Ingesting Custom Metrics from Applications - Complete Guide

Why Is This Important?

Custom metrics allow you to monitor application-specific data that Google Cloud's built-in metrics don't capture. This is essential for understanding application performance, business KPIs, and operational health. For the GCP Associate Cloud Engineer exam, understanding how to collect and use custom metrics demonstrates your ability to implement comprehensive monitoring solutions.

What Are Custom Metrics?

Custom metrics are user-defined measurements that you create to track specific aspects of your applications. Unlike built-in metrics that Google Cloud automatically collects (CPU, memory, network), custom metrics let you monitor:
- Application response times
- Business transactions per second
- Queue depths
- User session counts
- Any other application-specific data

How It Works

1. Cloud Monitoring API
The primary method for ingesting custom metrics is through the Cloud Monitoring API (formerly Stackdriver). You create metric descriptors that define the metric type, labels, and value type, then write time series data points.

2. OpenTelemetry
Google Cloud supports OpenTelemetry, an open-source observability framework. You can instrument your application with OpenTelemetry SDKs and export metrics to Cloud Monitoring.

3. Ops Agent
For applications running on Compute Engine or GKE, the Ops Agent can collect custom metrics from applications that expose them via protocols like StatsD or Prometheus format.

4. Client Libraries
Google provides client libraries in multiple languages (Python, Java, Go, Node.js) that simplify sending custom metrics to Cloud Monitoring.

Key Components

- Metric Descriptor: Defines the metric name, type (gauge, cumulative, delta), labels, and unit
- Time Series: The actual data points with timestamps
- Monitored Resource: The entity the metric is associated with (VM, container, etc.)
- Labels: Key-value pairs that add dimensions to metrics for filtering

Metric Types

- Gauge: Point-in-time measurements (current temperature)
- Cumulative: Values that only increase (total requests served)
- Delta: Change since last measurement

Exam Tips: Answering Questions on Ingesting Custom Metrics from Applications

Key Services to Remember:
- Cloud Monitoring is the primary service for custom metrics
- Custom metrics use the prefix custom.googleapis.com/
- Prometheus metrics use prometheus.googleapis.com/

Common Exam Scenarios:

1. When asked about monitoring application-specific data not available in built-in metrics, choose Cloud Monitoring custom metrics

2. For containerized workloads on GKE exposing Prometheus metrics, look for answers involving Google Cloud Managed Service for Prometheus or the Ops Agent

3. Questions about cost optimization should note that custom metrics incur charges based on the number of time series and data points ingested

4. For questions about instrumenting code, client libraries and OpenTelemetry are the recommended approaches

Watch Out For:

- Answers suggesting BigQuery for real-time metric ingestion (BigQuery is for analytics, not monitoring)
- Confusion between logs and metrics - they serve different purposes
- Rate limits: custom metrics have quotas on time series creation

Best Practices to Know:

- Use meaningful metric names following naming conventions
- Apply labels strategically but avoid high-cardinality labels
- Set appropriate metric kinds based on what you're measuring
- Create alerting policies on custom metrics for proactive monitoring

Quick Reference for Exam

- API: monitoring.googleapis.com
- Metric prefix: custom.googleapis.com/[metric_name]
- Required IAM role: roles/monitoring.metricWriter
- View metrics in: Cloud Monitoring Metrics Explorer
Test mode:
More Ingesting custom metrics from applications questions
26 questions (total)

Other Ensuring Successful Operation of a Cloud Solution concepts

Active Assist for resource utilizationAdding a subnet to an existing VPCAdding custom static routes in a VPCAdding, editing, and removing node poolsAdjusting application traffic splittingAutoscaling node poolsBacking up and restoring database instancesCloud ProfilerCloud TraceConfiguring and deploying Ops AgentConfiguring audit logsConfiguring autoscaling for Cloud RunConfiguring GKE to access Artifact RegistryConfiguring log bucketsCreating and viewing imagesCreating Cloud Monitoring alertsCreating Cloud Monitoring custom metricsDeploying new Cloud Run application versionsEstimating costs of data storage resourcesExecuting queries to retrieve dataExpanding a subnet for more IP addressesExporting logs to BigQueryExporting logs to external systemsGemini Cloud Assist for Cloud MonitoringGKE nodes, Pods, and ServicesGoogle Cloud Managed Service for PrometheusHorizontal Pod autoscalingIndex advisorKubernetes PodsKubernetes ServicesKubernetes StatefulSetsLog analyticsLog routersManaging GKE Autopilot Pod resource requestsManaging objects in Cloud Storage bucketsObject lifecycle management policiesPersonalized Service Health dashboardQuerying AlloyDBQuerying BigQueryQuerying BigtableQuerying Cloud SQLQuerying FirestoreQuerying SpannerQuery InsightsRemotely connecting to Compute Engine instancesReserving static external IP addressesReserving static internal IP addressesReviewing job status (Dataflow, BigQuery)Scheduling snapshotsSecuring objects in Cloud Storage bucketsUsing Database CenterVertical Pod autoscalingViewing and filtering logs in Cloud LoggingViewing GKE cluster inventoryViewing log message detailsViewing running Compute Engine instancesWorking with Cloud DNSWorking with Cloud NATWorking with GKE node poolsWorking with Kubernetes resourcesWorking with snapshots and images