Learn how we saved over $140k in Cloud Logging costs. By leveraging several practices, we were able to identify expensive logs.
TL;DR
- Cloud logging costs at scale can exceed $800 per day on GCP, driven primarily by high-volume workloads in Kubernetes clusters and global log ingestion from distributed components.
- BigQuery log sinks enable rich cost attribution by allowing teams to query log data by source, workload, and log payload structure, revealing which code components generate disproportionate log volume.
- GCP log exclusions are a high-impact optimization that stops noisy logs from being ingested entirely, while log sinks to Cloud Storage or BigQuery preserve audit trails at a fraction of the cost.
- Reducing log verbosity in the noisiest code paths, combined with exclusions and sinks, can cut cloud logging costs by 50% or more, translating to six-figure annual savings for high-scale environments.
In the public cloud world, nothing is free. Logging is sometimes viewed as an afterthought but an important piece to have the ability to audit and gain insights into a system. With the move to the public cloud, logs can get expensive if left unchecked; reasons could be that we are used to a certain level of log retention and verbosity on our own systems and expect the same in the public cloud.
At Harness, we use Google Cloud Platform (GCP) for hosting our production environment, and we are very proactive in optimizing our cloud usage and managing our Google cloud billing. As our services have been scaling, we noticed recently that our spend on logging has become a significant contributor to our total GCP bill. Before we took action, logging was costing us about $800 per day.
Understanding Our Logging Cost
With public cloud services, your final cost is usually aggregated across multiple dimensions of service such as storage and data transfer. With GCP, Cloud Logging prices are based on usage and it costs $0.50 per GiB of log ingested. We started looking into the major sources of the log volume to understand our Google cloud cost. The log ingestion dashboard pointed to the Google Kubernetes Engine (GKE) container and Global (our Delegate ingestion) as the top two sources of log volume. The third was Cloud HTTP Load Balancer.
This made sense since most of the Harness workloads are in GKE clusters which is the first category. The Global category is the logs we ingest from our delegates, aka Harness component which runs in customer environments.
To analyze further, we wanted to understand which workloads within the clusters were major contributors to this cost. For such kind of aggregation functionality, we configured Log Sink to BigQuery. Once we had a few days' log data in BigQuery, we were able to run rich queries and further breakdown the log data into its sources.
To further drill down the logs within these tables, we had to understand which components in our code are contributing/writing to the logs. We structured our log data into a JSON payload for each LogEntry. This allows us to have rich querying capabilities on the logs. In these payloads, we put information about source code classes, the source node, and customer account-specific information. Using this we were able to quickly attribute a cost to different areas of code which were most noisy.
With the cost data, we were able to start to be strategic around cloud cost optimization.
Optimizing Our Cloud Logging Cost
We took a few actions to optimize the Cloud Logging spend. Following are the most useful ones:
- Configured Log Exclusions. We did not know of this gem of a feature by Cloud Logging. We have stopped ingesting a major portion of noisy logs with this.
- Configured Log Sink to GCS/BigQuery for the logs which we need to keep for audit purposes, but this way we don’t have to pay for them in Cloud Logging.
- Reduce the logs in the particularly noisy components. Some of the source code was producing disproportionately high log volume. While it was not a problem with a lower scale, it's not useful, and in fact, diminishes our ability to effectively debug as service has grown. We followed up with individual developers to drop the unnecessary log lines.
The immediate results of this effort are evident. We have reduced the cloud spend on Logging to half since within a few days which amounts to annual savings of more than $140,000.
Hope our experience helps you in optimizing your own Cloud Logging spend. A similar approach can be taken with other public cloud vendors. Saving costs is a goal we have for our Cloud & AI Cost Management product and live those values every day at Harness. If you have not seen Cloud & AI Cost Management in action, feel free to request a demo today.
Frequently Asked Questions
How much does GCP Cloud Logging cost per GB?
GCP Cloud Logging costs $0.50 per GiB of logs ingested as of 2024. The first 50 GiB per project per month are free. Costs can add up quickly for high-traffic applications: a service ingesting 1,600 GiB per month would pay approximately $775 monthly after the free tier. [Source: Google Cloud, "Pricing | Google Cloud Observability", 2024, https://cloud.google.com/products/observability/pricing]
What tools can identify which workloads generate the most logs?
Configuring a Log Sink to BigQuery allows you to run SQL queries that aggregate log volume by source, namespace, pod name, or custom JSON payload fields. This reveals which services, workloads, or even specific code classes are producing the most logs, enabling targeted cost reduction.
What are log exclusions in GCP?
Log exclusions are filters that prevent specific logs from being ingested into Cloud Logging entirely. You define exclusion rules based on log content, severity, or source, and matching logs are dropped before ingestion, eliminating their cost while still allowing the application to generate them.
How much can log optimization typically save?
Organizations can typically reduce cloud logging costs by 30% to 60% through a combination of log exclusions, redirecting audit logs to cheaper storage (Cloud Storage or BigQuery), and reducing log verbosity in high-volume code paths. Harness achieved a 50% reduction, saving over $140,000 annually. [STAT NEEDS VERIFICATION: "30% to 60%" industry benchmark. Suggest sources: DORA State of DevOps Report, Gartner Cloud Cost Optimization studies, or cloud vendor case studies]