Challenge
A mobile-core platform handling session management and real-time charging had to sustain millions of concurrent connectivity sessions. Under high-load testing, the charging path became a throughput ceiling and nodes hit memory saturation — made worse by uneven pod scheduling across the cluster.
In real-time charging, every session event has to be metered correctly and cheaply. The metering layer cannot be the bottleneck.
Approach
- Measure before tuning. Flame-graph profiling of the charging path showed it was I/O-bound, not CPU-bound — which pointed the work at connections and batching rather than code-level optimisation.
- Remove the I/O ceiling. Widened HTTP connection pooling, tuned the AWS SDK retry and token-bucket behaviour, right-sized Kafka batch sizes, and redesigned the concurrency model of the NATS event bus.
- Cache with care. Added a read-through cache for slow-changing reference data — tariffs, routing, quota regions — with event-driven invalidation over NATS. Security-sensitive entities were deliberately excluded from caching to avoid correctness and race-condition risks.
- Fix scaling at the root. Diagnosed out-of-memory kills caused by eBPF map pre-allocation under burstable QoS, memory saturation on the in-memory store, and skewed pod distribution — then delivered a prioritised scaling strategy.
Outcome
The charging path was re-tuned around its real bottleneck, the reference-data cache took repeated lookups off the hot path without caching anything security-sensitive, and the cluster’s memory and scheduling problems were each traced to a specific root cause with a prioritised plan to address them.
