Hallmark
Thirty-plus Go microservices across commerce, accounts, gifting, and streaming — and the patterns that kept them coherent.
Stack
- Go
- Elixir
- Apache Kafka
- Elasticsearch
- Saga pattern
- Resiliency patterns
Problem
Four years of growth had produced an ecosystem spanning accounts, commerce, gifting, postage, and the Hallmark Movies Now SVOD platform, built by multiple engineering teams working in parallel. The individual services were sound. What was missing was the connective tissue: how a service finds another, what happens when a dependency is slow rather than down, and where the cache lives. Without shared answers, thirty services become thirty conventions. The earlier phase had a different shape of problem — the core commerce platform ran on Erlang and Elixir, and a purchase was not one transaction but a long-running sequence of them, spread across services that could each fail independently.
Approach
I architected the ecosystem around a small set of patterns every team could rely on: service discovery, centralized caching, and facade services fronting the SVOD platform. Resiliency was built into the facades — circuit breaking to fail fast when a dependency degrades, automated retries where they were safe, intelligent caching and graceful degradation where they were not. In the earlier core modernization, purchase and order fulfillment were orchestrated with the Saga pattern, which is what let a failed step compensate rather than leave an order half-committed across four services. That work also moved high-traffic services from Erlang/Elixir to Go, and tuned Kafka and Elasticsearch for the ingestion load.
The tradeoff
Circuit breakers trade correctness for availability. When a dependency degrades, the facade returns a cached or degraded response rather than the true one. For a streaming catalog that is the right call — a slightly stale list beats an error page. For a payment path it would have been indefensible, so those services never got the same treatment; they got sagas instead, which trade latency for the guarantee that money and orders end up agreeing with each other. The pattern is never the point. Where you refuse to apply it is.
Impact
30+ Go microservices · circuit breaking established across teams
- 30+ Go microservices architected into a coherent ecosystem with shared discovery and centralized caching
- Circuit breaking, automated retries, intelligent caching, and graceful degradation established across teams
- Purchase and order fulfillment orchestrated with the Saga pattern across independently failing services
- Postage Service and Gifting Service taken from prototype to production
- High-availability facade services delivered for the Movies Now SVOD platform
- Core services migrated from Erlang/Elixir to Go; Kafka and Elasticsearch tuned for high-load ingestion
What I learned
Patterns spread through review, not through documents. The circuit breaker did not become standard because I wrote it down; it became standard once it was in the facade every team already depended on, and the design review started asking why a new service was not using it. Architecture that lives only in a wiki has not been adopted — it has been filed.