How Kubernetes and Linkerd became Lunar's multi-cloud communication backbone
Challenge
Lunar, a Scandinavian online bank that embraced cloud native tech early on, has been running Kubernetes since 2017. Today, they have over 250 microservices distributed across three clouds. They wanted to implement quarterly production cluster failover testing to prepare for disaster recovery. After losing logs in their dev tests, they realized they needed a better way to manage their platform services.
Solution
To secure their logs, Lunar decided to centralize its log management system. After realizing the benefits of doing so, they decided to centralize all platform services, including their observability stack, release management, developer tooling, and cluster-API (they are still working on the latter). To connect the clusters, they implemented Linkerd, which also helped with gRPC load balancing and mTLS for all services.
Impact
The gains were substantial and came on many levels. Lunar is now fully prepared for disaster recovery while remaining compliant with government regulations. They significantly increased developer productivity and decreased drift, audits are now run for all clouds and services across groups and environments, and they can now integrate with newly acquired startups, an essential business requirement.
Projects used
By the numbers
40
Smooth production releases per day
250
Microservices across 9 Kubernetes clusters on 3 clouds
1 hour
Linkerd multi-cluster up and running
Founded in 2015, Lunar challenged the banking status quo by reinventing how people interact with their finances. Lunar is for those who want everything money-related in one place — 100% digital, right in their hands.
That meant offering customers a smarter way to manage their money with more control, faster savings, easier investments, and no meaningless fees. That’s how Lunar envisions the future of banking.
Those are big plans, and, to achieve them, Lunar must be able to grow and scale rapidly. In 2021, they acquired Lendify, a Swedish lending company, and PayLike, a Danish fintech startup. With these acquisitions, Lunar gained various new business-critical capabilities, but it also meant the engineering team had to integrate all these systems, ensuring they work together smoothly — a non-trivial task.
Lunar’s commitment to cloud native principles
Lunar’s team of 150+ full-time engineers push about 40 releases to production every day. Twenty of those engineers work across four platform teams. The platform team operates nine Kubernetes clusters across three cloud providers (AWS, Microsoft Azure, and Google Cloud Platform) on multiple availability zones. They run 250+ microservices plus a range of platform services that are part of their self-service developer platform. Lunar wants their teams — or Squads as they call them — to be autonomous and self-driven. To support this “shift left” mindset, a group of platform Squads builds abstractions and tooling to ensure developers can move their features fast, securely, compliant, and efficiently.
There are multiple reasons why Lunar chose the cloud native path. First, they needed a platform that allowed their teams to manage their services and be fully autonomous. Secondly, as a fintech company pioneering cloud-based banking, they had to clearly document how they would avoid cloud vendor lock-in — a regulatory requirement by the Danish FSA. Functioning as an abstraction on top of a cloud provider, Kubernetes helped them achieve both goals and was thus the perfect fit. With most dependencies removed, this autonomy allowed Lunar to scale easily and be perfectly positioned for achieving its ambitious goals.
The platform team also provides Squads with a mix of open source tooling, including Backstage, Prometheus, and Jaeger, and also some custom-built solutions which they have open-sourced, such as shuttle and release-manager.
“This multi-cloud strategy and work style support the company’s goal of scaling in terms of the number of employees and mergers and acquisitions. It also allows us to stay technology agnostic and choose the technologies that best fit our needs.”
Kasper Nissen, Lead Platform Architect, Lunar
Oh no, where are our dev logs?
The first platform service Lunar centralized was Humio, their log management system. At the time, Lunar was developing failover processes for their production clusters. During development, they realized logs in their log management system went missing. To avoid losing potentially critical data once in production, they removed the log system from their production cluster and centralized it before implementing the failover capability.
From logs to centralizing all platform services
After successfully centralizing their log management system, they embarked on a platform services centralization journey. Lunar has multiple environments, and its platform services, including the observability stack, were replicated in each environment — all fairly complex services that consumed lots of resources. Minimizing the number of stateful services in these environments would clearly help. Additionally, running nine replicated setups simply didn’t scale.
Lunar also had multiple endpoints to access tools like Grafana, resulting in duplication of effort managing users and dashboards. This confused the development teams who had to implement changes in multiple places, leading to drift between environments, among other challenges. Managing users in one system is much more efficient than doing so in nine (or more).
All this factored into Lunar’s decision to create a centralized cluster owned by the platform team that would eventually run the entire observability stack, release management, developer tooling, and cluster-API — an ongoing effort.
Today, Lunar’s log and release management run as centralized services along with Backstage and a handful of other tools. Next is their monitoring setup, a mix of Buoyant Cloud and Prometheus/Grafana.
The quest to connect their clusters
Once Lunar started centralizing platform services, they had to connect their clusters. At the time, they were only running clusters in AWS and considered VPC peering across accounts. Doing that was somewhat painful due to clashing CIDR ranges. They also evaluated VPNs but aren’t big fans of using technologies with two static boxes on each end. Besides, they wanted to move towards zero trust networking, following the principles of BeyondProd by Google.
Service meshes finally caught up with their needs!
During the 5+ years of running Kubernetes, Lunar continuously evaluated service meshes. In 2017, they even had Linkerd running as a PoC but decided against it. It was still the JVM-based Linkerd 1 and quite complex. They kept following the development and evolution of service meshes. When they heard the Linkerd 2.8 release included multi-cluster capabilities, they realized it was time to give service meshes another shot.
This decision was further reinforced by problems they were experiencing with gRPC load balancing (not natively supported by Kubernetes) and the need to switch to mTLS for all internal communication. A service mesh made a lot more sense now.
While they have always been big fans of Linkerd’s approach, starting with the basics and making that work well, they evaluated both Linkerd and Istio. They committed a week of time with two engineers, one playing with Istio and the other one with Linkerd.
“We had the Linkerd multi-cluster up and running within an hour! After a few days of struggling with Istio, we gave up on it. Linkerd did the job fast and easily — the perfect mesh for us. It had all the features we needed at the time, was easy to operate, had a great community, and solid documentation,” stated Nissen.
Since going live, Lunar also started using Buoyant Cloud for better visibility across all their environments.
Committed to the CNCF stack
A CNCF End User Member, the Lunar engineering team, trusts and uses many CNCF projects. Their stack includes Kubernetes, Prometheus, cert-manager, Jaeger, Core DNS, Fluent-bit, Flux, Open Policy Agent, Backstage, gRPC, and Envoy, among others. They’ve built an Envoy-based ingress/egress gateway in all clusters to provide a nice abstraction for developers to expose services in different clouds.
Prepared to scale and shake up the European banking market
From a technology perspective, Lunar has now achieved a fairly simple way to provide and connect clusters across clouds.
“Kubernetes allows us to run anywhere, Linkerd enables us to seamlessly connect our clusters, and GitOps provides an audited way to manage our environments across multiple clouds with the same tooling and process. From a developer’s perspective, the process is identical whether you deploy on GCP or AWS.”
Kasper Nissen, Lead Platform Architect, Lunar
Seamless integration with newly acquired startups
The business impact has been substantial. With Lunar’s new multi-cloud communication backbone, they are better positioned to support upcoming mergers and acquisitions — a key part of their business strategy. They can extend the Lunar platform in a cloud-agnostic way while selecting the provider that best fits their needs for each use case.
Fully prepared for DR while compliant with government regulations
With logs now centralized and no risk of losing logs during failover, they will soon implement quarterly production cluster failovers. That will allow them to know exactly how their system behaves in case of a failure and how to bring it back up. It’s important both from a regulatory perspective and a business perspective. If their customers lost access to their account information, it would have disastrous consequences for their business. That’s why they proactively train for the worst-case scenario. If something were to happen, they would know exactly what to do and how to avert any issues.
Centralized services and streamlined processes increased developer productivity
Centralizing their platform services has already streamlined many processes and improved developer productivity. All releases, metrics, logs, traces, etc., are properly tagged with fields such as Squad names, environments, and so on, making it easy for developers to find what they are looking for. It also ensures clear ownership of that particular piece.
Managing the team is also a lot simpler for Nissen. “I don’t have to set up dashboards, help search through logs, etc. — our Squads are truly independent. Because our platform is based on self-service, it is decoupled from the organization allowing our team to focus on implementing the next thing that will help our developers move faster, be more secure, or ensure better quality,” explains Nissen.
Easy audits and peace of mind for management
Audits have become a lot easier too. Since everything is centralized, they can run audit reports for all clouds and services across groups and environments. That is good for them and provides peace of mind in the highly-regulated financial services industry.
While they aren’t there yet, they expect to save significant time in engineering resources by not having to operate and maintain nine versions of the soon-to-be fully centralized stack.
Well-positioned to scale fast and smoothly
Overall, Lunar feels well-positioned for upcoming acquisitions and organic growth. With a platform able to extend anywhere, they’ve become a truly elastic organization. If your business expects to scale fast, Nissen recommends centralizing your platform services.
“It’s not painless, but once it’s all set up, growing your business becomes increasingly easy.”
Kasper Nissen, Lead Platform Architect, Lunar