Exposing services in private environments with Kubernetes and OpenELB
Challenge
CVTE, an electronics company that embraced cloud native tech early on, has been running services on Kubernetes for years. Today, they have a large number of microservices distributed across regions. For performance and security, most of their applications are running on bare metals and edge clusters in private environments. This increases the difficulty in accessing these applications from outside the Kubernetes clusters. As the number of applications grows, CVTE looked for a solution that could enable external access to LoadBalancer services in a graceful manner.
Solution
To find an enterprise-grade solution, the company evaluated several ways to expose Kubernetes pods or deployments, and decided to integrate OpenELB into their business systems. For native Kubernetes services such as NodePort and LoadBalancer, the use of NodePort is limited by the port range, while most LoadBalancer services are cloud-based, which is inapplicable to the company because most of their applications are running on bare metals. By using OpenELB, the company managed to expose LoadBalancer services on bare metals by using the Layer 2 mode.
Impact
“OpenELB enables external access to our Kubernetes clusters over TCP, working together with Ingress. Thanks to OpenELB, we can manage most of our web applications at ease,” says Zhenfei Pei, O&M Engineer at CVTE. OpenELB provides EIPs for external access and simplifies the management of IP address pools in private environments.
Projects used
By the numbers
Reduced complexity
In infrastructure and O&M
self-healing
New capabilities
Enhanced
Monitoring capabilities
Established in December 2005, Guangzhou Shiyuan Electronic Technology Company Limited (CVTE) is mainly engaged in the design, development, and sales of display control products such as LCD main control boards and intelligent interactive tablets. Since its establishment, the company has gained a leading position in the segmented market, and its products have been widely used in home appliances, education, and enterprise services.
“OpenELB is an ideal solution to LoadBalancer services in bare metals. We’ve also tried other LoadBalancer tools, but these tools are less stable during testing, and I often need to restart the controller to validate the configuration.”
Zhenfei Pei, O&M Engineer, CVTE
Before OpenELB came into being, the company used two native services NodePort and LoadBalancer to enable external access. However, the use of NodePort is limited by the port range. By default, the range of NodePorts is from 30000 to 32768. This range contains 2768 ports, which indicates that you can create a maximum of 2768 services with NodePorts. This is far from enough as the number of web applications of CVTE continues to grow. The LoadBalancer service is subject to cloud vendors, which is inapplicable to CVTE, where workloads are run on bare metal.
OpenELB, a sandbox CNCF project for LoadBalancer services, is dedicated to workloads in bare metals, edge clusters, and private environments. It serves as a LoadBalancer plug-in for Kubernetes, K3s, and KubeSphere to expose services to outside the cluster. OpenELB provides the following core features:
- Load balancing in BGP mode and Layer 2 mode
- ECMP-based load balancing
- IP address pool management
- BGP configurations using CRDs
OpenELB provides BGP and Layer 2 modes for networking. The BGP mode is applicable to high availability scenarios and requires support for BGP and ECMP routing. For systems that do not support BGP and ECMP routing, OpenELB provides the Layer 2 mode to achieve similar functionality. The Layer 2 mode requires the infrastructure to allow anonymous ARP/NDP packets.
“We use the Layer 2 mode more often because it’s simple and easy to use,” says Pei. To help developers focus on business logic, the company uses GitLab CI and ArgoCD to build automated pipelines and Nginx Ingress to deal with Layer 7 requests. In CVTE, when users visit a domain like argocd.cvte.com, the system forwards the request to the router that broadcasts an ARP request. Then, OpenELB responds to the request with a cluster NodeIP so that the router can know where to send data packets. If a node becomes unavailable, OpenELB can perform a failover to ensure business continuity.
“One of the most exciting things OpenELB brings to us is its friendliness to private environments. With it, we can enable external access to our LoadBalancer services in production environments, and its user experience is similar to that of any other cloud-based load balancers.”
Zhenfei Pei, O&M Engineer, CVTE