How Nexxiot runs integrated IoT hardware, software, and analytics with Cilium
Challenge
Nexxiot is a TradeTech pioneer with a mission to enable easier, safer, and cleaner global transportation for all stakeholders. They deploy track and trace IoT devices which work more than six years without power supply and allow clients to monitor the location, status and conditions of their assets and cargo in real-time, anywhere in the world. As an ISO/IEC 9001 and ISO/IEC 27001 certified company, they needed to keep their clients’ private asset data secure while managing service for hundreds of thousands of devices in the field.
Solution
Nexxiot needed highly-reliable, secure, performant, and cost efficient Kubernetes clusters. Their operations team looked for projects with simple day two operations and built in automation. They turned to Cilium as the CNI to lock down their clusters and enable resilient networking with reliable day two operations.
Impact
Nexxiot manages hundreds of thousands of devices in the field with their SRE team. Cilium helps them keep their clusters and data secure and available to meet their stringent customer demands.
Projects used
By the numbers
0 network outages
200,000+ IoT devices around the world
One solar panel
Powers an IoT device 6+ years
Every 15 minutes
Asset location shared
In today’s snarled supply chains, Nexxiot runs integrated IoT hardware, software, and analytics to create transparency, improve efficiency, and preserve value across supply networks.
The technology mitigates risks to people, infrastructure and cargo and reduces emissions and waste. It has grown from a startup to a company with over 200,000 devices in the field, ISO/IEC 9001 and ISO/IEC 27001 certifications, and many enterprises relying upon them to track their assets while keeping their data secure.
The IoT devices work for more than six years with only a solar panel and without a power supply while providing an assets’ location every 15 minutes. They can survive rough environments, traveling around the world on freight containers and rail cars. Data is transmitted from the device to the cloud platform over mobile networks with encrypted UDP packets to save resources. Once in the cloud, the messages are stored then undergo stream or batch processing and made available to the customer through an API. Sophisticated Big Data analytics delivers business intelligence at scale to drive efficiency, process automation, and achieve sustainability targets
To deliver their clients’ experience, the team started off with running just EC2 instances, then migrated to running container workloads on AWS without Kubernetes, and are now migrating fully to Kubernetes. At every step of the way, one of the key requirements, from both customers and as an ISO/IEC 9001 and ISO/IEC 27001 certified company, was security compliance. Nexxiot needed to prove to their clients that their private data was secure. The second key requirement was operational simplicity. They needed technology that would work well on day two and every day after and provided a good migration experience.
They originally built their clusters in CNI Chaining mode with AWS VPC CNI and Cilium for network policies thinking AWS VPC CNI would be easy to operate and well supported because it is the standard for EKS and they would just use Cilium for security. They found that the operational complexity of running two CNIs didn’t provide any benefits. By simplifying their stack to only Cilium, they were able to focus on their client’s needs rather than operational overhead.
“We chose Cilium because we were looking for a simple solution, a resilient solution, and one which is ready for day two operations.”
Alex Berger, Chief Architect at Nexxiot
When originally evaluating CNIs, they looked for open source solutions because they liked being able to dive into the code to find and resolve problems rather than waiting for months on customer support. They found the Cilium community to be responsive and helpful and liked the eBPF based networking model.
Using Cilium as their CNI provides reliable low-latency networking with support for Kubernetes NetworkPolicies and a rich feature set. NetworkPolicies are an important piece in their Kubernetes security toolbox to achieve workload isolation.
In addition, Cilium provides operational simplicity. Alex Berger, Chief Architect at Nexxiot, explained: “By simple, I mean that we do not have 500 SREs who can fix systems every minute. We are very much focused on self healing things that will usually fix itself.
When it comes down to day two operations, which is, of course, related to the simplicity of the architecture, it’s also relevant to be able to deploy new versions without bringing down clusters or having a significant interruption of service. We promise high availability to our clients and need to deliver on it.
If you read many success stories, people often boast about how fast they implemented it. What really matters is that that system has been running for two years now and had many releases without a breakdown and that’s what we are focusing on. Having software that is not only easy and fast to deploy but also to upgrade and maintain over the years. That is exactly what we have found with Cilium. We started off with 1.9.X and now we are 1.11.X with many releases in between and it hasn’t broken our clusters. This is an important aspect that I think people often forget about.”
What is next?
“Nexxiot is always looking for ways to simplify their tech stack. Each component, service, and workload that we can remove from our technology stack is very important. Everything we add is a daunting task to maintain or a potential source of errors, failures, or outages.”
Alex Berger
They currently run Linkerd2 and use it for workload authentication with cryptographic identity and transparent encryption via mutual TLS. Nexxiot is also looking to remove kube-proxy to reduce costs, RAM, and CPU consumption but are waiting for Amazon Linux to upgrade their kernel version. They also want to try minimal secure Kubernetes Linux distributions like AWS Bottlerocket when it has better support from all of the tools they use.
Nexxiot has built fantastic infrastructure at very low cost without requiring massive resources and that’s only possible because “we fully automate everything from the very beginning and we will continue to build this way going forward.”
