Kubernetes on Raspberry Pi: Challenges & Advantages
Lucas Käldström is a upper secondary school student from Finland. He has recently joined Weaveworks as a contractor. By tinkering with Raspberry Pis, he discovered Kubernetes, started kubernetes-on-arm, and then merged ARM support into...
Lucas Käldström is a upper secondary school student from Finland. He has recently joined Weaveworks as a contractor. By tinkering with Raspberry Pis, he discovered Kubernetes, started kubernetes-on-arm, and then merged ARM support into core Kubernetes. Since then, he has worked in sig-cluster-lifecycle with kubeadm, and also deployed Weave Net to ARM kubeadm clusters.
Learn more about his process: things that worked, things that didn’t, and why.
Q: What was your first project on a Raspberry Pi device?
A: It was a file server and an OpenVPN server. Combined with those I could access my files and my remote desktop from my phone, and it was pretty neat. I could, for example, control my desktop or family laptop from my phone and play Spotify. I guess that was the first project, and I did it from a tutorial. Was a little bit scared of piping anything else into the command line. I didn’t want to break something. After some months, that fear had gone away and I started experimenting with other things.
One thing that annoyed me was once I set it up I didn’t quite remember how I did it. If I wanted to start from scratch, I had to reread all those tutorials and guides and try to remember. Also, I thought it would be much better if I could isolate the OpenVPN server from the Samba server.
At that point in May 2015, I started using Docker and Docker containers, and that was what I’d been looking for. In that process I also found Kubernetes and got excited about it and wanted to try it, but etcd or Kubernetes weren’t available for Raspberry Pi. From there, I started converting everything and hacking my way around and then I suddenly had made myself a Kubernetes cluster!
Q: You started documenting everything that you did after the first time?
A: Yes, exactly. I made myself a small Git repo locally, which later evolved into the kubernetes-on-arm project. I wrote all the commands I typed into the command line down at the same time so I wouldn’t lose them.
Q: What was it that excited you about Kubernetes over other similar platforms?
A: One of the main things was it was created by Googlers. Of course they are kind of front figures. As a student interested in programming, you have a lot of respect for them. When you see good concepts from things Google does for real with Borg, I thought, “Well, now when this is open source, I can actually use it and try it out and see how they think. How do they reason about design decisions?” That was really exciting.
I thought it was fascinating that I could take part in something whose concept has been developed for many many years. That I actually had access to it. Of course, the Kubernetes platform itself was nice and exciting as well. So far I had been running a given program on only one Pi, but now I could make more advanced configurations with multiple nodes, so to combine all my small Pis to something stronger was very interesting.
Q: Why did you use Weave Net on a Raspberry Pi device over other container networking solutions?
A: In the beginning I used Flannel for the network layer, since I followed Brendan Burns’ Docker multinode guide, because Kubernetes wasn’t easy to set up. In August 2015 I had everything kind of working, but then I had to somehow choose how I used the binaries I had compiled. I was fascinated by the approach of running all Kubernetes components in Docker containers. Even the kubelet. I thought that was good because you don’t have to install anything on the host besides Docker itself obviously. That was the main reasons I chose to convert that deployment strategy to ARM, and Flannel was included in that package.
At the time I hadn’t heard of Weave Net. But the sig-cluster-lifecycle group was formed in July 2016, I heard of it from Luke and Ilya, and noticed the other networking solutions as well. I think the URL for the weave-kube DaemonSet is really easy to type and remember, compared to the other solutions which have very long kubectl apply -f URLs. Weave is the only one I remember by heart. I started reading about Weave and also thought Weave Mesh was an interesting solution for distributed networking.
Q: It was through kubeadm that you decided to try Weave Net on Raspberry Pi?
A: Yes, exactly.
Q: What are some of the challenges of running a Kubernetes cluster on a Raspberry Pi device?
A: Of course it [Raspberry Pi] doesn’t have too much power. It has sufficient power to run a full Kubernetes master plus maybe ten plain web servers. That works fine. The main limitation is the RAM, I would say. Because Raspberry Pi uses the VideoCore 4 graphics card it’s limited to one gigabyte of RAM, I’ve heard. The Raspberry Pi Foundation doesn’t want to break the backwards-compability towards the current versions, and bumping to 2GBs of RAM would break it. So right now, you have to use other credit card-sized computers if you want more than one gigabyte of RAM. It would be nice if the CPU power was a little bit higher, but it works absolutely fine when you’re running Kubernetes, so I don’t complain. But as you increase the number of nodes you have, the more power you will get obviously, and still within a reasonable price tag.
Q: There’s no credit-card-sized ARM board that has more than one gigabyte of RAM on the market now?
A: There is. For example, the PINE 64 costs $29 USD. It has got an 64-bit processor of type Cortex-A53 if I remember correctly. It has two gigabytes of RAM. That’s great, and the Odroid C2 also has two gigabytes of RAM and an even faster processor than the PINE 64.
When talking about speed and credit card-sized computers, it’s worth noting that the choice of disk (if you have a choice) is crucial. When you compare the normal hard drives to SSDs, you see a huge difference in speed and access times, and so you do when comparing SD Card and eMMC storage devices as well. I’d say the best thing with the Odroid C2 is that it has got an eMMC; it’s blazing fast in comparison!
Q: Do you have the same full functionality of a Kubernetes cluster with Weave Net on a Raspberry Pi?
A: Everything’s the same as any other amd64 bare metal cluster I’d say. I mean, there aren’t cloud provider integrations because we’re not running on a cloud provider as you might do with an amd64 cluster, but it works great anyway, and the cluster is super-easy to set up with kubeadm.
The best thing with my cluster is that it’s small and doesn’t draw much power. A Raspberry Pi’s mean power draw is around three, four, five watts or something. That’s not much. Those are the advantages, and I think the best thing with a credit-card-sized Kubernetes cluster is that it’s good for education, and since I went the hard way of setting up Kubernetes, I learned how Kubernetes actually works. You don’t do that with a “click to deploy” solution. I failed so many times I can’t count them all. And I think that’s why I’m so experienced with the project now.
Q: So how was it to try to set up Kubernetes the first time?
A:It was kind of crazy, because when I started in May 2015 to convert Kubernetes to Raspberry Pi, I didn’t even know how it would behave when it was done. I had a hard time initially because I wasn’t acquainted with the command line really, I just had copy/pasted some tutorials on it before. Also, Kubernetes changed very quickly since it was in the 0.x releases – 0.19 or something. Every time I found a guide for how to set up Kubernetes normally on amd64, it was for an older version which was incompatible with the version I tinkered with at the moment. That was kind of challenging, but with the 1.0 release I got it working. That was a lot of fun.
Q: What are some advantages and some use cases for running Weave Net on a Raspberry Pi device?
A: There could be many advantages. In industrial environments, running small devices that draw very small amounts of power could be used for automation of different kinds for example. If we consider the scenario that some company would deploy a lot of Raspberry Pi’s in our production process for automation, we have a few challenges to solve. We want to be able to update the software on the Raspberry Pi’s in an automated manner. We could do that by pushing the software to a central Docker image registry initially. Then we might want to update the application in a rolling-updated manner across the different nodes, and we could use the built-in features of Kubernetes for this.
Considering we want our nodes to be able to talk securely to each other as well, we can use the “crypto” feature in Weave Net. We could then run over semi-trusted WiFi networks for instance. That’s one possible use case. We want these small devices that are relatively low-cost and don’t draw a lot of power.
But in my opinion, the main and really big use-case for Kubernetes and Weave on ARM or credit-card-sized computers in general is education. Educating people about how Cloud Native distributed systems work in general, how the networking flows, how the cluster reacts on a node loss, how one can make the cluster secure etc. will be a very big topic in the future. In fact it already is.
For universities for example, having something concrete to show the students when talking about something abstract like the controller loops in Kubernetes can be a game-changer. Also, the lecturers at a given University may or may not have experience with Kubernetes, so that’s important to take into account as well. No one can say “I’ve got ten years of experience with Kubernetes.”, because it has only been around for about two and a half years.
Using the Raspberry Pi cluster is a good way of educating Kubernetes’ concepts, because it gets very physical and practical. You can actually see the cluster. You can touch it. You can draw the power cable and see what happens and so on.
There is a ~160 page master thesis written on this subject at Aarhus University in Denmark. Kasper Nissen and Martin Jensen tried educating Kubernetes, Docker, and concepts with stacks of Raspberry Pis by using my (now deprecated in favor for kubeadm) project kubernetes-on-arm. They had great success, please read the thesis, that’s a wonderful story as well.
Q: Have you spun up Kubernetes or Weave Net on ARM devices other than Raspberry Pi?
A: Yeah. I’ve used Kubernetes and Weave on PINE64s, ODROIDs, Banana Pros, and if we include cloud servers, I’ve used (and am using) Scaleway’s and Packet.net’s ARM bare metal cloud servers as well.
Q: What makes you prefer Raspberry Pi devices over other ARM devices, and if I’m wrong in assuming that, let me know.
A: Actually my favourite right now is the Odroid C2, it’s the best device I’ve tested, and is only about 10 euros more expensive. Raspberry Pi is the easiest alternative and the “de facto” way of running Kubernetes on ARM, because of the huge support for Raspberry Pi with regards to operating systems, the kernel, etc.
Q: What are some future Raspberry Pi or ARM projects that you have in store for yourself?
A: I want to try out Packet.Net and their ARM64 server offerings for real. They have killer machine called the 2A that has got 96 cores, and 128 gigabytes of RAM and more. That’s one of the things I want to play with and run Kubernetes on with many nodes. Tobias Sachs has tried it out and have run 90 Minecraft servers on the same Packet 2A machine, that’s really interesting in my opinion.
Q: Do you have any plans for running Weave Cloud or Weave Scope on a Raspberry Pi device?
A: Yes, I have. Scope’s visualization is interesting, and that’s something that might be worth trying to deploy on the Pi as well. The Kubernetes dashboard has been ARM-compatible from day one. And if I’m successful in bringing Scope to ARM, I might continue with other products in the Weave Cloud suite as well, who knows?
Q: Is there anything else that you want to say about Raspberry Pi devices or ARM devices and any challenges?
A: I hope the Raspberry Pi Foundation mentality and philosophy spreads to more parts of the world. I’ve seen they are teaching school kids how to code successfully in Great Britain and in some other countries as well, but there’s no such thing (yet!) here in Finland. Of course there are individuals with Raspberry Pis here, but not in the same great sense like in the UK. We could teach Kubernetes and Cloud Native technologies by using small clusters and make that way of demoing Kubernetes a success together as well. That’s also one of the things I’m hoping to look into in the future.
The recent Weave Net 1.9 release features faster, more secure networking and support for ARM builds. Read more.
