This post is no longer valid as the cluster runs Kubernetes. The build process is the same
Intro#
The Raspberry Pi from it’s first iteration has been a great hit. Adopted by most as the computer of choice for projects. The Pi is a fully fledged computer that fits in your pocket and the most recent edition boasts an ARM64 CPU and 8GB of RAM.
Now one Raspberry Pi 4 8GB is more than powerful enough for most workloads, so you may even get away with a single 4GB model. Taking into consideration their low price and power consumption why not go all out and build a cluster.
My excuse to build a Pi cluster: I would like to migrate the services running on a bare metal server that sucks a lot of power to a low power solution.
My Setup#
- 4 x Raspberry Pi 4 8GB
- 4 x 32GB SD cards
- 4 x PoE hats
- 1 x 5 port PoE Switch
Point of this post#
After following this post you will have setup Docker swarm on a cluster of Pis - stay even if you have one - and understand how to bend it to your will 🦾.
Why Docker Swarm 🧐#
While we can manage each docker node - Raspberry Pi - individually , Docker swarm enables us to manage all the devices from a single main node making managing the cluster a lot easier.
Think of Docker swarm as a set of 🤖 hands which will run tasks on each individual node for us. Some of these tasks are:
- Cluster Management
- Deployment and Scaling across the cluster
- Multi-host networking
- load balancing
Well why not Kubernetes#
Although Kubernetes is awesome, it adds a bunch of complexity I’m not ready for. As well as that, from my experience creating docker-compose files is a lot easier than writing manifests. In the future I will try Kubernetes
Required Stuff#
The links below are reccomendations/stuff I used
- Raspberry Pi 4B (Reccomend 4-8 @ 8GB) (Or newer)
- Raspberry Pi PoE Hats (PoE+ ) (Check compatibility with Pi Version)
- Cluster Case (Get any case you want)
- PoE switch (get one with enough 1Gb ports for the number of pis you want)
- RJ45 Ethernet Cable (Recommend short cables as it will look neater)
- Micro SD cards (Get enough cards for the number of Pis, Get the biggest cards you can afford)
This is an expensive project, but the best part is you can add to the cluster over time spreading the cost, which is how I managed to buy all the hardware. If I could start again though I would buy equipment that allows room to grow so an 8 port PoE switch instead of a 4 port if you know you will scale to 8 nodes
This cluster should be capable of serving most of your needs even with 2 nodes. Put it this way, if you have 4 x 8GB nodes then you have a combined total of:
- 32GB of RAM
- 16 physical CPU cores @ 1.5GHz (You can OverClock !!)
Money Saving Tips#
- 1-2 nodes to start
- Drop PoE and use normal power cables
- Raspberry Pis with less memory i.e 4GB
- Cheaper case or make one using these
- Smaller SD Cards try not to go below 32GB (Recommended 64GB If money saved elsewhere)
BUILD TIME !!#
Image the SD Cards#
The CPU in a RaspberryPi 4 is the ARM64 architecture. When selecting Raspbian pick the 64-BIT option. 32bit variants will work but not make use of the hardware capabilities.RaspberryPi Imager
when prompted with “Use OS Customisation” go with “Edit Settings”. General:
- Set the hostname
- Set a username and password
- Set Locale settings Services:
- Enable SSH
- Allow public-key authentication only
- Set your SSH Key
If you dont have an SSH key, read my post on SSH Securing SSH
Assemble the cluster#
Assembly depends on the case you bought, instructions should be included. Go ahead and piece together the cluster 🛠
There is an order of operations that I recommend:
- Install PoE hats onto the Pis
- Install the SD Cards into the Pis
- Assemble the case and install Pis
- plug ethernet cables in
Now allow the Pis to boot for a few minutes
Sort out the Network bits#
There are two methods you can use to set a static IP address, one is on your DHCP server and the other is on the device itself. Personally I prefer to set static IPs on the device itself so I will cover that here.
To find the current ip of your Pis you can go to your DHCP server and look for the DHCP leases for the Pis.
there are naming conventions for naming primary and secondary things but this is your cluster so be creative with it, I have my primary node named
Goob
and the secondary nodes namedDoris1
>Doris2
andDoris3
> reference
SSH to the Pi then open and edit the following files
- /etc/dhcpcd.conf _ knowing we will be using only static IPs here I delete everything in the file add the following
interface eth0
static ip_address=<your-Pi-ip>/24
static routers=<your-router-ip>
static domain_name_servers=<your-dns-ip-usually-same-as-router>
Now reboot the pi
sudo shutdown -r now
Provision with Ansible#
Ansible will allow us to setup our RaspberryPi Cluster from a remote machine where we only administer the commands once and Ansible will replicate across the cluster.
Make sure pipx is installed. Install Instructions
Install Ansible Install Docs
pipx install --include-deps ansible
A playbook defines steps Ansible needs to take on the remote machine to provision it. Playbooks are written in yaml so get ready to make sure your indents are correct 😬
Creating the playbook Copy what I have and it should work – after changing a thing or two – I encourage you to read the Ansible Docs and make any changes you feel are necessary.
Check this Gitlab Repo for the most up to date playbook.
This script assumes you are using SSH keys, which you should but if not you can skip the whole “Configure SSH” block, I RECCOMEND YOU USE SSH KEYS - read this
Create the deploy.yml
# Open the yaml file in a text editor
vim deploy.yml
---
- hosts: piCluster
become: yes
tasks:
- name: Configure SSH
block:
- name: Auth local SSH Keys
ansible.posix.authorized_key:
user: <pi_user>
state: present
key: "{{ item }}"
with_file:
- /home/<local_user>/.ssh/id_rsa.pub
- ./pi_rsa.pub
- name: Copy SSH Pi Intercoms Private Key
copy:
src: ./pi_rsa
dest: /home/<pi_user>/.ssh/id_rsa
mode: "0600"
owner: <pi_user>
group: <pi_user>
- name: Copy SSH Pi Intercoms Public Key
copy:
src: ./pi_rsa.pub
dest: /home/<pi_user>/.ssh/id_rsa.pub
mode: "0644"
owner: <pi_user>
group: <pi_user>
- name: Check for any updates and install them
apt:
update_cache: true
upgrade: dist
- name: Install Required build dependencies
apt:
update_cache: true
pkg:
- python3
- python3-pip
- vim
- curl
- name: Check Docker is installed
shell: "docker --version"
register: docker_version
- name: Install Docker
block:
- name: Download Docker script
command: curl -fsSL https://get.docker.com -o get-docker.sh
- name: Run Docker install script
command: sudo sh get-docker.sh
- name: Enable docker service
service:
name: "{{ item }}"
enabled: true
state: started
loop:
- docker
- containerd
- name: Add user to docker group
user:
name: <pi_user>
groups: docker
append: yes
when: docker_version.stdout.find('Docker version')
- name: check Docker compose is installed
shell: "docker-compose --version"
register: dc_version
- name: Install docker compose
command: sudo pip3 -v install docker-compose
when: dc_version.stdout.find('docker-compose version')
- name: wait for network on boot
command: raspi-config nonint do_boot_wait 0
- name: Reboot
reboot:
The steps this playbook takes are as follows:
- Copy your SSH keys to the pi authorized_keys
- Copy same Pi ssh key between the Pis for intercommunication
- Check for and Install updates
- Install any build dependencies
- Check if Docker is Installed
- Download the Docker Script
- Install Docker from script
- Enable the Docker service
- Add user to Docker group
- Check if docker-compose is installed
- Install docker-compose (This will take a while on each pi so be patient, grab a coffee)
- Change raspi config to wait for network before boot
- Reboot
take some time to read through the playbook, read the docs and try to understand whats going on. Change the usernames to your username.
Create an inventory hosts file in the same directory as the deploy.yml
# Open file in a text editor
vim hosts
[piCluster]
<ip address 1>
<ip address 2>
<ip address 3>
- [piCluster] - specifies a group of hosts
- change these to the Pis IP addresses
create the Pi Intercommunication keys
ssh-keygen
# Please note the ".pi_rsa" here
Enter file in which to save the key (/home/ahmza/.ssh/id_rsa): .pi_rsa
heres the command to run the playbook.
ansible-playbook -i hosts -kK deploy.yml
- ansible-playbook - the command to run the playbook
- -i hosts - specifies the host/inventory file
- -kK - asks you for password for the SSH connection(-k) and the sudo password(-K)
- deploy.yml - the Ansible playbook
You can do so much with Ansible, for example mount network shares etc, if you can do it in the cli you can do it in Ansible !!
Now that the Ansible script is done we can go ahead and actually configure all the Pis into the cluster.
Creating/Joining the swarm#
Creating and joining the cluster is easy - It is manual and tedious so this is something you could implement into the Ansible playbook using host groups.
Follow these steps to create the swarm:
- ssh to the main Pi, this will be the manager node.
- run
docker swarm init
- This will return a command and token. Make sure to copy them
- exit the main Pi
- ssh to one of the worker nodes
- paste and run the copied command
- do the same on the remaining worker nodes
You should now have a configured cluster !!
Optional Step - Portainer#
Portainer is a webgui that will allow us to manage the cluster, it will make deploying and destroying containers across the cluster a breeze.
Install Portainer
Firstly we want to download the yaml file that will be used to deploy the Portainer container on our cluster.
curl -L https://downloads.portainer.io/portainer-agent-stack.yml -o portainer-agent-stack.yml
Now we can actually deploy Portainer to the cluster using the following command:
docker stack deploy -c portainer-agent-stack.yml portainer
Once the stack is up and running navigate to:
https://raspberry-ip:9443
where raspberry-ip is the IP address of the manager node in the swarm.
Optional Step - Private Registry#
If you intend to build any of your own docker images then it may be useful to create a private repository to host any images that you need to run across the cluster.
to create this private repository create a file called docker-compose.yml in a folder new folder called repository and enter the following.
version: "3.9"
services:
registry:
image: registry:2
ports:
- 5000:5000
volumes:
- /home/<pi-user>/docker/registry/data:/var/lib/registry
Be sure to change pi-user to your pi user
create a file /etc/docker/daemon.json and eter the following
vim /etc/docker/daemon.json
{
"insecure-registries": ["localhost:5000", "<pi-address>:5000"]
}
where pi-address is the ip address of the main swarm node.
**restart the docker service
sudo systemctl restart docker
now we need to deploy the registry to the stack like we did with portainer
docker stack deploy -c docker-compose.yml registry
and we should have a private registry setup that we can use to push our custom docker images to.
Conclusion#
Well done, you have successfully setup a docker swarm cluster on a bunch of Pis and along the way you managed to learn a bit about SSH, Ansible, docker-compose, yaml and networking on Pis. The next step would be to setup a few containers but I’ll leave that to you, dont be affraid to play around and even ruin the cluster, you can always rebuild it have fun 😊