Want to skip straight to the commands? Click here.
After years of procrastination I finally set up a proper logging driver for the containers in my multi-arch swarm cluster. Gone are the days of command line cat and grep to debug issues, time to learn about Grafana, Loki, and logQL. All it took was a video from TechnoTim, and a very late night of tinkering.
The biggest roadblock by far was installing the plugin on the raspberry pi's in the cluster. When you follow the documentation on any arm device you will inevitably be met with this problem:
grafanaVery descriptive error, I know. What this boils down to is Grafana/Loki logging drivers not being compiled for arm devices as of the time of this post. So you'll need to build it yourself. For that you'll just need to have golang installed and working on your ARM machine and a little patience. You will need to make sure to build it on machine you're running it on.
Lets get building
If you dont already have golang installed on your machine you can download it from the official golang site or if you're looking for a convenient one liner for a linux arm64 build enter this into your terminal
wget -c https://dl.google.com/go/go1.17.3.linux-arm64.tar.gz -O - | sudo tar -xz -C /usr/local
Depending on your prompt you may find yourself starring at a stopped progress bar. Enter your sudo password and hit return, it should start the download.
Next you'll either need to add the /usr/local/go/bin to your path in bashrc/zshrc, or if you dont intend to use go in the future you can do it the quick and dirty way by entering
export PATH=$PATH:/usr/local/go/bin
You can verify that everything went according to plan with
go version
If you used the command above to install golang, you should get this:
go version go1.17.3 linux/arm64
Once you've got go configured it'll be time to start building. Series of commands goes like this:
git clone https://github.com/grafana/loki.git
cd loki
GOOS=linux GOARCH=arm GOARM=7 go build ./clients/cmd/docker-driver
And now comes the waiting. There is no progress bar, and its going to look like nothing is happening, especially on a lowered powered ARM device. Give it time and you should be left with a working binary! But you'll need to move that binary to the proper location in order to get it working.
Moving and installing your loki driver
If you already attempted to install the official Loki driver you're one step ahead, if not you'll need to attempt to install that driver knowing fully that it will fail. But what it'll do is create the necessary directories so you can easily copy your newly created driver
docker plugin install grafana/loki-docker-driver:latest --alias loki --grant-all-permissions
docker plugin enable loki
Next you'll need to find the right directory. It should look something like this: /var/lib/docker/plugins/e22f159c5f67916394f85a9e299954012118ac0490a082ba9f902f2ffffa351c/rootfs/bin
with that random directory being unique to each system you are doing this on. Once you find that path you can copy your driver and enable the plugin.
sudo mv docker-driver /var/lib/docker/plugins/e22f159c5f67916394f85a9e299954012118ac0490a082ba9f902f2ffffa351c/rootfs/bin/docker-driver
docker plugin enable loki
At that point you can add the logging configuration to your docker stack file:
services:
your-service:
logging:
driver: loki:latest
options:
loki-url: "http://<your-loki-server>:3100/loki/api/v1/push"
Or if you're using portainer you can configure it from the service configuration:
TLDR: Just the commands
Install GoLang
wget -c https://dl.google.com/go/go1.17.3.linux-arm64.tar.gz -O - | sudo tar -xz -C /usr/local
export PATH=$PATH:/usr/local/go/bin
Clone and build driver
git clone https://github.com/grafana/loki.git
cd loki
GOOS=linux GOARCH=arm GOARM=7 go build ./clients/cmd/docker-driver
Locate directory and move driver
sudo mv docker-driver /var/lib/docker/plugins/<random-alpha-directory>/rootfs/bin/docker-driver
Enable logging driver
docker plugin enable loki