detfalskested

Elasticsearch: reconcile-desired-balance

I've been struggling all day to get my GitLab pipelines running properly again.

It turns out the combination of Docker happily eating all your disk space and Elasticsearch being very cautious to not start if there is not plenty of available disk made things break down.

My setup is GitLab's runnners running inside Docker, which is testing a Django app with Elasticsearch (ES) attached as a service. This has been working flawlessly forever. But at some point recently, I started getting errors for my tests that depend on ES. And the weird thing is that it started during a quiet period where I did not touch the setup or the code.

From Python, I was getting the error message:

elastic_transport.ConnectionTimeout: Connection timed out

This seemed weird as I was perfectly able to connect to the ES container from the app container: I do a check at the beginning of my test script, by simply curling the ES host. And even the line of code right before the one that got a timeout was successfully connecting to ES. What those 2 lines do is:

  1. Remove the search index.
  2. Create the search index again.

With all the things running inside GitLab runners running inside Docker, it seemed a bit like a black box. But I figured out a way to get the logs from the ES container:

Looking at the log I found the very last entry to contain a clue:

{"@timestamp":"2025-06-06T11:47:27.110Z", "log.level": "INFO",  "current.health":"RED","message":"Cluster health status changed from [YELLOW] to [RED] (reason: [reconcile-desired-balance]).","previous.health":"YELLOW","reason":"reconcile-desired-balance" , "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[5a28270d5238][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.routing.allocation.AllocationService","elasticsearch.cluster.uuid":"dX-ZZeLEQ-OMCDKm2PjoCQ","elasticsearch.node.id":"r0CL-D0LT5u-tpTeHujQwQ","elasticsearch.node.name":"5a28270d5238","elasticsearch.cluster.name":"docker-cluster"}

It wasn't super clear what "reconcile-desired-balance" meant, but I fortunately found a forum post from someone having the same problem, suggesting it's because of lack of disk space.

Checking the disk (df -h), I had more than 30 GB free, but the usage percentage had crawled past 90%, which I assume could be a red flag for ES.

I do know that Docker will happily eat all your disk space over time. That has caused me problems before. And yes, it had also had a feast this time. Running docker system prune -a reclaimed 265 GB of disk.

After this, tadaaa! Elasticsearch no longer turns to a RED health status and thus does not time out: My tests are passing again. Oh, the joys of modern development.

Talk nicely

Amanda Bachman pretends to be the AI friendly tech CEO we all know too well, in A Company Reminder for Everyone to Talk Nicely About the Giant Plagiarism Machine:

I guess I understand. I, too, was once a little skeptical of the Giant Plagiarism Machine™. But that was before I attended The Conference for Big Boy Business Owners™. Here, I learned that my fellow titans of industry have been re-orging to “leverage plagiarism” and “minimize thought-waste.”

(...)

The way I see it, we’re family. It really does disappoint me that so many brilliant colleagues—whose genuine breakthroughs I’ve profited from for years—would be so quick to condemn this newer, stupider way that I and others like me can make money off your life’s work, through stealing.

Iris Meredith on LLMs

Iris Meredith, in Keeping up appearances:

LLMs provide a great deal of opportunities to increase your status and decrease other peoples': a core status marker in the corporate world these days is being "innovative" or "tech forward", which means that loudly and ostentatiously using LLM technology can be an excellent way of building your status in corporate spaces. The converse of this is that you can label LLM sceptics as being "irrational" or "frothing" and tell them that "if they don't use the technology they'll be left behind", which is an excellent way of lowering their status in the eyes of the corporate world that they function in. We've seen exactly the same pattern play out with blockchain and half a hundred other overhyped technologies.

Max Cooper

Last month, on a Sunday afternoon, on the train, on my way back home from the other end of the country, I accidentally discovered that Max Cooper would play a concert the same night in my hometown.

I almost didn't go. But, fortunately, I remembered the promise to myself to go to as many concerts as possible. So I bought a ticket and walked directly from Copenhagen Central Station to the venue. And then I was in for a treat I did not expect: A beautiful, 3 hour marathon of a show.

The Max Cooper live shows are marketed as audio visual experiences, which could easily just be a cliche. I already knew about the audio part – the music. But the visuals blew me away. It was some of the best I've seen in a long time, in the ranks of Moderat and Massive Attack. The setup was rather simple: A projection behind him, another one on a transparent screen in front of him and then a couple more on the walls adjacent to the stage. The effects of the layered projections was stunning.

Max Cooper released his new album, On Being, a week ago. Give it a listen. And if you happen to live near one of the cities remaining on his tour, I strongly suggest you go.

Self hosting the Fediverse

It's amazing times for self hosting the Fediverse.

While I'm not necessarily a big fan of Docker, if you're familiar with docker compose (which in itself of course raises the bar to a certain extend), you can easily spin up things in a matter of hours.

A few weeks ago I was trying out Owncast after dinner, and managed to get things running and start a stream from my webcam the same night.

Earlier this week, I set up GoToSocial along with Pinafore, Semaphore and later Phanpy as frontends for it, before I settled on Elk.

I highly recommend trying this out for yourself if you have the required computer skills. Or even if you dont, but are curious – ask your hacker friends for help. Remember, you can have this running to tinker with, while keeping your main accounts somewhere else that you consider more safe, if that's your worry.

Next up for me is to try out something that offers those sweet, sweet custom emoji reactions.