I don’t normally presume to be competent enough to do a write up of something as technically advanced as the xz backdoor, but this has been such an thrilling moment in IT history that I just had to write my thoughts down to ruminate on them.

At the end of the day I’m relieved that the impact was not greater, so now I’m mostly interested in who might be behind it. Instinctively it feels like a decently sized project by a state actor. It feels like they could even have had a little team with project manager, and their own milestones.

Because of how long it took to gain trust in a part of the open source community, and because of how the attackers used different kinds of social engineering pressure, and different types of indirect patches to achieve a long term goal.

Read the entire summarized timeline of the attack made by Russ Cox here for a quick introduction to what happened.

Short summary of my own perspective

In 2015 liblzma was linked into libsystemd in order to compress logs. Liblzma was a small and relatively new compression algorhithm, part of the xz-utils project maintained by just one person.

Many mainstream Linux distros link OpenSSH with libsystemd in order to use the sd_notify function. Which means that the sshd binary shares its address space with liblzma from xz-utils.

So a sophisticated hacker group saw this connection and decided to attack the xz-utils project. For 2 years they gained the trust of the maintainer by making regular patches and commits to the project.

They also applied pressure to the maintainer using different fake personas who wanted the maintainer to merge patches quicker, and focus less on the xz project and more on other projects. These personas rarely mentioned any of the other attacker personas, instead indirectly implying that they wanted their patches merged sooner.

All of this lead the xz maintainer to hand over the maintainer role to one of the attackers who had already proven to be a valuable contributor during the past 2 years.

In february 29 a pragmatical user suggested that libsystemd get rid of their dependency on many small compression libraries such as liblzma, and instead use the larger libarchive project to handle all compression formats. This change was actually merged into libsystemd on march 1st.

This put the attackers into a panic because it meant they had to speed up their plans of backdooring OpenSSH via libsystemd. It also lead to them opening up alternative attack vectors such as proposing Linux Kernel patches that would promote the use of xz to compress Linux kernel images, instead of the classic gz used today.

Fortunately for the entire world their panicked state caused them to write such bad code that the backdoor was discovered.

The China connection

Myself and probably many others reading the original mailing list post on openwall.com immediately reacted to what I call the main character in this drama, Jia Tan. An obviously Chinese-sounding name.

With the current media reporting today my instinctive thought was that China must be behind this.

But after talking about it with friends we quickly came to the conclusion that it could technically be anyone, having a Chinese-sounding name for the main character is quite distracting after all.

The other characters in the timeline all had names that seemed to be from all over the world, but all the commits were done by Jia and that was the first name anyone discovering this backdoor would encounter.

Evan Boehs on their blog claims to have sources with enough knowledge of Chinese languages and cultures to say that the various forms of Jia Tan used indicate it was falcified by someone who was in fact not from China.

On at least one occasion, buried in the git-commit logs, Jia Tan used the full name “Jia Cheong Tan”, and this was the combination that raised eyebrows among certain chinese experts.

Besides this sudden name change, the git logs also indicate that most work was done during regular business hours for the UTC+2/3 timezones.

So it seems to me the attackers made two crucial mistakes about keeping their identity hidden, they leaked a nonsensical full name of their main character, and worked during regular business hours.

The real origin

Some people reading this immediately said Russia, or even North Korea. I don’t agree with those guesses for two reasons. North Korea are indeed very loud and active when it comes to hacking, but they’ve never been sophisticated. They lack the morale required to attract disciplined and long term talent.

And Russia would never blame China for this attack now that they need missiles in their war against Ukraine. It’s already known they’re buying junk from North Korea just to keep murdering civilians in Ukraine, so China is their most valuable supplier.

No, my thoughts go to either USA or Israel. The USA has too much to lose with a backdoor that affects every single Linux server. They have the largest economy in the world and much of it runs on Linux servers.

Israel is a relatively small country, with very large resources, and nothing to lose by doing this. And they happen to be within the UTC+3 timezone where most of the commits were likely done.

Remember Stuxnet?

Israel was after all behind the notorious Stuxnet hack, where they gained such control over the Iranian nuclear enrichment program that they could have modified their systems to make them literally explode. Something that is often a myth purported by hacker movies.

And in the wake of Stuxnet being revealed there was little doubt that Israel was behind it. Some people might claim that Mossad is too advanced to leave such tracks behind them, I beg to differ. On the international stage they still always have plausible deniability.

Sophistication

Speaking of Stuxnet some have commented that the xz backdoor is almost as sophisticated as Stuxnet was. I beg to differ, it’s even more sophisticated!

I’d say that infiltrating a part of the open source ecosystem in this way takes even more patience, discipline and sophistication than Stuxnet.

Most likely Stuxnet was made possible by an insider, and perhaps a lot of bribes to that insider. Because it was a proprietary system being used so they first had to extract a lot of information from the Iranian plants, and then also inject their own virus back into their plants. Which is of course very risky, but not very hard when you already have an insider.

While infiltrating an open source project could not be done by any one insider, they had to first find a vulnerable project maintained by one person and then spend 2 years gaining their trust. And apply pressure on them using various fake personas.

And the only reason it failed was because the open source ecosystem is so volatile that someone proposed a patch that would have removed their attack surface.

I’d say this is slightly more sophisticated and complex than what I assume Stuxnet was like. But we’ll never know the true details of that attack.

Rizky biz podcast

An addendum after having listened to the rizky.biz podcast today on the 6th of april 2024.

Their arguments for Russia likely being behind this attack are as follows;

1. The “5 eyes” alliance would never use such weak encryption as RSA.
2. A russian guy felt that the e-mails looked like a russian trying to write in english.
3. A guy on the podcast felt that the deep knowledge of old build systems felt like “old unix” to them.

They didn’t even mention Israel, even though they were notoriously behind Stuxnet. Maybe they include Mossad in their high views of the “5 eyes” alliance.

After listening to the podcast I’m less sure of my case but I still feel that no one can be sure it was Russia either.

It’s just my gut feeling that Russian experts are not as loyal to the state as Israeli experts would be, and most of the skilled security experts in Russia have turned to quick money grabs instead of long term projects like this.