Fork me on GitHub

Given the importance of this: -- TL;DR: the JVM option was not sufficient to disable all lookup paths so installs with log4j2 2.10.0 - 2.14.1 with that option are still vulnerable. 2.16.0 is available now with the entire lookup machinery removed to prevent a potential DDoS possible through 2.15.0(!). Discussion in a thread please. Guess we'll be updating all our libraries again tomorrow!

👍 18
🙏 4

thanks! I released another version of

👍 1
Michael W15:12:27

If you can't upgrade you can strip JndiLookup.class from the jars, we have done that for several older systems with slow vendors at work and it appears to work great.

❤️ 2

@UAB2NMK25 - nice and simple solution for some situations

Cora (she/her)23:12:52

it looks like log4j 2.15.0 has a more serious vulnerability than just a denial of service, making an upgrade even more important

❤️ 4

That's why I posted the link to the log4j security page here last night and described this in the TL;DR note attached to it.


(CVE-45046 which Praetorian also mention)


Interestingly, I thought when I posted that link last night that the log4j security page explicitly said that 2.15.0 did not fix all lookup paths but now that I read it this afternoon, I can't see that mentioned.


Oh, here it is, way further down the page than I thought! "The reason these measures are insufficient is that, in addition to the Thread Context attack vector mentioned above, there are still code paths in Log4j where message lookups could occur: known examples are applications that use Logger.printf("%s", userInput), or applications that use a custom message factory, where the resulting messages do not implement StringBuilderFormattable. There may be other attack vectors." ( my emphasis )


This has been such a mess... I feel like I've been working on this issue in one form or another continuously since Thursday evening 😞

gratitude 4
♥️ 4

Today, our datacenter sent us a notification that "log4j2 2.14.1" has been detected on your servers. It was still in the local Maven cache, even though both 2.15.0 and 2.16.0 were there as well (and everything on those servers is using 2.16.0 only at this point!).


We're still waiting on Elastic Cloud to follow up (they already notified customers to restart their clusters to pick up the JVM option -- which we've since all learned is not sufficient).

Lukas Domagala00:12:41

ugh your situation sounds really exhausting, hope you guys get some well-earned rest after this week:+1:💪

Cora (she/her)00:12:05

this vulnerability goes beyond the one in the CVE, though?

Cora (she/her)00:12:11

in the link I posted

Cora (she/her)00:12:47

that sounds really frustrating

Cora (she/her)00:12:52

an entire week of work oh this


The log4j security page also goes beyond that CVE -- but it's a bit buried in the page. But the fact that 2.15.0 doesn't prevent all lookups was mainly why I posted that last night. The DDoS vuln was just icing on that particular cake.


It's good to have that Praetorian article posted too -- I suspect a lot of folks haven't read the log4j security page in detail (and, worse, it keeps getting updated so you have to keep going back and re-reading it over and over again!).


In another tech Slack, I've been arguing with people who continued to claim the JVM option was all you needed 😞

😬 2
😠 1

At this point, I'm sort of expecting 2.17.0 to appear with an explanation of why 2.16.0 doesn't solve everything either 🙂 (but, given 2.16.0 has removed the lookup machinery completely, I hope that's just me being cynical at this point!)...

thisisfine 1
Cora (she/her)00:12:40

all eyes are on java logging libraries for vulnerabilities now so it may even be something else entirely


At this point I’m just grateful that my company doesn't use Kronos for payroll


Both cloudflare and AWS have come up with firewall rules to drop log4j related attempts. I still see a few weird attempts reaching out nginx logs but otherwise it does a decent job. We still of course updated to 2.16.0


Given that requests can be broken down to ${${..}..} and not include the jndi string, I'm not sure how they're blocking it -- especially since it can occur in the body, the URL, or any header. But I guess if they're blocking the most obvious attempts that at least lessens the load on your app...


Yeah this log4j2 thing is a real pain due to its extent. Auditing has taken up a lot of our time this week. I’m hoping this will settle down now; but I’m also quietly hoping we’ll see log4j2 cut back and release a cut down version — moving any fancy/risky networking things into optional extras, and getting them off the classpath altogether. I don’t want to throw the baby out with the bathwater; but I’m also wondering if people have any recommendations for a more minimal java based logging framework that works with I’m not looking at jumping ship just yet; but I’ve since discovered tinylog, which might fit the bill.


Though I’d certainly appreciate any competing framework like this to have had a pretty thorough audit. At least with log4j2 we have many eyes right now.


Yeah I spotted that over the weekend too; this one seems less likely to be that big a deal for most folk though. Obviously YMMV