Fork me on GitHub
#clojure
<
2023-01-14
>
nooga17:01:47

I'm trying to figure out how dynamic vars work internally but having hard time to understand the Java impl. From what I can infer there is a tree (or a stack?) of thread-local environments (basically maps var->value) built via binding and the dynamic vars will traverse this tree to find their root upon deref. The thing I don't understand is how do we know which thread-local dynamic environment we are currently in while executing deref? How do we avoid dynamic lookups for non-dynamic vars? Could somebody eli5 the idea behind dynamic var implementation to me?

hiredman17:01:08

The compiler looks at the var metadata when generating bytecode and emits a call to getRawRoot (or a method that is named something similar) unless the var is marked as dynamic

nooga17:01:17

ok, so this is how we avoid dynamic lookup for non-dynamic vars

Alex Miller (Clojure team)17:01:53

originally all vars were dynamic, but the perf hit was not worth it when dynamic was not needed

Alex Miller (Clojure team)17:01:25

so the default was switched to non-dynamic (overridden with ^:dynamic)

nooga17:01:03

I went the other way and implemented only non-dynamic vars first, now I'm trying to implement binding and it gives me a headache 😄

Alex Miller (Clojure team)17:01:53

I'm not sure if the question on "how do we know which thread-local dynamic environment we are currently in while executing deref" was answered, but it's using a ThreadLocal for the current frame stack

Alex Miller (Clojure team)17:01:05

so it's inherently thread-specific

Alex Miller (Clojure team)17:01:36

there are some context where the binding environment is migrated to a new thread (like with futures, agents, etc)

👀 2
nooga17:01:42

I see, and since dvals is static it is shared across all vars, each thread having its own chain of Frames there

nooga17:01:54

so upon accessing a dynamic var we essentially perform lookup across bindings in those Frames and ThreadLocal makes sure we walk the right chain?

Alex Miller (Clojure team)17:01:24

yes, top frame in current thread and then on down

nooga17:01:50

ok, that makes total sense, thanks! 🙂

ghadi17:01:41

It doesn’t need to walk down the frames, just grab the current frame, which contains a map with all bindings

nooga18:01:50

ah, but pushThreadBindings seems to make a new frame with only the bindings passed to it we could have something like:

(def ^:dynamic *x* 1)
(def ^:dynamic *y* 9)
(binding [*x* 2] 
  (binding [*x* 3]
    (println *x* *y*))
  (println *x* *y*))
(println *x* *y*)
If I understand this correctly, *y* is only bound at the top here so whenever we use *y* here the lookup will go back to the top scope to find it

ghadi18:01:04

No. like i said, each frame contains all bindings

ghadi18:01:26

The bindings are persistent maps, added to with assoc

ghadi18:01:49

Well… all of the previous bindings

nooga18:01:02

Associative bmap = f.bindings;

nooga18:01:14

right, missed that one

nooga18:01:46

alright, I get it now

nooga18:01:23

thanks for explaining hiredman, Alex & ghadi 🙏

andy.fingerhut18:01:24

It gives the same result as a different implementation that walks the bindings in the stack, but takes advantage of persistent hash maps to optimize that.

2
nooga18:01:57

all good, I'm asking all those questions because I want to implement this myself and ultimately my impl will be different in many details like that. Mainly because I have my own VM that is totally different from JVM 😅

borkdude10:01:09

@UJEK1RGJ0 if you environment is single threaded, the impl can be much simpler. e.g. CLJS just uses mutation in place for dynamic vars

2
lilactown20:01:26

I typically reach for ring + jetty when developing web servers, but I'm struggling to find a definitive standard "here's how to add a web socket host" that doesn't involve switching to a different web server or bring in something like core.async. is my google-fu simply failing?

Ben Sless20:01:08

I think http kit exposes a callback interface for websockets

seancorfield21:01:24

sunng's jetty9 adapter includes websocket support (and actually uses Jetty 11, despite the 9).

seancorfield21:01:28

We switched from the default ring/jetty adapter to ring/jetty9 (i.e., 11) at work recently so we could have simple WS support and it works beautifully -- and everything else in our stack is still happy because it's still jetty, including all our monitoring etc.

lukasz21:01:46

There's also this Jetty wrapper, also supports WS https://github.com/factorhouse/slipway

lilactown22:01:05

this is great, thanks @U04V70XH6!

lilactown22:01:03

I also looked at slipway but was very confused about what it even does. It wasn't obvious to me until after much reading that it's a sort of replacement for ring + the ring jetty adapter

phronmophobic22:01:56

fwiw, if you search for "jetty websocket" on https://phronmophobic.github.io/dewey/search.html, the https://github.com/sunng87/ring-jetty9-adapter adaptor is the first result.

seancorfield00:01:09

Also, regardless of WebSocket support, Jetty 9 is well out of community support at this point and several CVEs have been reported against it lately so updating to Jetty 11 makes sense (although Jetty 9 is still getting critical security fixes I believe) -- and the default ring adapter is not likely to get updated to Jetty 10/11 per https://github.com/ring-clojure/ring/issues/439

lilactown03:01:50

I was unsure what the tradeoff was using the (newer, less battle tested from the outside) jetty9 adapter vs the official one. sounds like the official one ought to be deprecated in favor of the the jetty9+ one

rutledgepaulv04:01:04

i'd also recommend sunng's jetty9+ adapter and have been using it happily for years. in case it's helpful here's an example of building a registry of connections and adapters for core.async on top of the jetty9 websockets: https://github.com/RutledgePaulV/websocket-layer

igrishaev13:01:57

Aleph provides a web-server which is compatible with Ring handlers and middleware with WS support as well.

seancorfield20:01:58

One thing to consider for production usage is observability: you need to endure whatever you select is well-supported by your existing monitoring services and tools. We went from Jetty to http-kit but New Relic doesn't support it so we went back to Jetty. We also have a Netty-based service and New Relic struggles to give useful information with that. Aleph is Netty-based so that would be a concern for me.

lilactown16:01:08

that's a good point sean. thanks for sharing

lilactown16:01:47

we have our own hand-rolled observability system at work (because aleph+netty), but that gives me food for thought when thinking about what I base my OSS projects on

2