This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2020-05-05
Channels
- # announcements (7)
- # babashka (20)
- # beginners (130)
- # bristol-clojurians (1)
- # cider (14)
- # clj-kondo (7)
- # cljdoc (14)
- # cljs-dev (15)
- # cljsrn (16)
- # clojars (11)
- # clojure (190)
- # clojure-dev (4)
- # clojure-europe (7)
- # clojure-italy (9)
- # clojure-nl (3)
- # clojure-romania (6)
- # clojure-uk (51)
- # clojurescript (44)
- # component (4)
- # conjure (28)
- # cursive (1)
- # data-science (4)
- # datascript (1)
- # datomic (30)
- # duct (4)
- # emacs (1)
- # figwheel (4)
- # fulcro (56)
- # graalvm (4)
- # helix (51)
- # jackdaw (2)
- # jobs-discuss (12)
- # joker (4)
- # lambdaisland (1)
- # local-first-clojure (1)
- # meander (73)
- # mid-cities-meetup (2)
- # nrepl (4)
- # off-topic (43)
- # pathom (56)
- # re-frame (37)
- # reagent (26)
- # shadow-cljs (161)
- # slack-help (9)
- # spacemacs (1)
- # tools-deps (18)
- # xtdb (18)
I just asked this on #re-frame but since it's not entirely related, I will ask here for general advice as well: Has anyone used `re-learn` (https://github.com/oliyh/re-learn/)? I'm having trouble integrating it into my application.
Is there a technique for overlaying 2 partial protocol implementations onto each other? I think I'd want a way to go from something implementing a protocol to a map from {:method (fn ...)}
like the one given to extend
and then merge those together & attach as metadata.
My current idea is to have the default impl return "::not-implemented" and then try the next impl in that case.
@dominicm hmmm, extend
is a function, can you just write another function that extends your types in a way you want?
@vlaaad I want to combine two instances, rather than two types. I want to do this at runtime.
do you control the protocols? If your instance is IObj
(can have meta), you can use :extend-via-metadata
to do this on value level
Tbh, if I separate the implementations to functions, I don't even need the overlay function
found it -was using the wrong method:
(defmethod print-method (class (transient {}))
[c ^.Writer w]
(let [v (persistent! c)]
(.write w (str "transient: " (pr-str v)))
(transient v)))
(prn (transient {})))
(this is only for debugging)or is it for physics optics?
I’m thinking about classpath isolation of libraries. Would it theoretically be possible to load two different versions of a library without conflicts. I can imagine this is possible for pure clojure sources as we could hack into the ns
macro and prefix the namespace with something unique. For java class this might be harder. Anyone, know of people that, thought about this before?
there are big projects that do this kind of thing, this is part of why classloaders are so complex
https://stackoverflow.com/questions/9576207/what-is-an-isolated-classloader-in-java
the tricky thing would be setting up clojure to be able to use any of those isolated loaders - there was someone trying that, he had to fork clojure itself to make it work
I don't know how far he got
You can override the ns macro with some hackery?
And then there is now Sci which would be able to natively override the ns
macro
for clojure namespaces, absolutely, but this was about being able to use different versions of a class
yeah exactly that is unknown territory for me, but maybe that’s possible to given what you mentioned about isolated loaders
Thank you. I might give it a try later
but then how do you want that to interact? clojure currently assumes that as you make new definitions, you want all previous loaders avaialble (it chains them as you define new classes via defn etc.)
I think that's where things get trickier - do you have two clojures for the different isolations? one clojure that somehow chooses between one isolation and the other?
I think a "split timelines" metaphor applies: once you jump into an isolated loader, you can see shared history, but your new definitions won't cross into the parallel timeline(?)
to have a single clojure loader, that can choose isolations (jump between timelines, since each loader sees parent loaders / the past), you need to change clojure more fundamentally I think
ok so let’s make a difference between libraries 1) ones you load directly and 2) ones that are loaded by your libraries
ones that are loaded directly need to be unique, period, no double versions
the ones that are load indirectly are loaded in a known context, that of your library. When loading this library all externally loaded could be prefixed by the hash of the artifact name and it’s version
that doesn't address the fundamental problem - classloaders work by saying "I know how to look up class x,y,z, and on failure I look to my parent"
This way if different libraries load the same dependencies they can still reuse the loading of this, because they end up with the same prefixes
this works cleanly if you have a chain from every new loader to parents up to the top
OK this is fine for namespaces, I agree that's easy
I'm talking about the thing that's hard, classes
Yeah i’m not sure how it would work exactly with the classloader. I’m mostly thought about the clojure side
So I guess I barely touched the real problem 🙈
I also have no idea how expensive a seperate classloader per library would be
clojure creates a new classloader each time you call defn
ok sounds cheap
ok let me read upon classloaders and rethink what i’m suggesting
the basic idea is a linked list, current classloader delegates to the one it was created with as a parent, you ask a classloader for a class, and it either finds and returns it, or gets one from the parent
I guess technically other behaviors might be possible...
One classloader per (nested) library? And within these libraries the fn’s created would use their library classloader I guess
Ok i have no idea what i’m talking about. I’ll have to study a bit
that's why clojure creates one per defn - so that each time you call defn it can refer to the one defined previously
This may be off the topic of what you are looking for, but there is a technique sometimes called "shading" where you load Clojure code into a different namespace than it was originally written to use, so that you can potentially load multiple versions of that Clojure code simultaneously, using namespaces. One library to assist in this is called mranderson: https://github.com/benedekfazekas/mranderson
I have not used that library myself, but have heard others have used it with good effect. I wrote a hacked-up thing in order to shade some libraries that the Eastwood linter uses internallly.
There is no classloader trickery involved here, but that means it only works for Clojure source code, not for arbitrary Java libraries.
Thank you!
hey all, i'm looking to do some serialization of a data structure that holds closures, aka functions with stored state. in python, i would use pickle, but I'm struggling to find an equivalent. there's serializeable-fn and nippy, but those don't handle "stateful" functions
well, i found a blogpost saying that nippy does handle this, but their documentation doesn't, so i'll have to do some more research about it
@nbtheduke what I've used for this is instead of a closure, a defrecord
implementing IFn
that means you can call it like a function, but instead of closed over hidden state, it has an explicit immutable hash-map
so you use conj
or update
or assoc
etc. to get a new record, with the same methods but new "state"
this is easy to serialize - clojure makes sure records act like maps as far as data is concerned
Huh, interesting. I don’t know that that’ll work for my use case but it’s a clever way of doing things
(defrecord F [x]
clojure.lang.IFn
(invoke [this arg]
(* arg x))
(applyTo [this [arg]]
(.invoke this arg)))
(ins)scratch=> (def f (->F 2))
#'scratch/f
(ins)scratch=> f
#scratch.F{:x 2}
(ins)scratch=> (f 3)
6
(ins)scratch=> (apply f [3])
6
(ins)scratch=> (def g (update f :x inc))
#'scratch/g
(ins)scratch=> (g 3)
9
@nbtheduke if that doesn't work because F needs mutable state, atoms are available
that just makes your serializer slightly more complex - it needs a deref
i think there was a recent attempt at this? not sure wht the constraints were but someone was doing serializable functions
hell yeah, thank you
Serializing functions is a bad idea, and yet there are about 2 projects a year that come out purporting to do it
you can also capture closed over state, but the function still needs to opt in
(defmacro locals
[]
(into {}
(map (juxt (comp keyword name)
identity))
(keys &env)))
(ins)scratch=> (def f (let [x 2] (fn ([] (locals)) ([y] (* x y)))))
#'scratch/f
(ins)scratch=> (f 3)
6
(ins)scratch=> (f)
{:x 2}
some environments make it hard to live without. given redplanetlabs and the word "nippy" I'm guessing this has something to do with a distributed environment like spark or flink or whatever
it for sure causes problems in those environments but it comes with a lot of benefits too
I like doing it via record rather than function+macro, because the record comes with its own composition utilities (creating a new of "itself" with updated bindings), which also acts as a deserializer of that state (with all the caveats of data / state, eg. you should be using immutable values to get the most from this)
yeah I wrote clj-headlights (defunct) which is clojure in a distributed env (apache beam), and I went with capturing vars and a list of serializable params intead. it worked better and was "safer" but it was not as nice to use
I think that expecting to take normal functions and serialize in a way that includes all relevant closure data is monkey-patch-complete
it is impossible without the ability to monkey-patch and comes with all the problems monkey-patching introduces
I think deliberately exposing immutable and effectively closed data is a reasonable compromise
for sure, but if you're doing something like writing spark jobs you tend to have the possible issues that could come up in the back of your head and so just make sure they don't come up
of course, you can implement any language feature by convention ifyou have the intellect and discipline
I think I lack both, in the aggregate
like, I don't generally pass a lambda into a spark dataset function... unless I'm in some tiny static function and I can see everything that's going to go across so I know it's good
I have a game engine that has to be restarted to update. I’d like to be able to serialize the state of a game so I can update the engine without losing existing games
the "simplest" thing there is to "lift" the state out of function closures into an explicit argument passed through the programming logic - that's easy to capture and reuse
I know putting simplest and lift in quotes looks pretentious but I want to be clear I'm using very specific domain meanings of those terms
it's slightly tedious to code that way, but it's harder to make big mistakes
if you want to work bottom up you can trade function closures for records over IFn that expose their internal state
if you want to work top down you can implement a big hash map with your state, pass it to an updater / program logic, and have that return the new state
Sadly, the game engine is quite stateful, using an atom hash map, and things like card abilities are represented as closures
the atom hash-map can be traded for a regular hash-map if you use the return value of every function that currently acts on it
the closures can be made transparent if instead of just closing over locals, they explicitly own them / expose them
of course this is harder with bigger projects, it's a rewrite, but it's a bunch of local rewrites in most clojure codebases, rather than one big global change
it can be done piece by piece, bottom up or top down as you prefer
That would be great, but it’s not feasible with the size of the code base lol. For example, https://github.com/mtgred/netrunner/blob/37b7225582f635d44bff54f8be2564414de46b21/src/clj/game/cards/ice.clj#L610 Bloom is a card with a closure. req
is a macro that expands to a function definition, so the use of the this
variable isn’t easily handled another way
I’d love to have a more pure game engine, but it is very much not and would require a nearly full rewrite
I'm not saying it wouldn't, but I am saying that this rewrite can be incremental - you can make a version of define-card
where the effect doesn't call swap!, but rather takes a state and returns a new state - now the caller calls swap!
and smaller conversions like this would keep the current behavior, but move closer to purity, and purity is where you get trivial serialzation, so these kinds of changes should each make serialization easier
(and I think generally, whether you are doing things functional / data oriented or not, serializing state is easier if you implement it early and shape your design around it)
yeah, agreed. i wish every day the engine wasn't so stateful, lol
thanks for the input
also you might need state for perf reasons in some cases, but you can still "firewall" and be explicit about what's in scope of that state and how things cross the boundary from the serialized system to the stateful one (you see this in games a lot, certain things about your character are never saved - older games would eg. only let you save from a certain room and if you weren't mid combat etc. so the amount of data to store would be much smaller, and the restoring code could be simpler)
this was the approach we used when I worked on a MUD implemented in C (multiplayer networked text based game) - there were explicitly known savable properties, and restrictions that prevented relying on properties outside that set
but even that is easier if you start with saved state of a character as an initial feature
yeah, that all makes sense
thanks
question for people who are familiar with Clojure’s STM / Java threading stuff: I’ve started playing with implementing a sort of MVCC STM framework for (CL)JS, and I’m finding myself drifting from the way that it’s implemented in Clojure notably, `dosync` is eager and synchronous (in the thread it’s executed in), but what I’m finding in JS that’s a significant downside if your transaction is CPU bound at all since in JS you can only achieve concurrency by yielding (using some sort of async timer + callback), I’m finding it better to split up the work in your transaction into discrete units and then schedule them to be run, yielding to the main thread between each unit of work. would this paradigm at all be useful in Clojure as well? is there anything similar out there that I could use to relate my field notes to?
I'm not sure I'd ever invoke future
, or async/go
inside a dosync
.
(if that's a valid analogy?)
My reasoning boiling down that transactions should be thin (fast), and therefore not dependent on IO (threading can be considered IO)
Thick transactions are subject to be retried endlessly
Personally I'm not super sure I'd use dosync
for a CPU-heavy transaction either. dosync
txs can be retried, so that CPU work would be retried as well
Seems slow/problematic
So, to split work transparently, I think you'd need to give CLJS a go-loop runtime so you can instrument each invocation with an :interrupt
signal. But after a specific invocation is fired, I still don't think you'll be able to interrupt the JS that runs inside that context, unless it calls back into a runtime-managed fn at some point and the interrupt signal can be checked again.
I think you'd have to store interrupts in SABs for the different contexts to communicate them synchronously :thinking_face:
I doubt people actually want to use STM directly in applications; rather, it is something that can be leveraged by a framework similar to core.async, or an in-memory database, etc.
I'm building it because I need it for another project that's at that higher level of abstraction
@U45T93RA6’s point about CPU-heavy transactions is valid, though. a highly contentious, CPU bound transaction is a terrible thing
I wasn't talking about io, though, but similar to what you're saying @U050PJ2EU about allowing transactions to be interrupted and resumed later. I'm just not super picky about the syntax for that atm
I guess core.async is a reasonable example for this sort of scheduled, interruptable programming being useful. although without transactional semantics
yeah, so in order to prevent transactions from blocking the main thread, you write your transaction as a sequence of functions that will run in order
and in between invoking each function, a scheduler runtime can check to see if there's more important work to do and schedule the next function to be run
ah, I'm only talking about the differences between clojure's STM and what's in my brain
the STM part is that all the code running in the context of the transaction sees their changes throughout the execution - but the outside world does not until the transaction is committed
but presumably, different callers are dropping calls into that bucket and the transaction takes care of running them in the correct order, or re-running them if necessary?
the transaction executes each function in the order it's received and makes sure that the functions see the local values of each ref correctly, in the context of that transaction. and re-running them, etc.
there can be multiple transactions happening at the same time, which is how contention happens
so you create two transactions:
(def foo (ref 0))
(def txA (transaction))
(def txB (transaction))
(with-tx txA
(ref-set! foo 9))
(with-tx txB
(alter foo + 1))
@ref ;; => 0
(doseq [thunk txA]
(thunk)) ;; in txA, `foo` is 9
(doseq [thunk txA]
(thunk)) ;; in txB, `foo` is 1
;; sets `foo` to 9 in the global context
(commit txA)
;; drift has occurred; `txB` retries and computes 10 as the new value of `foo` and commits that
(commit txB)
@foo ;; => 10
Trying to imagine... Perhaps if you also monitored the datastructure inside the bucket... And Bob dispatches an update on :a in {:a 1 :c 2} and Alice is subscribed to :a, then if the results of her last computation depended on :a, the transaction will re-run her dispatch to :a... But, it's single threaded, so Bob and Alice would have already had the latest :a from Alice, before Bob started... So I'm confused
you could potentially yield after each call, or batch them together, however you like
in the example above I just synchronously call them using doseq
. a much more advanced scheduler could e.g. prioritize the calls to each thunk
based on some criteria set at the beginning of the transaction
I guess I'm curious about how it goes about solving a coordination problem between two refs, to the extent that we have coordination problems between atoms in CLJS
well, all of this is in service of creating a slightly more advanced reactive framework similar to reagent
that's what drove me to needing this ref/STM stuff... if you can't/don't want to compute the entire dataflow graph in one loop, then you need a way of scheduling changes to the graph while keeping data coherent
I see what you mean.... I'm not sure I'd call it an STM. Because a single JS thread is atomic, it has the same guarantees as an STM, so if you batch fns into a single synchronous context of execution, it would provide the same guarantees, sure.
yeah, I guess the point of what I’m building is to give atomic guarantees to a transaction that crosses multiple frames
here’s some psuedo-code that shows the kind of API I’m finding myself trending towards in CLJS
what are the semantics for conflicts? for clojure’s refs, if there are conflicting transactions, then the transaction will automatically be retried. my intuition is that if the work is split up, then transactions can’t automatically be retried. is that right? having transactions that don’t automatically get retried seems like it might have benefits for some use cases. although I don’t know what they would be off the top of my head.
interesting. does
(doseq [thunk tx]
(thunk)) ;; execute each unit of work in the tx
get reran when you call (commit tx)
?the example shows only one ref being modified in the transaction. can transactions modify multiple refs? if so, that’s pretty neat.
the transaction in the psuedo-code above is mutable, so by executing each unit of work it would move that thunk from "pending" to "done" a conflict would reset each unit of work back to pending and clear any local ref alterations trying to commit a transaction that conflicts could either throw or automatically retry, depending on what behavior you want
this approach seems to make a lot of sense for an environment with a single thread. I’m not sure there’s much benefit for an environment that already does have support for multiple threads unless your library can help schedule the transaction work to avoid conflicts*.
clojure’s stm relies on optimistic concurrency which works well with low contention (i believe). an alternate approach that schedules transaction work to avoid conflicts might work well for a workload where you do expect a lot of contention.
the only benefit is that you can potentially find out about contention sooner... e.g. when running each unit of work, the transaction could check to see if any ref version have drifted since the last one
makes sense. to be fair, there aren’t really any multi-threaded approaches that work well with a lot of contention. thanks amdahl
but I think you still end up with potentially lots of retries with lots of contention. I'm not sure if scheduling would help... unless you could sort transactions into groups based on contention
which is ultimately my goal: to have the ability to order transactions by priority and allow a low priority transaction to be paused so that the scheduler can run a higher priority transaction
yea, just trying to think of what the tradeoffs might be compared to a implementation with multiple threads on the jvm. it seems like the main difference is the scheduler (ie. you manually working on parts of the transaction vs the OS scheduler picking which threads to run which determines which transactions get worked on)
working on threads transactions* by priority sounds interesting
it's pretty key for UIs to prioritize state changes, e.g. keyboard input needs to be handled almost immediately while a websocket can probably wait a bit
@U4YGF4NGM, using refs for ui state seems really intriguing, but afaik, it’s largely unexplored. do you know any good examples or resources for using refs for ui state?
nothing that I’ve seen explicitly stating it’s using STM/refs, but lots of things start to exhibit similar behavior if you squint
React is working on a concurrent, prioritizing schduler for rendering that uses immutable data + ref-like semantics, which is what inspired me to see what a more explicit translation of STM would look like that would fit that use case
i’m building desktop user interfaces. I’ve been using atoms to hold all of the app state and that generally works well. it seems like there might be some performance and architectural advantages to using refs, but most desktop design is based on old OO principles and the functional designs in the browser are under different constraints.
yeah, I think that using an atom leads one to eventually storing all state in a single atom to coordinate updates
the idea behind using refs is that it allows you to store state in separate containers, and ensure that they are updated in a coordinated fashion to avoid flickering / needless re-renders
another way of solving this is batching at the renderer, but you also have to solve the same problem anywhere you are doing side effects based on state changes
the tactic I’ve been using is to have to wrap expensive rendering calculations. basically, if the expensive calcuation is ready, it shows the result, otherwise, you can choose to show something else (eg. a partial result, the most recent finished result with a spinner, an empty view with a spinner)
the main reason I’m interested in refs are potential performance benefits and more flexibility when modeling your UI state