Fork me on GitHub
#clojure
<
2020-05-05
>
Setzer2207:05:57

I just asked this on #re-frame but since it's not entirely related, I will ask here for general advice as well: Has anyone used `re-learn` (https://github.com/oliyh/re-learn/)? I'm having trouble integrating it into my application.

oliy09:05:57

hi, i'm the author

oliy09:05:13

there are two example apps in the repo, have you looked at those?

dominicm09:05:56

Is there a technique for overlaying 2 partial protocol implementations onto each other? I think I'd want a way to go from something implementing a protocol to a map from {:method (fn ...)} like the one given to extend and then merge those together & attach as metadata.

dominicm09:05:30

My current idea is to have the default impl return "::not-implemented" and then try the next impl in that case.

vlaaad10:05:28

@dominicm hmmm, extend is a function, can you just write another function that extends your types in a way you want?

dominicm10:05:45

@vlaaad I want to combine two instances, rather than two types. I want to do this at runtime.

vlaaad10:05:12

do you control the protocols? If your instance is IObj (can have meta), you can use :extend-via-metadata to do this on value level

dominicm10:05:23

@vlaaad I do. But how do I know which object implements what?

vlaaad10:05:54

If all objects implement your protocols using meta, you can just look at the meta

dominicm12:05:29

Ah, subset of protocols. Makes sense.

dominicm12:05:12

Tbh, if I separate the implementations to functions, I don't even need the overlay function

dvingo12:05:18

Is there a way to override the print-dup method for clojure types?

dvingo13:05:00

found it -was using the wrong method:

(defmethod print-method (class (transient {}))
  [c ^.Writer w]
  (let [v (persistent! c)]
    (.write w (str "transient: " (pr-str v)))
    (transient v)))
  (prn (transient {})))
(this is only for debugging)

Rucy12:05:09

Is there any clojure library for optics?

vlaaad13:05:49

like get-in/`assoc-in` but more powerful?

Old account13:05:45

or is it for physics optics?

Rucy14:05:27

@vlaaad wow! that seems very interesting thanks. And no haha not physics optics 🙂

orestis05:05:48

Also meander

jeroenvandijk13:05:25

I’m thinking about classpath isolation of libraries. Would it theoretically be possible to load two different versions of a library without conflicts. I can imagine this is possible for pure clojure sources as we could hack into the ns macro and prefix the namespace with something unique. For java class this might be harder. Anyone, know of people that, thought about this before?

noisesmith13:05:21

there are big projects that do this kind of thing, this is part of why classloaders are so complex

noisesmith13:05:08

the tricky thing would be setting up clojure to be able to use any of those isolated loaders - there was someone trying that, he had to fork clojure itself to make it work

noisesmith13:05:13

I don't know how far he got

jeroenvandijk13:05:11

You can override the ns macro with some hackery?

jeroenvandijk13:05:51

And then there is now Sci which would be able to natively override the ns macro

noisesmith13:05:00

for clojure namespaces, absolutely, but this was about being able to use different versions of a class

jeroenvandijk13:05:40

yeah exactly that is unknown territory for me, but maybe that’s possible to given what you mentioned about isolated loaders

jeroenvandijk13:05:17

Thank you. I might give it a try later

noisesmith13:05:46

but then how do you want that to interact? clojure currently assumes that as you make new definitions, you want all previous loaders avaialble (it chains them as you define new classes via defn etc.)

noisesmith13:05:23

I think that's where things get trickier - do you have two clojures for the different isolations? one clojure that somehow chooses between one isolation and the other?

noisesmith13:05:24

I think a "split timelines" metaphor applies: once you jump into an isolated loader, you can see shared history, but your new definitions won't cross into the parallel timeline(?)

noisesmith13:05:30

to have a single clojure loader, that can choose isolations (jump between timelines, since each loader sees parent loaders / the past), you need to change clojure more fundamentally I think

jeroenvandijk13:05:23

ok so let’s make a difference between libraries 1) ones you load directly and 2) ones that are loaded by your libraries

jeroenvandijk13:05:42

ones that are loaded directly need to be unique, period, no double versions

jeroenvandijk14:05:08

the ones that are load indirectly are loaded in a known context, that of your library. When loading this library all externally loaded could be prefixed by the hash of the artifact name and it’s version

noisesmith14:05:40

that doesn't address the fundamental problem - classloaders work by saying "I know how to look up class x,y,z, and on failure I look to my parent"

jeroenvandijk14:05:52

This way if different libraries load the same dependencies they can still reuse the loading of this, because they end up with the same prefixes

noisesmith14:05:03

this works cleanly if you have a chain from every new loader to parents up to the top

noisesmith14:05:20

OK this is fine for namespaces, I agree that's easy

noisesmith14:05:28

I'm talking about the thing that's hard, classes

jeroenvandijk14:05:44

Yeah i’m not sure how it would work exactly with the classloader. I’m mostly thought about the clojure side

jeroenvandijk14:05:09

So I guess I barely touched the real problem 🙈

jeroenvandijk14:05:42

I also have no idea how expensive a seperate classloader per library would be

noisesmith14:05:00

clojure creates a new classloader each time you call defn

jeroenvandijk14:05:16

ok sounds cheap

jeroenvandijk14:05:58

ok let me read upon classloaders and rethink what i’m suggesting

noisesmith14:05:47

the basic idea is a linked list, current classloader delegates to the one it was created with as a parent, you ask a classloader for a class, and it either finds and returns it, or gets one from the parent

noisesmith14:05:58

I guess technically other behaviors might be possible...

jeroenvandijk14:05:02

One classloader per (nested) library? And within these libraries the fn’s created would use their library classloader I guess

jeroenvandijk14:05:15

Ok i have no idea what i’m talking about. I’ll have to study a bit

noisesmith14:05:57

that's why clojure creates one per defn - so that each time you call defn it can refer to the one defined previously

andy.fingerhut15:05:39

This may be off the topic of what you are looking for, but there is a technique sometimes called "shading" where you load Clojure code into a different namespace than it was originally written to use, so that you can potentially load multiple versions of that Clojure code simultaneously, using namespaces. One library to assist in this is called mranderson: https://github.com/benedekfazekas/mranderson

4
andy.fingerhut15:05:10

I have not used that library myself, but have heard others have used it with good effect. I wrote a hacked-up thing in order to shade some libraries that the Eastwood linter uses internallly.

andy.fingerhut15:05:35

There is no classloader trickery involved here, but that means it only works for Clojure source code, not for arbitrary Java libraries.

Noah Bogart16:05:08

hey all, i'm looking to do some serialization of a data structure that holds closures, aka functions with stored state. in python, i would use pickle, but I'm struggling to find an equivalent. there's serializeable-fn and nippy, but those don't handle "stateful" functions

Noah Bogart16:05:14

well, i found a blogpost saying that nippy does handle this, but their documentation doesn't, so i'll have to do some more research about it

noisesmith16:05:17

@nbtheduke what I've used for this is instead of a closure, a defrecord implementing IFn

noisesmith16:05:50

that means you can call it like a function, but instead of closed over hidden state, it has an explicit immutable hash-map

noisesmith16:05:12

so you use conj or update or assoc etc. to get a new record, with the same methods but new "state"

noisesmith16:05:32

this is easy to serialize - clojure makes sure records act like maps as far as data is concerned

Noah Bogart16:05:04

Huh, interesting. I don’t know that that’ll work for my use case but it’s a clever way of doing things

noisesmith16:05:20

(defrecord F [x]
  clojure.lang.IFn
  (invoke [this arg]
    (* arg x))
  (applyTo [this [arg]]
    (.invoke this arg)))

(ins)scratch=> (def f (->F 2))
#'scratch/f
(ins)scratch=> f
#scratch.F{:x 2}
(ins)scratch=> (f 3)
6
(ins)scratch=> (apply f [3])
6
(ins)scratch=> (def g (update f :x inc))
#'scratch/g
(ins)scratch=> (g 3)
9

noisesmith16:05:40

@nbtheduke if that doesn't work because F needs mutable state, atoms are available

noisesmith16:05:57

that just makes your serializer slightly more complex - it needs a deref

dpsutton16:05:42

i think there was a recent attempt at this? not sure wht the constraints were but someone was doing serializable functions

Noah Bogart18:05:27

hell yeah, thank you

ghadi16:05:43

Serializing functions is a bad idea, and yet there are about 2 projects a year that come out purporting to do it

ghadi16:05:02

serializing the identity of a function is sound, and people do that all the time

noisesmith16:05:06

you can also capture closed over state, but the function still needs to opt in

(defmacro locals
  []
  (into {}
        (map (juxt (comp keyword name)
                   identity))
        (keys &env)))

(ins)scratch=> (def f (let [x 2] (fn ([] (locals)) ([y] (* x y)))))
#'scratch/f
(ins)scratch=> (f 3)
6
(ins)scratch=> (f)
{:x 2}

bfabry16:05:19

some environments make it hard to live without. given redplanetlabs and the word "nippy" I'm guessing this has something to do with a distributed environment like spark or flink or whatever

ghadi16:05:00

who knows, every language that does it lives to regret it

ghadi16:05:13

pickle is a security nightmare same like Java serialization

bfabry16:05:16

it for sure causes problems in those environments but it comes with a lot of benefits too

ghadi16:05:30

we'll see

ghadi16:05:57

you can call requiring-resolve on a symbol, receive a var/function in return

noisesmith16:05:48

I like doing it via record rather than function+macro, because the record comes with its own composition utilities (creating a new of "itself" with updated bindings), which also acts as a deserializer of that state (with all the caveats of data / state, eg. you should be using immutable values to get the most from this)

bfabry16:05:31

yeah I wrote clj-headlights (defunct) which is clojure in a distributed env (apache beam), and I went with capturing vars and a list of serializable params intead. it worked better and was "safer" but it was not as nice to use

noisesmith16:05:00

I think that expecting to take normal functions and serialize in a way that includes all relevant closure data is monkey-patch-complete

noisesmith16:05:17

it is impossible without the ability to monkey-patch and comes with all the problems monkey-patching introduces

noisesmith16:05:59

I think deliberately exposing immutable and effectively closed data is a reasonable compromise

bfabry16:05:05

for sure, but if you're doing something like writing spark jobs you tend to have the possible issues that could come up in the back of your head and so just make sure they don't come up

noisesmith16:05:41

of course, you can implement any language feature by convention ifyou have the intellect and discipline

noisesmith16:05:01

I think I lack both, in the aggregate

bfabry16:05:12

like, I don't generally pass a lambda into a spark dataset function... unless I'm in some tiny static function and I can see everything that's going to go across so I know it's good

Noah Bogart16:05:21

I have a game engine that has to be restarted to update. I’d like to be able to serialize the state of a game so I can update the engine without losing existing games

noisesmith16:05:21

the "simplest" thing there is to "lift" the state out of function closures into an explicit argument passed through the programming logic - that's easy to capture and reuse

4
noisesmith16:05:48

I know putting simplest and lift in quotes looks pretentious but I want to be clear I'm using very specific domain meanings of those terms

noisesmith16:05:37

it's slightly tedious to code that way, but it's harder to make big mistakes

noisesmith16:05:56

if you want to work bottom up you can trade function closures for records over IFn that expose their internal state

noisesmith16:05:24

if you want to work top down you can implement a big hash map with your state, pass it to an updater / program logic, and have that return the new state

Noah Bogart16:05:26

Sadly, the game engine is quite stateful, using an atom hash map, and things like card abilities are represented as closures

noisesmith16:05:21

the atom hash-map can be traded for a regular hash-map if you use the return value of every function that currently acts on it

noisesmith16:05:51

the closures can be made transparent if instead of just closing over locals, they explicitly own them / expose them

noisesmith16:05:47

of course this is harder with bigger projects, it's a rewrite, but it's a bunch of local rewrites in most clojure codebases, rather than one big global change

noisesmith16:05:30

it can be done piece by piece, bottom up or top down as you prefer

Noah Bogart16:05:32

That would be great, but it’s not feasible with the size of the code base lol. For example, https://github.com/mtgred/netrunner/blob/37b7225582f635d44bff54f8be2564414de46b21/src/clj/game/cards/ice.clj#L610 Bloom is a card with a closure. req is a macro that expands to a function definition, so the use of the this variable isn’t easily handled another way

Noah Bogart17:05:04

I’d love to have a more pure game engine, but it is very much not and would require a nearly full rewrite

noisesmith17:05:29

I'm not saying it wouldn't, but I am saying that this rewrite can be incremental - you can make a version of define-card where the effect doesn't call swap!, but rather takes a state and returns a new state - now the caller calls swap!

noisesmith17:05:32

and smaller conversions like this would keep the current behavior, but move closer to purity, and purity is where you get trivial serialzation, so these kinds of changes should each make serialization easier

noisesmith17:05:12

(and I think generally, whether you are doing things functional / data oriented or not, serializing state is easier if you implement it early and shape your design around it)

Noah Bogart17:05:12

yeah, agreed. i wish every day the engine wasn't so stateful, lol

Noah Bogart17:05:17

thanks for the input

noisesmith17:05:35

also you might need state for perf reasons in some cases, but you can still "firewall" and be explicit about what's in scope of that state and how things cross the boundary from the serialized system to the stateful one (you see this in games a lot, certain things about your character are never saved - older games would eg. only let you save from a certain room and if you weren't mid combat etc. so the amount of data to store would be much smaller, and the restoring code could be simpler)

noisesmith17:05:51

this was the approach we used when I worked on a MUD implemented in C (multiplayer networked text based game) - there were explicitly known savable properties, and restrictions that prevented relying on properties outside that set

noisesmith17:05:26

but even that is easier if you start with saved state of a character as an initial feature

Noah Bogart18:05:09

yeah, that all makes sense

lilactown23:05:26

question for people who are familiar with Clojure’s STM / Java threading stuff: I’ve started playing with implementing a sort of MVCC STM framework for (CL)JS, and I’m finding myself drifting from the way that it’s implemented in Clojure notably, `dosync` is eager and synchronous (in the thread it’s executed in), but what I’m finding in JS that’s a significant downside if your transaction is CPU bound at all since in JS you can only achieve concurrency by yielding (using some sort of async timer + callback), I’m finding it better to split up the work in your transaction into discrete units and then schedule them to be run, yielding to the main thread between each unit of work. would this paradigm at all be useful in Clojure as well? is there anything similar out there that I could use to relate my field notes to?

vemv13:05:56

I'm not sure I'd ever invoke future, or async/go inside a dosync. (if that's a valid analogy?) My reasoning boiling down that transactions should be thin (fast), and therefore not dependent on IO (threading can be considered IO) Thick transactions are subject to be retried endlessly

vemv13:05:30

Personally I'm not super sure I'd use dosync for a CPU-heavy transaction either. dosync txs can be retried, so that CPU work would be retried as well Seems slow/problematic

john16:05:19

So, to split work transparently, I think you'd need to give CLJS a go-loop runtime so you can instrument each invocation with an :interrupt signal. But after a specific invocation is fired, I still don't think you'll be able to interrupt the JS that runs inside that context, unless it calls back into a runtime-managed fn at some point and the interrupt signal can be checked again.

john16:05:45

Might be able to do it with sci

john16:05:08

haven't actually looked at the sci code yet

john16:05:49

I think you'd have to store interrupts in SABs for the different contexts to communicate them synchronously :thinking_face:

lilactown16:05:28

I think that's way farther up the abstraction tree than I'm thinking in atm

lilactown16:05:45

I'm perfectly fine with being explicit about yielding

john16:05:26

Maybe I'm misunderstanding your intention

lilactown16:05:51

I doubt people actually want to use STM directly in applications; rather, it is something that can be leveraged by a framework similar to core.async, or an in-memory database, etc.

lilactown16:05:14

I'm building it because I need it for another project that's at that higher level of abstraction

lilactown16:05:12

@U45T93RA6’s point about CPU-heavy transactions is valid, though. a highly contentious, CPU bound transaction is a terrible thing

lilactown16:05:52

I wasn't talking about io, though, but similar to what you're saying @U050PJ2EU about allowing transactions to be interrupted and resumed later. I'm just not super picky about the syntax for that atm

lilactown16:05:27

I guess core.async is a reasonable example for this sort of scheduled, interruptable programming being useful. although without transactional semantics

john16:05:22

well, what do you mean about breaking your work up into chunks?

lilactown16:05:57

I posted some psuedo code in the message below this one

john16:05:12

I saw it

lilactown16:05:32

yeah, so in order to prevent transactions from blocking the main thread, you write your transaction as a sequence of functions that will run in order

lilactown16:05:05

and in between invoking each function, a scheduler runtime can check to see if there's more important work to do and schedule the next function to be run

john16:05:33

how do we achieve STM?

john16:05:46

with that though?

lilactown16:05:06

ah, I'm only talking about the differences between clojure's STM and what's in my brain

lilactown16:05:01

the STM part is that all the code running in the context of the transaction sees their changes throughout the execution - but the outside world does not until the transaction is committed

john16:05:46

but presumably, different callers are dropping calls into that bucket and the transaction takes care of running them in the correct order, or re-running them if necessary?

lilactown16:05:14

the transaction executes each function in the order it's received and makes sure that the functions see the local values of each ref correctly, in the context of that transaction. and re-running them, etc.

lilactown16:05:26

there can be multiple transactions happening at the same time, which is how contention happens

lilactown16:05:58

so you create two transactions:

(def foo (ref 0))

(def txA (transaction))

(def txB (transaction))

(with-tx txA
  (ref-set! foo 9))

(with-tx txB
  (alter foo + 1))

@ref ;; => 0

(doseq [thunk txA]
  (thunk)) ;; in txA, `foo` is 9

(doseq [thunk txA]
  (thunk)) ;; in txB, `foo` is 1

;; sets `foo` to 9 in the global context
(commit txA)

;; drift has occurred; `txB` retries and computes 10 as the new value of `foo` and commits that
(commit txB)

@foo ;; => 10

john16:05:03

Trying to imagine... Perhaps if you also monitored the datastructure inside the bucket... And Bob dispatches an update on :a in {:a 1 :c 2} and Alice is subscribed to :a, then if the results of her last computation depended on :a, the transaction will re-run her dispatch to :a... But, it's single threaded, so Bob and Alice would have already had the latest :a from Alice, before Bob started... So I'm confused

lilactown16:05:07

hmm there's no subscriptions here. just refs

john16:05:31

yeah, that was just way of communicating the idea

john16:05:37

I see what you mean

john16:05:04

Just batching updates to a value so other readers see a consistent view

john16:05:32

Or no... You're yielding to the main thread after each call

lilactown16:05:14

you could potentially yield after each call, or batch them together, however you like

lilactown16:05:03

in the example above I just synchronously call them using doseq. a much more advanced scheduler could e.g. prioritize the calls to each thunk based on some criteria set at the beginning of the transaction

john16:05:25

I guess I'm curious about how it goes about solving a coordination problem between two refs, to the extent that we have coordination problems between atoms in CLJS

lilactown16:05:55

well, all of this is in service of creating a slightly more advanced reactive framework similar to reagent

john16:05:09

And replacing React's batcher/fiber stuff?

lilactown16:05:45

I actually want to directly use Reacts scheduler for it 🙂

john16:05:51

hmm, like vdom, but for atoms 😉

lilactown16:05:55

that's what drove me to needing this ref/STM stuff... if you can't/don't want to compute the entire dataflow graph in one loop, then you need a way of scheduling changes to the graph while keeping data coherent

lilactown16:05:01

yeah kinda! 😄

john16:05:50

I see what you mean.... I'm not sure I'd call it an STM. Because a single JS thread is atomic, it has the same guarantees as an STM, so if you batch fns into a single synchronous context of execution, it would provide the same guarantees, sure.

lilactown17:05:18

yeah, I guess the point of what I’m building is to give atomic guarantees to a transaction that crosses multiple frames

john17:05:49

Seems useful

lilactown23:05:12

here’s some psuedo-code that shows the kind of API I’m finding myself trending towards in CLJS

phronmophobic00:05:19

what are the semantics for conflicts? for clojure’s refs, if there are conflicting transactions, then the transaction will automatically be retried. my intuition is that if the work is split up, then transactions can’t automatically be retried. is that right? having transactions that don’t automatically get retried seems like it might have benefits for some use cases. although I don’t know what they would be off the top of my head.

lilactown00:05:28

transactions can be retried similarly in this model

phronmophobic00:05:22

interesting. does

(doseq [thunk tx]
  (thunk)) ;; execute each unit of work in the tx
get reran when you call (commit tx) ?

phronmophobic00:05:23

the example shows only one ref being modified in the transaction. can transactions modify multiple refs? if so, that’s pretty neat.

lilactown00:05:22

the transaction in the psuedo-code above is mutable, so by executing each unit of work it would move that thunk from "pending" to "done" a conflict would reset each unit of work back to pending and clear any local ref alterations trying to commit a transaction that conflicts could either throw or automatically retry, depending on what behavior you want

lilactown00:05:42

yes, it works with multiple refs. I think it's pretty cool too! 😄

phronmophobic00:05:22

this approach seems to make a lot of sense for an environment with a single thread. I’m not sure there’s much benefit for an environment that already does have support for multiple threads unless your library can help schedule the transaction work to avoid conflicts*.

phronmophobic00:05:49

clojure’s stm relies on optimistic concurrency which works well with low contention (i believe). an alternate approach that schedules transaction work to avoid conflicts might work well for a workload where you do expect a lot of contention.

lilactown00:05:25

yeah this approach still will not perform well with lots of contention AFAICT

lilactown00:05:47

the only benefit is that you can potentially find out about contention sooner... e.g. when running each unit of work, the transaction could check to see if any ref version have drifted since the last one

lilactown00:05:55

and restart the transaction

phronmophobic00:05:24

makes sense. to be fair, there aren’t really any multi-threaded approaches that work well with a lot of contention. thanks amdahl

lilactown00:05:43

but I think you still end up with potentially lots of retries with lots of contention. I'm not sure if scheduling would help... unless you could sort transactions into groups based on contention

👍 4
lilactown00:05:09

but I think there are more useful ways to sort transactions, e.g. priority

lilactown00:05:17

which is ultimately my goal: to have the ability to order transactions by priority and allow a low priority transaction to be paused so that the scheduler can run a higher priority transaction

phronmophobic00:05:01

yea, just trying to think of what the tradeoffs might be compared to a implementation with multiple threads on the jvm. it seems like the main difference is the scheduler (ie. you manually working on parts of the transaction vs the OS scheduler picking which threads to run which determines which transactions get worked on)

phronmophobic00:05:33

working on threads transactions* by priority sounds interesting

lilactown01:05:22

it's pretty key for UIs to prioritize state changes, e.g. keyboard input needs to be handled almost immediately while a websocket can probably wait a bit

phronmophobic19:05:52

@U4YGF4NGM, using refs for ui state seems really intriguing, but afaik, it’s largely unexplored. do you know any good examples or resources for using refs for ui state?

lilactown20:05:00

nothing that I’ve seen explicitly stating it’s using STM/refs, but lots of things start to exhibit similar behavior if you squint

lilactown20:05:03

in clojurescript we often use atoms for storing state

lilactown20:05:34

React is working on a concurrent, prioritizing schduler for rendering that uses immutable data + ref-like semantics, which is what inspired me to see what a more explicit translation of STM would look like that would fit that use case

phronmophobic20:05:33

i’m building desktop user interfaces. I’ve been using atoms to hold all of the app state and that generally works well. it seems like there might be some performance and architectural advantages to using refs, but most desktop design is based on old OO principles and the functional designs in the browser are under different constraints.

lilactown20:05:27

yeah, I think that using an atom leads one to eventually storing all state in a single atom to coordinate updates

lilactown20:05:18

the idea behind using refs is that it allows you to store state in separate containers, and ensure that they are updated in a coordinated fashion to avoid flickering / needless re-renders

lilactown20:05:48

another way of solving this is batching at the renderer, but you also have to solve the same problem anywhere you are doing side effects based on state changes

phronmophobic21:05:26

the tactic I’ve been using is to have to wrap expensive rendering calculations. basically, if the expensive calcuation is ready, it shows the result, otherwise, you can choose to show something else (eg. a partial result, the most recent finished result with a spinner, an empty view with a spinner)

phronmophobic21:05:00

the main reason I’m interested in refs are potential performance benefits and more flexibility when modeling your UI state