Clojurians Log v2

Just to avoid an XY problem - what exactly do you want to achieve by doing that?

I'm looking to be able to serialize classes and send them over the wire for a replicated run-time environment to build a distributed repl for Clojure. You know, just a casual Monday over here 😉

I want to see if I can avoid having to package and distribute JARs of the runtime environment and instead be able to replicate from one environment to another (ie. I want to be in my editor and be able to run (.submit my-distributed-scheduler (fn [] ...)) and have it "just work"

just brainstorming here: would it be easier to send every clojure form except .submit forms and similar to a remote repl?

I'm pretty sure you can't do it, at least not in general, due to any sort of local state that "threads through" different compilation units. For example, gensym. You can have some #() that gets compiled to a fn__2 class or whatever, and remotely you also have #() that gets compiled by a different process, with its own gensym state, also to fn__2. Different bytecode, same class name.

I make my suggestion because clojure is already doing the work (via vars and namespaces) to make things replicable

@p-himik Yes, I suspect that once I have the bytecode I would have to do an additional pass to make sure subsequent gensyms end up with the same class name (or perhaps I compile it into a path and see all classes generated)

what @p-himik is saying is two gensyms on two vms with the same name are totally different objects

gensym uses a predictable scheme, so these clashes will happen constantly

I'm looking to build on the ideas in https://blog.redplanetlabs.com/2020/01/06/serializing-and-deserializing-clojure-fns-with-nippy/ - and I have been able to do it across processes if both processes are evaluating across disk (ie. my editor process loads a server namespace, and I invoke a -main from a clj -m client namespace - I was able to serialize a function from common (which both server and client require)

(again, this is at the boundaries of both my Clojure and JVM knowledge so apologies for ignorance here)

Regarding the clashes, and not sure if it helps, but the client process would only ever be invoking anything with instruction from the server. ie. if it helps, we know the server would be serializing everything and sending it to the clients and that the clients would never evaluate anything that didn't go through the server first)

@noisesmith Regarding your idea of sending all of the forms to child distributed processes - would that result in consistent naming on gensyms? ie. if I eval the same forms in the same namespace do I get the same classes generated (such that a function instance could be sent through the wire)?

if lookups go through namespaces and vars the gensyms don't matter, those are local form specific and the clojure compiler keeps them sorted

Right, I suspected it was something to that effect since the common namespace worked fine (which had a var). But so much of Clojure's power is in the repl and its dynamicism so being able to get all of that distributable is the goal.

vars are what make the dynamism work, sending the definitions for namespaces and vars is the simpler option here IMHO

anything else will reach the irreducible problem of dynamism and either use or replicate vars

you already need all of clojure's core and RT on the other side for the compiled code to work, so just use it on the other side

While I can appreciate that, that somewhat forces everything be a defined function right? ie. if I have a notebook namespace I sort of have to defn everything so that we get some sort of var to resolve things to, right? I'm trying to get something working one layer higher of prototyping. ie. These are ML pipelines that are very experimental and so a lot of the work is just map (fn some-anon-fn [...]) while we are experimenting on things. Having to move everything into definitions gets rid of a lot of the plasticity of Clojure

send the form then? if it's being sent to the same ns in the same order it will just work

My idea, and it's maybe half-baked, is to have a macro that intercepts map to run it in a distributed fashion... Eg.

(->> (range 1000000)
     (map inc)
     (reduce +)

;; To run on a cluster

(distributed
  (->> (range 1000000)
       (map inc)
       (reduce +))

what about: a function with the call signature of map, but a new namespace and name, that uses a namespace level boolean to decide whether it executes or not. turn the boolean on on a remote machine

what this saves is the complexity of pruning the world of your bytecode, and reassembling the needed pieces

perhaps a macro that becomes a no-op on one host, and normal code on another

Yeah I sort of imagined the macro hijacking map and replacing it with my-lib.distributed/map anyway - so perhaps there's just something there

hijacking map provides superficial benefit with huge risks of bad things happening, which is why I suggested some other thing that acts like map

On the host it becomes something that enumerates those items and fans it out to the RPC calls

that's still more complex than what you need

the easier thing is sending all the forms to all the peers

(range 1000000) might be a bad example

The point is that the source only exists on the leader

ie. I can't evaluate the whole sexp, I need to transmit data from the source to the children

> I can't evaluate the whole sexp you can, and it's less work than avoiding doing so is

sending the source is not a harder problem than sending clojure.RT, and all the compiled bytecode for clojure.core etc.

It's just that the whole sexp might not be evaluatable on the other machine

(->> (csv/read "file_that_only_exists_on_my_machine.csv") ;; happens locally
       (distributed-map (comp parse-int #(nth % 1)) ;; happens on many machines
      (reduce +) ;; happens locally)

That's what I want to be able to work

the only languages I know of where this is a "beginner" task are erlang and julia :D best of luck

Haha #beginners just has the most people to ask questions to. If there's a better channel I'm all for it

with two macros local and remote and some helpers you could get something like your example above, with explicit marking of what happens where

(->> (local (csv/read "file_that_only_exists_on_my_machine.csv"))
     (remote->> (map (comp parse-int #(nth % 1))))
     (local->> (reduce +))

local / remote / local->> / remote->> would either evaluate, or fan out or fan in contextually

Erlang is exactly what I'm looking to bring to Clojure though - so your analogy is spot on the money. Distributed runtime. I don't expect it to be a small undertaking, but something that would be a great addition to the ecosystem if it existed

Yeah that's a very cool idea.

or maybe, local, fan-out, fan-in

and since you'd want to generalize, each would take a unique host identifier as an arg

I think local might just be the default. dmap and dreduce (distributed), where dreduce could have a monoidal reducer for reducing across nodes and instances

the point of (local host-id forms...) is if you are not host-id you generate code to receive the data, rather than doing the things in the forms

Yeah, per host evaluation (via node IDs) and partitioning probably

I see, that's a good thought

Alright it gives me a direction to play with - thanks for the ideas!

my big criteria with my suggestions here (many subtly or not so subtly wrong in the particulars): localize changes to a place that belongs to your project, reuse the abstraction tools of the language as much as possible

Definitely good advice - and if I can get the ergonomics that I want (or close enough to them) with it, then its absolutely the right approach. The question will be how much will the abstraction leak. "What color is your function" is one of my favorite papers - if possible I want to avoid coloring my functions for the sake of making things distributable

I think it makes sense to "color" code that executes remotely or concurrently in general, because it intrinsically breaks in ways other code doesn't

everything that tries to hide that color difference creates new color differences

similar to how clojure colors mutation - it is segregated and doesn't work cleanly with the core functions, because it violates the assumptions that make them effective

I'm not sure I agree. I think loom is a great example of what you can achieve by having a runtime handle what previously required treating concurrent code "differently". Yes, I can have exceptions, that might be different, but map and reduce are the primitives of distributed computation and functional programming alike