This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2019-07-02
Channels
- # announcements (1)
- # beginners (119)
- # calva (2)
- # cider (40)
- # cljsrn (14)
- # clojure (145)
- # clojure-dev (122)
- # clojure-europe (4)
- # clojure-italy (9)
- # clojure-nl (5)
- # clojure-spec (2)
- # clojure-uk (32)
- # clojuredesign-podcast (1)
- # clojurescript (10)
- # cursive (44)
- # data-science (1)
- # datomic (53)
- # defnpodcast (6)
- # emacs (6)
- # fulcro (13)
- # garden (25)
- # graalvm (3)
- # graphql (7)
- # jobs (3)
- # liberator (4)
- # nrepl (1)
- # off-topic (21)
- # quil (27)
- # reagent (4)
- # reitit (2)
- # remote-jobs (1)
- # ring-swagger (3)
- # shadow-cljs (3)
- # spacemacs (24)
- # sql (29)
- # tools-deps (68)
- # xtdb (2)
Curious whether gfredricks’s idea might work. For literal check for fn
, for var use metadata.
Hard to see what having to tag vars improves over the sam macro above, in both cases you have to specify the Sam type.
don't have to specify the method in the var tag case
which might be a bigger deal than just less typing -- the interface name might be a lot more memorable than the method name
has there been any work done on making clojure objects (e.g. functions, anything reified) efficiently serializable between processes which are running the same code?
seems like this would need deep compiler integration in order to correctly account for closures
like create an object in one process and send it to another process on another machine
that's running the same code
not very familiar with that
specifically, I need to be able to take a function instance and send it to another process
or an instance of an object defined via reify that has some random closure
that seems too inefficient, and how would you send the closure?
yes, my case would be distributed processing
anyway, just wondering if there was some branch of clojure somewhere tackling this or any related work
1) have to setup the same context around it (maybe not trivial), 2) I’ve hit issue with max method size errors trying to eval
across as well - depending on what you need to send.
I remember doing some extensions and improvements over this https://github.com/technomancy/serializable-fn in a private repo in the past
but it was based around just eval
’ing forms later, but could get a bit more sophisticated with it.
(also, I’ve only did work on any of this for function instances, not things liek reify
)
i'll take a look at portkey
it is quite slow to eval
a bunch of times. you want to “batch eval
” but in a way that can not break method size limits
the efficient way to do this would be to serialize an id for the class and the values in its closure
It was super slow to eval
tons of “fn’s back again”, so we made a big data structure to hold them all in to do a “batch eval
and then relink them to their appropriate places after”
Sometimes, the customer really wants a machete, and all you can do is point out the sharp edges and hope they don't return it later with a bandage wrapped around their hand.
> Sometimes, the customer really wants a machete, and all you can do is point out the sharp edges and hope they don’t return it later with a bandage wrapped around their hand. eh, I don’t really consider this constructive here
There are several clj libs out there doing things around this. Ones built around Hadoop stuff/Spark etc
But yeah, I’ve specifically targeted fn’s before, not just “any object”. I know there will be limitations
and I’m not even sure I think there is a “good solution” for clj alone. Basically I’ve always came back to the eval
approach, but then have to do things like batching forms together etc
I was actually working at Terracotta at this time, helping Paul, before I was a Clojure user :)
that might have been my first intro to Clojure actually
which is an interesting question, if you want to send closures around, why not just turn all your machines in to a single shared memory space
even with the very general problem statement expressed, I still think you shouldn't pass around closures
if you want to send code, have a control plane be in charge of worker machines or classloaders from above
roll the code forwards or what not, but limit what's passed in between workers to be data
unless that's impossible for the SLA, in which case send the code to the data like Datomic Ions do
working on sending function instances is not something we're likely to do in Clojure proper anytime soon
working on being able to send rehydratable var refs is something we're working on
vars are actually serializable now (since 1.9? 1.10? can't remember) and when deserialized, they will re-resolve in the target
what we're looking at is making the reader support that too
all seems reasonable. I just chimed in based on some past experiences anyways. fortunately for me, I haven’t been fighting with this situation in recent times.
the other place this pops up (thorny serialization issues) is image based development
the var serialization stuff is still a reference to the var, not the actual contents
@nathanmarz I've been working on a similar thing in CLJS [tau.alpha](https://github.com/johnmn3/tau.alpha/tree/master/src/tau/alpha) that I've recently been working on porting to nodejs.
And I've been re-working the fn serialization stuff and it works pretty much like shipping around byte code
And in a web worker env or a tightly controlled cluster, I think it makes most sense to use a fully connected mesh so they appear to have a single shared memory space
the more recent version for node is aware of locals and grabs them too, if necessary, so the code looks a little more traditional. In the code above, that version uses a more explicit binding conveyance mechanism for the on
macro. Anyway, ping me if you have any questions about it. I'll hopefully have a rough nodejs version out soon
@john cool, my case though is specifically that the code itself is shared between processes
so no need to send byetcode, since it's already there
just need to send an id for the class and whatever the fields are, which would be the closure
since you're not dynamically generating code, you don't need to do any of this: which ever class that generates the closures should be the target
(aside: this is no different semantically than passing around RPC-style maps with a target "op" key)
@nathanmarz that's sort of how I'm doing it though. The reason what I'm doing mostly works is because the same compiled code is on both sides
You can just use tagged literals for the IDs and hydrate them with the read function on the other side
@ghadi how do you ask instance of a function to generate its closure?
you mean with reflection?
do you mean to write or generate the functions that need to be serializable in a special way?
not sure what you mean by that example
the code I need to be able to write would be like this:
(defn foo [a] (fn [] a))
(def f (foo 1))
(defn bar [a] (reify SomeInterface (someMethod [this] a)))
(def r (bar 2))
...
(send-to-other-process (serialize f))
(send-to-other-process (serialize r))
I get the impression that you're focused on mechanism and might need to step back into the problem space
yes, that's been established
the question is how do I take an instance of an arbitrary function and send that data
the data would be the class and the fields that comprise that instance (the closure)
or an instance of some object made with reify
so how would you imagine my code sample working then?
foo
has to annotate its return with metadata as to what is in its closure?
If foo and bar can be custom types, you can customize the print writer so that the other side knows to call foo and bar constructors on them on the other side. Assuming you control foo and bar.
no, that's not the case
this needs to work for any function instance or reify instance
it sounds like the answer to my question is there has been no experimental work on the clojure compiler for this