There's Apache Ignite I believe you could checkout. It defaults to TCP/IP multicast for node discovery, or you can just statically tell it the other IPs and ports of the nodes, both are relatively easy to setup. You can also scale it to thousands+ nodes by switching to ZooKeeper discovery, but that's more annoying to setup. It's brokerless, nodes talk to each other directly. Not sure about a nice Clojure wrapper, but I tend to use interior directly with stuff like this, and find it just as easy most of the time as using a Clojure wrapper, or I guess I tend to wrap the parts I use myself haha. There's also Hazelcast, but not sure if it's free.

For something like this I might try just dropping both machines on to a https://tailscale.com/ network and use a CRDT to distribute the data instead of reaching for a message queue or something with a more traditional client/server setup https://github.com/smothers/cause https://github.com/aredington/schism

You might even get away with just using something like unison if you prefer to rely more on the filesystem https://www.cis.upenn.edu/%7Ebcpierce/unison/

I actually don't have super good grasp of the requirements. I expect less than 1000 msg per second per machine and msg size less than 200 bytes.

It's for distributed crawling. So the messages are essentially urls to be crawled by a different worker.

If you're in a cloud environment this might be a good use case for a queue service like AWS SQS. Your workers just read messages off the queue and delete them from the queue after successfully processing the task

Certainly easier if you can avoid having your processes coordinating with each other and instead pull tasks from a shared queue

sequence returns a lazy-seq of the application of the transducer to the collection. However, it realises the first element on initialisation, which a normal lazy-seq doesn't do.

if you (count xs), for example, it'll realise the whole thing

Yes, sorry, imprecise question — I'm wondering specifically about realizing the first element on initialization.

That is, why sequence does that when normal lazy seqs don't.

I think that's to do with the implementation of transducers ... I think it was explained in one of Rich's transducer talks, but I forget the actual reason right now

the whole point of it is to lazily compute

but sequence with a transducer is NOT a lazy seq

it is incrementally computed but not lazy in the intermediate steps like other lazy seqs

> When a transducer is supplied, returns a lazy sequence of applications of the transform to the items in coll(s) from the docstring of sequence.

Hmm. Does that mean the docstring for sequence is wrong?

(Jinx.)

it means there's too much to explain in a docstring :)

Haven’t dug into the RT bits but i suspect that the reason you are seeing the first element “computed” is the logic (or (seq x) ()) somewhere in the code.

or some analog to that.

([xform coll]
 (or (clojure.lang.RT/chunkIteratorSeq
      (clojure.lang.TransformerIterator/create xform (clojure.lang.RT/iter coll)))
     ()))

https://youtu.be/4KqUvG8HPYo?t=2165 ... I think this is the bit where Rich explained it

@U11BV7MTK Right, that makes sense, thanks. :thumbsup:

@U0P0TMEFJ thanks, will take a look.

(I'm pretty sure I've watched that talk, but I must've forgotten much of it.)

yeah ... me too ... I find the details leaking out of my brain these days ... I'm clearly getting old 😉

But that bit does explain it nicely. :thumbsup:

I always forget the difference, every time lol. If I remember, instead of having each chunk go through each step one by one. It has each element of the chunk go through all steps. So lazy-seq goes: do step1 for next-chunk do step2 for next-chunk And sequence goes: for each element in next chunk: do step1 for element do step2 for element

the easiest example to think about is something like mapcat range - lazy seq will just compute elements as needed where as each intermediate range will be fully computed for the transducer sequence

and in the extreme case, an infinite intermediate sequence is fine for a lazy seq, and very not fine for transducers :)

Ya, I think that's the case that makes more sense. Because the lazy-seq will force realize each chunk as well. So the real difference in behavior from a user point of the view is short-circuit no?

it's not short circuit as much as actually lazy vs not lazy

in practice, infinite intermediate seqs is pretty unusual :)

Ya, and I think that concept confuses me too much haha. I tried with a (range) as intermediate, and than a take 10 at the end, and transducer didn't go in an infinite loop. And then I thought, well that make sense, since each element go through the whole chain one by one. So I'm not sure how to construct an infinite intermediate that transducer chokes on.

what did you try?

(->> [1 2 3]
  #_=>      (map inc)
  #_=>      (mapcat #(do (println %) (range)))
  #_=>      (map inc)
  #_=>      (take 10))

(into []
  #_=>  (comp
  #_=>      (map inc)
  #_=>      (mapcat #(do (println %) (range)))
  #_=>      (map inc)
  #_=>      (take 10))
  #_=>       [1 2 3])

so that doesn't use sequence, which is what we're talking about

(take 10 (sequence (comp (map inc) (mapcat #(do (println %) (range))) (map inc)) [1 2 3]))

will spin

Oh... ok for some reason I thought sequence would still be lazy chunk by chunk, except each chunk is realized through the transducer pipeline. You're right though, that spins.

From the discussion I understand that sequence realizes at least one element (chunk) on initialization But I was still surprised that this realized first 64 elements (2 chunks):

;; this prints 0 .. 31
(def xs-seq (sequence (map inc) (map prn-inc (range 128))))

;; this prints  32 .. 63
(first  xs-seq)

I think that's different, try

(def xs
 (sequence
  (comp
   (map println)
   (map inc)) 
  (map
   #(do
     (println ":: " %)
     %)
   (range 128))))

What happens is that sequence will pull the first chunk from the lazy-seq. Then when you grab elements from the sequence, it will run the transducer on the full chunk, and since it ran over that chunk, it will grab the next chunk from the lazy-seq

So you'll see (first xs) will print the first 32 from the lazy-seq, then it'll print the first 32 from the transducer, and finally the next chunk, 32-64 from the lazy-seq will print, but they won't go through the transducer.

I think that's creating an anonymous child class no? So maybe with proxy?

(-> (proxy [TypeToken] []) .getType)

That is the goal of my library: https://github.com/RutledgePaulV/kube-api . The core module is usable at this point but still work to be done in the others and I haven't touched it in a while

This readme is the best example: https://github.com/RutledgePaulV/kube-api/tree/master/kube-api-core

for reference google's android publisher api description is about 4k lines of json (but it isn't a completely valid spec)

Using your build tool, build and deploy to some repository

Thanks, but can I do it locally?

Both lein and clojure CLI support local install

Have such code in separate project

(ns trivial-library-example.core
  (:gen-class))

(defn add-doc [db e doc] [[:db/add e :db/doc doc]])

run in its wd lein do pom, jar, install , get /home/ivana/pet-projects/Clojure/trivial-library-example/target/trivial-library-example-0.1.0-SNAPSHOT.jar

In main project Dockerfile added ENV DATOMIC_EXT_CLASSPATH=/home/ivana/pet-projects/Clojure/trivial-library-example/target/trivial-library-example-0.1.0-SNAPSHOT.jar

rebuild container, run @(*d/transact* conn [[trivial-library-example.core/add-doc 17592186045423 "this is foo's doc"]]) - get Syntax error (ClassNotFoundException)

What do I wrong?

Is my .jar file correct? How can I test it without datomic&docker interaction? Just for localising the problem. Or I should fight for absence SNAPSHOT in its name or smth else?

Just checked it via :resource-paths in project.clj - it works, at least in repl. Then seems that .jar is right, will continue to fight with docker/datomic

@U0A6H3MFT what does the separate Jar get you?

@U3JH98J4R ability of using my own util & helper functions inside datomic transactions, not only stdlib and core

Oh my, seems that it works!... Just trying not to forget all the steps how to reproduce it 🙂

Finally found usable way of using datomic classpath functions as standalone tx-creators or database transactor function helpers. That was all I wish 😊

@U0A6H3MFT Would you consider sharing this mini library?

@U01KQ9EGU79, looks like it's not a case for library in a sense of code, actually it's a step-by-step process for involving your custom clojure code into datomic transactor and all the ways of using it - directly or inside db-functions. Maybe I'm not smart enough, but I coudn't do it reading docs only, and had to spent several hours on experiments and googling, but finally I got it.

Either way, the knowledge is valuable. I’d love to see this shared online.

Maybe it's a good case for blogpost or short youtube stream. I'l think how to share it, but if not only you are interested in it, I'd like to see the all of participants - maybe I'l organize a small stream with online code & screen sharing and realtime feedback with all the participants. What do you think about it? Or this theme is so special and not actual for other guys?