Fork me on GitHub
#meander
<
2021-03-08
>
jjttjj14:03:02

Hi I'm trying to implement an advanced "pub/sub" for core.async. I have an incoming stream of data. I'd like to be able to subscribe to "queries". Basically, for each message that comes in, check which of the registered subscriptions matches. But I'd like it to be done efficiently, so if multiple subscriptions share a match condition, this would only be applied once to the incoming message. I have an almost working example here: https://gist.github.com/jjttjj/ad9c6355d8b6741b43f23e90c7c77442 But I'm wondering if I could use meander here, though I haven't fully wrapped my head around how that would look.

(sub pub
  {:venue ?venue
   :type (me/or :done :open)
   :price (me/pred #(< % 100000) ?price)}
  ch)
if I have something like this, is there a way to compile that data such that i just get a set of all bindings and match conditions in a way that they can be appended to a larger set of match conditions? And then apply the whole set of conditions to some incoming data, and then get back a set of matching conditions that could be compared for equality to the conditions in the "subscription" match query?

jjttjj16:03:24

After learning a bit more meander, I think a more direct question might just be, is it possible to m/search based from a map of pattern->clauses?

(def pattern->clause (atom {}))
(swap! pattern->clause assoc '[?x 1] :x)
(swap! pattern->clause assoc '{:a ?a} '?a)
Then do something like:
(let [x (mapcat identity @pattern->id)]
  (eval `(me/search
           {:a 1}
           ~@x)))
which currently doesn't seem to be working edit: ok this does work sometimes, maybe not with dynamic clauses?

Jimmy Miller17:03:10

So I didn’t get to process all of this yet. For dynamic things we just started exposing an interpreter. But It is definitely not something we have focused on for performance. And it is no where near as battle tested as the non-interpreted version.

Jimmy Miller17:03:25

In fact, I just pushed up a fix for pred that someone discovered. If you are looking for efficiency as your primary attribute, I’m not sure it if is the right answer for you. But also, if you are looking for efficiency, you might want to think about how dynamic you want to be.

jjttjj17:03:59

Thanks! I'll look into the interpreter stuff a bit. And I still have a bit more thinking about my problem to do, and learning how meander works.

borkdude14:03:13

@noprompt @jimmy Could any of you explain why you need an iterator function in meander, here: https://ask.clojure.org/index.php/10303/interop-clojure-pattern-clojure-consider-adding-iter-clojure?show=10305#a10305 I'm trying to convince that clojure.core should expose an iter function like CLJS does.

noprompt16:03:01

Its used for memory variables. Two things: 1. I don’t like this 🙂 2. I think Clojure should expose iter as CLJS does. 🙂

borkdude16:03:25

Can you explain in the issue to Alex Miller instead of me? He is the one that needs this data

Jimmy Miller17:03:03

If @noprompt can’t find the time, I will try to explain. Though I will have to learn why myself 🙂. Any advice on the angle to explain it? I can’t imagine the core team will found the meander case very compelling. I’d imagine they would just suggest not doing it the way we do it.

borkdude17:03:51

My angle is: not have to include clojure.lang.RT in babashka but still enable people to use meander in babashka

noprompt17:03:57

> I can’t imagine the core team will found the meander case very compelling. I’d imagine they would just suggest not doing it the way we do it. I agree and, on top of that, I don’t even want to be using iterator this way.

borkdude17:03:27

In meander this got fixed, but now I came across the same issue with pathom(3)

noprompt17:03:49

There is a larger problem though in that Clojure and ClojureScript have no core library “standard” of any kind between the two of them.

noprompt17:03:02

“Not very clojure-ry” is a really poor argument.

noprompt17:03:09

> They are stateful and generally not concurrency friendly. However, neither are arrays and yet make-array, etc.

borkdude17:03:43

The same argument was given for index-of: this will force a linear scan. Yes, so will filter ...

noprompt17:03:11

> It’s not in core to promote better suited data types This is also a really shitty argument.

borkdude17:03:10

Maybe I should just release a babashka.utils lib that has all of this and you can also run it on the jvm ;)

👍 6
noprompt17:03:55

It, in fact, doesn’t address the actual argument which is “I have a vector and I don’t want to call .indexOf; I would like cross platform support for this perfectly useful thing that exists that has these measurable performance characteristics.”

noprompt17:03:50

Theres a history with CLJS being treated more as a dialect of CLJ rather than a CLJ for JS which has caused many of these problems. I think what has become clear over time is that host specific functions probably should be placed in their own namespace if there is ever going to be a day where core is platform agonistic.

noprompt17:03:03

I think the argument for iter is that it is practical when you need to do interop which requires an iterable instance, is probably one of the best.

noprompt01:03:32

@U04V15CAJ I replied about iter as diplomatically as possible. Mostly, I would like Alex to be clear for the sake of conversation. If he’s going to say it’s not a good idea to add iter but that the cases where iter has been implemented independently are “perfectly fine”, then I think he should also explain to the rest of the audience what criteria iter would need to meet such that it should be a core function.