Fork me on GitHub
#clojure
<
2020-12-28
>
GGfpc13:12:27

Has anyone here used Langhor for RabbitMQ? If so, how can I have multiple threads consuming in parallel? I have tried the :executor option with a fixed thread pool but that results in connection refused for all but one consumer. Supposedly you can just create multiple consumers and they are executed in a threadpool by default, but my messages seem to be processed in a single thread. 😕

(defn init
  [f]
  (if (false? @started)
    (let [conn (rabbit/connect {:uri      (System/getenv "RABBITMQ_URL")})
          chnl (channel/open conn)
          queue-name "goodstats.queue"]
      (timbre/info "Starting AMQP subscriber at: " (System/getenv "RABBITMQ_URL"))
      (queue/declare chnl queue-name {:exclusive false :auto-delete false})
      (consumers/subscribe chnl queue-name f {:auto-ack false})
      (consumers/subscribe chnl queue-name f {:auto-ack false})
      (consumers/subscribe chnl queue-name f {:auto-ack false})
      (consumers/subscribe chnl queue-name f {:auto-ack false})
      (reset! started true))))

emilaasa13:12:39

Do you want each consumer in a different thread?

GGfpc13:12:15

They can share a thread pool. Really what I want is for each consumer to be able to process one message at a time, but N consumer to process N threads at once

emilaasa13:12:27

When using Java I've used one channel per thread, I think I read it in the java api docs.

emilaasa13:12:11

I saw that you could provide something like a threadfactory to langhor which might do something, but I wouldn't count on it handling concurrency (well) by itself

lukasz15:12:15

Apologies for a shameless plug - but it's coming from a good place 😉 My team's RabbitMQ client comes with multi-threading (and adjustable prefetch) support out of the box - and it's based on the Java client directly, rather than use Langhor (which we started with years ago). One caveat is that it has a hard-ish dependency on Component, but you can use it without it with relatively small effort: https://github.com/nomnom-insights/nomnom.bunnicula

emilaasa20:12:23

Good on you for open-sourcing it! Looks good docs wise too. I'll give it a shot! 😃

emilaasa20:12:40

My org quite recently moved towards using rabbitmq to "just do pub-sub" but it seems amqp + rabbit has quite a bit of complexity to handle, so having libraries with opinions is great.

lukasz21:12:08

Yeah, the amount of configuration options is quite overwhelming on the client and server side. We have basically set everything to be as safe as possible, while trading off throughput a bit (although we push 10s (or 100s now) of thousands of messages every hour and it's been fine. One thing to watch out for - there's v3 coming which will not be backwards compatible

Norman Ziebal15:12:46

Would you prefer Clojure or ClojureScript for FaaS? Personally I am concerned about the start up times for JVM.

valtteri16:12:00

Depends on use-case. If you’re expecting fast response to infrequent HTTP-requests then I’d probably pick CLJS. However I think the startup time is one of the smallest problems to solve when entering FaaS territory. 🙂

valtteri16:12:16

If you’re worried, you can write most of your code in cljc and easily swap between runtimes. 🙂

3
Norman Ziebal17:12:17

@U6N4HSMFW could you elaborate on the other problems when using clojure with FaaS, I would be curious to know.

valtteri19:12:04

@U01HN2UCH9Q oh, I didn’t mean that Clojure would be a problem with FaaS. Actually I think the language itself is a great fit for FaaS. What I wanted to say is that in my experience people are too worried about startup penalty, which I perceive as a minor (technical) thing that can be worked around. I think there are more fundamental problems to be worried about, such as splitting the problem domain into distinct deployable artefacts and managing dependencies between them.

👍 6
souenzzo17:12:51

About cognitect.anomalies: (not sure if it's the right channel) There is a field :cognitect.anomalies/message When I'm using anomalies in a ex-info, should I use the same value for ex-info msg and for :cognitect.anomalies/message? Something like (let [msg ...] (throw (ex-info msg {:cognitect.anomalies/message msg ....}))) ? Or they have different meanings? EDIT: I just see that :cognitect.anomalies/message is :opt, so may I skip it in ex-info?

didibus06:12:53

I don't think you are supposed to put anomalies inside an ex-info

didibus06:12:14

You're supposed to return them instead. so you would not throw an anomaly, just return one

ghadi17:12:47

I use anomalies as an alternative to exceptions

souenzzo17:12:04

Do you return it? So, you don't use "let it crash"?

souenzzo17:12:03

There is any good oss example of usage?

ghadi18:12:52

I make my functions return data->data. The caller still needs to check for success

ghadi18:12:02

Or let it crash if necessary

ghadi18:12:48

Or whatever policy you want

souenzzo18:12:15

right. so caller may write it own msg to ex-info with a possibly generic error, and :cognitect.anomalies/message will came from "the cause"

ghadi18:12:51

What’s your goal?

p-himik21:12:57

@U050ECB92 "The caller needs to check for success" sounds a lot like C return codes. I haven't really dealt with such an approach in Clojure. Does it not get overwhelming, to have to check the result in every single function that can get an "anomaly" from one of its callees?

👀 3
souenzzo13:12:42

result, _ := op();

dominicm09:12:43

Fwiw, datomic api uses anomalies and can be used for inspiration

souenzzo12:12:06

I know, i catch some exceptions from there and I can dig into it. But as a non-oss, is harder to see

frodeaux23:12:15

Does anybody have a recommendation or pointers for a tutorial/into on cljfx? I've got the basic example at https://github.com/cljfx/cljfx running, but I'm not sure how to move past this. I'm relatively new with only moderate Clojure + Leiningen experience.

phronmophobic23:12:03

you can try the #cljfx channel. they're pretty helpful there.