proletarian 2023-08-18 | Slack Archive

lwhorton16:08:01

hey, this is an awesome tool and i love how it “just works”. kudos to the creators/maintainers. data-driven and simple, it feels as clojure-ey as you could ask of any lib.

lwhorton16:08:42

i ran into something the other day that confused me, a little, but i haven’t yet found the time to dig into the issue. for now i found a different way to invoke the library that avoids this issue… but i’m going to leave this as a note for anyone else in the future who might get equally confused: the normal worker creation looks something like this:

(let [ds (jdbc/get-datasource db/pgres)]
    (worker/create-queue-worker ds handle-job! #:proletarian{:log logger
                                                             :worker-threads 1
                                                             :retry-strategy-fn retry-job!
                                                             :failed-job-fn failed-job!
                                                             :install-jvm-shutdown-hook? true}))

i have structured my codebase so that individual modules that need to generate a worker can provide their own retry-job and failed-job functions. the wrong way to do it (i think), is to provide function refs from the module.

... ns my.app.has-work-to-do
(my.app.workers/create-worker my.app.has-work-to-do/retry-job! my-app.has-work-to-do/failed-job!)

... ns my.app.has-more-work-to-do
(my.app.workers/create-worker my.app.has-more-work-to-do/retry-job! my-app.has-more-work-to-do/failed-job!)

... ns my.app.workers

(defn create-worker [retry-job! failed-job!]
  (let [ds (jdbc/get-datasource db/pgres)]
    (worker/create-queue-worker ds handle-job! #:proletarian{:log logger
                                                             :worker-threads 1
                                                             :retry-strategy-fn retry-job!
                                                             :failed-job-fn failed-job!
                                                             :install-jvm-shutdown-hook? true})))

what i noticed when running workers is that failed jobs in my.app.has-work-to-do could actually invoke the my.app.has-more-work-to-do/failed-job handler. after a few hours of exploration i couldn’t really figure out why. it might be a conceptual misunderstanding of the documentation + library’s intent. this could also potentially be due to some bad state in my system layout (i am using mount, for example). but… you can sidestep this issue entirely via multimethods for the retry-job and failed-job handlers.

... ns my.app.workers
(defmulti retry-job! (fn [job-type _payload] job-type))

... ns my.app.has-work-to-do
(defmethod retry-job! ::has-work-to-do [type payload] ...)

... ns my.app.has-more-work-to-do
(defmethod retry-job! ::has-more-work-to-do [type payload] ...)

this seems entirely more sensible considering multimethods are also the recommended way to dispatch on the workers’ handle-job!.

2023-08-18

Channels