missionary

braai engineer 2024-05-16T10:40:28.031559Z

Has anyone tried to build an async task runner / workflow system with dependent tasks using Missionary? Like Airflow or Celery.

adamfrey 2024-05-17T12:02:13.028509Z

my apologies, it's public now.

Eric Dvorsak 2024-05-16T11:33:22.953109Z

I'll need (to buils) one soon

Ben Sless 2024-05-16T14:27:40.918799Z

I've been thinking about something similar from another angle, but this also relates to distribution of work and state persistence. If we can distribute work between client and server, how can we distribute work among servers in a similar manner? I want (via remote ...)

Eric Dvorsak 2024-05-16T15:21:46.730579Z

what I've done in the past (without missionary) is have the servers pick up tasks when they have capacity. For that I have a job table in postgres, and use a skip lock in a transaction to process the task:

(postgres/execute-one! postgres
                               {:delete-from :scheduled-tasks
                                :where [:=
                                        :id
                                        {:select :id
                                         :from [:scheduled-tasks]
                                         :order-by [:execute-at]
                                         :where [:<= :execute-at now]
                                         :for [:update :skip-locked]
                                         :limit 1}]
                                :returning [:*]})
if the server fails to execute the task (rolling back or timing out the transaction) then the task is picked by another

nivekuil 2024-05-16T19:52:45.763229Z

I would just use https://github.com/manetu/temporal-clojure-sdk

adamfrey 2024-05-17T00:47:11.423439Z

I started working on something like that. I didn't quite get it to a usable state for lack of time or focus, but you can look at it if you are interested https://github.com/AdamFrey/missionary-pipelines

Ben Sless 2024-05-17T05:14:39.683299Z

Link 404s

Eric Dvorsak 2024-05-17T05:30:18.058419Z

@adamfrey seems like a private repo