Fork me on GitHub
#missionary
<
2024-05-16
>
braai engineer10:05:28

Has anyone tried to build an async task runner / workflow system with dependent tasks using Missionary? Like Airflow or Celery.

Eric Dvorsak11:05:22

I'll need (to buils) one soon

Ben Sless14:05:40

I've been thinking about something similar from another angle, but this also relates to distribution of work and state persistence. If we can distribute work between client and server, how can we distribute work among servers in a similar manner? I want (via remote ...)

Eric Dvorsak15:05:46

what I've done in the past (without missionary) is have the servers pick up tasks when they have capacity. For that I have a job table in postgres, and use a skip lock in a transaction to process the task:

(postgres/execute-one! postgres
                               {:delete-from :scheduled-tasks
                                :where [:=
                                        :id
                                        {:select :id
                                         :from [:scheduled-tasks]
                                         :order-by [:execute-at]
                                         :where [:<= :execute-at now]
                                         :for [:update :skip-locked]
                                         :limit 1}]
                                :returning [:*]})
if the server fails to execute the task (rolling back or timing out the transaction) then the task is picked by another

adamfrey00:05:11

I started working on something like that. I didn't quite get it to a usable state for lack of time or focus, but you can look at it if you are interested https://github.com/AdamFrey/missionary-pipelines

Eric Dvorsak05:05:18

@U06060CK0 seems like a private repo

adamfrey12:05:13

my apologies, it's public now.