This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2020-05-04
Channels
- # announcements (25)
- # babashka (7)
- # beginners (52)
- # calva (29)
- # clara (1)
- # clj-kondo (4)
- # cljs-dev (55)
- # clojure (86)
- # clojure-europe (5)
- # clojure-finland (1)
- # clojure-france (1)
- # clojure-italy (1)
- # clojure-nl (1)
- # clojure-uk (57)
- # clojurescript (33)
- # conjure (107)
- # cursive (20)
- # datomic (37)
- # emacs (23)
- # events (13)
- # fulcro (67)
- # helix (73)
- # jobs-discuss (22)
- # lambdaisland (1)
- # leiningen (32)
- # malli (2)
- # meander (9)
- # mid-cities-meetup (1)
- # observability (1)
- # off-topic (14)
- # overtone (3)
- # pathom (39)
- # re-frame (22)
- # reagent (13)
- # reitit (13)
- # shadow-cljs (52)
- # sql (15)
- # tools-deps (29)
- # vim (11)
What should I use to make a background worker with persistent queue in clojure? Once a user signs up for my service I want to make a bunch of requests to twitter api, but they have to be dispersed in time not to go over twitter's restrictive request limits — think one every ten seconds. The task is not urgent and there won't be a lot of users at the start, so performance is not a priority. But I don't want to lose any tasks in progress if the server restarts. I am already using postgres so ideally the queue would be there.
Postgres offers a few primitives to help with queues- SELECT FOR UPDATE SKIP LOCKED https://tnishimura.github.io/articles/queues-in-postgresql/
Doesn’t scale greatly but it’s a good start. Polling the db every few seconds should be ok
Assuming a default mode of writing software is to aim for simplicity by separating concerns, are there examples where intentionally complicating things leads to an improvement?
Since "complication" is the opposite of "simplification", it can't happen. But it's possible with "complecting" - if the separation of concerns was ill-advised in the first place.
I am pretty sure that complicating things could be done in some cases to improve performance, trading better performance for more entanglement/dependencies between different parts of the code.
Yes, it's somewhat common to inline/intertwine things for performance e.g. for a program that parses an input, analyzes it and then presents a report, you might be tempted to analyze or report stuff as soon as it's parsed
Or a later step could be written in a way that assumes some other additional data or intermediate results have been computed earlier, and passed along, and the format of that extra data becomes a separate thing from the bare minimum data that could be passed from step A to step B that makes A and B less easy to change independently.