Fork me on GitHub
#off-topic
<
2020-05-04
>
Gleb Posobin00:05:52

What should I use to make a background worker with persistent queue in clojure? Once a user signs up for my service I want to make a bunch of requests to twitter api, but they have to be dispersed in time not to go over twitter's restrictive request limits — think one every ten seconds. The task is not urgent and there won't be a lot of users at the start, so performance is not a priority. But I don't want to lose any tasks in progress if the server restarts. I am already using postgres so ideally the queue would be there.

orestis04:05:48

Postgres offers a few primitives to help with queues- SELECT FOR UPDATE SKIP LOCKED https://tnishimura.github.io/articles/queues-in-postgresql/

orestis04:05:25

Doesn’t scale greatly but it’s a good start. Polling the db every few seconds should be ok

p-himik07:05:23

Polling can probably be avoided with listen + notify used in triggers.

gibb06:05:13

I can recommend the queues in postgres approach too, works great.

mloughlin09:05:24

Assuming a default mode of writing software is to aim for simplicity by separating concerns, are there examples where intentionally complicating things leads to an improvement?

p-himik09:05:52

Since "complication" is the opposite of "simplification", it can't happen. But it's possible with "complecting" - if the separation of concerns was ill-advised in the first place.

aisamu12:05:57

Monoliths (in some cases)?

andy.fingerhut14:05:47

I am pretty sure that complicating things could be done in some cases to improve performance, trading better performance for more entanglement/dependencies between different parts of the code.

vemv14:05:36

Yes, it's somewhat common to inline/intertwine things for performance e.g. for a program that parses an input, analyzes it and then presents a report, you might be tempted to analyze or report stuff as soon as it's parsed

andy.fingerhut14:05:16

Or a later step could be written in a way that assumes some other additional data or intermediate results have been computed earlier, and passed along, and the format of that extra data becomes a separate thing from the bare minimum data that could be passed from step A to step B that makes A and B less easy to change independently.

👍 4
vemv15:05:27

tr-8s and a modular synth setup, nice!

vemv16:05:32

definitely liked it, I appreciate world sounds :drum_with_drumsticks: