Hello! I’m fairly new to core.async and I’m hoping to improve some of my code! I’ve uploaded some code that I’m currently using to create somewhat reusable pipelines for a series of variables that I need to scrape off some public websites. If anyone could look at the code and suggest improvements I’d greatly appreciate it! (https://gist.github.com/adairdavid/84932f9e77eb1f62b63ed606b898a214). Specifically, I’m looking to remove some of the duplicated boilerplate for each scraping process. In the gist, some duplicated boilerplate would be the launch function. I feel like my code currently creates an unnecessary amount of channels, and I’d like to streamline that so that these processes are more easily started/stopped when integrated into the Component Lifecycle. Thanks!

@dadair core.async has functions for all your utils

realtime would be pipe

process is pipeline

persist is pipeline-blocking with a curried or partial function that closes over the db connection in the transducer

In terms of reducing the number of channels a) there’s no way around that and b) it’s ok channels are cheap

One thing I tend to do is wrap the pipeline* functions to change their signatures so that they take an input channel but create/return the output one, this way the channels can be threaded with ->

Hello!

I want to "time-buffer" some scrapping code - make a 1sec delay between get requests, but if more than 1 sec passed from last get request, execute imediately

sth like: (delay-expr (http/get url))

@dimovich there’s a timeout fn in core.async

yeah, but how to use it 🙂

just starting to write this, but maybe someone already has some solution 🙂

if you have (a/<! (a/timeout 1000)) in a go block it’ll wait 1 sec

@jfntn perfect thanks! I’ll look into those functions 😄 cheers!