Hey Clojourians!! Nice to meet you all! Is anyone in Europe here? I have a remote long term project with a leading client where I’m looking for an experienced Clojure/ClojureScript with Java Microservices BE and AWS skills too. If anyone is interested in a high paid remote role, please get in touch with me ASAP 🙂 Have a nice day, hope everyone is staying safe. Many Thanks Daniel

any recommendations on a mature http client that supports core.async by default? its server side so it dont need to be compatiblle with CLJS

@plins it's kind of a gap

requests with cognitect.http-client return a channel, but cognitect.http-client is designed for a very specific use-case

I had a wrapper of http://java.net.http that returns you a channel...

Separately, I've been working on adapting the client within Cognitect Labs' aws-api to support broader use-cases.

you can always take one of the existing clients and adapt it to return you a channel

but IMHO existing clients have a lot of dependencies

Apple's Service Talk seems like a robust foundation https://apple.github.io/servicetalk/servicetalk-examples/SNAPSHOT/http/index.html

what’s the use case for an http client that returns a channel?

sorry if that’s a #stupid-question

when you think http requests are cheaper than threads?

most of the time you want IO to happen in the background

well, that's a bad generalisation. but it's often helpful

if I request data from 8 different web services and the requests don't depend on each other it's nice if something goes off and gets all that concurrently

@lilactown real world use case: to get a result for a client, I need data from 5 apis in parallel - none of the requests wait on results from the others

there's a paradigm where all IO is async by default, and you have to be explicit about what things you want to wait for. it's pretty popular

sometimes it's actually a good idea and not just js stockholm syndrome :D

haha. yeah vert.x is not quite the same thing as node.js 😆

right, but shouldn’t how the http call gets handled (asynchronous, synchronous, channels, which thread, etc) be left to the consumer of the http library rather than bundled with the http library itself?

there's a decent argument that syncing on something async has fewer gotchas than going async on something sync

we do have good concurrency tools (futures, core.async, thread pool executors) but there are gotchas

yeah, I’m just pattern matching on the rule of thumb that “io inside of a go channel is bad”

channels don't go inside go, go goes inside channels

sounds like having async IO that returns a channel representation of that async process is what we’re talking about, not actually running the request inside of a go?

got it

it's easy enough to return a channel and deliver to it without using a go thread

I was recently looking at guile fibers, it's an interesting compare / contrast (instead of go recompiling a state machine, the can use continuations; just like us they need to worry about blocking calls eating up their thread pool)

fibers/continuations at the language level seem like such a good idea for the 80% case, we just need some behemoth to throw a bunch of money at making it good enough at the 20% where it kinda sucks rn

well, they have continuations at the language level, fibers are userspace just like our go blocks are

they fixed multiple bugs in their compiler (and more to come) due to problems exposed by using fibers as intended, which is cool

one thing i like - instead of our go which puts your code into a singleton async scheduler, fibers have run-fibers which creates a new scheduler to run the body in - so for example two libraries can each use their own fiber scheduler if they are independently async and don't need to coordinate

I'm not sure how practical that is in practice. The reality is that you really want a single thread pool in your application really. You can't reserve the number of cores*2 to infinity. In theory I'm guessing that core async could do that by not using a Singleton?

probably - I'm not sure how much (if any) of the current code relies on there being a single instance of the thread pool - eg. what would happen if one channel was used in two go blocks inside different schedulers? since the continuation of a go block is expressed as a channel callback, I'm not sure how that should work

in fact, the least disruptive thing would be channels attached to schedulers, not go blocks, but that's very counterintuitive

yes that makes sense to me

go/thread isn't the dilemma here

can avoid using either of them and submit normal functions to an ExecutorService, that yield things on channels when done

(let [ch (async/chan 1)]
  (.submit executor 
    (fn []
      (put! ch (http/request ....))))
  ch)

is the general pattern

aren't you just re-creating go ?

but inside your own executor

I suspect grouping fibers like guile does is more about the ability to wait for them all to finish then it is about scheduling

and to cancel them as a group even?

maybe, dunno

since their vm allows canceling from the outside

@lilactown I don't understand your question

I think guile is single thread, so having multiple "schedulers" on top of a single thread would be really odd

having a process return a channel allows you to use all the coordination facilities in core.async

which are very very useful

but other than being a big fan of http://wingolog.org I don't know much

here is an example of shelling out to ffmpeg and returning a channel:

returns you a clojure map on the channel

util/future->ch is a couple lines:

I believe you; I'm very #beginners in my understanding of Java threads / Clojure core.async.

I was thinking of sort of the general core.async case where you want to throw some computation to be scheduled to be run, and it might await other channels

you can't do blocking IO calls within async/go blocks, must communicate with channels

yeah

i get future->ch but what is future-map?

future-map is basically fmap with the CompletableFuture

I thought we were talking about the general case - I put some code in a go block and it gets scheduled to run on some thread pool or what have you

isn't it more you can do blocking IO there, but you should not, because that thread pool has been sized specifically for doing compute?

and when you said that go/thread isn't relevant, and you can create your own function/macro that runs the code on the executor

it sounded like you were saying the solution was to essentially create your own go construct that runs on the executor of your choice

which is fine, but seems pretty onerous

that is not what I'm saying

use whatever thread pools you want, but I coordinate processes using channels

I wrote a process that crawls a filesystem, looking for all video files, and launches ffmpeg using the above code to extract metadata from all the videos. the process runs max 20 ffmpeg calls in parallel

it lights up the cores:

the clojure process doesn't even register on the map

one thread walks the filesystem and pumps all videos encountered onto a channel

another process in the middle takes video files and shells out to ffmpeg, subject to the concurrency limit

and it places its result maps on a third channel

that channel contains datomic transaction data

so I can answer questions like "What files have h.264 streams that are more than 720p?", etc.

wherever I said "ffmpeg" I could replace that with "http request"

HTTP is a little bit complex though. If you're sending JSON or receiving JSON, in which thread pool does the JSON get written/read?

the HTTP client's pool? caller's thread?

a lot of libraries muddy the waters

I'm still a little confused, since it didn't sound like we were talking about I/O before

it does make sense paired with the previous part of the convo when I was confused about a question about http requests and core.async

I thought you were replying to > in fact, the least disruptive thing would be channels attached to schedulers, not go blocks, but that's very counterintuitive

which it actually sounds like is what happens in practice, but channels returned by go blocks are inherently tied to whatever thread pool core.async comes with by default, which it sounds like is different than guile's fibers that allow you to pick the scheduler at runtime

that's right, go blocks run within the core.async managed pool

go translates channel ops into code that attaches callbacks to channels

the jvm has had a default threadpool for a while now too (https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/ForkJoinPool.html#commonPool--) which core.async and clojure don't use

is it go or the channel that decides where the execution happens?

the channel

I thought you could do all of the channel operations on whichever thread you want with the blocking( !! suffix) operations

those compile to the non-blocking calls internally

the channels directly add thunks invoking callbacks to core.async's threadpool, but an alternative implementation would be the channels directly invoking the callbacks and relying on them to do work on whatever threadpool they want

my understanding was that if you want to use a default threadpool, then use go blocks. if you want more control over scheduling and threads, use the blocking operations (`!!` suffix)

I imagine it was just chance of the draw that channels ended up putting stuff on dispatch instead of the other way

but there would even be some benefits to having the callbacks control the threadpooling queing

right now the way real blocking operations are implemented is a callback is attached to the channel which fulfills a promise when run, and the operation blocks on that promise

that callback still gets run on the core.async pool, so even if you are only ever doing real blocking stuff you are running lots of small little tasks on the core.async pool

if the callback got to decide if it ran on an executor or not it could skip the executor for just fulfilling the promise

@smith.adriane https://groups.google.com/d/msg/clojure/yUIl2tl4HoM/awXms4n4AAAJ > In general, you should never directly or indirectly use blocking IO operations in go block threads. The go block threads are a fixed pool of (by default 8) threads. If you block enough of these threads, you lock up the pool, potentially in unrecoverable ways. > This release contains a new Java system property (intended primarily for development use) that will throw if core.async blocking operations (anything ending in "!!") are used in a go block. The exception will be thrown in a go block thread which by default will bubble up to the ThreadGroup's uncaught exception handler and get printed to stderr. You can also set Thread.setDefaultUncaughtExceptionHandler() if you want to do something else. Note that this only catches one set of blocking calls, other blocking IO is equally as problematic and will not be caught with this flag.

(That's not the latest release anymore, since @alexmiller released more good stuff this morning)

:thumbsup:. I wasn’t recommending using blocking operations inside of go blocks. Is it still reasonable to use your own threads with blocking operations if want more control over scheduling and threads?

ok cool, just clarifying. And, BYO threads/executors when desired

In the system I mentioned above, the filesystem pump uses async/thread the ffmpeg stuff is a CompletableFuture that dumps onto a channel the concurrency limiter in the middle is a single go block the main thread reads the results of video metadata extraction

outside of cljs, I typically avoid using go blocks except for temporary solutions

@ghadi do you mind sharing what the concurrency limiter looks like? I'm trying to think of what that looks like with completeable futures

sure. the concurrency limiter is a variant of core.async/pipeline

it knows nothing about completable futures

its input, like clojure.core.async/pipeline-async, is a function that returns a channel

docstring:

"Runs asynchronous function 'af' on each input from channel 'in',
   producing results to channel 'out'. af is presumed to return a channel

   Input order is *not* preserved.

   Runs af with maximum 'max' concurrency. max can be an integer
   or a function returning integer (allowing dynamic concurrency
   control)

   close?, default true, controls whether the output channel is closed
   upon completion"

args [max af in out]

the af that I pass in, is https://clojurians.slack.com/archives/C03S1KBA2/p1589230848408600

which uses a CompletableFuture internally, because that's what java offers you when shelling out

(it's a Java 9+ thing)

why not use clojure.core.async/pipeline-async? 1) I don't care about preserving input order 2) there is a tiny bug (maybe?) in pipeline-async's concurrency limit, where you might get N+2 tasks in flight

the limiter looks very similar to pmax in here https://stuartsierra.com/2013/12/08/parallel-processing-with-core-async

i've never looked at the source of pipeline. that's quite nice

the key here is not caring about anything about af besides it takes an input and returns you a channel with eventual result

that way a specific af is free to do whatever particular resource mgmt it needs

hope that helps!

always love to glean from your examples. thank you!

you can directly use a completablefuture as a readport (the read half of a channel)

https://gist.github.com/hiredman/1788aa052f26d127c00a1679656026f0

that is a very neat gist. I had to do some blocking IO on the process stdout/stderr right as soon as it was completed, so I stayed in future land before converting to channel