Clojurians Log v2

We (http://atomist.com) are heavy users of PubSub via Clojure. It's been a bit of a journey as we've moved from SQS to PubSub for a lot of our use. We like to keep things in order so use a lot of FIFO queues and found AWS had an issue at about 20k queued messages where a noisy neighbour could ruin fairness. Anyway, our approach is a partly custom one. We're using the Google client library (https://github.com/googleapis/java-pubsub) but we found that the async subscriber classes weren't helpful (due to limitations with NACKing messages for particular periods of time) so we've wrapped the client library calls up with some core.async (`pipeline-blocking` , pipeline-async are great) and it's worked really well for us. Any questions, feel free to ping me!

I should mention that we started by just using clj-http and making the raw calls ourselves. I really like Google's APIs for that. However, we found some weird 503 responses and lots of retry code that we had to add. That has gone away by using the client library which hides all those retries etc. Definitely worth a bit of interop in the short term to use the library in my opinion.

@neilprosser during dev, do you guys just use default credentials or set the credentials via the environment variables? Or is there some way to build the credentials programatically and set it like that instead of via env. We currently just use the http endpoint. This makes switching between emulator and different projects quite easy right from the repl. Ideally we would prefer to use the java lib, but we would need to be able to do the switching.

A bit late but this found this https://github.com/iprally/jonotin

+1 thanks

@koornhof.pieter When I was working with the library locally I was using my credentials (so whatever has been stored for auth while using the CLI). That made it nice and easy. When we're running in production on GKE we use workload identity for auth so fortunately for us the same code works and we never have to manually inject credentials. However, we have a need to talk to PubSub via Datomic (which has to run in AWS) so for that this method (https://github.com/googleapis/google-auth-library-java/blob/main/oauth2_http/java/com/google/auth/oauth2/GoogleCredentials.java#L138) is your friend! (let [credentials (GoogleCredentials/fromStream (ByteArrayInputStream. (.getBytes value)))] ...) where value is a string containing the JSON version of the credentials. Does that make sense? For raw HTTP access you can just do gcloud auth print-access-token and use that in your Authorization header until it expires.

Thanks. After pasting this I did some digging and found you can construct and provide custom credentials to the lib via some builder. This either did not exist a few years back or I was just terrible at reading java docs. probably the latter 😄

This is handy as we where looking at using the http requests mainly to avoid interop with java as it often feels much nicer to read and write, I wonder if those kind of issues are resolved, but making me think perhaps we should take a look at the google java library

Personally, I think it's worth a couple of hours (maybe) of writing interop and type hints to use the client library and make it idiomatic. I can't remember whether I saw any existing open-source libraries which look alive and already attempt to do this. My journey of doing this went from prototyping and fiddling with the client library to having it working pretty swiftly.

This was my experience using http client its very easy to knock something together working out auth and permissions was the part that took the time, will give both a try things like the auto retry would be nice to have

I’m curious why you needed retry code? if something fails pubsub will deliver the message again after the ack deadline. We leverage this quite often.

Sorry. I probably wasn't clear. The retry code we needed to add was for random failures in publishing rather than subscribing. There were still occasional failures in subscribing but the nature of subscribing means you just try again!

once its on the queue but if the http request to place it on the queue fails then you may want to retry, @neilprosser mentioned getting 503 errors I assume thats with publishing events and the java lib does the rety, although looking at http-client it looks like it has a retry mechanism as well as its backed by apache http client

That's right. I think it was 50x (so might not have necessarily been 503s). Some publish requests would fail pretty quickly because of those and we'd retry. I've never investigated clj-http's retry-handler functionality. We've always used diehard (https://github.com/sunng87/diehard) when we've wanted retries.

Just looking at the readme for clj-http that retry-handler looks nice. I must've not scrolled down that far before!

diehard is an interesting one to know about

Definitely. Serves a general retry purpose and with some other goodies thrown in too!

@neilprosser ah ok makes sense.