@xeqi: cool, thanks!

anybody know if it's possible to reify or deftype an interface that has both overloaded methods and void return types? (https://hc.apache.org/httpcomponents-client-ga/httpclient/apidocs/org/apache/http/conn/ssl/X509HostnameVerifier.html)

@danboykis: does type hinting the arguments help? I've had to do that before for proxy

@xeqi: I tried type hinting but it doesn't work

(reify X509HostnameVerifier
    (verify ^void [_ ^String h ^javax.net.ssl.SSLSocket s] nil)
    (verify ^void [_ ^String h ^java.security.cert.X509Certificate s] nil)
    (verify ^void [_ ^String h #^"[Ljava.lang.String;" a "[Ljava.lang.String;" b] nil))

@xeqi: I haven't tried proxy (yet) but I don't see why it would be any different

@danboykis: https://github.com/dakrone/clj-http/blob/master/src/clj_http/conn_mgr.clj#L30

@xeqi: thanks, great find!

also watch out for the # in #^"[Ljava.lang.String;" incase that wasn't a copy/paste error

it was an error, I just noticed it, I had it in my code

I mean it was correct in my code, but i pasted wrong

@danboykis: Thanks!

np

A tip request please: How can I concat a set of large csv files into one. I need rows identified as duplicates removed (i.e. filter (some #{s} (get row 1) ) Each file is has no duplicates, actually, only between the files can duplicate rows appear. The order of the final outputs isn't crucial, but matching a sequential scan of the files would be preferred.

hi all - is there any way to make a project using cljc backwards compatible with Clojure 1.6?

@mikera there is not, some people just switched back to cljx for this reason

@dnolen thanks for confirming my suspicion. Was looking at using cljc to make core.matrix work with ClojureScript, but I guess it will have to wait until we can drop 1.6 support... maybe after the Clojure 1.8 release

What's currently the most active / wide spread web framework? I used plain ring before but was wondering if there's something better out there

@dvcrn: If you’re looking for something that is batteries included then I would say it is probably https://github.com/pedestal/pedestal

that looks interesting

I'll check it out!

Hi! Question about Loom graphs. I’ve got a collection [:origin A :targets [B C D]] , and graph takes a collection [node [edges]] … how do I transform my map to the right data structure? There’s gotta be an elegant way to do this that I’m missing out on.

Here’s an example data structure: https://gist.github.com/LusciousPear/febbd458c4e1e35979de

@dvcrn: I've used liberator a lot http://clojure-liberator.github.io/liberator/

Does anybody know if it’s possible, with yesql to construct queries where you have more than one set of values?

Ohhh I’m close, (map (juxt :origin :targets) (:edges test-graph))

Im trying to download a bunch of iTunes Connect Sales reports quickly so I can assemble an overview of the mobile apps on an iTunes app store account. (Im doing this because Apple doesn't seem to provide and API to just query the list of mobile apps and their properties, like release dates of their versions, etc) It means I need to do a bunch of HTTP GETs in parallel to get it in a few seconds. Serially it would take minutes (for an account with 50+ apps of which many might have 3+ years history). What tools would you recommend to approach this problem, if a, I want to handle errors b, so I can retry downloads c, persist the state of the downloads, so I can pick it up later (and also retry the failed ones later)

I know there is https://github.com/ztellman/manifold/ which would allow me to wire together some download queues, but I suspect the simplest result could be achieved by using core.async.

d, for later use I would also cache the report files in S3 and/or disk, so to have it close to the CPUs processing it

manifold works very well

you can do something like (->> (map get-async urls) (apply deferred/zip))

is there any example which does a similar "mirroring" or some website with adjustable HTTP request parallelism and something like a backing off retry? i would think this is an often solved problem

and then loop this until all the values are successful

haven't heard of a tool that would do that

what im not clear about is where and how can i insert some throttling... like how can i say i want max 50 http request to be active at once

you would partition the urls above by 50

nah, that would be rigid. the 50 should be a sliding window over N urls

yeah, that requires a stream then (or a chan)

something like eachLimit from https://github.com/caolan/async#each > <name>Limit - the same as <name> but runs a maximum of limit async operations at a time

@onetom: this is an async client I help maintain, it has connection limiting: https://github.com/cch1/http.async.client

it's a wrapper around https://github.com/AsyncHttpClient/async-http-client

but in this case the order doesnt matter, however things i put in a core.async chan would be in order, no?

dm3: it's easy to do with core.async/alts! too

@pbostrom: thx, checking

mpenet: ah, alts!, thanks for the reminder

@onetom: sorry, the documentation is a bit sparse

an agent could work - have it keep the open conn count and a queue of requests

(I think :))

this is a nice lib with multithreaded utilities, especially for doing unordered parallelism https://github.com/TheClimateCorporation/claypoole

i suspect i dont really need threads in this case, since the problem is not CPU but IO bound and separate threads would just increase mem usage, no?

but of course ideally it would be great to utilize multiple cores too

yeah, it might not buy you much if you use an async client

depends on what you do with the response of course

i might even need to slow requests down to make sure im not hitting some per-IP address throttling on iTunes Connect side....

the sales reports im downloading are just few 10s of kilobytes of gzipped tsv which i want to a, pump into Datomic b, backup the original gzip into S3

yeah, I take back the claypoole suggestion, if you use an async client, the callbacks are going to run on their own thread anyway

it's fairly easy to do with core.async, you can use a channel for rate-limiting (it provides request "token" when you can perform one, handles the pauses etc)

if you google core async throttling, rate limiting etc you ll find a few starting points

then if your client has a core.async interface it's should be very quick to put together

or just use a blocking http client and a fixed threadpool (or a client that allows you to specify these things)

@pbostrom: how does https://github.com/cch1/http.async.client compares with http://www.http-kit.org/client.html ?

I would stay away from http-kit (it's barely maintained) and I have no experience with http.async.client

there's also manifold that has an http client I think

there is aleph which is mentioned together w manifold usually if u need some networking, but iirc it was just interfacing with clj-http?

jet too

http.async.client uses a promise-based model. I think http-kit is callback-based. There are also big differences in the underlying libs. http.async.client is based on ning’s async Java client which in turn uses netty.

I think http kit uses web kit.

http-kit is custom written

or maybe aleph just said they provide a clj-http compatible api

no dependencies

all these clients have a mostly ring compatible api

aleph does not have what I would call an traditional http client -it’s more of an HTTP client tool kit.

That being said, you could “get there from here"

And, aleph too is based on netty.

looks like a full featured client to me :]

@mpenet: Good to know about http-kit not having any dependencies. That’s pretty cool.

yeah but it's broken

I have had some hairy experiences with http-kit

check also the issue tracker

(both the cilent/server)

also saving a few kbs from what will certainly be a multi mb jar in the end, with a gigantic runtime in the background, sounds a bit useless.

I’ve had overall good experience with http-kit in production but updates are infrequent and the issue tracker is swamped. Once things slow down around work I’m thinking about trying to contribute / fork it.

On that note (active development), I can definitely vouch for the underlying libs behind http.async.client -the ning client is very actively maintained -as is netty. The clojure wrapper itself (the lib @pbostrom and I are maintaining) had been neglected until earlier this summer when we ramped up its support again.

tbh we also had some issues w http-kit. my colleagues said it's better to look for http-kit documentation in some other project's examples (forgot what it was) then its own docs.

But, while it had not been actively maintained, it was rock solid in production for us for years.

I have had deadlocks with cpu spinning 100%

with http-kit

I think the issue mentioning this is still open btw

@cch1: pbostrom: well, thanks guys for the introduction to your lib then!

np

jetty9 also includes a decent http client

and there's a clojure wrapper for it (shameless plug). but it's fairly young

mpenet: dont be shy, whats the url for the wrapper?

https://github.com/mpenet/jet

mpenet: where is your avatar from? https://avatars2.githubusercontent.com/u/106390?v=3&amp;s=400

it's from Pablo Bernasconi

anyone have an example application using danielsz/system? im probably just missing something, but i want to see how other folks tackle injecting stub dependencies for unit tests

currently im doing something like (alter-var-root #’system #(assoc % :component stub)) in my test, but im not sure this is the right way

pssst how do I change [:a 1 :b 2] into {:a 1, :b: 2} … into {} isn’t working how I’d expect

@bradford They need to be pairs

(apply hash-map [...])

oo

@mpenet: Thank you 🙏:skin-tone-2:

BTW, the answer to my question, how do I turn https://gist.github.com/LusciousPear/febbd458c4e1e35979de into a graph? it’s

(graph (apply hash-map (reduce into
                                 (map (juxt :origin :targets) (:edges a-graph)))))

Sooooo hawt

i'm suspicious about reduce into + apply hash-map

seems like this can be done in one step

reduce-kv?

can you show an example of a-graph?

dunno abou the shape of your data

bradford: (graph (into {} (map (juxt :origin :targets) ...))

@ghadi: oh, rad! thanks.

@jstokes: check out with-redefs

yaknow, when I wrote a SQL Database, I wish I had done the query planner/parser/optimizer in clojure instead of java. prolly woulda saved me 90% LOC 😛

@onetom: you mentioned retrying failures, picking up where they left off. It sounds like hara.event could be useful there. It's a bit of a paradigm shift to use "events" (a la "conditions" in Common Lisp) instead of exceptions, but it sounds like it addresses your need well. http://docs.caudate.me/hara/hara-event.html

@jeff.terrell: thanks for the pointer. (not directly related, we are using http://docs.caudate.me/adi already, btw

Hmm, I hadn't seen that, looks interesting, thanks!

chouser will be doing a talk about conditions at the Conj btw http://clojure-conj.org/speakers#chouser

Oh, very cool!

I'm writing a macro and I receive a list of arguments, and I need to apply some postprocessing to them (a when check). What's the right way to do that? I want to not have a loop in resulting code ideally

@jstokes: Not that I've spent much time on this, but to me it sounds you just want to define a test-sytem, along a dev-system or a prod-system, and start it as first thing in your test runner, and destroy at the end.

@jstokes I hope this makes sense.

@danielsz: that’s almost what i was thinking, exactly

@jstokes: 🍻

Is there anything like this in the standard lib? (defn split-by [pred coll] [(filter pred coll) (remove pred coll)])

(Obviously a better implementation would only traverse the collection once.)

Kinda like a group-by, but with a predicate function.

@camdez: maybe you want partition-by

partition-by maybe … but that splits every time the predicate changes

Right.

Alternate implementation: `(fn [pred coll] (let [m (group-by pred coll)] (map m [true false])))`

probably would just do it with a reduction if you want to only traverse the collection once

Yeah. It’s not hard to implement, just wondering if there’s an existing function.

Naming—as is so often the case—is the trickiest part. I’m not sure it’s quite a split, the way Clojure uses the term. But I’m not sure what I’d call it as partition function.

Rolling with this for now:

(defn pred-partition [pred coll]
  (map (group-by pred coll) [true false]))

predtition. 😉

Why do you need to wrap group-by? Is it that you explicitly want the true values first in the resulting sequence?

hello guys, when I create codox documentation, can i specify a lein profile?

mdaines: It doesn’t have to be first, per se. Sample use case: splitting xml nodes into those of a certain tag, and all others. I want both lists. So I’m destructing into two bindings. Something like (let [[links nonlinks] (pred-partition nodes (tag= :link))] ,,,).

(Err, with the argument order reversed on the call to pred-partition.)

It feels a little odd to use the boolean as a key, but this works:

(let [{odd false even true} (group-by even? (range 23 43))]
   odd)

Haha, was about to paste this:

(let [{odds true evens false} (group-by odd? (take 4 (range)))]
  [odds evens])

Thanks. I always forget about that style of destructuring.

heh heh, yeah.

I’ve finally got to a point when I don’t have to look it up to remember the format.

Yeah, it takes a while. Nested map destructing still takes me a second...

@camdez: regarding a name, maybe separate-by? Or maybe bifurcate?

splitmap as opposed to zipmap.

@asolovyov: not really sure what you're trying to do.

@jeff.terrell: I’m kind of fond of bifurcate, thanks. But as per discussion with @mdaines, I think I might just roll without.

Destructuring the group-by has a hint of cognitive overhead, but not enough that I’m yet ready to introduce a new name (/ concept).

Fair enough. :thumbsup:

PODS: Plain Ol' Data Structures

POCO: Plain old Clojure object

Is there a convenient means of mocking a db instance with monger? The web site states that there’s “unit test support” but I’ve yet to find any documentation that goes further into this.

Is there a version of walk that only operates on the values themselves? I thought specter would have this, but it doesn't, which is a shame.

Take a look at the source for clojure.walk/stringify-keys — is that along the lines of what you’re trying to do?

@clem: Problem with that is... derp of course that works. I just have to reverse it. Thanks. I need to stop working such long hours.

heh, no problem