Fork me on GitHub
#clojure
<
2015-10-27
>
bensu00:10:48

@xeqi: cool, thanks!

danboykis00:10:49

anybody know if it's possible to reify or deftype an interface that has both overloaded methods and void return types? (https://hc.apache.org/httpcomponents-client-ga/httpclient/apidocs/org/apache/http/conn/ssl/X509HostnameVerifier.html)

xeqi00:10:33

@danboykis: does type hinting the arguments help? I've had to do that before for proxy

danboykis00:10:26

@xeqi: I tried type hinting but it doesn't work

(reify X509HostnameVerifier
    (verify ^void [_ ^String h ^javax.net.ssl.SSLSocket s] nil)
    (verify ^void [_ ^String h ^java.security.cert.X509Certificate s] nil)
    (verify ^void [_ ^String h #^"[Ljava.lang.String;" a "[Ljava.lang.String;" b] nil))
Here's the error: CompilerException java.lang.IllegalArgumentException: Mismatched return type: verify, expected: void, had: java.lang.Object

danboykis00:10:37

@xeqi: I haven't tried proxy (yet) but I don't see why it would be any different

danboykis00:10:00

@xeqi: thanks, great find!

xeqi00:10:17

also watch out for the # in #^"[Ljava.lang.String;" incase that wasn't a copy/paste error

danboykis00:10:35

it was an error, I just noticed it, I had it in my code

danboykis00:10:07

I mean it was correct in my code, but i pasted wrong

kegunder01:10:01

A tip request please: How can I concat a set of large csv files into one. I need rows identified as duplicates removed (i.e. filter (some #{s} (get row 1) ) Each file is has no duplicates, actually, only between the files can duplicate rows appear. The order of the final outputs isn't crucial, but matching a sequential scan of the files would be preferred.

mikera02:10:58

hi all - is there any way to make a project using cljc backwards compatible with Clojure 1.6?

dnolen02:10:46

@mikera there is not, some people just switched back to cljx for this reason

mikera04:10:00

@dnolen thanks for confirming my suspicion. Was looking at using cljc to make core.matrix work with ClojureScript, but I guess it will have to wait until we can drop 1.6 support... maybe after the Clojure 1.8 release

dvcrn05:10:26

What's currently the most active / wide spread web framework? I used plain ring before but was wondering if there's something better out there

rok305:10:53

@dvcrn: If you’re looking for something that is batteries included then I would say it is probably https://github.com/pedestal/pedestal

dvcrn05:10:27

that looks interesting

dvcrn05:10:30

I'll check it out!

bradford10:10:27

Hi! Question about Loom graphs. I’ve got a collection [:origin A :targets [B C D]] , and graph takes a collection [node [edges]] … how do I transform my map to the right data structure? There’s gotta be an elegant way to do this that I’m missing out on.

pupeno10:10:46

Does anybody know if it’s possible, with yesql to construct queries where you have more than one set of values?

bradford10:10:53

Ohhh I’m close, (map (juxt :origin :targets) (:edges test-graph))

onetom12:10:49

Im trying to download a bunch of iTunes Connect Sales reports quickly so I can assemble an overview of the mobile apps on an iTunes app store account. (Im doing this because Apple doesn't seem to provide and API to just query the list of mobile apps and their properties, like release dates of their versions, etc) It means I need to do a bunch of HTTP GETs in parallel to get it in a few seconds. Serially it would take minutes (for an account with 50+ apps of which many might have 3+ years history). What tools would you recommend to approach this problem, if a, I want to handle errors b, so I can retry downloads c, persist the state of the downloads, so I can pick it up later (and also retry the failed ones later)

onetom12:10:57

I know there is https://github.com/ztellman/manifold/ which would allow me to wire together some download queues, but I suspect the simplest result could be achieved by using core.async.

onetom12:10:38

d, for later use I would also cache the report files in S3 and/or disk, so to have it close to the CPUs processing it

dm312:10:20

manifold works very well

dm313:10:55

you can do something like (->> (map get-async urls) (apply deferred/zip))

onetom13:10:02

is there any example which does a similar "mirroring" or some website with adjustable HTTP request parallelism and something like a backing off retry? i would think this is an often solved problem

dm313:10:29

and then loop this until all the values are successful

dm313:10:45

haven't heard of a tool that would do that

onetom13:10:50

what im not clear about is where and how can i insert some throttling... like how can i say i want max 50 http request to be active at once

dm313:10:07

you would partition the urls above by 50

onetom13:10:50

nah, that would be rigid. the 50 should be a sliding window over N urls

dm313:10:07

yeah, that requires a stream then (or a chan)

onetom13:10:35

something like eachLimit from https://github.com/caolan/async#each > <name>Limit - the same as <name> but runs a maximum of limit async operations at a time

pbostrom13:10:58

@onetom: this is an async client I help maintain, it has connection limiting: https://github.com/cch1/http.async.client

onetom13:10:32

but in this case the order doesnt matter, however things i put in a core.async chan would be in order, no?

mpenet13:10:39

dm3: it's easy to do with core.async/alts! too

onetom13:10:17

mpenet: ah, alts!, thanks for the reminder

pbostrom13:10:58

@onetom: sorry, the documentation is a bit sparse

dm313:10:23

an agent could work - have it keep the open conn count and a queue of requests

dm313:10:40

(I think :))

pbostrom13:10:16

this is a nice lib with multithreaded utilities, especially for doing unordered parallelism https://github.com/TheClimateCorporation/claypoole

onetom13:10:45

i suspect i dont really need threads in this case, since the problem is not CPU but IO bound and separate threads would just increase mem usage, no?

onetom13:10:59

but of course ideally it would be great to utilize multiple cores too

pbostrom13:10:23

yeah, it might not buy you much if you use an async client

pbostrom13:10:57

depends on what you do with the response of course

onetom13:10:07

i might even need to slow requests down to make sure im not hitting some per-IP address throttling on iTunes Connect side....

onetom13:10:27

the sales reports im downloading are just few 10s of kilobytes of gzipped tsv which i want to a, pump into Datomic b, backup the original gzip into S3

pbostrom13:10:33

yeah, I take back the claypoole suggestion, if you use an async client, the callbacks are going to run on their own thread anyway

mpenet13:10:37

it's fairly easy to do with core.async, you can use a channel for rate-limiting (it provides request "token" when you can perform one, handles the pauses etc)

mpenet13:10:07

if you google core async throttling, rate limiting etc you ll find a few starting points

mpenet13:10:24

then if your client has a core.async interface it's should be very quick to put together

mpenet13:10:10

or just use a blocking http client and a fixed threadpool (or a client that allows you to specify these things)

mpenet13:10:16

I would stay away from http-kit (it's barely maintained) and I have no experience with http.async.client

mpenet13:10:31

there's also manifold that has an http client I think

onetom13:10:08

there is aleph which is mentioned together w manifold usually if u need some networking, but iirc it was just interfacing with clj-http?

cch113:10:15

http.async.client uses a promise-based model. I think http-kit is callback-based. There are also big differences in the underlying libs. http.async.client is based on ning’s async Java client which in turn uses netty.

cch113:10:27

I think http kit uses web kit.

mpenet13:10:36

http-kit is custom written

onetom13:10:36

or maybe aleph just said they provide a clj-http compatible api

mpenet13:10:44

no dependencies

mpenet13:10:08

all these clients have a mostly ring compatible api

cch113:10:19

aleph does not have what I would call an traditional http client -it’s more of an HTTP client tool kit.

cch113:10:35

That being said, you could “get there from here"

cch113:10:06

And, aleph too is based on netty.

mpenet13:10:49

looks like a full featured client to me :]

cch113:10:50

@mpenet: Good to know about http-kit not having any dependencies. That’s pretty cool.

mpenet13:10:58

yeah but it's broken

mpenet13:10:08

I have had some hairy experiences with http-kit

mpenet13:10:15

check also the issue tracker

mpenet13:10:24

(both the cilent/server)

mpenet13:10:30

also saving a few kbs from what will certainly be a multi mb jar in the end, with a gigantic runtime in the background, sounds a bit useless.

rok313:10:37

I’ve had overall good experience with http-kit in production but updates are infrequent and the issue tracker is swamped. Once things slow down around work I’m thinking about trying to contribute / fork it.

cch113:10:38

On that note (active development), I can definitely vouch for the underlying libs behind http.async.client -the ning client is very actively maintained -as is netty. The clojure wrapper itself (the lib @pbostrom and I are maintaining) had been neglected until earlier this summer when we ramped up its support again.

onetom13:10:08

tbh we also had some issues w http-kit. my colleagues said it's better to look for http-kit documentation in some other project's examples (forgot what it was) then its own docs.

cch113:10:14

But, while it had not been actively maintained, it was rock solid in production for us for years.

mpenet13:10:22

I have had deadlocks with cpu spinning 100%

mpenet13:10:38

with http-kit

mpenet13:10:50

I think the issue mentioning this is still open btw

onetom13:10:25

@cch1: pbostrom: well, thanks guys for the introduction to your lib then! simple_smile

mpenet13:10:29

jetty9 also includes a decent http client

mpenet13:10:56

and there's a clojure wrapper for it (shameless plug). but it's fairly young

onetom13:10:38

mpenet: dont be shy, whats the url for the wrapper? simple_smile

mpenet14:10:35

it's from Pablo Bernasconi

jstokes14:10:52

anyone have an example application using danielsz/system? im probably just missing something, but i want to see how other folks tackle injecting stub dependencies for unit tests

jstokes15:10:48

currently im doing something like (alter-var-root #’system #(assoc % :component stub)) in my test, but im not sure this is the right way

bradford15:10:06

pssst how do I change [:a 1 :b 2] into {:a 1, :b: 2}into {} isn’t working how I’d expect simple_smile

jgdavey15:10:48

@bradford They need to be pairs

mpenet15:10:54

(apply hash-map [...])

bradford15:10:18

@mpenet: Thank you 🙏:skin-tone-2:

bradford15:10:40

BTW, the answer to my question, how do I turn https://gist.github.com/LusciousPear/febbd458c4e1e35979de into a graph? it’s

(graph (apply hash-map (reduce into
                                 (map (juxt :origin :targets) (:edges a-graph)))))

bradford15:10:52

Sooooo hawt

mishok1315:10:14

i'm suspicious about reduce into + apply hash-map

mishok1315:10:22

seems like this can be done in one step

mpenet15:10:44

reduce-kv?

mishok1315:10:48

can you show an example of a-graph?

mpenet15:10:52

dunno abou the shape of your data

ghadi15:10:58

bradford: (graph (into {} (map (juxt :origin :targets) ...))

bradford15:10:08

@ghadi: oh, rad! thanks.

exupero15:10:33

@jstokes: check out with-redefs

bradford15:10:13

yaknow, when I wrote a SQL Database, I wish I had done the query planner/parser/optimizer in clojure instead of java. prolly woulda saved me 90% LOC 😛

jeff.terrell15:10:22

@onetom: you mentioned retrying failures, picking up where they left off. It sounds like hara.event could be useful there. It's a bit of a paradigm shift to use "events" (a la "conditions" in Common Lisp) instead of exceptions, but it sounds like it addresses your need well. http://docs.caudate.me/hara/hara-event.html

onetom16:10:45

@jeff.terrell: thanks for the pointer. (not directly related, we are using http://docs.caudate.me/adi already, btw simple_smile

jeff.terrell16:10:33

Hmm, I hadn't seen that, looks interesting, thanks!

alexmiller16:10:29

chouser will be doing a talk about conditions at the Conj btw http://clojure-conj.org/speakers#chouser

jeff.terrell16:10:22

Oh, very cool!

asolovyov16:10:03

I'm writing a macro and I receive a list of arguments, and I need to apply some postprocessing to them (a when check). What's the right way to do that? I want to not have a loop in resulting code ideally

danielsz16:10:36

@jstokes: Not that I've spent much time on this, but to me it sounds you just want to define a test-sytem, along a dev-system or a prod-system, and start it as first thing in your test runner, and destroy at the end.

danielsz17:10:11

@jstokes I hope this makes sense.

jstokes17:10:56

@danielsz: that’s almost what i was thinking, exactly

camdez17:10:49

Is there anything like this in the standard lib? (defn split-by [pred coll] [(filter pred coll) (remove pred coll)])

camdez17:10:01

(Obviously a better implementation would only traverse the collection once.)

camdez17:10:55

Kinda like a group-by, but with a predicate function.

mdaines17:10:27

@camdez: maybe you want partition-by

tyler17:10:37

partition-by maybe … but that splits every time the predicate changes

camdez17:10:34

Alternate implementation: `(fn [pred coll] (let [m (group-by pred coll)] (map m [true false])))`

tyler17:10:42

probably would just do it with a reduction if you want to only traverse the collection once

camdez17:10:58

Yeah. It’s not hard to implement, just wondering if there’s an existing function.

camdez17:10:55

Naming—as is so often the case—is the trickiest part. I’m not sure it’s quite a split, the way Clojure uses the term. But I’m not sure what I’d call it as partition function.

camdez17:10:49

Rolling with this for now:

(defn pred-partition [pred coll]
  (map (group-by pred coll) [true false]))
If anyone has a great idea for a name, I’d appreciate it. simple_smile

camdez17:10:12

predtition. 😉

mdaines17:10:54

Why do you need to wrap group-by? Is it that you explicitly want the true values first in the resulting sequence?

richiardiandrea18:10:31

hello guys, when I create codox documentation, can i specify a lein profile?

camdez18:10:52

mdaines: It doesn’t have to be first, per se. Sample use case: splitting xml nodes into those of a certain tag, and all others. I want both lists. So I’m destructing into two bindings. Something like (let [[links nonlinks] (pred-partition nodes (tag= :link))] ,,,).

camdez18:10:35

(Err, with the argument order reversed on the call to pred-partition.)

mdaines18:10:28

It feels a little odd to use the boolean as a key, but this works:

(let [{odd false even true} (group-by even? (range 23 43))]
   odd)

camdez18:10:41

Haha, was about to paste this:

(let [{odds true evens false} (group-by odd? (take 4 (range)))]
  [odds evens])

camdez18:10:59

Thanks. I always forget about that style of destructuring.

mdaines18:10:25

heh heh, yeah.

mdaines18:10:39

I’ve finally got to a point when I don’t have to look it up to remember the format.

camdez18:10:15

Yeah, it takes a while. Nested map destructing still takes me a second...

jeff.terrell18:10:08

@camdez: regarding a name, maybe separate-by? Or maybe bifurcate?

akiva18:10:46

splitmap as opposed to zipmap.

jeff.terrell18:10:57

@asolovyov: not really sure what you're trying to do.

camdez18:10:27

@jeff.terrell: I’m kind of fond of bifurcate, thanks. But as per discussion with @mdaines, I think I might just roll without.

camdez18:10:27

Destructuring the group-by has a hint of cognitive overhead, but not enough that I’m yet ready to introduce a new name (/ concept).

jeff.terrell18:10:27

Fair enough. :thumbsup:

ghadi20:10:10

PODS: Plain Ol' Data Structures

danielcompton21:10:30

POCO: Plain old Clojure object

clem22:10:04

Is there a convenient means of mocking a db instance with monger? The web site states that there’s “unit test support” but I’ve yet to find any documentation that goes further into this.

dominicm22:10:52

Is there a version of walk that only operates on the values themselves? I thought specter would have this, but it doesn't, which is a shame.

clem22:10:44

Take a look at the source for clojure.walk/stringify-keys — is that along the lines of what you’re trying to do?

dominicm23:10:10

@clem: Problem with that is... derp of course that works. I just have to reverse it. Thanks. I need to stop working such long hours.

clem23:10:37

heh, no problem