Fork me on GitHub
#clojure
<
2015-09-03
>
tel00:09:55

A reducer is a function like (a -> r -> r). Technically you also have it’s “default” form, but we can handle that in the notation by just passing a default r value along too

tel00:09:05

a reducer is (a -> r -> r, r)

tel00:09:14

that’s exactly what you pass to reduce, too

tel00:09:32

a transducer is a function between reducers

tel00:09:41

(a -> r -> r) -> (a -> r -> r)

tel00:09:53

technically you can change the type of the a if you like (and more things)

tel00:09:19

so you might write Transducer a b = (b -> r -> r) -> (a -> r -> r) where I’ve written it to demonstrate the “backwardness” of it

tel00:09:38

identity is a function with a type like x -> x for any choice of x

tel00:09:46

so it can definitely be a transducer

tel00:09:51

a Transducer a a

tel00:09:03

by letting x be filled in as (a -> r -> r)

tel00:09:28

I worked some of this stuff out a while back

tel00:09:43

very Haskell-y which makes various things harder to do

tel00:09:57

but the idea is very clear if you know how to read the types

tel00:09:51

the clever bit if you want to skip much of the haskell

tel00:09:10

is that a transducer is essentially “as powerful as” mapcat

tel00:09:45

a “pure” function of the form (b -> r -> r) -> (a -> r -> r) is equivalent to a -> [b]

tel00:09:02

but transducers do two or three tricks extra

tel00:09:41

they let you intermingle effects, they compose functions and therefore avoid construction of intermediate data structures, and they wedged an early termination effect inside

marcus01:09:30

Howdy. How do I realize a LazySeq returned to me by Korma? doall didn't appear to do it, so I simply used (list thing). What's the 'right' way?

amacdougall02:09:31

According to the docs, in-ns doesn't include clojure.core, so a lot of basic functions will not be available. I'm curious why that's the case, but at least it's documented. There's something that confused me a bit more, though: I have a test namespace:

(ns foo.test.db.core
  :require [clojure.test :refer :all]) ; various requires omitted
...but when I use (in-ns 'foo.test.db.core) from the REPL, I can't access the clojure.test stuff either. Particularly (run-tests). If I execute the full ns form in the REPL, I get access to all the stuff. Is this expected? Or am I doing something weird?

amacdougall02:09:12

Whoops, just figured it out. I'll leave this question up in case it ever helps some other duffer. The answer is that I was in-ns-ing to a namespace that had not been loaded yet! That is, since none of its code had been exercised as part of the standard REPL startup, it didn't "exist." It is necessary to execute the entire contents of the namespace, actually. 😦

Pablo Fernandez10:09:30

I need to have a jar file somewhere predictable when I deploy, because I need to pass it as a command line. Is it possible to copy a jar from dependencies to the uberjar when it’s being built?

martinklepsch10:09:56

@pupeno: you could put that jar into resources as part of your build process

Pablo Fernandez10:09:45

@martinklepsch: that sounds like a good approach, any ideas what to search for?

martinklepsch10:09:49

Not really tbh

martinklepsch11:09:11

you could always write some shell scripts or manually copy it

martinklepsch11:09:28

I assume you’re using Leiningen?

Pablo Fernandez11:09:22

That’s still not trivial to do. I don’t know what’s the appropriate way to invoke shell scripts either and I don’t know how to write a shell script that can run on both unix and windows and reach out to the maven repositories, wherever they happen to be on the computer and copy the files.

Pablo Fernandez11:09:26

Yes, I’m using Leiningen.

Pablo Fernandez11:09:48

But maybe I need some plug in.

martinklepsch11:09:00

Maybe you could download it during startup using something like https://github.com/cemerick/pomegranate

Pablo Fernandez11:09:51

@nberger: indeed, that looks like what I’m looking for, I don’t know how to refer to my maven repo and deal with version numbers.

nberger11:09:44

yeah, I don't know how to do that either... have you exhausted all the other options and are you really sure you need to do the manual copy?

Pablo Fernandez11:09:25

nberger: I don’t know, I rather not do the manual copying, but my other options didn’t work: http://stackoverflow.com/questions/32350244/leiningen-java-agents-not-working-in-uberjar

tcrayford11:09:36

@pupeno for the reference, Yeller has a few jars like this, and I just throw them in the filesystem in my deploy tool (which grabs em from s3)

tcrayford11:09:47

on heroku you might be able to do something similar with a custom buildpack

tcrayford11:09:10

(specifically: you want the multi buildpack, combining the default clojure buildpack and another custom buildpack that downloads the new relic jar)

Pablo Fernandez11:09:16

tcrayford: I generally prefer more self-contained solutions. I’d like my app to be more portable. My biggest problem with having the jar in a build-pack is that lein ancient won’t upgrade that jar.

tcrayford11:09:35

as far as I know it's a limitation of the jvm that you can't uberjar an agent

Pablo Fernandez11:09:37

That jar will be stuck in that version until someone happens to remember to check for an upgraded version.

tcrayford11:09:40

but that's a guess 😉

Pablo Fernandez11:09:36

tcrayford: oh, but I already did it. I put the jar in my source control and the passed -javaagent:newrelic/newrelic.jar and it worked. I don’t want the jar in my source control for the same reasons, it’s out of the build system and thus won’t get updated.

tcrayford11:09:57

right - I meant "uberjar in the same jar as your app" 😉

Pablo Fernandez11:09:05

I just need something to automatically copy it from $MAVEN_REPO/om/newrelic/agent/java/newrelic-agent/$LATEST_VERSION/newrelic-agent-$LATEST_VERSION.jar to $SOME_DIR/newrelic-agent.jar at uberjar build time.

tcrayford11:09:56

yeah, so a buildpack or a lein plugin could do that for sure (pretty sure java can copy some files on the filesystem), but you'd have to write either of those yourself I think

Pablo Fernandez11:09:07

I find it surprising that New Relic itself advocates having a jar on the source tree. In Ruby/Rails it’s a gem that gets updated with all the other gems, it’s part for the checks for security vulnerabilities, etc.

tcrayford11:09:25

it's a java agent limitation thing - agents have to be seperate jars from the app

Pablo Fernandez11:09:40

tcrayford: lein plug in seems to be the way to go. https://github.com/m0smith/lein-resource looked promising, but it’s not working.

tcrayford11:09:42

(and if they didn't write it as an agent it couldn't get at certain information because of the jvm's security model)

tcrayford11:09:08

fork it and fixit 😉

Pablo Fernandez11:09:37

@tcrayford: of course. Two things are stopping me: I don’t understand the error yet (when the plug in is present, my code fails to compile) and I’m not sure it’s actually a viable solution as it’s not clear whether it can copy files from the maven repository.

Pablo Fernandez11:09:31

Oh, Heroku is not picking it up from the uberjar, it’s picking it up from a copy of the source code 😞

bernhard12:09:02

Hey clojurians, anyone managed to get a web service running with ring that accepts gzipped request bodies? Wondering if it only works in our Dropwizard services because they implemented it themselves... (http://stackoverflow.com/questions/31555633/jetty-responds-with-400-bad-request-when-request-content-type-is-gzip)

nberger12:09:28

Sorry I'm not following everything but perhaps :scope "provided" would help in the sense that it won't be able to "pick it from the source code"?

Pablo Fernandez12:09:01

nberger: in Heroku, in my app, I can see the source code of my app with a target directory with uberjar. When the jar is my source control, it’s naturally on my Heroku app as well, but I’m trying to avoid that.

Pablo Fernandez12:09:11

nberger: does that make sense?

Pablo Fernandez13:09:32

I almost got my jar-copying plug in.

roberto13:09:36

any reason why you can’t pull the jar from maven?

roberto13:09:00

Or push the jar to your own registry using something like artifactory or nexus?

Pablo Fernandez13:09:39

roberto: I could download it from maven, but maven already downloaded it to the local repo, why should I download it again and figure out how to talk to maven? About the second question, I’m not sure I follow, do you mean my own maven repository?

roberto13:09:25

hmmm, I don’t think I understand what the issue is, if you are already using maven to download the jar

roberto13:09:29

why are you copying it around?

Pablo Fernandez13:09:32

roberto: I don’t want to have my own repository, I want to pick the jar from the central maven repository. One of the reasons why I’m doing all of this is to make sure I’m picking the latest jar, so that lein ancient upgrades it. Besides, having my own repository still leaves me figuring out everything else (doesn’t solve the problem of having the jar in a well known location at runtime)

Pablo Fernandez13:09:24

roberto: I need the jar to be present in my production server in a well known location so I can pass -javaagent:path-to-the-library.jar so that the JVM knows where it is to pick it up and load it as an agent.

Pablo Fernandez13:09:45

This code feels horrible: https://github.com/carouselapps/jar-copier/blob/master/src/leiningen/jar_copier.clj#L20-L28 It essentially checks that :jar-copier {:java-agents: true, :destination “blah”} exists in the project.clj. Any advice on how to make it more idiomatic, easier to read, etc?

Pablo Fernandez14:09:23

I'm doing lein release for this library that I just wrote: https://github.com/carouselapps/jar-copier/blob/master/project.clj and I'm getting the error gpg: skipped "J. Pablo Fernández <[email protected]>": secret key not available, which is confusing, because that’s not the email address in my key which I specified in project.clj: http://keys.gnupg.net/pks/lookup?op=vindex&amp;search=pupeno%40carouselapps.com. Any ideas what’s going on?

borkdude15:09:44

suggestions how to generate PDFs in clojure from html, similar to https://github.com/mileszs/wicked_pdf ?

borkdude15:09:06

I was thinking of calling http://wkhtmltopdf.org/ myself

borkdude15:09:14

unless someone has a nicer solution

ragge15:09:32

@pupeno: have you uploaded your key to clojars?

Pablo Fernandez15:09:23

It’s failing well beyond that, as far as I know. It’s not finding the private key when signing because it’s looking for one that doesn’t exist.

borkdude16:09:20

@robert-stuttaford: that doesn't really use html. I need to render the same html/css as in the browser (sort of) but to pdf. I'm now trying https://github.com/gberenfield/ring-wicked-pdf

yogthos16:09:40

@borkdude I think some people also used enlive with clj-pdf

borkdude16:09:08

@yogthos: what is the advantage of this library compared to others?

yogthos16:09:59

I find using hiccup style syntax works really well for describing PDF docs

borkdude16:09:38

@yogthos: I can also write hiccup and render to pdf using html. Does your approach offer anything else that I'm missing?

borkdude16:09:05

@yogthos: I'll try and see how far I get with both libs

yogthos16:09:37

clj-pdf is explicit about what gets generated, but you can just look at the examples in the readme to see if it's a good fit or not :-)

nathan.schile17:09:31

@borkdude: I’ve used wkhtmltopdf by simply calling the executable. Has worked well so far.

jstaffans18:09:24

I find myself writing this a lot:

(if (vector? y)
  (into [] (map #(assoc {} :x %) y))
  (assoc {} :x y))

jstaffans18:09:23

Basically I want the apply the same operation to each element in y if y is a collection as to y itself if not.. Anyone got a better idea than using if?

cigitia18:09:42

(into base-map xform source-collection) is the idiomatic way to apply a transducer to a source-collection and, assuming that the transducer yields map entries, yields a new map based on the given base-map. This will overwrite any vals in base-map that share keys with any entries that the transducer outputs. But what if I want to merge vals in base-map with those in the entries that the transducer outputs, rather then overwriting them? I could write a custom reducing function (one that uses transients, like into does) and use transduce, but I suspect that there’s an easier, more idiomatic way.

jonpither18:09:09

@jstaffans: I haven't used it , but I wonder if Spectre would help. Can't wait to use it myself

darwin18:09:35

@cigitia: maybe producing a map from transducer and then using merge-with?

darwin18:09:05

but writing custom reducing function would be probably better optimized

darwin18:09:20

@jstaffans: why don’t you make a helper function out of that snippet?

darwin18:09:58

#(assoc {} :x %) can be an op parameter

exupero18:09:24

That may give you some ideas for abstracting it. @darwin's suggestion is good.

amacdougall18:09:05

I could use some advice on converting from clj-time to SQL timestamps when using YeSQL. It's easy to convert to the timestamp format:

(let [timestamp (clj-time.core/now)]
  (db/create-user! {<other data>, :created_at (clj-time.coerce/to-sql-time timestamp)}))
...but since YeSQL generates a get-user function, the only way I see to get the corresponding from-sql-time is to write a bunch of functions wrapping the ones automatically generated by YeSQL, which seems to kind of defeat the purpose. I have a feeling I'm missing something that would make my problem go away.

meow18:09:39

@cigitia: something like? (merge base-map (into {} xform source-collection))

meow19:09:23

but you want it to merge during the transducing, not as a separate step

cigitia19:09:45

It’s fine; I wrote my own reducing function. Thanks for the help!

meow19:09:55

wait, you want to merge values when there are duplicate keys, right?

cigitia19:09:02

Yes, that’s right.

meow19:09:20

yeah, seems like a custom reducing function is the way to go

meow19:09:24

since what it means to "merge" the duplicate values needs to be determined somehow

exupero19:09:34

merge-with?

meow19:09:55

I was thinking that update might be useful https://clojuredocs.org/clojure.core/update

meow19:09:26

Then you just supply update with a custom function that knows how to do the merging.

meow19:09:31

And since update was added to 1.7 that might be a good sign that it's useful as the reducing function in tranducers

meow19:09:13

@cigitia: Have you looked at update?

meow19:09:12

@exupero: I believe that update is the reducing function form that corresponds to merge-with which operates on the entire map.

meow19:09:18

But now I'm not so sure...

meow19:09:43

Just when you think you've got things all figured out... 😞

meow19:09:49

This really shouldn't be that hard to figure out since this is clearly the bailiwick of transducers.

meow19:09:51

I'm so used to just relying on conj as the reducing function.

meow19:09:50

Now I believe I'm wrong, and update is merely filling in the gap of needing a single-level version of update-in

meow19:09:45

But the question remains: how to get update/update-in functionality within a tranducable process without having to write a custom reducing function?

meow19:09:03

Which is what merge-with would do, and I'm just being stupid, right?

meow19:09:32

I just hope there is some entertainment value in all the stupid things I say or do...

borkdude20:09:04

where should one look if you get an error 500 from a ring application, but there is no exception printed in the console, nor do I get it back in html?

borkdude20:09:23

I'll try it, thanks @akiva

meow20:09:40

I still haven't cracked this nut. Anyone got a better idea? This didn't work:

(transduce identity (partial merge-with +) {:a 1 :b 2} {:b 3 :c 4})
ClassCastException clojure.lang.Keyword cannot be cast to java.util.Map$Entry  clojure.core/key (core.clj:1496)

borkdude20:09:01

I found the exception by adding ring.middleware.stacktrace to the mix.

borkdude20:09:09

in lib-noir I got that one for free I think, never thought about it

ecelis20:09:57

borkdude: I use timbre logging to print to the console, stacktraces or any other messages I want printed

borkdude20:09:51

@ecelis: ring.middleware.stacktrace does that too

ecelis20:09:08

maybe I should take a look at that too

borkdude20:09:38

well, it's better to turn off ring.middleware.stacktrace in production, so your solution is probably better

aengelberg20:09:58

@meow: how about (transduce identity (partial merge-with +) {:a 1 :b 2} [{:b 3 :c 4}])

aengelberg20:09:13

Recall that reduce and transduce take a list of elements to reduce onto the starting value

meow20:09:29

@aengelberg: aha! thank you! simple_smile

meow20:09:39

(transduce identity conj {:a 1 :b 2} [{:b 3 :c 4}])
=> {:a 1, :b 3, :c 4}
(transduce identity (partial merge-with +) {:a 1 :b 2} [{:b 3 :c 4}])
=> {:a 1, :b 5, :c 4}

meow20:09:27

What threw me was this worked:

(transduce identity conj {:a 1 :b 2} {:b 3 :c 4})
=> {:a 1, :b 3, :c 4}

meow21:09:45

More fun with transducers. They keep getting easier.

aengelberg21:09:01

One of the reasons I don't see why conj can operate on tuples or maps.

aengelberg21:09:27

Rather, conj can operate on maps and tuples, or maps and maps.

bronsa21:09:55

i regularily ask myself that, find the answer and forget

bronsa21:09:13

all i remember is that there is a good reason for that

tel22:09:24

is it really the case that some and every?

tel22:09:35

or is there something I’m missing

ghadi22:09:31

tel: I didn’t follow your question

tel22:09:57

some and every? seem to be parallel functions

tel22:09:04

despite the difference in naming convention

tel22:09:11

whereas some? is different?

ghadi22:09:29

also some-fn and every-pred

ghadi22:09:38

some? is more recent

roberto22:09:55

they are different

ghadi22:09:06

are you wondering about the question mark?

roberto22:09:08

some stops as soon as one value is true

roberto22:09:16

every? if all of them

tel22:09:17

and just the general naming convention

bronsa22:09:39

@tel some returns a value, every? returns a boolean

tel22:09:02

that’s the logic

jtackett22:09:46

Looking for the best clojure finite state machine library, any advice?

beppu23:09:59

Is anyone here using RabbitMQ with Clojure? Any library recommendations (or warmings)?