Fork me on GitHub
#clojure
<
2016-02-18
>
Chris O’Donnell00:02:15

@george.w.singer: I think people generally mean the type of the input arguments and return value when they refer to the signature of a function.

alexisgallagher00:02:03

Is @stuartsierra 's component framework very widely used?

alexisgallagher00:02:42

I am using it for the first time and I'm having trouble understanding some aspects of it, or wondering why they are not a bit simpler. In particular, the component vs system dichotomy feels a bit odd. And I don't understand why components don't declare their dependencies by implementing something that's part of the Lifecycle protocol, rather than using metadata and special functions.

alexisgallagher00:02:18

Are there particular reasons for these aspects of the design? Any good pointers on how to use it well? Simpler variations that are now all the rage?

schmir00:02:02

alexisgallagher: yes, it's widely used.

alexisgallagher00:02:47

ok. good to know.

schmir00:02:47

alexisgallagher: https://github.com/tolitius/mount looks like an interesting alternative...though I didn't use any of them

alexisgallagher01:02:06

The discussion on that reddit thread is quite insightful.

alexisgallagher01:02:10

Actually, scratch that. It's super duper insightful.

ghadi01:02:19

IMHO the mount proponents have missed the point, but that thread covers it better than I can summarize here.

ghadi01:02:58

we're back to using global variables everywhere with mount

tolitius03:02:09

@ghadi: I would not jump the global variables everywhere misconception gun, and that is not at all what this reddit thread concludes. simple sometimes takes time to understand and appreciate. Clojure vars are not bad, they are good. They are general purpose. And if the one does not like keeping states in Clojure vars there is always an alternative: https://github.com/tolitius/yurt

alexisgallagher03:02:04

tbh I find the fundamental nature of the var to be one of the things that puzzles me most, despite that I've been using clojure on and off for years. They're one of the concurrency primitives. But they also seem to be used to define pretty much every single constant thing in the language (like function definitions). I think it's the number of indirections in chains like name -> symbol -> var -> atom -> value that confuses me.

alexisgallagher03:02:29

Perhaps I need to process it in meditation.

dmitrig0103:02:45

Have had some troubles recently with macros. Reduced it down to a very simple test case:

dmitrig0103:02:51

app-application.core=> (let [foo inc]  (eval `(~foo 1)))
2
app-application.core=> (let [foo (memoize inc)]  (eval `(~foo 1)))

IllegalArgumentException No matching ctor found for class clojure.core$memoize$fn__5708  clojure.lang.Reflector.invokeConstructor (Reflector.java:163)

dmitrig0103:02:08

am I doing something wrong here?

tolitius03:02:16

@alexisgallagher: very true. they can be abused, as anything else can. meditation will definitely help to be the one with the force simple_smile

kul03:02:08

Anyone using Aleph http handler in production?

kul03:02:09

I dont see how to configure backpressure for requests?

kul03:02:59

Is there something like :queue-length in http-kit

tolitius03:02:26

@kul: yes. http-kit puts all the requests to the bounded queue

tolitius03:02:48

it's size is controlled with a :queue-size option

kul03:02:08

I mean is there something similar in aleph

tolitius03:02:26

there is also :thread that is the number of threads to process responses

tolitius03:02:59

ah.. I am not sure about aleph

kul03:02:06

Http-kit is no longer maintained

kul03:02:39

So for http-kit is it a dropping queue?

kul03:02:16

Or sliding?

tolitius03:02:42

I believe it is just a LinkedBlockingQueue, but this can be checked in sources..

kul03:02:52

interesting, i will check it out. Thanks @tolitius

kul03:02:05

Hurmm weird, so the excess requests will be blocked

tolitius03:02:39

right, but that is only if the queue fills up, which would mean something is off with the balance of writes vs. reads

kul03:02:12

Yep take a busy server for instance

tolitius03:02:19

i.e. if it did not block, there is a risk of blowing up the heap, getting out of file handles, etc..

tolitius03:02:45

busy does not mean reads are not catching up, just means there are lots of writes and reads simple_smile

tolitius03:02:38

that's why there is also a :thread option that could scale the "readers"

tolitius03:02:08

similar to netty's number of worker threads

kul03:02:18

So my problem here is that there is no option of dropping requests here

kul03:02:23

With a timeout code may be from the handler itself

shaun-mahood03:02:02

@jonahbenton: thanks, didn't think of that one!

tolitius04:02:28

@kul, I remember there is a timeout option (on the server start) that is applied to requests..

tolitius04:02:21

@kul, do you have a problem, or you are anticipating one?

tolitius04:02:22

have not explicitly used them with http-kit, which means that its defaults worked quite well for me simple_smile

tolitius04:02:24

@kul, actually in case the queue fills up, requests won't be blocked, they will be rejected with HTTP 503: https://github.com/http-kit/http-kit/blob/0e53d3d883901a229c189c263b2a4f6c07e66a0e/src/java/org/httpkit/server/RingHandler.java#L170

kul04:02:13

Nice thanks @tolitius i will have a look

mhjort05:02:00

@kul it's true that original author doesn't have time to maintain http-kit project at the moment. However, there is some activity to get the project back to live.

mhjort05:02:14

I work as a collaborator for http-kit and really hope people still send PR s

seancorfield06:02:52

That's good to hear @mhjort as we'd looked at http-kit but were a bit concerned about that...

ordnungswidrig08:02:41

{[] :c '() :b} => IllegalArgumentException Duplicate key: [] clojure.lang.PersistentArrayMap.createWithCheck (PersistentArrayMap.java:71)

schaueho08:02:32

@alexisgallagher: you might want to check out weavejester's approach to using component, cf. https://www.booleanknot.com/blog/2016/01/24/how-i-use-component.html

jethroksy08:02:07

anyone using yesql/hugsql with a n:m relationship on postgres that I can look at?

kul09:02:54

is there a ring middleware for logging req in standard formats

kul09:02:02

e.g. catalina

slotkenov10:02:05

Someone experience with the client side generated clojure code from swagger? I’m trying to see how to incorporate it into my project. Should I use it as a separate library somehow, or would it be better to put the code directly into my project?

slotkenov10:02:17

I just copied the result into the src dir of my project. Works for now. I can look into automating that process

bronsa12:02:54

@ordnungswidrig: that's a result of the unfortunate decision of having vectors compare equal to lists

nberger12:02:53

@kul I don't know much about Catalina, so I don't know if/how it's similar to it, but I wrote https://github.com/nberger/ring-logger (started as a fork of http://github.com/pjlegato/ring.middleware.logger, it's quite different nowadays)

larhat12:02:05

@nberger: is it known bug/feature that ring-logger throws exception on urls like "smth/?hello"? https://gist.github.com/910d3696a06efef269c9 — here is the diff with offending request

nberger12:02:13

@larhat it was not known to me. Feel free to create an issue/send PR, I'll take a look into it soon anyways

alexisgallagher14:02:27

@schaueho: thanks. Will check it out. I spent longer than I care to admit wrestling with the components + jetty + routes issue, and I feel I always learn something from wesvejester's taste.

timgilbert16:02:03

Hey, I'm looking for a way to make a short bit of code a little more elegant/idiomatic. I have a function that will return either nil or a value, and I want to map it over some input and collect the non-nil values in a seq. So for purposes of an example, the function could be defined as (defn f [x] (rand-nth [x nil])) What I've got right now is (remove nil? (map f [:a :b :c :d])), which doesn't seem terrible and is pretty clear, but I keep thinking there's a way to make it more concise by using reduce.

jswart16:02:55

you can do (filter identity (map f …) I believe

jswart16:02:48

with reduce just make f look at the incoming input value and if it isn’t nil conj it onto the accumulator

ddellacosta16:02:55

reading the article http://danboykis.com/?p=2263 ("The Beauty of Clojure”), and I’m struggling to understand something—why do people think protocols and deftype/defrecord are OO?

timgilbert16:02:58

Yeah, that's what I was thinking, I guess I was looking for a version of conj that doesn't append nil

ghadi16:02:01

(keep f coll)

ghadi16:02:14

timgilbert: ^

timgilbert16:02:29

Aha! That sounds perfect

timgilbert16:02:56

That is perfect! Thanks @ghadi

jswart16:02:31

And if you want a version of keep that returns only truthy values:

ghadi16:02:11

>Returns a lazy sequence of the non-nil results of (f item). Note, this means false return values will be included. f must be free of side-effects.

ghadi16:02:24

note keep's semantics.

timgilbert16:02:50

Hmm, I actually do have side effects inside the definition of f, I'm doing some database work over a batch of inputs and then trying to accumulate errors

jswart16:02:01

you could use what I pasted above in the snippet wrapped in a call do doall, (doall (reduce …) or you may want to use doseq and collect the results by swap!ing into an atom

jonahbenton16:02:48

hey @ddellacosta - there are some superficial resemblances between protocols and records/types and Java interfaces and classes and if you are coming from an OO mindset where those specific features are all you see, then protocols and records/types can look OO to you.

ddellacosta16:02:35

jonahbenton: I see. They have always seemed a lot more similar to typeclasses in Haskell to me

ddellacosta16:02:04

as the types they are applied to are data types, not objects, and state is clearly separated

roberto16:02:51

yeah, I’ve had similar impressions. I found it easier to reason about it if I thought of them as typeclasses.

ddellacosta16:02:36

seems to render the point of that article kind of defunct

roberto16:02:11

yeah, I think so too. They miss the point of OO. It was never about classes and interfaces in the C++/Java way.

roberto16:02:34

I believe Allan Kay even said that Lisp, Erlang and SmallTalk were the only OO languages. So implying that defrecords and protocols are the equivalent of Classes and Interfaces in Java, in my opinion, make clojure less OO.

ddellacosta16:02:12

I dare not dive into the quagmire of “what is and isn’t OO” 😄

danboykis16:02:53

@ddellacosta I wrote that article, why do you think it's defunct?

ddellacosta16:02:15

danboykis: well, if you’re starting with the premise that Component is not the right choice because it is too OO, and it’s not actually OO, then the rest of the argument seems to lose it’s motivation.

ddellacosta16:02:18

but! I will be clear that I don’t actually know what OO is. So maybe it is too OO.

ddellacosta16:02:59

more generally—not specific to your article—I find a lot of these articles on Mount vs. Component to be uncompelling

ddellacosta16:02:33

component and mount both seem to allow a ton of flexibility, so it seems hard to evaluate on the grounds of how “truly Clojure-like” one or the other is

roberto16:02:51

yeah, I’m generally wary of articles that try to highlight how great some library is by casting aspersions on another one

ddellacosta16:02:57

and it can descend into No true Scotsman territory really easily

danboykis16:02:00

@ddellacosta: I agree OO is not one single thing (kind of like FP isn't either) but once you start creating data types with methods and Dependency Injection graphs, I call that OO

ddellacosta16:02:43

danboykis: fair enough, but I’m referencing your original article, where you say > However, after using component for a project at work, I noticed that my code stopped looking like idiomatic Clojure code and more like OO Java I used to write. While features like reify, defprotocol, deftype, and defrecord exist they exist for the purposes interop with Java, type extensions and library APIs. In my opinion the bulk of Clojure code should strive to utilize functions and be data oriented.

danboykis16:02:59

@ddellacosta: if you like component I'm not advocating you don't continue using it, if it's not compelling to you that's fine

ddellacosta16:02:14

and the implication there is that reify, defprotocol, defrecord are OO somehow—that’s all.

danboykis16:02:24

@ddellacosta: that blog post reflected my experience, I even said it in the beginning that I am biased

ddellacosta16:02:00

I’m not trying to be contentious here. Furthermore, I’m completely undecided on any of these things—I haven’t actually used either significantly, although we are trying out component now to see if it helps with structuring our system.

ddellacosta16:02:56

Mostly I wanted to point out that defprotocol, deftype and etc. are very much data oriented and functional, and as such not grounds for dismissing something as OO. That’s it!

ddellacosta16:02:20

…at least, as far as I understand. That’s why I asked why people considered those features to be OO

danboykis16:02:27

I consider ApplicationContext like DI to be OO, you may disagree but I think it's generally accepted as a OO pattern

ddellacosta16:02:56

haha, I have no idea what ApplicationContext is

roberto17:02:36

DI isn’t OO, everytime you pass an argument to a function, that is DI

roberto17:02:10

if you are talking about spring’s magical DI, then that is another thing. And component is nothing like spring.

danboykis17:02:12

it's about classes and methods

ddellacosta17:02:52

danboykis: I’m not saying you can’t use it for something that can be called OO. The point is that those Clojure language features very much support functional programming.

ddellacosta17:02:29

and yeah, talking about patterns is not going to convince me we’re talking about OO—pattern is too vague of a notion to tell me anything

tolitius17:02:11

the article does not claim that the use of records and protocols is not idiomatic, although it is quite an easy angle of attack when paraphrasing (@ddellacosta nothing you've said, but a few often do). In fact both mount and yurt use deftypes, records and protocols which are great

ddellacosta17:02:54

tolitius: hmm…the chunk I quoted seems to be saying that pretty explicitly.

danboykis17:02:06

ddellacosta: I think you can do FP in a lot languages that doesn't mean they are FP

ddellacosta17:02:31

I want to be clear here though—I’m not advocating component over mount. I have no dog in this fight.

tolitius17:02:44

@ddellacosta: no, it is not talking about records and protocols, but rather about component. there is a difference

danboykis17:02:48

@ddellacosta: I agree, I am not talking about them eitehr

ddellacosta17:02:49

danboykis: think we’re well off the point I started with, so I’m taking my leave of this debate.

ddellacosta17:02:03

sorry to start an argument—forgive me. cheers all 😄

roberto17:02:24

yeah, I’m confused at this point too about what was the point of that statement. Have to get back to work.

tolitius17:02:33

clojure protocols and records are general purpose tools. they can be misused as anything else can.

jonahbenton17:02:33

hey @danboykis i missed the discussion but thought it was a good piece, it's always useful to highlight differences between approaches with real code. thanks for writing it.

chrisn17:02:55

(defn dense-element-multiply!
  [alpha a-data a-offset x-data x-offset beta y-data y-offset op-len]
  (let [^long data-len op-len
        ^long a-offset a-offset
        ^long x-offset x-offset
        ^long y-offset y-offset
        ^double alpha alpha
        ^double beta beta
        ^doubles a-data a-data
        ^doubles x-data x-data
        ^doubles y-data y-data]
    (dotimes [idx data-len]
      (aset y-data (+ idx y-offset)
            (+ (* beta (aget y-data (+ idx y-offset)))
               (* alpha (aget x-data (+ idx x-offset))
                  (aget a-data (+ idx a-offset))))))))
public static void alphaAXPlusBetaY( int op_len, double alpha,
					 double[] a, int a_offset,
					 double[] x, int x_offset,
					 double beta,
					 double[] y, int y_offset)
    {
	for (int idx = 0; idx < op_len; ++idx) {
	    int y_off = y_offset + idx;
	    y[y_off] = alpha * x[x_offset + idx] * a[a_offset + idx] + beta * y[y_off];
	}
    }
What do you suppose the performance difference is between these two version of the same function, one in java and one in clojure?

echristopherson18:02:47

you're asking what accounts for actual measured difference?

chrisn18:02:16

Well, I was first asking for guesses as to the performance difference across a range of vector lengths 10-10000.

pbostrom18:02:27

if I had to guess they would be pretty close

chrisn18:02:55

OK, that is one guess. Any others?

chrisn18:02:45

And more productively I was wondering if someone saw something I did that would dramatically change this comparison.

dm318:02:31

the best I got on tight loops and array ops was within ~5x of a java while

chrisn18:02:57

So are you proposing that a while loop will perform better than this dotimes?

cmcfarlen18:02:09

perhaps something is getting boxed?

chrisn18:02:45

That is reasonable but you will notice the code compiles clean with these warnings:

(set! *warn-on-reflection* true)
(set! *unchecked-math* :warn-on-boxed)

chrisn18:02:08

sorry, with these compiler directives.

chrisn18:02:48

Here is the test code:

(defn perf-test-element-multiply
  []
  (doseq [elem-count [10 100 1000 10000 100000]]
    (println "elem-count" elem-count)
    (let [^netlib_ccm.core.DenseVector y (m/array :netlib (repeat elem-count 1))
          ^netlib_ccm.core.DenseVector x (m/array :netlib (repeat elem-count 2))
          ^netlib_ccm.core.DenseVector a (m/array :netlib (repeat elem-count 3))]
      (print "clojure:")
      (time (dotimes [iter 1000]
              (nc/dense-element-multiply! 1.0
                                          (.data a) 0
                                          (.data x) 0
                                          1.0
                                          (.data y) 0
                                          elem-count)))
      (print "java   :")
      (time (dotimes [iter 1000]
              (Ops/alphaAXPlusBetaY elem-count 1.0
                                    ^doubles (.data a) 0
                                    ^doubles (.data x) 0
                                    1.0
                                    ^doubles (.data y) 0))))))

chrisn18:02:42

elem-count 10
clojure:"Elapsed time: 0.263458 msecs"
java   :"Elapsed time: 0.124357 msecs"
elem-count 100
clojure:"Elapsed time: 1.909401 msecs"
java   :"Elapsed time: 0.259004 msecs"
elem-count 1000
clojure:"Elapsed time: 18.399427 msecs"
java   :"Elapsed time: 1.52352 msecs"
elem-count 10000
clojure:"Elapsed time: 182.669039 msecs"
java   :"Elapsed time: 14.149774 msecs"
elem-count 100000
clojure:"Elapsed time: 1845.416527 msecs"
java   :"Elapsed time: 143.310439 msecs"

jonahbenton18:02:46

hey @chrisn would be interestin to see the disassembly, e.g. http://insideclojure.org/2014/12/15/warn-on-boxed/

roberto18:02:13

You are also doing the same calc more than once in some cases

chrisn18:02:37

Do you mean the index calculation?

chrisn18:02:14

for Y? You propose that using a let in the loop will speed it up by a factor of 10?

chrisn19:02:25

@jonahbenton: there it is. The clojure version is a lot lot more complex.

chrisn19:02:34

Definitely due to casting/boxing it looks like.

jonahbenton19:02:35

@chrisn yeah- lots of what should be unnecessary casting. have to jump off, will try to re-engage later

Alex Miller (Clojure team)19:02:08

why are you letting the primitives rather than typehinting them in the signature?

chrisn19:02:59

You can't type hint function signatures longer than 4.

Alex Miller (Clojure team)19:02:16

there are workarounds for that

chrisn19:02:36

OK, sounds plausible.

Alex Miller (Clojure team)19:02:01

that's definitely affecting your times to some degree

Alex Miller (Clojure team)19:02:14

you can group all the longs into a long[] for example

Alex Miller (Clojure team)19:02:23

and there are other more tedious paths (like declaring an interface with primitive methods, and a deftype that holds primitives and implements that interface)

chrisn19:02:54

OK, so to work around what appears to be a compiler bug with let you can do some work arounds like definterface and reify?

chrisn19:02:06

I have used the arrays-of-longs before.

Alex Miller (Clojure team)19:02:53

it's not a compiler bug, it's an implementation choice

chrisn19:02:29

I would expect a let with type hints to work the same as a type hinted function once past the let.

Alex Miller (Clojure team)19:02:37

IFn has one arity for every combination of Object, double, and long - this is a combinatorial explosion of arities

chrisn19:02:44

That is beside the point.

Alex Miller (Clojure team)19:02:53

3^4 was the pragmatic stopping point

chrisn19:02:05

Yes, I agree with that logic.

Alex Miller (Clojure team)19:02:08

or whatever the math is

chrisn19:02:45

If you read the assembly you will see that the casting happens in the loop after all variables have been type hinted.

Alex Miller (Clojure team)19:02:15

yeah I saw that, I was trying to figure out what was causing that

Alex Miller (Clojure team)19:02:34

instead of using ^long on the let bindings, you might try instead doing "x-offset (long x-offset)" and see if that yields a different result

chrisn19:02:06

OK, trying that.

Alex Miller (Clojure team)19:02:51

there could be a bug here, not sure

chrisn19:02:01

That got a lot closer...

chrisn19:02:23

elem-count 10
clojure:"Elapsed time: 0.168278 msecs"
java   :"Elapsed time: 0.150324 msecs"
elem-count 100
clojure:"Elapsed time: 0.648329 msecs"
java   :"Elapsed time: 0.307831 msecs"
elem-count 1000
clojure:"Elapsed time: 5.586705 msecs"
java   :"Elapsed time: 1.514433 msecs"
elem-count 10000
clojure:"Elapsed time: 44.18003 msecs"
java   :"Elapsed time: 14.140744 msecs"
elem-count 100000
clojure:"Elapsed time: 433.436932 msecs"
java   :"Elapsed time: 143.524602 msecs"

Alex Miller (Clojure team)19:02:29

can you post the disassembly for that? just the method part

chrisn19:02:53

going to lunch

bronsa19:02:56

no bug there, type hints are not casting ops

bronsa19:02:57

(let [^long x x] ..) is semantically different than (let [x (long x)] ..). in the first we're telling the compiler "treat x as a long", meaning it will be cast to long when it's going to be used, in the second we're actually casting x t long

cmcfarlen19:02:12

@bronsa: If its type hinted, should it still cast?

cmcfarlen20:02:26

If I type hint a var should the compiler still cast to that type it when its used?

bronsa20:02:57

I can't reply to the "should it" question, my opinion probably differs from that of Rich. I can only tell you that this is what the implementation does

bronsa20:02:30

when you type hint a local, that will result in a cast at the call site. one cast for each time that local is used in an interop/prim call

cmcfarlen20:02:36

hmm, interesting. That is not what I would expect to happen. Thanks for the info

bronsa20:02:56

you're not alone in that assumption 😉

lsankar403320:02:25

what’s the preferred way to apply a transducer to a channel? right now I’m creating the channel to be returned, passing in the transducer and then just pipe into it, but that seems a bit janky

jr20:02:06

you can use map if you’re just transforming data. If the transducer performs a filter then piping is the preferred method

chrisn20:02:26

@bronsa, @alexmiller thanks for looking at this and I think bronsa's description of the difference between ^long and (long x) is very clear. That is a very useful bit of knowledge along with the use of no.disassemble will really help a lot.

codemartin21:02:16

What is the rationale for not having strings in Clojure be seqs, like lists, vectors, etc? I don't see why it could not work, nor that it would be a performance issue. Curious to learn

Alex Miller (Clojure team)21:02:39

it would be a performance issue

Alex Miller (Clojure team)21:02:59

strings are seqable and so can be used to produce a seq if desired, but are far more efficient being stored using Java strings in the JVM

Alex Miller (Clojure team)21:02:09

The Google Summer of Code 2016 organization application is due tomorrow and David Nolen and I worked on completing it today. However, this requires community help to get anywhere. Right now we need: 1) project ideas http://dev.clojure.org/display/community/Project+Ideas+2016 2) mentors for those ideas Presuming Clojure is accepted as an org, students will then be able to propose those projects in March.

Alex Miller (Clojure team)21:02:45

there is also now a #C0N1QHE3W channel