Transducing processes like clojure.core/transduce, into, sequence, etc. are supposed to be able to handle Reduced objects that are returned by their reducing functions’ step arities. However, is the same true for Reduced objects that are returned by their RFs’ initialization and completion arities?

Why this matters for me: I’m making a transducer, and the RFs that it creates are intended to insert a bunch of items into the process when it completes. When the upstream RF is completed and calls my transducer’s RF’s completion arity, my transducer’s RF will then call its downstream RF’s step arity multiple times, using it to reduce the items it wants to insert into the upstream accumulation—creating another accumulation—before it finally calls the downstream RF’s completion arity on that accumulation.

Of course, the downstream RF’s step arity might return a Reduced—for instance, if the downstream RF was created with (take 2). This means that my transducer’s RF’s completion arities might in turn return a Reduced, if their internal reductions got cut off by the downstream RF. Nowhere in http://clojure.org/transducers is this explicitly allowed or disallowed, but it seems to make sense. However, clojure.core/transduce, into, and sequence do not seem to handle Reduceds when they are returned by completion arities.

Is this a bug of transduce, etc.? Should they be able to handle Reduceds from completion arities, not just step arities? Or should completion arities never return Reduceds? There are similar questions about initialization arities. I personally think it does make sense for completion arities to be able to return Reduceds, even if it is a little redundant, but either way, clarification would be wonderful.

In short, no.

This has come up in the past even with just reduce and generally it is only checked on the return of the transformation function/arity, not on initialization or completion

I have discussed this with Rich in the past so consider this "as designed"

For completion this was at one point relied on (not unrolling reduced) for an expanding transducer which does an inner reduce but that aspect changed a couple of times so can't remember what's necessary now from memory

I would be happy to look at specific code next week when I'm off vacation to give more specific advice (and double check what I'm saying when looking at source and notes)

I'd also be happy to clarify the docs on the site - these are good and tricky questions

So ping me next week if you want

@cigitia: ^^

@alexmiller: Thanks super much for the explanation, gosh; that clarifies a ton. I'll ping next week.

hi guys is there anyway that we can configure clojure.tools.logging to output to nRepl ?

@nxqd: A similar question has come up in the mailing list: https://groups.google.com/d/msg/clojure/iRDFUzRF5Ek/lQiQxdQfBQAJ This might help

@kimsnj: thanks for pointing out. I did read the discussion, but there is particular answer to output log to nRepl.

Hey everyone!… I am stuck with a routing problem using the middlewares macro of compojure-api. I want to wrap a route with an auth middleware, which requires a config/setup map and I can’t find a way to pass this map using the middlewares macro. The code is super simple:

Of course, this (wrap-set-auth-user auth-config) is not working.

@rcanepa: What's the problem? It should work.

@juhoteperi: Nevermind, It was a compile error because I was passing the wrong config map (one that was not visible from that part of the code).

Also, you use :middlewares metadata on context*

Mmm that sounds even better

This :middlewares [(wrap-set-auth-user auth-tokens) wrap-authentication] throws a compile error while if remove the (wrap-set-auth-user auth-tokens) it doesn’t.

Is the :middlewares metadata keyword a shorcut to the middlewares macro?

@rcanepa: Yes. Though I remember seeing a compile error few times which is probably caused by some macro magic.

@juhoteperi: Should I create an issue on github?. I could create an easy to replicate case.

If the error is about arity, a workaround might be to add additional no-op arity to the middleware. If there is a problem, the wrong arity is only called compile time.

@rcanepa: Yes please.

@juhoteperi: Ok, I will. However, the error trace doesn’t say nothing about an arity problem.

@rcanepa: Okay, might be different error than one I remember.

<!everyone> happy new year from siberia !!!

@ir4y: happy new year!

(inc 2015)

@ir4y: Happy new year ! (from France )

@ir4y: happy newyear!

Happy New Year from Ukraine! 🎄

this is somehow the only slack group where people dont get slammed for calling channel or everyone

Happy New Year from St. Petersburg 🎄

which is quite refreshing

Huh, I'm kind of surprised though - I thought that was disabled.

one does not slam the russians on new years

everyone knows that

hahaha

Well, the last time that happened they were in fact responded to angrily

Happy New Year from Indonesia! 🍺

Thank you guys !!!

For me, there is a marked difference between “Attention everyone please help me” and “Let’s spread some good cheer"

so yeah. shrug Happy new year!

A functional new year ; d

is there a "compile w/ optimization" kind of thing for clojure?

in 1.8 there will be direct linking

will direct linking + AOT improve startup time?

I'm mostly interested in speeding up runtime

AOT looks like it will help startup (but not runtime) http://clojure.org/compilation

there are other things committed that will help with startup time

lots of further improvement possible

i guess you don't eliminate the vars with direct linking, it's just that some code doesn't see them?

rust has "default compile, optimize for compile speed", and then you can do "-O3", where compile takes longer but you get a huge runtime speedup

obviously apples/oranges, v different models

Yeah, but it's a statically typed language that has a lot of information to base the optimizations off.

I've only experienced ~0.2s speed gain:

> time java -cp /Users/onetom/.m2/repository//org/clojure/clojure/1.8.0-RC4/clojure-1.8.0-RC4.jar clojure.main
Clojure 1.8.0-RC4
user=>
        1.42 real         2.39 user         0.15 sys
        1.32 real         2.37 user         0.13 sys
        1.31 real         2.35 user         0.14 sys

> time java -cp /Users/onetom/.m2/repository//org/clojure/clojure/1.7.0/clojure-1.7.0.jar clojure.main
Clojure 1.7.0
user=>
        1.49 real         2.55 user         0.15 sys
        1.51 real         2.76 user         0.14 sys
        1.51 real         2.83 user         0.14 sys

also, HNY, from Hong Kong!

@ghadi certainly apples & oranges, but one can dream

we've measured about 15% on core startup -- pretty close to what you're seeing

jaredly: you can't dream clojure to be static

😉

tons of type info can be inferred. most types don't change

(there will be CNY (Chinese New Year) in ~2 month too, then Thai New Year (Songkran) too)

10%/15% is what to expect, yes. although I think that speedup is caused by the slimmer static initializers rather than by DL

^ that

DL should only affect runtime performances

direct linking does help to have fewer static inits tho

hmm, direct linking also causes every function to have more bytecode that needs to be loaded though

so clojure.core will be compiled with direct linking in 1.8.0 regardless of the system property setting?

yes

it wasn't clear to me from the changelog how that would work

more bytecode but less <clinit> code

@micha: just like you can't opt-out of inlining in clojure.core or opt-in of *unchecked-math*

gotcha

lots can be inlined, optimized away. although I guess people want to be able to repl into their running program?

it's kind of hard to do those kinds of optimizations in clojure, partly because the current compiler impl is not modular & lack of build profiles, but also partly because stuff like multimethods/extend require reified vars

yeah

it seems like the multimethods/extend would be doable w/ whole-program analysis, though

But only for a subset of well behaved programs. For most Clojure programs JITs are probably still the way to go.

We’ve run with this in production for a while: https://clojurians.slack.com/archives/clojure/p1451586943010378

Our next production build bumps to 1.8.0 RC 4 and direct linking of our whole code base.

(we do run a REPL inside our app but using it to live replace code is so rare that we aren’t too worried about losing that ability with direct linking)

@jaen: any good JITs to recommend?

Well, HotSpot is one ; d

What I mean is in dynamic languages best optimisations are those you can infer at runtime - just see how fast things like V8 or LuaJIT can go.

@jaredly: the clojure evaluation model doesn't really work well with whole-program analysis/compilation

And with how Clojure is structured you are going to get most mileage from just-in-time compilation, especially if you want to keep the language's dynamic nature.

you'd have to make several non-trivial compromises, and you'd end up compiling a static subset of clojure. @arrdem has played with the idea for a while

There's also http://clojure-android.info/skummet/, though it's mostly concerned with load time.

Say, my user/create! function hashes the supplied password, which is great and all, but it makes tests run damn slow. What’s the best way to mock the hashing feature when running unit tests? Is this somewhere where I’d use with-bindings, and just rebind fully-qualified path.to.namespace/fname vars? i.e. replace buddy.hashers/encrypt with rot13 or something...

(with-redefs [ns.hashing/function identity] …)

Oh, nice, I didn’t know about with-redefs. Thanks!

@jr is a font of useful tricks

In common lisp, if using SLIME, you can C-c C-d h and pull up hyperspec documentation. Is there something similar for Clojure, e.g., in Cider hopefully, that pulls up https://clojuredocs.org/ by any chance?

are there any situations in which -Xmx wouldn’t be respected by the JVM? I think I’m setting it properly to 1024m, but I’m seeing RSS of 2.8g

Why not set -Xms to the same (1024m)?

That way the JVM will try to grab that much memory from the start, rather than relying on need.

cjmurphy: I don’t care if java uses less, I do care if it uses more

Oh I see you want to set a lower bound.

I want to set an upper bound

Sorry yep!

when using prismatic/schema I received an error: (not (sequential? :id)) what does it mean exactly I have been searching for an explanation of this error message

@arohner: resident set size (RSS) is the portion of memory occupied by a process that is held in main memory (RAM). Note that the JVM uses more memory than just the heap. For example Java methods, thread stacks and native handles are allocated in memory separate from the heap, as well as JVM internal data structures.

Well, schema's errors are not always exactly clear, but it's trying to say that :id is not a sequence.

ah

(s/validate [s/Int] :id) will result in the same error you are having.

Clojure newbie here, trying out Enlive (https://github.com/cgrand/enlive) Been reading through one of the tutorials, but I’m not too sure how to explore the API. Is there a common convention/platform for online API docs? Or for browsing in an editor?

Depends on the maintainer. For the most part folks have been using weavejester/codox. You could clone enlive, add codox to the plugins in the project.clj, and run it to get API docs.

https://github.com/weavejester/codox

@tom hmm yeah see that for ring (http://ring-clojure.github.io/ring/), nothing for Enlive, thanks.

another thing — is Clojure more or less like other dynamic languages where you’re never quite sure the type(s) that a function takes without looking at the docs?

when I looked at this I was like, “OK, what’s a resource?”

net.cgrand.enlive-html/html-resource
([resource] [resource options])
  Loads an HTML resource, returns a seq of nodes.

Well, it is a dynamic language after all.

@jaen true, just wondering if there’s anything to mitigate one of the big downsides of dynamic languages.

There is core.typed if you’ve got a fondness for sharp edges

But I don’t reckon it’s really a wise addition to an introductory clojure curriculum

Also, idiomatic Clojure is untyped, so you have problem typing idiomatic Clojure things.

@aaronk: the closest you can get is using prismatic schema as contracts, I suppose.

I’ve spent time in Python and Go. Python is nicer once you’re familiar with an API, but it is quite nice when looking at API docs (in Go) that you can easily see the type of every argument.

Well, prismatic's schema can let you do something similar

(s/defrecord StampedNames
  [date :- Long
   names :- [s/Str]])

(s/defn stamped-names :- StampedNames
  [names :- [s/Str]]
  (StampedNames. (str (System/currentTimeMillis)) names))

It's verified only at runtime (and optionally so) but go-level static type system doesn't give much guarantees anyway, so not much is lost.

sure it does, as long as you’re not using interface{} all over. It’s a big help in refactoring and understanding APIs IME. I’m not here to sing Go’s praises or anything… the reason I’m looking at Clojure in the first place is because of dissatisfaction with Go.

Well, you don't have to use interface to have loose guarantees either, as far as I understand nil is a valid value for a pointer for example and there's nothing to enforce nil-checking.

But about types being a huge help in refactoring and understanding APIs I agree

It's just I see go as not doing enough in the way of that when you can have Rust, Haskell, Idris, F# and so on.

Out of curiosity what would you be looking in Clojure that go didn't deliver on?