Fork me on GitHub
Alex Miller (Clojure team)00:06:23

RELEASE dependencies are not officially supported (because they are pseudo versions and break the cache)

Alex Miller (Clojure team)00:06:48

You can add -Sforce to force the cp to be recomputed


anybody know off-hand where Ring adds a session token (cookie GUID) to the response? I can't find it in wrap-session


doing some security auditing and need to confirm it's securely generated


it's called ring-session


after some more digging it seems to be in ring.middleware.session.memory, where a UUID/randomUUIDis added to Ring's memory store as a map key

👍 2

I'm trying to track down a memory leak using VisualVM and have a few questions 1) Any hints what it might mean when there are a lot of java.lang.reflect.Methods allocated? 2) Other than that, the memory sampler is just showing me Doubles and PersistentVectors and Objects. Is there a good way to narrow this down further? I do know I'm creating a lot of these, I'm rendering a visual that adds a lot of points to several time series every second. 3) Any other VisualVM tips welcome

Alex Miller (Clojure team)12:06:12

Seems likely you have a reflective call due to 1, but that itself is unlikely to be a memory leak

Alex Miller (Clojure team)12:06:36

Tools like yourkit let you do things like “find paths to root” which can tell you reference path from an object back to how it’s being held, not sure if visualvm has that

👍 2

Is there a reason a reflective call might not be detected with (set! *warn-on-reflection* true) ?


@jjttjj it can happen when the code isn't loaded at all if it's already AOT-ed for example

👍 2


$ clj
Clojure 1.10.1
user=> (set! *warn-on-reflection* true)
user=> (require 'clojure.core :reload)
user=> (load-string (slurp ( "clojure/core.clj")))
Reflection warning, null:1717:7 - call to java.lang.IllegalArgumentException ctor can't be resolved.
Reflection warning, null:3631:22 - call to java.math.BigInteger ctor can't be resolved.
Reflection warning, null:3645:14 - call to java.math.BigInteger ctor can't be resolved.
Reflection warning, null:3659:14 - call to java.math.BigDecimal ctor can't be resolved.

Alex Miller (Clojure team)13:06:12

there might be other things that create Method objects too, just seems like the most likely candidate


next.jdbc states this in their docs: > Note: you need to be careful when using stateful transducers, such as partition-by, when reducing over the result of plan. Since plan returns an IReduceInit, the resource management (around the ResultSet) only applies to the reduce operation: many stateful transducers have a completing function that will access elements of the result sequence -- and this will usually fail after the reduction has cleaned up the resources. This is an inherent problem with stateful transducers over resource-managing reductions with no good solution. Is there a more detailed explanation of this problem; and perhaps a minimal example that illustrates it?


Reproducing this is a bit tricky: you generally need a resource that is batched/buffered under the hood and to trigger the issue you need a large enough input set that it will cross that batch/buffer boundary — and the way that the completing function processes elements needs to actually trigger an error if performed on a resource that has already been “closed” (not all access will trigger an error — so it very much depends on the system that is fetching data from that resource). So it’s only going to happen with a resource that is lazily computed (with side effects) and can fail if you attempt to realize items after turning the resource “off”.


If you can reduce the whole thing first, and then run the stateful transducer over that result, you’re safe.


that doesn't really make sense to me -- is the "connection/resource" not closed as the cleanup procedure for the IReduceInit operation?


Yes it is: but the completing function only runs after the reduction is completed.


So if the completing function does anything that would cause realization of elements it has in hand, it can fail.


got a concrete example?


As I said above, it’s hard to repro because of the conditions under which it fails.


(transduce (partition-all 500) conj [] (sql query.......)) wouldn't be an example of this, right?


Using plan, yes, maybe it would fail. On some databases, with a sufficiently large query result, and certain fetch size (batching) settings.


It’s because in the stateful transducers, you can have elements stored (in the state) that have not yet been fully-realized, then the reduction completes (and closes the resource), and then the completion function processes those elements and attempts to realize them.


is the "elements not fully-realized" some optimization that avoids touching the ResultSet?


fake maps

Alex Miller (Clojure team)16:06:28

seems like you have to ensure that anything in the state does not require an active connection

✔️ 2
Alex Miller (Clojure team)17:06:17

or you could defer the transducer calling into the completion of the rs until after it's processed? I mean, it's a stack and the transducer surrounds the rs and controls the completion call right?

Alex Miller (Clojure team)17:06:48

maybe the rs is being closed on last iteration instead of on completion? (sorry, haven't looked at any code)


Well, transduce itself does the full reduction (which closes the resource at the end) and then it runs the completion after that:

([xform f init coll]
     (let [f (xform f)
           ret (if (instance? clojure.lang.IReduceInit coll)
                 (.reduce ^clojure.lang.IReduceInit coll f init)
                 (clojure.core.protocols/coll-reduce coll f init))]
       (f ret))))


is there a good resource for uberjar creation? Specifically how to deal with multiple licenses from deps. Our uberjar seems to have random dist/LICENSE and LICENSE and LICENSE.txt from deps. I want to be a good jvm citizen and ensure that a) our license it top level and in a canonical spot, and that all of our deps licenses are in an appropriate spot

Alex Miller (Clojure team)17:06:37

there are N tools for this and they make N choices afaict


yeah. and i think its the defaults of lein uberjar that i need to be fighting. But i would like to find out the guidelines for what to do so I know a goal to shoot for

Alex Miller (Clojure team)17:06:58

the proper place (imo N+1 opinion) is that META-INF is the explicit place marked out to put descriptive meta stuff orthogonal to the jar itself


depstar concatenates all the license files, as I recall, assuming they’re recognizable as such in the input files.

Alex Miller (Clojure team)17:06:27

Maven will for example put stuff under META-INF/maven/<index by group/artifact>/stuff

Alex Miller (Clojure team)17:06:47

that is imo a sane strategy


yeah i looked at depstar and uberjar. and they look for license files at certain spots and not root level ones


so i'm not sure if that's a missing part of theirs or if the ibm/icu jar is being naughty with a top level LICENSE file about unicode

Alex Miller (Clojure team)17:06:32

yeah, there's really two questions here where do I find stuff in jars where do I put stuff in uberjars

seancorfield17:06:46 is the issue where I tried to deal with this — see my comment about “I couldn’t find much information on how Maven shade handles various license files and I examined the source and could not deduce what it does from that either” 😐


exactly. what should a jar look like, what should i do when creating my uberjars, what surgery should i do to correct misbehaved dep jars

Alex Miller (Clojure team)17:06:41

because there is no standard, there will likely be conflicts, particularly at jar root


The practical answer is that there’s only so much you can do, and depstar mostly “does the right thing” (now).


yeah. i'm bundling all licenses up by inspecting all jars on the classpath, so i think i'm comfortable nuking all licenses in the resulting jar but ours and placing those rollups in there

Alex Miller (Clojure team)17:06:22

I think it makes sense to grab stuff at jar META-INF and put it under uberjar META-INF/<indexing-strategy>/...


depstar does some of that too, yes.


“It’s complicated”.


i've got to run to a meeting. thank you both for your experiences

Alex Miller (Clojure team)17:06:10

I have not finalized a strategy/process for this in yet, but I would like it to be a pluggable/directable thing


it is because the "completing" is part of transduction, and not part of the IReduceInit interface, so the IReduceInit that is backed by the ResultSet closes things down and cleans things up when the "reduce" is done, because it can't clean up at a point later than that, and then transducer comes along and calls the completing arity


So put another way… IReduceInit(ables) need to close the resource themselves because they’re usable independently of transduce… e.g. you can supply them as a “collection” for reduce (and things built on top of it). transduce introduces a new notion of completion bespoke to the composed xform stack, and it has no knowledge of the independent resource management that’s happening in the implementation of IReduceInit. So could you not hack a “fix” for this case (i.e. not a general solution) by closing in the completion function, and not in IReduceInit?


Also just trying to understand the conclusion of this… re @U064X3EF3’s point: > seems like you have to ensure that anything in the state does not require an active connection @U0NCTKEV8 I’m wondering if you think this is because mapify-result-set works on the active collection which means you’re delaying the realisation of values to outside of the reduction. If instead those were values were fully realised as values (e.g. maps) would the problem not go away? I’m not claiming these aren’t valid trade offs — I just want to understand if the problem is really with how transduce / IReduceInit work; or if it’s just that they can’t reliably be used in all cases with things like mapify-result-set, that delay construction of values from mutable connections. In which case it’s not really a general problem, but more of a problem with next.jdbc Is that a fair assessment?

Alex Miller (Clojure team)13:06:09

reduce (and transduce) are eager operations so I'm not sure that they should even be in the job of closing resources. if you consider the "steps" to be: • open resource and make a reducible from it (run your sql query get a result set) • reduce/transduce • safe to close original resource, the work is done

Alex Miller (Clojure team)13:06:18

I don't feel satisfied that we've really gotten to the problem

Alex Miller (Clojure team)13:06:35

"IReduceInit(ables) need to close the resource themselves" seems maybe wrong to me


Yeah — for context I’m creating something similar to next.jdbc and was wanting to base it off transducers too. I’d been looking at next.jdbc for inspiration, and stumbled across this, and was unsure what it meant. To be fair I’d also always assumed one of the stated benefits of the transducer / reducer work was that you can control resource usage more effectively than with laziness; and that, that meant that you could essentially move the closing of resources into the implementation. However it might be a step too far to assume that it means you can hide the closing of the resources away like that.


It feels to me like the issue is with next.jdbc though, not clojure — though that might be because the stated contract for reducible and transducers isn’t entirely obvious or clear to implementers.


However I also don’t feel like I understand the problem deeply enough. I don’t use next.jdbc and I’ve also not knowingly experienced it in my code… so it’s hard to know if my code is susceptible to the issue or not. I’d like to avoid recreating the issue if possible though


i think racket has a reduce type function which offers a completion step which seems quite nice


racket has so many helpful functions, love that language


@seancorfield IMHO it's because the funny maps returned are entangled with the live resources, not because of how the completing step works


wouldn't it would still be a problem if there was no such thing as completing?


the problem with completing is a transducer with a partition-by step in the middle, everything after the partition-by might run as part of the completion


so your pipeline that only pulls what it needs from the database as it needs and it produces a nice concrete result will still fail


I’m working with Selmer and I have a hash map that I want to loop over the keys of in a particular order — I have a vector with the keys in order but it’s not clear to me from the Selmer docs whether it’s possible to do that sort of get-in style variable reference where a key in the middle of a path is a variable, rather than hardcoded? i.e., something like {{[email protected]}} where k is the for loop var — the equivalent of (get-in foo [:bar k :quux]) /cc @yogthos


I don't know the answer for sure, but I assume it's best to do some preprocessing on the data before feeding it into the template


yeah that would be my recommendation as well


Thanks @yogthos I had a feeling that would be the case.

👍 2