by preferring (into [] ...) to mapv you get all the perf benefits of transducers (and transients)

Nice! I would have imagined the impl of mapv would have been upgraded after the introduction of transducers

mapv/filterv are vestigial functions now that transducers exist

That's good to know. I never got that message

Nor did I! Perhaps clj-kondo should highlight it?

I wouldn't put it as strongly as deprecate

mapv is certainly more concise and in cases where you know the data is small, you're not stacking transformations, etc I still use it

Ok, thanks.

mapv and filterv were yet another small step on the "why do we keep building the same family of functional transformations in new contexts" journey

That's interesting. It preserves the laziness, up to a point. One could call it semi-laziness :)

Isn't that wrapper just equivalent to calling seq ? Given that it checks that the collection is non-empty?

👀 Got you, one could (->> xs (map f) (seq)) for being lazy and fail-fasting. Maybe sequence would be better, for avoiding nil values.

Probably not, as clojure.lang.ILookup as an interface (as opposed to a protocol, which can be extend -ed)

the lookup behavior of Keywords is in the java implementation (implementing IFn or AFn), so there's no good way to shadow or replace it it's not a syntactic feature of clojure that keywords do lookup, it's just that clojure tries to invoke things and IFn tells you what happens if you do that

Yeah, it’s a strange Java class file format internal detail that leaks into Clojure.

https://github.com/stuarthalloway/clojure-from-java/blob/master/src/java/example/Main.java

I’m not sure about nrepl, but clojure supports socket repls out of the box via system properties, this is as close to “minimal impact” as it gets

does that java app use something like maven or gradle? clojure is “just a jar” that you can add using your build tool’s way to specify dependencies

depending on the jdk, one of the following may work: https://github.com/dkz/liverepl (this version has some nrepl support iiuc) the original is: https://github.com/djpowell/liverepl -- may have need some tweaking

the original does not provide nrepl support and i think it predates the official socket repl (it provides its own method iiuc)

i think they are more likely to work out of the box with jdk <= 8. i think i got one of them working with a more recent jdk, but i'd have to dig a bit to confirm -- i don't tend to do nrepl-related things much though, so even if this worked it would be via a socket repl.

Hey @U47G49KHQ and @UG1C3AD5Z -- thanks the replies. Sorry it took me a while to get back to you. We're using Gradle on OpenJDK 11. Thanks for the links. If I understand correctly, my options are: 1. Link Clojure's jar, add a system property to the process invocation and connect via Socket-REPL 2. Link to Clojure's jar, NREPL and CIDER, and add java code to start a CIDER-enabled NREPL server into the existing codebase 3. Use Liverepl, which if I understand correctly allows the Clojure code to live in a normal Clojure project, but "remote connects" to the Java process. It's not clear to me how this would work, if there's synchronization needed, or whether it eventually "joins" java processes. I appreciate your insight! I feel like I have a bit more to go on now.

iirc, option 3 injects clojure into the running java process -- the java process then runs a socket repl of sorts and one connects to that. if you are just poking around, this can be a short-term option. however, the liverepl option is a bit dated and getting it working with recent jdks may require some tweaking. also if the java application is using java 9 modules or something "different" regarding the classpath, it may introduce additional complications. if you have aspirations of eventually using clojure within the application, i think vlaaad's approach makes sense to start with. another reason to go that way is that the application is using jdk > 8.

you're holding on to the head of r, and then building a 1e9 long sequence to get the last value

so the whole sequence is instantiated in memory

I don't know what decompile is

user=> (doc decompile) ------------------------- clj-java-decompiler.core/decompile ([form]) Macro Decompile the form into Java and print it to stdout. Form shouldn’t be quoted. nil

My understanding is that, it is the =r= that holds the sequence. So both forms should OOM. However, the first form doesn’t.

well the first form is only 139 elements, which is not much memory?

vs 1000000

Sorry. I pasted wrong. The number should be the same 1e9.

in the first one, the last that walks the sequence is in tail position - the Clojure compiler can clear the r reference before evaluating it

and it does

https://bpaste.net/PQ5Q It is the de-compiled java code. Is it more of the runtime java gc, or due to the clojure compiler? I don’t see a much difference between the decompiled code of the first and second form.

This is more workaround than solution. Ignoring flushes is potentially dangerous. I think I can get this right for my particular use case, but if you're not careful about flushing at the right time, the end of your response may be cut off. I'll likely be reading the source for the output stream I'm receiving to ensure that it's .close method doesn't utilize .flush.

Which also means my solution isn't general purpose.

The description change is made. I'm happy to make a repro in the way that is most useful to someone 🙂

combine the lists with concat?

how would that help? concat would just give me one big list

(map-indexed (fn [idx [a b c]] ...) (map vector a-list b-list c-list))

(map (fn [id x y z])
     (range)
     coll
     coll
     coll)

ooh, clever

@dpsutton hell yeah I like that 😄

has the benefit of not having to remember where the index is. i always forget if its first or last

Complexity of using it or reading codebase of dokku? Dokku have good "git push" story.

The codebase / size / deps

It handles many operations which we'll have to deal manually otherwise, won't an alternative with same features have similar size?

Dokku codebase is largely composed of bash scripts.

What alternative ways do you recommend to Dokku?

there’s little to recommend. Piku doesn’t work as well as it looks.

I posted because I don’t know good alternatives

So, do you prefer bash scripts over these solutions?

Ah true. We're only doing 10s of deploys a day at the moment. You're referring to data transfer fees adding up, right?

the most likely problem for that scenario yeah

drop-while will walk over the whole sequence, realising it, while data-rows is still retaining the head

From VisualVM, I can see it's holding on to a bunch of strings.

Oh, duh

drop-while doesn't realize it, the call to vec does

Right. So the issue is it gets realized and the let is holding onto the head?

no

call vec is realizing it

you are reading all the data in and turning it in to a vector

An empty vector

yes sorry the call to vec is realising the lazy seq created by drop while which contains the lazy seq from data.csv

ah, wasn't paying attention to the drop while predicate

just because the strings are in memory that doesn't mean they are being held

that could also just mean they haven't been gc;ed yet, how large is your max heap, and have you actually seen an out of memory error?

No oom. Max heap is 30gb.

Actually the original code seems fine.

jvm will basically use whatever you give it, so by giving it such a large heap, it may delay gc'ing until it has allocated close to 30gb

Forcing a gc does clean up all that. To be clear, vec is realizing the lazy seq but data-rows is not holding on to the entire csv.

seems I need to re-read some stuff

something something locals clearing

have you tried reading with a 512MB max heap?

that will confirm both hiredman's reasoning and your correct implementation

Good idea. Will try that

keep in mind that as soon as you remove the artificial drop-while argument, your call to vec becomes a liability

Yep

I do get an OOM when running with 512m.

Same with 1g

2g as well. Perhaps holding on to rows in the let is pulling everything into memory?

This does not oom

(with-open [rdr (io/reader "data.csv")]
  (vec (drop-while (constantly true) (csv/read-csv rdr))))

It's definitely the rows in the let binding. Removing that fixed this.

There have been changes to Clojure in the past several years that eliminate some cases of locals clearing being done in cases where it did not used to, to avoid some 'holding onto head' cases similar to this. It isn't clear to me whether this is a case of "should hold onto head" or "locals clearing should prevent that".

I think with-open could kill locals clearing

Hmm, rereading what I wrote, I meant to say "that add (not eliminate) some cases of locals clearing"

Don’t guess; look at the bytecode

Need to know what clojure version if someone wants to repro

1.10.1

I don't spend as much time looking at the compiler as I have done in the past, but I think the way it determines when to clear locals might be the most inscrutable part of it. I have a toy re-implementation of the core.async's go macro that does some pretty standard dataflow analysis which results in always clearing locals right after last use, but is iterative (I think I read some where if you arrange things right the algorithm will be linear, which I have not done).

what the clojure compiler does is supposed to be better in some way (faster? simpler?) and it generates a tree of usages and compares paths in that tree, and it always seems to need tinkering with

public static Object invokeStatic() {
        final Object rdr = ((IFn)user$fn__287.const__0.getRawRoot()).invoke("data.csv");
        Object invoke;
        try {
            final Object rows = ((IFn)user$fn__287.const__1.getRawRoot()).invoke(rdr);
            final Object header = RT.nth(rows, RT.intCast(0L));
            final Object data_rows = ((IFn)user$fn__287.const__4.getRawRoot()).invoke(rows);
            invoke = ((IFn)user$fn__287.const__5.getRawRoot()).invoke(((IFn)user$fn__287.const__6.getRawRoot()).invoke(((IFn)user$fn__287.const__7.getRawRoot()).invoke(Boolean.TRUE), data_rows));
        }
        finally {
            ((Reader)rdr).close();
        }
        return invoke;
    }

would locals clearing, if it were happening, look like a setting of those variables to null before one of those steps in invoke?

(I dunno how accurate it is but clj-java-decompiler is damn cool)

I dunno exactly what it would look like in whatever you are using to turn the bytecode in to java, maybe something like rows = null; after the last usage

> To make the output clearer, clj-java-decompiler by default disables https://clojuredocs.org/clojure.core/*compiler-options* for the code it compiles. You can re-enable it by setting this compiler option to false explicitly, like this:

haha

Object rdr = ((IFn)user$fn__196.const__0.getRawRoot()).invoke("data.csv");
        Object invoke2;
        try {
            Object rows = ((IFn)user$fn__196.const__0.getRawRoot()).invoke(rdr);
            RT.nth(rows, RT.intCast(0L));
            final IFn fn = (IFn)user$fn__196.const__3.getRawRoot();
            final Object o = rows;
            rows = null;
            Object data_rows = fn.invoke(o);
            final IFn fn2 = (IFn)user$fn__196.const__4.getRawRoot();
            final IFn fn3 = (IFn)user$fn__196.const__5.getRawRoot();
            final Object invoke = ((IFn)user$fn__196.const__6.getRawRoot()).invoke(Boolean.TRUE);
            final Object o2 = data_rows;
            data_rows = null;
            invoke2 = fn2.invoke(fn3.invoke(invoke, o2));
        }
        finally {
            final Object target = rdr;
            rdr = null;
            Reflector.invokeNoArgInstanceMember(target, "close", false);
        }
        return invoke2;

is what it looks like if you don't turn off locals clearing (and in fact locals are being cleared)

indeed

you can it is confused by the bytecode that isn't directly translatable to java

(the o = rows right before rows = null)

so data_rows is being cleared. but o2 is defined as data_rows and is not

oh you're saying that's garbage, interesting

so my guess is @U083D6HK9 is using some tooling that turns locals clearning off

it is, the bytecode that is emitted loads the value on to the stack (no direct representation in java source) from the local and then nulls out the local

Typing this in my repl does indeed sound like that is true

*compiler-options*
=> {:disable-locals-clearing true}

mystery solved

Next question is who is setting that & why is it being set to that 🙂

my guess would be you are using cursive

You would be correct

cursive likely turns it off by default for the debugger

oh you can disassemble to bytecode instead, nice

Ah, I am using the debugger-enabled repl.

Yup, that is it. I see a little switch for it. That clears things up 🙂 Thanks

( a little bit meta: finding re-find in an app called re-find: https://borkdude.github.io/re-find.web/?args=%23%22foo%22%20%22foobar%22&amp;ret=%22foo%22 )