Provided you don’t need all maps to start, and provided you’re streaming them, and not sending them as one blob; yes.

Right. Currently it's all coming back as a single http response. I'm wondering if/how I can stream it using the existing setup, or if I should try something with websockets instead.

Not sure it matters logically. You could do multiple POSTs as well as multiple messages on WS.

single http response isn't the salient part

it's whether the response is maps wrapped by a collection, or if it's a series of maps

(series of bare maps)

e.g. like JSONL

Assume I have the best structure possible; I'll go from there 😉

{} {} {} is streamable by calling transit/read three times

[{} {} {}] is not

Makes sense

[{} {} {}] is also technically streamable, but you'd have to make changes to the Transit library.

right, it's about having a streaming parser

which lowers the abstraction level to event/token level, which we can probably all agree is unpleasant.

If I understand correctly, I need to use the ByteInputStream from the http response (or something similar) and parse it... but how can it be done in parts?

As in, how can I parse each item individually to get a lazy coll, or to put them on an async channel one at a time?

The input data has to have separate top-level objects on separate lines. You read the input buffer till you get to a new line, feed that buffer to Transit, repeat. And wrap it all in a lazy coll if you need. Alternatively, you can use a multipart response. It's probably supported by your HTTP client, you just have to tell it that each part must be decoded with Transit.

IF it is a series if maps then tmdjs and tmd's transit encoding pathway is going to be a lot faster.

In terms of encoding time, decoding and overall size although with gzip compression the win from size is decreased. That is specifically the pathway they are designed for.

Likely fast enough that decoding each map isn't going to help you at all and will overall be much slower.

@UDRJMEFSN Currently it's only between clojure (jvm) instances, so I don't thinnk tmdjs would apply. My maps are also nested, so I'm not sure if tech.v3.dataset could be used. But it has popped a few times when I was looking on ways to increase the processing speed, so I'll keep an eye on it for sure. @U2FRKM4TW thanks, I'll look into the multipart one!

No

thanks, that was my guess but it's counterintuitive

Code is evaluated by compilation (compiled then executed)

Using eval should still generally be considered a "code smell" -- it is very rarely needed.

When you load code from a file it is loaded by reading a form at a time from the file and calling eval on it(it doesn't actually call clojure.core/eval but it calls the same static method on the compiler that clojure.core/eval calls)

A better approach is to use requiring-resolve. Then if clients of your system don't want the clojure compiler booted up they can require the extensions or whatever at a high level (like in their main function) and then requiring-resolve is just resolve of an already compiled namespace.

Compared to that yes, eval is slower and is not optimizable. Requiring-resolve is at least optimizable.

this contradicts the previous statements that no optimization is happening either way

they are doing different things

once code is evaled the performance is the same either way

it's the loading of the initial code that is being sped up in crisn's suggested approach?

e.g. if you eval (fn [a] (* a a)) and call it, the function object returned by eval will have the same peformance as any other function object

but actually evaling (fn [a] (* a a)) to produce the function object has some cost too

@UDRJMEFSN is assuming you are using eval to approach a particular problem and is proposing a particular approach that can solve that problem using less work than eval does, but I don't know what problem you are trying to solve with eval

If it’s about exposing scripting capabilities, https://github.com/babashka/sci will give you an effectively sandboxed environment to run code. Comes at a performance price.

I've never tried to reach into multiple levels of Java nesting in Clojure, but I think this will help you : https://ericnormand.me/article/tricks-for-java-interop

INFO is a value in the inner Level enum inside the inner Logger interface of the System class.

it will depend

each of those dots in the java could be a field, a static field, or an inner class, and you need to know what each is to translate to clojure

Maybe you could :import the enum itself directly.

(import 'System$Logger$Level)

Did you try System.Logger.Level/INFO ? Or just Level/INFO if you imported it

you'll need to figure out where System is coming from

System is java.lang

System is java.lang.System in clojure, but you need to know what System is in that java expression

and it is not java.lang.System

ah!

(import 'java.lang.System$Logger$Level)

public final class System {
    public static interface Logger {
        public static enum Level {
            INFO
        }
    }
}

whoops, pardon me, java.lang.System grew a System.Logger at some point

JDK 9 apparently

Java logging 🥵

wow thanks for all that. It demystified several things for me. Which means I'll be back with more questions later 😆