does GenericContainer have side effects on construction?

I believe not on Construction. It only interacts with docker upon .start and .end

Of course the builder pattern used by the lib manipulates the instance with every call

@U011J2CQT0F Mayhaps, you are looking for reducing with https://clojuredocs.org/clojure.core/merge-with ?

something like this: [{:name "A" :age 11} {:mame "B" :age 12}] => {:name ["A" "B"] :age [11 12]}

(merge-with
  (fn [v1 v2]
    (let [v1' (if (vector? v1) v1 [v1])]
      (conj v1' v2)))
  {:name "A" :age 11}
  {:name "B" :age 12}
  {:name "C" :age 13})

It will not work if one of the existing values is a vector already.

Well… 🙂 we assumed they are strings initially… if you have such irregular data, that’s a problem in itself I would say… I would make sure my inputs are consistent (if I have control over it)

If they are all vectors to start with, it’s somewhat simpler

(apply
  merge-with
  into
  [{:name ["A"] :age [11]}
   {:name ["B"] :age [12]}
   {:name ["C"] :age [13]}])

A generic solution based on the code of merge-with:

(defn merge-into-vecs [maps]
  (when (some identity maps)
    (let [merge-entry (fn [m e]
                        (let [k (key e) v (val e)]
                          (update m k (fnil conj []) v)))
          merge2 (fn [m1 m2]
                   (reduce merge-entry m1 (seq m2)))]
      (reduce merge2 {} maps))))

Thank you very much. That's great.

Maybe identical? will help you? https://clojuredocs.org/clojure.core/identical_q

Right, thanks! Completely blanked out on that one.

The valid use cases for identical? are pretty unusual in Clojure, since = is often what you want. But I won't belabor the point if you know you need identical? for some reason.

pretty sure everyone uses into {}

now that I see the question on the screen and out of my head it all makes sense. I feel a bit silly now 🙂

Clojure uses more memory by design, but not that much more. I can go over the code if you post it.

Thank you very much for your reply. I have posted the code the uestion already. Here it is again.

I don't see anything obviously wrong here. One thing that might help is skipping the joining of the ngrams

that should save some allocation of new strings

(defn ngrams  [n N sent]
  (let [s (str/split sent #"\s+")]
    (for [i (range n (inc N))]
      (partition i 1 s))))

in ng-count (map ...) (apply map concat) can be replaced with (mapcat ...)

(defn ng-count [n N filename]
  (with-open [rdr (io/reader filename)]
    (->>(line-seq rdr)
        (mapcat (partial ngrams n N ) )
        (map frequencies )
        doall)))

Probably does not matter much for perf. or mem usages though

How many ngrams do you expect in the text file, and what is your -Xmx setting?

Every JVM string object has an overhead of about 40 bytes, regardless of its length, so representing N-grams as vectors of N strings, rather than creating a new string, might save memory. There is still some memory allocated for each vector object, too, so those two approaches might end up near the same memory, after all.

Thank you very much for your replies, Chrisblom and Andy. That is very helpful. I actually tried many values for -Xmx from 1000 to 2000.

I noticed something else, which I have no clear information of. I noticed that the program runs in multithreaded manner. Are the threads created for the processing executed by the program or they are rather Java's utilities to handle internal tasks?

There are multiple different garbage collection methods that modern JVMs implement. Some of them use separate threads, and can take advantage of separate CPU cores from those that execute your main code, to perform garbage collection.

Determining whether that is what is happening in your case is possible, with appropriate examination of the JVM's operation, e.g. using some kind of JVM profiler tool that can attach to a running JVM.

You can also type Ctrl-\ in a terminal from which the JVM was started to cause it to dump out info about all threads, maybe more detailed than you would like, which might help you discover what the threads are called, at least, and a snapshot of what their current stack trace is for the code they are executing.

Perfect, great info... many many thanks

@U015D447PSL took a stab at your question and only now realized that Slack showed me a 7 day old thread. 🙂 Anywho, it may still be relevant - I think you can save a lot on memory, by not creating so many temporary thunks and using transducers. Here's a gist as an example. Probably some things could be even more elegantly described, but hopefully this gives you some ideas: https://gist.github.com/pithyless/44ce8908cee196fd3f5ace685b8db2f4

(or (seq coll) [1])

not sure is it more idiomatic or not, but that is the form that I’m using a lot

I was trying to go for an or but I was missing the seq function

thank you 😃

Be careful though if there are type checks.

=> (vector? (seq []))
false

Needing this function might indicate you are doing https://stuartsierra.com/2015/06/10/clojure-donts-heisenparameter

Not sure if this helps but:

(defmacro foo [m]
  `(defn ~(symbol (:k m)) [] (+ 1 1)))

(defmacro doseq-foo [a-macro & args]
  `(do
     ~@(map (fn [arg] `(~a-macro ~arg)) args)))

(doseq-foo foo {:k “f20”} {:k “f21"} {:k “f22”} {:k “f23"})

“generates” all of those functions

Yeah that’s pretty close to what I want to do

I agree with other though that there’s probably a simpler way to achieve whatever you’re trying to do…

The problem I guess is that the route data actually looks more like:

["/foo"
 ["/bar" ...]
 ["/baz" ...]]

This kinda of macro black magic is veeery rarely the only solution 🙂

Yeah, agree

I don’t see another way to get the stuff I listed above though

you can flatten the route tree with reitit.

If you know of one I’d be all ears

(-> ["/foo"
     ["/bar" ...]
     ["/baz" ...]]
    (r/router)
    (r/routes))

internally the trees are always flattened, enables huge perf wins.

@U055NJ5CC okay cool thanks, I was actually trying to do something like that

specifically I’m trying to do this:

(doseq [[path data] (reitit.impl/resolve-routes my-routes (reitit.core/default-router-options)),
        :let [path-params (get-in data [:parameters :path])]
        [method spec] (dissoc data :parameters)]
  (defn-my-binding path method spec))

I guess your suggestion is cleaner though 🙂

either way, my problem of “how do I actually define the bindings” is still kind of unsolved

Not sure if this helps it:

(defmacro doseq-foo [a-macro args]
  `(do
     ~@(map (fn [arg] `(~a-macro ~arg)) args)))

(doseq-foo
  foo
  [{:k "f40"}
   {:k "f41"}
   {:k "f42"}])

But unless you have the

[{:k "f40"}
   {:k "f41"}
   {:k "f42"}]

Hmm yeah, the thing is that I do have all the routes at compile-time, so I suppose maybe it could work?

does it matter if the vars are available in both clj/cljs or is it mostly for convenience at the clj repl?

Well ideally I'd like to use the bindings in both cljs and cljs

If the repl experience isn't as good in cljs then that's fine

@wombawomba I believe the cljs macros need to be in .cljc file (not sure if you’ve dealt with that)

Yeah I do have all of this in cljc

cool

if the routes are available at compile time. I would probably do something like

(def-routes-and-schemas-and-fns 
   ;; routes
   [[route-definition-1]
    [route-definition-2]
    ...])

and have that macro generate all the stuff you need

Okay yeah that sounds like a good solution to me

I assume that'd work even if the args would actually be (flatten-routes (generate-routes))?

(the routes are defined as a tree, so I need to flatten them first)

macros allow for syntactic abstraction, so I would just describe the routes however is most appealing and have the macro figure out how produce the right code (flattening/etc)

the way I like to think about it, is use whatever form I would use if I was writing pseudo code on paper or at the whiteboard and then create the macro to match that syntax

just be aware that demunge is best effort and will not work in all cases (importantly both - and munge to and demunge can't know which is right)

there may be other cases that don't work

(also, spec 2 no longer has this function because specs are always resolved in terms of symbols, not as function objects)

changes may need to be made in the future wrt to mappings from fns to classes (class names can get too long for the filesystem due to nesting) and at that time, this hack will also stop working for those classes

as long as you understand all those caveats, have at :)

before you go for gRPC, make sure you really can explain why you need it; in my little experience with it, it doesn’t work well with the Clojure philosophy of doing things; it’s more of a Golang tool

I do not know how many, but one named protojure shows up in a Google search for "clojure grpc". There are also several examples of using Java interop from Clojure to call Java gRPC methods, which for some Java libraries is a very reasonable thing to do in a Clojure program.

just to know, in my company we're having a discussion to know if it's possible to adopt gRPC and I was suggesting some red flags, I saw that only had protojure and java API integration.

Clojure is hosted on the JVM as a design choice, not an accident, specifically to make it straightforward to take advantage of the huge number of Java libraries that exist.

yup, but not too much stuff are made in Java using gRPC

That I have no knowledge to share.

and I thought that gRPC was not a good choice, maybe some day Avro?

but thanks @U0CMVHBL2 @U050KSS8M 😃

I don't know if that is much of a minus, if the Java support works, but I also have no experience reports to share with you, good or bad, in that regard. I mean, one of the purposes of gRPC of course is to have a language-independent data communication mechanism, but that doesn't mean that all languages support it equally well.

@U9ABG0ERZ no problem! If your company is mostly Clojure, I would stick with Clojure’s transit where possible

yup

“it doesn’t work well with the Clojure philosophy of doing things” I disagree with this, gRPC is language independent and has a really solid way of defining service contracts. That being said I probably wouldn’t write a gRPC service in a non-statically typed language like Clojure, but a client would work just as fine.

@UDBM9Q0VC have you used gRPC from Clojure?

just as a consumer of a grpc service

I think the primary complaints I have heard (and some people really hate it) have been about the need to edit and re-generate the generated code. This is fundamentally because people are just wrapping the java libraries. From what I have read(and at one point I was thinking to implement a "native" protobuf library in clojure, so I have read a bit) there is nothing stopping an inherently more dynamic experience on clojure, just no one has done the work yet, so you likely will run into all the same annoyances as others

We have a home grown RPC library, based on HTTP and Avro - the lack of recompilation step is a huge win. I've looked at gRPC and it has a really big surface, requires ProtoBuf recompilation and if your frontend (js/cljs) has to call your API/RPC server - there is (was? ) no good answer apart from a half-baked proxy. Oh, and also - it requires using HTTP2, which might not work in your setup (for example AWS ELB (classic) doesn't support it)

Yeah, I totally agree the compilation step is a pain. And there are similar issues using protobuf outside of gRPC with Clojure or Java. But at the cost of having a really solid contract, it makes total sense for some use cases. In our case we were building out a replacement for an old https://docs.geoserver.org/latest/en/user/services/wfs/reference.html that traditionally uses RPC and XML. I think there are some great use cases for it but at the same time I could understand the frustration of using it when there are simpler more popular solutions.