Fork me on GitHub
#beginners
<
2020-10-14
>
SoV400:10:00

oontz oontz oontz

SoV401:10:11

the sound of techno

seancorfield02:10:26

In the #beginners channel? Perhaps more appropriate for #music or #off-topic ?

jumar03:10:34

@mario.cordova.862 the first thing I would think about is to upgrade your JDK and garbage collector. Since you're still using concurrent mark and sweep (which has already been removed in recent JDK versions) it suggests you have something older - I suppose JDK 8?. JDK 11+ is perfectly fine to use these days and even if you cannot upgrade JDK I'd suggest you try G1 Gc (available in JDK 8 too. Also, are all the errors you mentioned happening about the time when the app gets unresponsive? For lightweight investigation I suggest using jcmd (much better in recent jDK versions) or similar tools to quickly inspect JVM (e.g. class histogram). You can also configure JVM to make a heap dump on OOM via -XX:+*HeapDumpOnOutOfMemoryError* (https://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot/clopts001.html) and examine it later e.g. via VisualVM (or Yourkit, Eclipse MAT, etc.)

Mario C.15:10:55

Yes, we are using JDK 8. And the errors all occur at the same time. All API calls seem to fail. Most request handlers just barf. The CPU spikes. Locally I added the -XX:+*HeapDumpOnOutOfMemoryError* and set the Metaspace to a low value so I can replicated the issue. Now I am using the heap analyzer tools but honestly I can barely make any sense out of it. I see references to some of our code but I can't really make sense of the big picture.

jumar19:10:28

Note that even a simple clojure application demands at least 30-40 MB - depends on how much code & dependencies you load. If your app consumed gigabytes then that might be a real problem - would be good to have a continuous monitoring of various aspects of the JVM memory consumption; it's not super-specific to your problem but I always recommend reading this SO question to get a better understanding of JVM memory consumption: https://stackoverflow.com/questions/53451103/java-using-much-more-memory-than-heap-size-or-size-correctly-docker-memory-limi/53624438#53624438

jumar19:10:40

And here some nitty-gritty details about Metaspace: https://stuefe.de/posts/metaspace/sizing-metaspace/

Jim Newton07:10:56

Does anyone know how local variables are handled w.r.t. the GC in the case that clojures are maintained for a long time? What do I mean. I mean that if I have a function which has local variables whose values are HUGE data structures, and the function returns a clojure which DOES NOT reference those variables, is the GC able to deallocate them or is the entire stack-frame maintained. e.g., in this contrived example, the function returned only references p whose value is an integer, so m and q, can be deallocated. foo returns a function. Suppose that I hold onto that function, in an global variable. Is it my responsibility to deallocate m and q or will the GC do it for me?

(defn foo [f]
  (let [m (range 10000000)
        q (map inc m)
        p 12]
    (doseq [x q]
       (if (f q) (throw (ex-info ...))))
    (fn [g]
      (g p))))

Jim Newton07:10:56

Does anyone know how local variables are handled w.r.t. the GC in the case that clojures are maintained for a long time? What do I mean. I mean that if I have a function which has local variables whose values are HUGE data structures, and the function returns a clojure which DOES NOT reference those variables, is the GC able to deallocate them or is the entire stack-frame maintained. e.g., in this contrived example, the function returned only references p whose value is an integer, so m and q, can be deallocated. foo returns a function. Suppose that I hold onto that function, in an global variable. Is it my responsibility to deallocate m and q or will the GC do it for me?

(defn foo [f]
  (let [m (range 10000000)
        q (map inc m)
        p 12]
    (doseq [x q]
       (if (f q) (throw (ex-info ...))))
    (fn [g]
      (g p))))

leonoel08:10:08

A closure only retains values bound to its free variables. Both sequences are candidate for GC when foo returns.

jumar09:10:25

There's also an interesting version of fn* using :once true metadata for one-shot closures: https://stackoverflow.com/questions/23586710/what-is-fn-and-how-does-it-differ-from-fn

Jim Newton10:10:08

@U053XQP4S, so closures contain only the values bound to free variables, not to the environment frames which contain those variables? If that's so, that's terrific.

leonoel14:10:24

I don't think this behavior is documented anywhere but that's what the compiler actually does and it's very reasonable to rely on that IMO

Jim Newton08:10:17

I wonder whether excessive use of lazy sequences might be the culprit. A raw sequence only contains final values, even if very many of them. However, a lazy sequence must contain references to all values necessary to compute the rest of the sequence. It could be that in extreme cases lazy sequences occupy much more memory than the sequences themselves.

Jim Newton08:10:41

(defn foo [x]
  (let [very-long-sequence (compute-very-long-sequence-of-sequences x)]
    (map count very-long-sequence)))
For example, foo returns a lazy sequence which must retain a pointer to x and also to very-long-sequence .... however, the final sequence is just a list of integers. Right?

Jim Newton09:10:47

so as I understand, I can wrap all generations of such lazy sequences with doall for force them to be non-lazy. But finding all such lazy generations may be challenging. Not really sure how to do that.

leonoel09:10:02

or use eager constructions like vectors, transducers, reductions

Jim Newton10:10:38

sorry, don't know how to do that. And it sounds like a difficult and invasive refactoring of my codebase, especially since it is not guaranteed to solve the problem.

leonoel10:10:14

then doall sounds good

Jim Newton10:10:54

map is documented as returning a lazy sequence. but it is not clear whether mapcat returns a lazy sequence or not.

Jim Newton11:10:14

after a bit of testing, it is promising but not conclusive. I get far fewer out of memory errors after forcing all calls to map and mapcat to be non-lazy. My tests now run in about 2 minutes, whereas they previously took a half hour to run, But occasionally still give out-of-memory errors. But less often.

Jim Newton11:10:21

very interesting results.

Jim Newton11:10:54

AND the cpu meter I'm using on my Mac was showing huge stress, now is much happier and the cooling fan is much quieter.

Clark Urzo09:10:20

Hello there. I'm trying to deploy a simple reagent app to heroku and I'm lost.

practicalli10:10:46

Is this the sample app from the Heroku guide or your own reagent app?

Clark Urzo11:10:05

I essentially copied everything over

Clark Urzo11:10:40

The sample reagent app deploys correctly, btw.

practicalli11:10:32

My only thought is how bootstrap is being pulled in. I use a CDN for CSS libraries. If using local libraries then check the path to them is correct when deployed on Heroku

Clark Urzo09:10:21

I get this Error: browser bootstrap used in incorrect target when I run heroku logs --tail

Clark Urzo09:10:15

Wait, let me post more from the logs

2020-10-14T09:15:45.000000+00:00 app[api]: Build succeeded
2020-10-14T09:15:46.930655+00:00 heroku[web.1]: Starting process with command `node public/js/compiled/main.js`
2020-10-14T09:15:50.116752+00:00 app[web.1]: Setting JAVA_TOOL_OPTIONS defaults based on dyno size. Custom settings will override them.
2020-10-14T09:15:50.323232+00:00 app[web.1]: /app/public/js/compiled/main.js:1564
2020-10-14T09:15:50.323261+00:00 app[web.1]: throw new Error("browser bootstrap used in incorrect target");
2020-10-14T09:15:50.323262+00:00 app[web.1]: ^
2020-10-14T09:15:50.323263+00:00 app[web.1]: 
2020-10-14T09:15:50.323263+00:00 app[web.1]: Error: browser bootstrap used in incorrect target
2020-10-14T09:15:50.323264+00:00 app[web.1]: at Object.<anonymous> (/app/public/js/compiled/main.js:1564:11)
2020-10-14T09:15:50.323264+00:00 app[web.1]: at Object.<anonymous> (/app/public/js/compiled/main.js:1695:3)
2020-10-14T09:15:50.323264+00:00 app[web.1]: at Module._compile (internal/modules/cjs/loader.js:1015:30)
2020-10-14T09:15:50.323265+00:00 app[web.1]: at Object.Module._extensions..js (internal/modules/cjs/loader.js:1035:10)
2020-10-14T09:15:50.323265+00:00 app[web.1]: at Module.load (internal/modules/cjs/loader.js:879:32)
2020-10-14T09:15:50.323266+00:00 app[web.1]: at Function.Module._load (internal/modules/cjs/loader.js:724:14)
2020-10-14T09:15:50.323266+00:00 app[web.1]: at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:60:12)
2020-10-14T09:15:50.323266+00:00 app[web.1]: at internal/main/run_main_module.js:17:47
2020-10-14T09:15:50.376140+00:00 heroku[web.1]: Process exited with status 1
2020-10-14T09:15:50.412256+00:00 heroku[web.1]: State changed from starting to crashed
2020-10-14T09:15:51.886696+00:00 heroku[router]: at=error code=H10 desc="App crashed" method=GET path="/" host= request_id=b6bb93ae-1e8d-4b0f-ae8c-168effede9dc fwd="152.32.111.228" dyno= connect= service= status=503 bytes= protocol=https
2020-10-14T09:15:52.666612+00:00 heroku[router]: at=error code=H10 desc="App crashed" method=GET path="/" host= request_id=8031aa23-9a08-4d57-839f-0864b8b33081 fwd="152.32.111.228" dyno= connect= service= status=503 bytes= protocol=https

practicalli10:10:10

Posting a link to the logs or adding them as a thread would be a preferred way to share, avoiding taking up lots of space in the main thread. Thanks

Michaël Salihi11:10:14

@signup867 You say a "simple Reagent" so I suppose you only target the front-end. If so you don't need NodeJS. The error seems your trying to run the Reagent/React compiled file with node which that cannot works. > 2020-10-14T09:15:46.930655+00:00 heroku[web.1]: Starting process with command node public/js/compiled/main.js If you tried this command on localhost, you'll get the same error. Node don't know about JS targeting the browser https://github.com/thheller/shadow-cljs/blob/b05b8f41afd45564ef7b9831a07201d09a5c4a07/src/main/shadow/boot/browser.js#L9

Michaël Salihi11:10:37

@signup867 The third part of the guide your mentionning talks about targeting NodeJS so you can run an CLJS app like this one https://github.com/PrestanceDesign/cljs-sample-express but not a Reagent one.

Clark Urzo11:10:34

Ahh, hmm so does that mean I have to roll out my own server?

Michaël Salihi11:10:40

No you have a bunch of options like Netlify, Github pages, Surge, even Netify if you want to stay with: https://create-react-app.dev/docs/deployment/

Michaël Salihi12:10:34

I think I remember having exchanged with you last month about Reagent on Netlify, right? > Ahh, hmm so does that mean I have to roll out my own server? Maybe i didn't fully understand your needs for Heroku, do you need a backend?

Clark Urzo10:10:11

Not really. In fact it’d be great if I can just copy paste my public folder and let it serve my files

Clark Urzo10:10:20

The reason why I think I need a backend is because I need to store input from users. I’ll need a database for this, right? My impression is that you can’t do this with Netlify and other static hosts.

Clark Urzo10:10:22

(Sorry, I’m really new to web dev but otherwise not that new to programming in general. Hope you understand.)

Dave Russell14:10:49

Hey folks! I'm trying to monkey-patch a function in one of my deps. I've tried both (intern 'dep-ns dep-fn (defn ...... as well as:

(in-ns 'dep-ns)

(defn dep-fn ....

(in-ns 'my-ns)
But I'm getting a java.lang.ClassNoFoundException: dep-ns Any ideas? 🙂 I can resolve dep-ns functions from my-ns, so it's not some issue where I can't require the dependency.

dpsutton14:10:48

clj
Clojure 1.10.1
user=> (require 'clojure.set)
nil
user=> (in-ns 'clojure.set)
#object[clojure.lang.Namespace 0x5a772895 "clojure.set"]
clojure.set=> (defn union [x y] (prn "i've changed this"))
#'clojure.set/union
clojure.set=> (in-ns 'user)
#object[clojure.lang.Namespace 0x1c12f3ee "user"]
user=> (clojure.set/union #{:a} #{:b})
"i've changed this"
nil
user=>
did you require 'dep-ns ?

Dave Russell14:10:23

(ns my.ns
   (:require [clojure.set])
(in-ns 'clojure.set)
(defn union [x y] (prn "i've changed this"))
(in-ns 'my.ns)
(clojure.set/union #{:a} #{:b})
This gives me an arity exception saying 3 args are passed to union. Note that I'm not running at the REPL

dpsutton14:10:14

can you show the stack trace?

Dave Russell14:10:35

But when I move it into a function (e.g. wrap the above lines inside a function) and then evaluate that function at the REPL, it works

Dave Russell14:10:55

So it would seem that union is being called elsewhere after I redefine it, when the reader goes through?

Dave Russell14:10:58

Would that make sense?

dpsutton14:10:58

what does the stack trace say?

dpsutton14:10:45

it should tell you where its called with three args. and union has a much more rich signature than what we did just for example purposes

dpsutton14:10:49

([] [s1] [s1 s2] [s1 s2 & sets])

Dave Russell15:10:21

Sorry I have to run off but I appreciate the help and will let you know if I get to the bottom of it :)

Cj Pangilinan14:10:27

Hi, clojure beginner here. i also use java and javascript at work.

Cj Pangilinan14:10:08

i am on windows 7. i use lein.bat. but i read this new other tool, clj, which needs powershell. which one should i use? i am now learning the basics of clojure. i plan to create web apps and games in the future. the goal is to use clojure in the back end and clojurescript in the front end.

seancorfield17:10:33

If you're stuck on Windows 7, you probably can't install a recent enough Powershell to be able to use the new Clojure CLI.

seancorfield17:10:15

But ask in #clj-on-windows in case folks there know more about Windows 7 and Powershell than I do (very likely -- I'm a lot more familiar with Windows 10).

Cj Pangilinan23:10:49

Thanks. Hmmm upgrade to Windows 10 with the new laptop will be next year. I guess lein.bat is sufficient for now to learn clojure basics.

seancorfield23:10:54

A lot of the tutorials out there still use Leiningen as well as many of the books, so you should be fine with lein for learning. Windows is a bit of a second-class citizen in the Clojure world, so even with Windows 10, it's a lot easier to do everything using WSL2 which is how I work with Clojure: WSL2 with Ubuntu, Clojure CLI, use VcXsrv (Xlaunch) on Windows to support X11 apps, then run everything on WSL2, including my editor.

practicalli00:10:15

The Clojure code itself will be the same regardless of if using Leiningen or Clojure CLI tools. The difference is is the details of how you configure Clojure projects and even then there is much cross-over.

Cj Pangilinan08:10:00

Thanks Sean and John.

Cj Pangilinan08:10:46

John, i found your Youtube channel just now. I subscribed to it. Watching 4clojure guides playlist now.

practicalli09:10:10

Feel free to ask any questions about my content in the #practicalli channel or send me a direct message.

Pablo20:10:56

Hello everyone! Is there a more ‘elegant’ way to write this?

(defn remove-empty-key [m k]
  (if (empty? (k m))
    (dissoc m k)
    m))

(-> (get-from-db {:some :person})
	(remove-empty-key :kids))

manutter5120:10:15

I suppose the more elegant way would be not to add the key to the map in the first place if the value is empty.

Pablo20:10:12

It comes from DB. It returns things like:

{:name "Maria"
 :kids []}

manutter5120:10:25

But remove-empty-key is simple and straightforward, so I’d say that’s as elegant as anything else.

Michaël Salihi20:10:59

Hello everybody! I add one more Clojure implementation on the website http://www.todobackend.com/. This time using a more recent "trending" stack with Reitit and Next-jdbc. You can check the repo directly here: https://github.com/PrestanceDesign/todo-backend-clojure-reitit I think that can be useful. Cheers!