Fork me on GitHub
#clojure
<
2023-01-04
>
kanwei16:01:25

how do you do REPL driven development with Integrant? we use mount currently and we can just (mount/start) and then use the vars in the REPL. In integrant, you have to pass the system into the function, so how do you pass in the system? Do you have to spin up a new system for each REPL query, or use a global system var? In which case that feels just like using mount?

p-himik16:01:00

There's a integrant.repl namespace with some useful functions.

p-himik16:01:33

And you can store systems however you like, and then use the right ones in the right context.

p-himik16:01:49

There's also integrant.repl.state namespace, but nothing forces you to use it.

p-himik16:01:33

Ah, those namespaces aren't built into Integrant itself, they're in a sister project that Integrant's README mentions: https://github.com/weavejester/integrant-repl

vlaaad16:01:18

Yeah, so the idea is that during dev you have a system saved somewhere easily accessible by the repl

kanwei17:01:24

hmm okay, i'll take a look thanks!

markbastian17:01:26

Here's a fragment from a project I'm currently working on that shows how I do it:

(def config
  {::conn          {:schema chat-schema}
   ::clients       {}
   ::jetty9/server {:host             "0.0.0.0"
                    :port             3000
                    :join?            false
                    :clients          (ig/ref ::clients)
                    :conn             (ig/ref ::conn)
                    :handler          #'web/handler}})

(defonce ^:dynamic *system* nil)

(defn system [] *system*)

(defn start
  ([config]
   (alter-var-root #'*system* (fn [s] (if-not s (ig/init config) s))))
  ([] (start config)))

(defn stop []
  (alter-var-root #'*system* (fn [s] (when s (ig/halt! s) nil))))

(defn restart
  ([config] (stop) (start config))
  ([] (restart config)))

(comment
  (start)
  (stop)
  (restart)
  (system)
  )
With the optional config arg you can have multiple configs in your project and call restart on any of them.

markbastian17:01:38

And if you want to use certain elements of your system in the REPL (e.g. the conn from above), you can do something like this in your REPL:

(let [conn (::conn (system))]
  @conn)

(let [clients (::clients (system))]
  clients)

(let [conn (::conn (system))]
  (d/entity @conn [:username "C"]))

markbastian17:01:31

Also, if you var quote your top level handler (such as the web handler from above) you don't need to bounce the system if you are working on handler logic.

kwladyka17:01:02

{:env/variables {:version (app-version)
                   :k_revision (System/getenv "K_REVISION")}
   :db/postgresql {:uri (System/getenv "postgresql")}
   :data-providers/binance {:auth {:api-key (System/getenv "binance_api_key")
                                   :secret-key (System/getenv "binance_secret_key")}}
   :logs/google-json-payload {:level (keyword (System/getenv "log_level"))
                              :is-loggable? (fn [{:keys [^String logger-name ^Level level]}]
                                              (let [level-int (.intValue level)
                                                    log? (fn [{:keys [logger level]}]
                                                           (and (.startsWith logger-name logger)
                                                                (< level-int (google-json-payload/JUL-levels->int level))))]
                                                (not
                                                  (some log?
                                                        [{:logger "org.postgresql" :level :config}
                                                         {:logger "jdk.event.security" :level :config}
                                                         {:logger "okhttp3.internal" :level :config}
                                                         {:logger "sun.net.www.protocol.http.HttpURLConnection" :level :config}]))))
                              :pretty? (nil? (System/getenv "K_REVISION"))}
   }
I have a config which get values from env variables or from other place. The config is the same code. But values changes depends on environment.

Nundrum18:01:05

Is there a specific place to take memory leak questions? I have a long-running Clojure2D-based project. After about a day on an rPi it crashes with an OOM. Looking at the project on my desktkop, it looks like one of the send-off threads just keeps growing.

jumar18:01:35

I think here it's fine. There's #C03L9H1FBM4 but that's much smaller group

👍 2
jumar18:01:26

What tool have you used so far to monitor the process? Also, there's `-XX:+HeapDumpOnOutOfMemoryError` that you can use to create a heap dump on OOM and then investigate it separately in VisualVM or something similar

Nundrum18:01:21

Thus far visualvm

Nundrum18:01:23

I have a task running every 10 seconds, and the send-off thread is active and growing on that schedule. So that seems to be the likely culprit.

Nundrum18:01:50

It uses clj-http to fetch data and instaparse to parse it.

jumar18:01:06

You can take a heapdump at any time in VisualVM and investigate it

Nundrum18:01:08

But I'm starting to wonder if the future is the problem.

jumar18:01:44

Hard to tell without seeing the code.

Nundrum18:01:16

(defn periodically [func millis]
  (let [p (promise)]
    (future
      (while 
        (= (deref p millis :timeout) :timeout)
        (func)))
    #(deliver p :cancel)))

Nundrum18:01:14

Yeah, when I cancel it, the heap size drops by several GB!

rutledgepaulv19:01:01

a good way to investigate memory leaks is to take multiple heap dumps over time and then focus on the objects that exist in consecutive dumps that you would have expected to get garbage collected instead. once you identify those you can work backwards to figure out what's hanging onto the references and why

jumar19:01:07

I'd focus on func - it seems it's somehow holding the data in memory.

rutledgepaulv19:01:23

Yeah either func or something accumulating in thread local on the future thread

Nundrum19:01:09

How do you tell what's what in this heap dump?

Ben Sless19:01:55

You can analyze heap dumps with visual VM or with eclipse MAT

Nundrum19:01:38

I'm using visualvm right now, but it's just a bunch of Object or Byte. I don't see how to match those to the Clojure data structures I have.

jumar19:01:22

It can be challenging- what does the profile look like?

Ben Sless19:01:29

MAT is slightly better at making sense of heap dumps https://www.eclipse.org/mat/

Nundrum20:01:33

I tried MAT but still don't get it. I see a bunch of vars and agents, but still have no idea which one is which. Even worse, the reports don't work.

Nundrum20:01:10

@U06BE1L6T which profile are you talking about?

jumar20:01:44

I mean the heap dump. Maybe you can at least share a screenshot Ideally, try to create a minimal public reproducer - then it would be easier for people to help you

Ben Sless20:01:56

you're probably holding on to lots of strings

Ben Sless20:01:05

probably in a map

Nundrum20:01:20

Is there some guide as to what to look for? I haven't been able to find anything.

Ben Sless20:01:26

Sadly, not so much. You can try exploring the reports generated by MAT, they are interactive, or if you can, share the heap dump

Ben Sless20:01:41

If that's not possible, I can try to hop on a call tomorrow to help you analyze it if you'd like, but it's mostly heuristics, unless you want to start writing weird queries on the heap in some abomination language

😄 4
Nundrum20:01:27

I can't get MAT to generate any reports. It pops up a "save this index.html" dialogue, but I never get a report.

Nundrum20:01:11

Have a good place to drop the heap dump?

Ben Sless20:01:26

you can even upload it here :man-shrugging:

Ben Sless20:01:17

if it's big you might want to zip it first

Nundrum22:01:24

I removed a spot where a var was being re-defed in the periodic function and replace that with an atom. Had put it there for debugging and forgot about it. Looks like it is more stable for the moment, but I'll have to let it run awhile to be sure.

Nundrum02:01:34

With that simple change, it has been stable for a day now. Wah hoo!