Fork me on GitHub
#clojure
<
2022-01-31
>
wombawomba12:01:32

Is there an EDN parser that can output a full AST including line and column numbers?

p-himik12:01:14

EDN is a subset of Clojure syntax, right? If so, when it comes to parsing Clojure code while getting info on line/column numbers people always mention rewrite-clj.

wombawomba13:01:28

perfect, thanks!

Lucas Jordan17:01:07

I am trying to optimize a clojure function. When I use Java VisualVM to look at the running process. I see low CPU utilization, however, about 1000 classes are created per call. The function in question is recursive (non-tail), is that the likely culprit? Or is there another known reason why crap tons of classes would be created? Thanks in advance for any advice!

ghadi17:01:02

Classes created or instances created?

ghadi17:01:15

Share raw data if possible

ghadi17:01:21

Sources/ graphs

Lucas Jordan17:01:29

graph coming…

ghadi17:01:03

The function in question would be really useful too

Lucas Jordan17:01:32

I can share that but not the data (without obfuscating it). one sec. thanks!

Lucas Jordan18:01:18

Green circles are func in question, when run from repl

Lucas Jordan18:01:04

(defn- process-message
  [hl7 parsed segment-defs func]
  (let [_ nil]
    (loop [parsed (remove-unknown parsed (all-known-segments segment-defs))
           remaining segment-defs
           result {}
           lines-read 0]
      (if-let [seg-def (first remaining)]
        (if-let [seg (first parsed)]
          (cond
            (some? (seg-def :pea.hl7/id)) (let [seg-def-id (seg-def :pea.hl7/id)
                                                seg-details (get-in hl7 [:pea.hl7/segments seg-def-id])
                                                ;_ (println (str "seg-details: " seg-details))
                                                title-key (keyword (.replace (.toLowerCase (seg-details :pea.hl7/desc)) " " "-"))
                                                seg-id (select-first [0 0 0 0] seg)
                                                not-repeating (= :pea.hl7/none (seg-def :pea.hl7/repeatability))
                                                _ nil]
                                            (if (= seg-def-id seg-id)
                                              (if not-repeating
                                                (recur (rest parsed) (rest remaining) (assoc result title-key (func seg seg-def)) (+ lines-read 1))
                                                (let [previous (or (result title-key) [])
                                                      value (func seg seg-def)
                                                      values (conj previous value)]
                                                  (recur (rest parsed) remaining (assoc result title-key values) (+ lines-read 1))))
                                              (recur parsed (rest remaining) result lines-read)))
            (some? (seg-def :pea.hl7/segments)) (let [title-key (keyword (.replace (.toLowerCase (seg-def :pea.hl7/segment-group)) " " "-"))
                                                      segments (seg-def :pea.hl7/segments)
                                                      not-repeating (= :pea.hl7/none (seg-def :pea.hl7/repeatability))
                                                      sub-proc (process-message hl7 parsed segments func)
                                                      sub-group (sub-proc 0)
                                                      lines-to-drop (sub-proc 1)
                                                      _ nil]
                                                  (if not-repeating
                                                    (recur (drop lines-to-drop parsed) (rest remaining) (assoc result title-key sub-group) (+ lines-read lines-to-drop))
                                                    (let [previous (or (result title-key) [])
                                                          values (conj previous sub-group)
                                                          the-remaining (if (= 0 lines-to-drop) (rest remaining) remaining)]
                                                      (recur (drop lines-to-drop parsed) the-remaining (assoc result title-key values) (+ lines-read lines-to-drop)))))
            :default (throw (Exception. (str "Don't really know how we got here: " remaining))))
          [result lines-read])
        [result lines-read]))))

Ben Sless18:01:43

Interning lots of keywords? :thinking_face:

ghadi18:01:56

that doesn't create classes

Lucas Jordan18:01:26

Thats what I thought 🙂

Lucas Jordan18:01:47

dynamic classes I guess?

Lucas Jordan18:01:48

clojure.lang.DynamicClassLoader.defineClass() is using the majority of time when “sampleing” the process in Java VisualVM… screenshot coming

Ben Sless18:01:05

right, that's just instances of singletons

ghadi18:01:08

I'm more concerned about Compiler$ObjExpr

ghadi18:01:36

are you sure something isn't calling eval?

ghadi18:01:22

dynamically making defrecords, etc

Ben Sless18:01:26

I'm wondering if this includes results from requireing in the ns form

Ben Sless18:01:04

If the profile is since JVM startup and the function was not AOT compiled, is it possible?

ghadi18:01:48

having started out my career doing HL7, my money's on dynamic defrecords

Lucas Jordan18:01:03

no defrecords, just maps. Ill look for calls to eval I might not be remembering.

Lucas Jordan18:01:19

I am on java8 (macOS) is that a hnt?

ghadi18:01:52

from the profile, the application is definitely compiling clojure things

ghadi18:01:06

the -Xlog statement will help you

Lucas Jordan18:01:20

jvm failed to start with that

ghadi18:01:21

or grepping for eval

ghadi18:01:33

oh on java 8 use -verbose:class

Lucas Jordan18:01:41

thanks, trying that

Lucas Jordan18:01:16

So thats interesting…. the verbose

Lucas Jordan18:01:26

gives me lines like this:

Lucas Jordan18:01:26

[Loaded pnx.hl7.discharge_window$eval22184 from JVM_DefineClass__] [Loaded pnx.hl7.discharge_window$eval22188$fn__22189 from JVM_DefineClass__]

Lucas Jordan18:01:23

‘discharge_window’ is the namespace I am using in repl to explore this

Lucas Jordan18:01:56

is the slowness just from running in a repl?

Lucas Jordan18:01:28

when I use this: (set! warn-on-reflection true)

Lucas Jordan18:01:15

I get this: Reflection warning, /Users/lucasjordan/pea/workspaces/pnx-api/siderail/pnx/hl7/discharge_window.clj:47:1 - call to method replace can’t be resolved (target class is unknown).

Lucas Jordan18:01:20

A similar number of times

Ben Sless18:01:09

If you're running with Leiningen you might have a flag which disables JIT which will make things slower Reflection is also expensive, try type hinting .replace with ^String I'm not sure how Clojure's reflector works, does it define classes?

ghadi18:01:14

it does not

ghadi18:01:52

look inside pnx.hl7.discharge-window namespace for eval

ghadi18:01:39

do you have some automatic reload middleware? or similar

ghadi18:01:46

if so, crush that

1
Lucas Jordan19:01:13

It was a deeply buried eval I thought was being called before my testing, but was lazyliy being called…. thanks everyone for the help!!

ghadi18:01:48

use a snippet if possible (CMD-Shift-Enter, you can even have it highlight clojure syntax)

ghadi18:01:33

are any of the helper fns calling eval? dynamically generating regexes?

Lucas Jordan18:01:00

the intention here is to take a list like [{},{},{},{},{},{},{},{}] and turn it into this: [{}, {}, [{} {}], {:a {} :c [{} {}]}] based on some rules

leif18:01:38

I posted this a few weeks back, but since we're still looking for more responses, I'll post it here one more time for those who missed it: The Programing Research Laboratory at Northeastern University is conducting a study on the usability of visual and interactive syntax for ClojureScript. Participation involves filling out a questionnaire and is expected to take about 30 minutes. The questionnaire consists of multiple choice and free form responses. All questions are optional and responses are anonymous. If you are interested in participating, please visit: https://study.visr.pl (Chrome Recommended) To learn more about visual and interactive syntax for ClojureScript, please visit: https://prl.ccs.neu.edu/blog/2022/01/06/introducing-visual-and-interactive-syntax-realized-visr-for-clojurescript-and-javascript If you have any questions, please contact: <mailto:[email protected]|[email protected]>

ghadi18:01:40

@lucasjordan another thing you can do is turn on logging when classes are loaded: -Xlog:class+load=info

👀 1
annarcana23:01:01

Working on finally updating the https://github.com/jarcane/clojurice template, and running into the dreaded "illegal reflective access operation" errors, and even after updating pretty much every dependency possible in the entire application I'm still getting it. It's not even the same one all the time: in local dev, I get the error on the XNIO library, while on Travis, the CI logs show an error in dynapath. I managed to find an old issue about the latter but it indicated it should've been fixed years ago. Will attach boot show -d in the thread, would welcome any ideas; I've scared up and down the tree and I'm not sure where the culprit is or if I can fix it.

hiredman23:01:30

you need to look at the error message and see where it occurred and fix that, blindly updating deps isn't guaranteed to fix anything

hiredman23:01:52

I would ignore travis, fix local, then make sure local and travis match

Alex Miller (Clojure team)23:01:18

I would strongly recommend using the suggestion at the end there to use the debug setting

hiredman23:01:37

immutant is, I believe, no longer maintained, so there are much newer releases of the underlying libraries there

annarcana23:01:45

OK. Digging deeper again it looks like the problem is immutant yeah

annarcana23:01:59

the warning I get is this:

WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.xnio.nio.NioXnio$2 (file:/home/annarcana/.m2/repository/org/jboss/xnio/xnio-nio/3.3.6.Final/xnio-nio-3.3.6.Final.jar) to constructor sun.nio.ch.EPollSelectorProvider()
WARNING: Please consider reporting this to the maintainers of org.xnio.nio.NioXnio$2
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release

annarcana23:01:33

digging in the tree, it looks like XNIO is used in Undertow 1, which is used in immuntant/web

hiredman23:01:39

yeah, the most recent xnio stuff is, I believe 4.x

annarcana23:01:54

the immutant dep is there because of system; I wonder if I can't replace that then

annarcana23:01:50

haha, yes. http-kit serves as an alternate and fixes the issue completely. no more immutant. thank you!