Fork me on GitHub
#beginners
<
2020-08-30
>
vlad_poh06:08:25

Howdy y'all! I have a little clojure web application and i'm trying to embed jasper reports. When i run my application with lein run or lein repl it works! When i create an uberjar and run it it throws a null pointer exception on some reports

java.lang.NullPointerException
        at java.base/java.io.File.<init>(File.java:278)
        at net.sf.jasperreports.engine.JasperFillManager.fill(JasperFillManager.java:317)
        at net.sf.jasperreports.engine.JasperFillManager.fillReport(JasperFillManager.java:850)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:167)
        at clojure.lang.Reflector.invokeStaticMethod(Reflector.java:332)
        at vgs.api.report$get_pdf$fn__17267.invoke(report.clj:80)
and a jruntime exception on others
at net.sf.jasperreports.engine.util.JRQueryExecuterUtils getExecuterFactory "JRQueryExecuterUtils.java" 103
  at net.sf.jasperreports.engine.fill.JRFillDataset createQueryDatasource "JRFillDataset.java" 1251
  at net.sf.jasperreports.engine.fill.JRFillDataset initDatasource "JRFillDataset.java" 726
  at net.sf.jasperreports.engine.fill.BaseReportFiller setParameters "BaseReportFiller.java" 457
  at net.sf.jasperreports.engine.fill.JRBaseFiller fill "JRBaseFiller.java" 584
  at net.sf.jasperreports.engine.fill.BaseReportFiller fill "BaseReportFiller.java" 414
  at net.sf.jasperreports.engine.fill.JRFiller fill "JRFiller.java" 120
  at net.sf.jasperreports.engine.fill.JRFiller fill "JRFiller.java" 103
  at net.sf.jasperreports.engine.JasperFillManager fill "JasperFillManager.java" 530
  at net.sf.jasperreports.engine.JasperFillManager fillReport "JasperFillManager.java" 954
  at jdk.internal.reflect.NativeMethodAccessorImpl invoke0 "NativeMethodAccessorImpl.java" -2
  at jdk.internal.reflect.NativeMethodAccessorImpl invoke "NativeMethodAccessorImpl.java" 62
  at jdk.internal.reflect.DelegatingMethodAccessorImpl invoke "DelegatingMethodAccessorImpl.java" 43
  at java.lang.reflect.Method invoke "Method.java" 566
  at clojure.lang.Reflector invokeMatchingMethod "Reflector.java" 167
  at clojure.lang.Reflector invokeStaticMethod "Reflector.java" 332
  at vgs.api.report$get_pdf$fn__17267 invoke "report.clj" 80
but not a single one works from the jar'ed app. Any suggestions on troubleshooting the problem.

phronmophobic06:08:53

they're both failing on get-pdf in report.clj (on line 80). where are the arguments for that call coming from?

phronmophobic06:08:13

my guess is that it's trying to find a file, but the environment changed in way that makes it not find the file it's looking for. what's the working directly when you run from the repl? is it the same working directory when you try to run the jar?

vlad_poh07:08:40

how do get the pwd in clojure? trouble translating the answer here https://stackoverflow.com/questions/320542/how-to-get-the-path-of-a-running-jar-file

return new File(MyClass.class.getProtectionDomain().getCodeSource().getLocation()
    .toURI()).getPath();

phronmophobic07:08:32

I was referring to whatever directory you're running the jar from

vlad_poh06:08:33

yep its in the same folder as the jar file

phronmophobic07:08:12

usually, the file path is relative to the current working directory (ie. pwd), which may be different than the location of the jar fiile

vlad_poh07:08:56

Line 80

(try (JasperFillManager/fillReport
                            (get @jasper-reports (keyword reportname))
                            (map->java params)
                            (jdbc/get-connection db))
                           (catch JRRuntimeException jrrx (vgerror :get-pdf|fill (seq (.getStackTrace jrrx))) (JasperPrint.))
                           (catch JRException jrex (vgerror :get-pdf|fill (seq (.getStackTrace jrex))) (JasperPrint.)))
the first parameter is a JasperPrint which is loaded in memory

phronmophobic07:08:58

what about params?

vlad_poh07:08:22

has report parameters which i convert to a java borrowed the function

(defn map->java
  "Convert Clojure hash-map to Java HashMap object"
  [data-map]
  (let [hashmap-reducer (fn [m [k v]] (doto m (.put (name k) v)))]
    (reduce hashmap-reducer (java.util.HashMap.) data-map)))

phronmophobic07:08:48

right, but what is the value of params before it's passed to fillReport?

vlad_poh07:08:59

depending on the report it is a hashmap of dates and ints

phronmophobic07:08:17

ok, then I would check what the value of (get @jasper-reports (keyword reportname))

phronmophobic07:08:23

it's either one of the 3 arguments to fillReport that is different or the environment is somehow different

dpsutton07:08:30

and are you sure that (get @jasper-reports (keyword reportname)) is non-nill when run in a jar?

vlad_poh07:08:24

positive. I added some printlns to count the reports loaded in memory but let me try that call on its own and see.

dpsutton07:08:38

how do you populate that?

Dimitar Uzunov09:08:59

So I wrote this program to parse some multiple GB json log files.

(ns loghell.main
  (:require [cheshire.core :as cs])
  (:require [ :as io])
  (:require [clojure.pprint :as pp]))          

(def log13h "ut5-2020-08-28_13H.log")


(defn parsed [line]
  (cs/parse-string line true)) ; for keywords instead of strings


(defn pred [i]
  "AND of predicates to filter the logs"
  (and 
       (not= true (get-in i [:jsException :isJoi])) ;ignore joi validations
       (= (:level i) 50))) ;get only errors

(defn needed [i]
  "what properties/keys are needed from the log"
 (select-keys i [:msg :name :time]))


(defn loginv [pred? needed log]
  "investigate logs by doing an AND of predicates"
  (with-open [r (io/reader log)]
   (let [s (line-seq r)]
     (-> needed
         (pmap
          (filter #(pred? %)
                  (pmap parsed s)))
         doall))))

(time (loginv pred needed log13h))

Dimitar Uzunov10:08:44

I’m a bit unsure about doall - it puts the whole seq in memory; is there a way to avoid it?

Timur Latypoff10:08:24

I think dorun is what you’re looking for instead of doall — it does not retain the whole seq in memory.

Dimitar Uzunov10:08:37

I tried dorun, but I guess I don’t understand how it works. I think I need to process everything in one pass because it doesn’t retain the head of the seq

Timur Latypoff10:08:24

You should use dorun only in the very end, to “force” the lazy sequence. @ULE3UT8Q5

Dimitar Uzunov10:08:51

I tried to just replace doall with dorun but instead of the lazyseq with maps I get a nil

Dimitar Uzunov10:08:11

so it makes me think I’m using it wrong somehow 🙂

Timur Latypoff10:08:13

Actually, I read your code once again, and yeah, my mistake — you need the resulting sequence

Timur Latypoff10:08:09

Since you’re filtering, I don’t think the whole source file is retained memory, don’t worry about that

Dimitar Uzunov10:08:24

is it a map vs filter difference? anyways I will try to do a parse, filter and select keys in one function and see how it goes

Dimitar Uzunov10:08:47

I mean to do all these actions per line

Timur Latypoff10:08:33

Actually, can you try not keeping the seq in the local variable s, but running it directly in-place? I believe right now the sequence s cannot be garbage-collected because you keep the head, so actually the whole file is in-memory

Dimitar Uzunov15:08:24

Hmm running it directly in place doesn't seem to change anything

Dimitar Uzunov15:08:01

I tried visualvm on the process but the heap doesn't grow at all when I run the function

Timur Latypoff15:08:26

@ULE3UT8Q5 I think in this case, Clojure’s compiler “sees” that you don’t use the local variable after some point, and nulls it behind the scene (which allows the GC to clean it up): https://clojure.org/reference/compilation#_locals_clearing So I was wrong again, and Clojure compiler is smarter than I thought :)

Dimitar Uzunov10:08:15

also is it the final seq after the pmap and filter, or does it load the whole multi-GB file in memory?

Dimitar Uzunov10:08:38

the time I got is 5 seconds for a 1 gb file, but I want to parse and analyse even longer ones

Dimitar Uzunov10:08:01

I’m also concerned that I’m doing pmap, filter and pmap again, should I use transducers so I can do it in one pass?

Timur Latypoff10:08:54

If I am not mistaken, nesting pmaps is superfluous. Also, I believe pmaps are no magic bullets, and they will help mostly only if individual item processing is rather heavy.

Dimitar Uzunov10:08:41

for what it is worth these pmaps give a 5s processing time compared to the 8s maps 🙂

Timur Latypoff10:08:20

That’s certainly more than I expected :)

Dimitar Uzunov10:08:05

this file has lines that are pretty big, with nested json 🙂

Mark Gerard11:08:15

Hello, I know there is a better way to do this: [`(.getYear timestamp) (.getMonthValue timestamp) (.getDayOfMonth timestamp)]`

jumar12:08:02

If they were clojure fns and not java methods you could use juxt easily; also depends on what you want to do with the result - maybe you can leverage date formatter instead

jumar14:08:24

You'd have to do something like this since java methods aren't first class Clojure fns:

((juxt #(.getYear %) #(.getMonth %) #(.getDay %)) (java.util.Date.))
;;=> [120 7 0]

Mark Gerard14:08:15

Thanks a bunch

jumar14:08:15

No problem; as I said, there might be a better approach depending on how you are going to use this representation

Mark Gerard14:08:45

Just wanted to get a date (year month and day)

Mark Gerard12:08:31

I want to remove the repetition. Do you mean bean the first func call and then thread it?

Alex Miller (Clojure team)12:08:45

You could bean, then juxt

Alex Miller (Clojure team)12:08:36

Although I think what you were doing is probably the most efficient answer

Mark Gerard08:08:52

Thanks @alexmiller the closest I came to was this: (select-keys (bean date) [:year :month :day])

Mark Gerard08:08:44

That versus ((juxt #(.getYear %) #(.getMonth %) #(.getDay %)) date)

Mark Gerard08:08:39

I am not sure how to combine those two as you suggested - but these seem to work. I am just curious how to combine them. I am assuming that date is an instance of (java.util.Date.)

Mark Gerard12:08:09

The repetition is triggering my OCD

Mark Gerard12:08:19

Or rather DCD

kennytilton20:08:56

Ooooh, this is scary, I come up with a bean-ish question and this channel is discussing beans! Anyway, here I am messing with datomic seriously for the first time and using the low-level API I am retrieving #datoms such as #datom[83562883711053 10 :orange-ish 13194139533330 true] and I want to take it apart. bean complains about being unable to process keywords or sth unhelpful. I put up a hail mary and (:a <datom>) works. Yay, But (:t d) does not. So what are the other keys (I wondered before I found the java doc). :tx I had been able to guess, but the active value? OK, I sorted it out thx to the doc, but why did not bean work, and what else could I have used. Thx!

seancorfield21:08:46

@hiskennyness Out of curiosity, did you try calling keys on that #datom object? fails with Don't know how to create ISeq from: datomic.core.db.Datum

seancorfield21:08:45

(my assumption would be that it is probably a record, so I would expect keys to work...) wrong... I guess it's a Java type or a deftype

seancorfield21:08:24

You can use org.clojure/java.data to get the datoms as hash maps:

$ clj -Sdeps '{:deps {com.datomic/dev-local {:mvn/version "0.9.195"} org.clojure/java.data {:mvn/version "RELEASE"}}}'
...
user=> (def y (last (d/datoms db {:index :eavt})))
;; in my case that is:
#datom[83562883711058 10 :yellow 13194139533320 true]
user=> (require '[clojure.java.data :as j])
nil
user=> (j/from-java-shallow y {:exceptions :omit})
{:v :yellow, :e 83562883711058, :tx 13194139533320, :t 8, :p 19, :a 10, :assertion true}
user=>
^ Just in case that helps @hiskennyness

kennytilton22:08:58

Cool, thx, I will give it a try! Yeah, I had tried keys when I saw to my surprise things like :a work. Surprised again when it did not. 🙂 And that error you show is reminiscent of what bean failed with.

Alex Miller (Clojure team)22:08:38

keys and bean both expect certain things about maps that Datomic datoms don't provide. The keyword lookup protocol is ILookup, and that one is supported. I'd refer to the Datomic docs for what expectations they provide (or ask in #datomic)

seancorfield22:08:58

Looks like a fairly limited set of keys are supported:

user=> (:v y)
:yellow
user=> (:e y)
83562883711058
user=> (:tx y)
13194139533320
user=> (:t y)
Execution error (IllegalArgumentException) at datomic.core.db.Datum/valAt (db.clj:374).
No matching clause: :t
user=> (:p y)
Execution error (IllegalArgumentException) at datomic.core.db.Datum/valAt (db.clj:374).
No matching clause: :p
user=> (:a y)
10
user=> (:assertion y)
Execution error (IllegalArgumentException) at datomic.core.db.Datum/valAt (db.clj:374).
No matching clause: :assertion
user=> 
Which makes me curious what T and P are in a Datom?

seancorfield22:08:45

The rest are all obvious.

seancorfield22:08:44

Things not behaving like hash maps is partly what I added from-java-shallow (and from-java-deep) to java.data with the options to control what happens if a "getter" doesn't actually behave like a getter. You can also omit "properties" that line up with get* methods that "do bad things" (are mutating / do I/O / etc).

kennytilton22:08:23

Thx all! It felt like a beginner Java interop question, but I see after playing with from-java-shallow that (as you say, @alexmiller) datoms march to the beat of a different drummer. I'll go see about these "expectations" ... 🙂