Fork me on GitHub
#clojure
<
2021-10-14
>
Kris C06:10:31

How can I read a text file in batches of n lines? Is there a simple way of doing this?

seancorfield07:10:27

You can use line-seq to get a lazy sequence of lines. That's probably a good start.

Kris C07:10:25

and then write a recursive function with take n and drop n?

seancorfield07:10:51

It would depend on how exactly you want to consume/process it. What are you trying to do with those batches of lines?

seancorfield07:10:53

You can partition-all the line-seq, for example.

seancorfield07:10:05

So, what problem are you actually trying to solve here?

Kris C07:10:29

I have a file containing a large dataset and want to import that to a db

Kris C07:10:29

so, read n lines -> process transaction -> read next n lines...

seancorfield07:10:28

OK. So I'd probably structure that as

(run! transact-batch (partition-all n (line-seq source)))

seancorfield07:10:03

transact-batch would be invoked with a sequence of n lines at a time (except for the last batch which may be smaller).

seancorfield07:10:31

It could then use next.jdbc/execute-batch! to insert a block of n rows into the database.

Kris C07:10:42

uh, thanks a lot.

seancorfield07:10:25

If you need each line parsed into data suitable for the DB:

(run! transact-batch (partition-all n (parse-line (line-seq source))))

seancorfield07:10:52

(it's morning for you but after midnight here so I'm off to bed but feel free to post any follow-up Qs and I can answer when I'm awake again)

Jakub Holý (HolyJak)09:10:36

This is a mystery:

(do
  (def p (java.util.Properties.))
  (def ^:const btst4 {:k p})
  (:k btst4))
; => IllegalArgumentException: No matching ctor found for class java.util.Properties
But if I remove the ^:const it starts working. Any idea what is happening? (Clojure 1.10.1 and 1.10.3)

javahippie10:10:33

The exception (for me) is thrown in this line: https://github.com/clojure/clojure/blob/b8132f92f3c3862aa6cdd8a72e4e74802a63f673/src/jvm/clojure/lang/Reflector.java#L308 My interpretation is that it finds multiple constructors with the same arity, but none matches the parameter type?

javahippie10:10:35

I’m wondering about what args are assumed there, but I don’t have a setup which can debug into Java Code

vemv10:10:50

last time I checked ^:const isn't documented in http://clojure.org, it's simpler and better documented to use direct linking

Jakub Holý (HolyJak)10:10:56

Same here, I couldn't find it there. Worth direct linking you mean ^:inline?

p-himik10:10:23

^:const is mentioned in the API cheatsheet at least.

Jakub Holý (HolyJak)11:10:21

Thank you. If direct linking is a compiler flag then it isn't really relevant here. I am trying to define a map with rather static config options, that's why I marked it with const. I guess I will try to debug it as @U0N9SJHCH proposed to see what really is happening and whether or is a Clojure bug

p-himik11:10:24

^:const does not prevent the referenced object from changing. It simply tells the compiler that the var itself will not be changed.

p-himik11:10:45

So in your case, seems like you don't need it at all.

valerauko11:10:56

it's weird because it works with classes like HashMap

(def x (java.util.HashMap.))
(def ^:const bar {:foo x})
(:foo bar)

valerauko11:10:50

With Properties you don't even need to call it, hell you can't even wrap it in try-catch. I think it dies somewhere in the bowels of the compiler

(def x (java.util.Properties.))
(def ^:const bar {:foo x})
So far so good, but then
(defn check [] (:foo bar))
dies without ever calling (check)

javahippie11:10:36

Also, it works if you wrap it in a (defn…)

javahippie11:10:00

And Java 8 returns a different error message than Java 11

Alex Miller (Clojure team)12:10:26

This is not a place where I would expect const to work

Alex Miller (Clojure team)12:10:05

Use const when you have something like a number or a string you want to inline during compilation. Almost anything beyond that is probably a misuse

💯 1
Jakub Holý (HolyJak)12:10:06

Ah, ok, good to now :-) I thought it would make the field static final or something.

valerauko12:10:26

The inconsistency bugged me. https://github.com/clojure/clojure/blame/master/src/jvm/clojure/lang/Compiler.java#L7329-L7330 Seems like const stuff get quoted? Then I assume evaled at some later point? Thing is that even (eval {:k (java.util.Properties.)}) results in the same error. I added a ton of debug statements in the Reflector and for some reason it can't find the no-args constructor of Properties?

ctor public java.util.Properties(java.util.Properties)
params class java.util.Properties
args {}

ctor public java.util.Properties(int)
params int
args {}
I can't seem to replicate this with other java.util classes (and I tried with a few that aren't used in the Clojure source), so no idea why it only breaks with Properties

❤️ 1
valerauko12:10:36

Just some debug output from Compiler/analyze and Reflector

Alex Miller (Clojure team)13:10:56

These are two different things, but I think you're misunderstanding both of them.

Alex Miller (Clojure team)13:10:01

With const, you're saying that the value is a conpile-time constant and the compiler can replace the var reference in the bytecode with the literal value

Alex Miller (Clojure team)13:10:23

In the eval case, I'm not sure exactly what you're seeing there, but it feels like the fallback case the compiler uses to round trip arbitrary objects through bytecode by using print/read

valerauko13:10:29

Hm that makes it more clear. Any idea why it behaves differently with Properties and for example BitSet?

Alex Miller (Clojure team)13:10:10

no, without spending time I don't really have atm to find out. Properties is one of the worst classes in the JDK and it has a lot of eccentricities.

👍 1
Mateo Difranco12:10:10

Hmmm... what could be wrong here?

(map key t)
=> (:4 :1 :2 :5 :3)
(name :4)
=> "4"
(map (name key) t)
Execution error ...
clojure.core$key cannot be cast to clojure.lang.Named

opqdonut12:10:07

keywords are functions, strings aren't

Mateo Difranco12:10:00

You're right, that was silly.

(map #(name (key %)) t)
🙂

👏 1
opqdonut12:10:06

oh, and (name key) is applying the name function to the function key

Ed12:10:33

do you mean (map (comp name key) t) ??

☝️ 1
Mateo Difranco12:10:29

That's a nice solution 🙂

Muhammad Hamza Chippa13:10:01

is there any way to lexicographically sort list ("john" "harry" "anne") into ("anne" "harry" "john")

Johan Thorén13:10:37

(sort '("john" "harry" "anne"))
;; => ("anne" "harry" "john"

🙌 1
pavlosmelissinos14:10:55

Be careful when you sort collections of strings:

(sort ["John" "anne" "Harry"])
;; => ("Harry" "John" "anne")
Depending on the desired result you might want to sort-by lower-case

🎯 1
🙌 1
Daniel Ziltener14:10:13

Is there any library to "memoize" a function using Redis as the caching backend? I found the "crache" library, but it's broken

emccue14:10:47

I wouldn't imagine so since there are a few competing redis libraries and no clear winner

emccue14:10:56

but it shouldn't be that hard to write - just decide how you want to serialize your function's arguments and result to store in redis

seancorfield16:10:16

You could probably write a Redis backend to core.cache -- it's designed to be extensible.

Daniel Ziltener16:10:25

I am currently trying to do so, by trying to fix "crache", but so far I am failing, the last argument passed to CacheProtocol/miss seems to be a function from core.memoize...

seancorfield17:10:14

core.cache does not depend on core.memoize -- but core.memoize does depend on core.cache.

Apple18:10:09

deps.edn:
  :build {:paths ["src"]
          :deps {io.github.clojure/tools.build {:git/tag "v0.6.2" :git/sha "226fb52"}}
          :ns-default build}

build.clj:
(ns build
  (:require [clojure.tools.build.api :as b]
            [shadow.cljs.devtools.api :as shadow]))
(defn uber [_]
  (clean nil)
  (shadow/release :app)
  (b/copy-dir {:src-dirs ["target" "resources"]
               :target-dir class-dir})
  (b/compile-clj {:basis basis
                  :src-dirs ["src"]
                  :class-dir class-dir})
  (b/uber {:class-dir class-dir
           :uber-file uber-file
           :basis basis}))
Hi I have success in combining the cljs compilation part with clj to get a uber jar file. Question: is there a way to default the .jar to a main class instead of
java -cp xyz-1.2.2-standalone.jar clojure.main -m xyz.core
I'd like to use just
java -jar xyz-1.2.2-standalone.jar

Alex Miller (Clojure team)18:10:10

yes, you can use the :main key in b/uber

Alex Miller (Clojure team)18:10:50

(and btw, you might find #tools-build to be good for future questions like this)

Apple18:10:39

(b/uber {:class-dir class-dir
         :uber-file uber-file
         :basis basis
         :main 'xyz.core})
is this correct?

vncz18:10:12

Is there also an option to AOT compile the code? I have not been able to find anything

seancorfield18:10:39

There's a compile-clj function in tools.build

seancorfield18:10:15

It's shown in @zengxh’s build.clj above (further up).

vncz18:10:30

Oh so compile-clj would do the Ahead of Time compilation

vncz18:10:55

Interesting; I was receiving an error from graalvm when trying to create a native image even though I was using AOT. I might have done something else incorrectly.

vncz18:10:58

Thanks for the clarification

seancorfield18:10:58

Well, it's "just" compilation, but it is being done before building the uber JAR so it is "AOT" in that respect.

Apple18:10:07

$ java -jar xyz-0.1.4-standalone.jar 
Error: Could not find or load main class xyz.core
Caused by: java.lang.ClassNotFoundException: xyz.core
$ java -cp xyz-0.1.4-standalone.jar clojure.main -m xyz.core
<OK>

vncz18:10:12

Is there also an option to AOT compile the code? I have not been able to find anything

seancorfield18:10:21

@zengxh You also need to ensure that your main class is compiled and that it has (:gen-class) in its ns form for that namespace.

seancorfield18:10:31

(I think a lot of people miss that last part)

Apple18:10:06

compile-clj with that i thought i could get away without gen-class . i never really understand gen-class 😳

seancorfield18:10:17

:gen-class causes -main to become a static main method in the class compiled from the namespace.

Alex Miller (Clojure team)19:10:08

really, it causes all -foo methods to become static, just most importantly -main :)

seancorfield19:10:23

Yes, I was being deliberately vaguely/over-simplified here 🙂 :gen-class is a veritable Swiss Army Knive!

seancorfield18:10:52

(otherwise it's just compiled Clojure code that can be called from Clojure, rather than directly from Java)

Apple18:10:14

perfect. gen-class is what i missed.

Apple18:10:34

thank you!!

seancorfield18:10:58

I tend to miss it almost every time I write a new "main" namespace. Then I compile and build the uber JAR and go "Oh, rats! I forgot :gen-class again!" 🙂

seancorfield19:10:18

I often wish it was a special case that defn -main always became a static main method 🙂

dharrigan20:10:34

maybe there should be a linter for that 🙂

seancorfield20:10:34

Yeah! @borkdude if clj-kondo sees defn -main in a namespace that doesn't have (:gen-class) in the ns form, that could be a useful warning! 🙂

borkdude20:10:32

hmm, makes sense! :)

borkdude20:10:01

although you can still invoke this with -M -m foo as well without having to have :gen-class necessarily - any other heuristics that should point at :gen-class ?

j3vs21:10:31

I'm refactoring some code that uses a dynamic db connection built using mount, it looks like

(ns service.test-fixtures
  (:require [service.db :refer [*db*]])) ;;the dynamic var

(defn db-tx-rollback-fixture [f]
  (jdbc/with-db-transaction [tx *db*]
    (jdbc/db-set-rollback-only! tx)
    (with-redefs [*db* tx]
      (f))))
this code has been copy pasted in numerous repos and i wanted to pull out a bunch of test fictures we have into a common lib. Refactoring this as simply
(defn db-tx-rollback-fixture [db]
 (fn [f]
   (jdbc/with-db-transaction [tx db]
     (jdbc/db-set-rollback-only! tx)
     (with-redefs [db tx]
       (f)))))
and calling with
(ns service.test-fixtures
  (:require [service.db :refer [*db*]]))

(use-fixtures :each (db-tx-rollback-fixture *db*))
obviously doesnt work, I'm trying to wrap my head around how i can write a macro to solve this.

seancorfield21:10:54

Instead of with-redefs, use binding. *db* is dynamic.

Muhammad Hamza Chippa21:10:09

how to generate uuid from string ?

seancorfield22:10:24

Are you fairly new to Clojure @U02DQ45FQF9? If so, maybe ask Qs in #beginners where folks have opted in to helping in depth with basic questions like this.

🙌 1
borkdude22:10:11

(logged an issue about gen-class + -main here: https://github.com/clj-kondo/clj-kondo/issues/1417)

1