Fork me on GitHub
#beginners
<
2021-07-01
>
practicalli-johnny10:07:29

I would like to understand all of this Clojure error message (not just the first line)

class java.lang.Long cannot be cast to class clojure.lang.IFn (java.lang.Long
   is in module java.base of loader 'bootstrap'; clojure.lang.IFn is in unnamed
   module of loader 'app')
The error is from evaluating (1) as an expression, which I expect to fail as 1 is not defined as a function anywhere in the code. So this explains the java.lang.Long is not a clojure.lang.IFn part of the message. I assume the second line is referring to the name of the class loader used to include java.lang in the JVM of the REPL. Is app a specific class loader for all code in the REPL? I see this last part of the message in many error messages and its about time I understood what it means :)

Alex Miller (Clojure team)13:07:26

it's also including module information - since Java 9, there is a module loading system in Java and the java.base module includes the core set of modules needed to start Java (other modules can be added as needed). for backwards compatibility you can still mostly ignore the module stuff but then you're loading stuff in the "unnamed" module

👍 2
popeye15:07:07

I have a scenario, where my function will execute more than 10 mins which is expected, I want to stop the execution if it reaches more than 10 mins, How can I achieve it in clojure ?

Fredrik15:07:41

What is your function doing? Listening on a socket, processing data in a loop?

popeye15:07:45

My function doing caluclation on huge data, if it take more time I need to return nil , - which is my requirement

delaguardo15:07:01

https://clojuredocs.org/clojure.core/future-cancel

(def job (future (Thread/sleep 10000)))

(future-cancel job)

noisesmith15:07:06

cancelling threads preemptively is technically possible but risks crashing the vm if your job periodically sleeps, or waits on IO, or explicitly checks the cancellation status of its thread and exits if cancelled, future-cancel will work, but it won't do anything to a "hot loop"

noisesmith15:07:50

(future-cancel is not using a preemptive cancellation, it's relying on one of the "cancellable" methods being called)

noisesmith15:07:21

also the deref function (what @ expands to) takes an optional time-out number and an alternative value to return if the time-out is exceeded

noisesmith15:07:28

which might match what you want

noisesmith15:07:17

also, on some operating systems, it might be simpler to spawn a second vm, and force that vm to shut down

noisesmith16:07:47

the "accepted answer" is from a source I usually trust - amalloy is a smart guy I learned a lot from (we were both very active on IRC back in the day), but this answer is dangerously wrong. it uses the stop method on Thread, which is extremely dangerous and can crash the vm: https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/lang/Thread.html#stop() future-cancel is safe, you just need to make sure the code you are cancelling is inside a future and that it periodically waits on IO, or calls Thread/sleep, or in some other way makes itself cancellable details here: https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/lang/Thread.html#interrupt()

noisesmith16:07:31

to be extra safe you could make sure the loop executing in the thread checks the return value of the interrupted method

ghadi16:07:42

again, Thread.stop is not safe

ghadi16:07:03

what is your work doing for more than 10 minutes?

ghadi16:07:10

if it's IO, it can be interrupted()

popeye16:07:22

My function doing caluclation on huge data, if it take more time I need to return nil , - which is my requirement

noisesmith16:07:29

writing code like that is a lot more complex than just using a future and calling future-cancel after some time period, that code you linked is what I was referring to being unsafe because of stop

ghadi16:07:33

please describe the calculation?

ghadi16:07:40

pure math? I/O?

popeye16:07:09

it calls the lambda service

noisesmith16:07:18

then it's waiting on IO (network)

noisesmith16:07:26

then future-cancel will work

ghadi16:07:36

is the Lambda the thing you're trying to stop? or are you trying to stop the thing calling the lambda?

popeye16:07:22

lamda taking more time to perform calculation. my requirement is timeout if more than 10 mis

ghadi16:07:21

AWS Lambda?

popeye16:07:17

@U051SS2EU future-cancel how to pas the minutes?

noisesmith16:07:35

you need something else (eg. (future (Thread/sleep (* 10 60 1000)) (future-cancel worker)))

popeye16:07:18

worker is function-name here?

ghadi16:07:33

why not just let the lambda timeout and be killed?

noisesmith16:07:33

here (* 10 60 1000) turns minutes into seconds (multiply by 60), then turns seconds into ms (multiply by 1000)

noisesmith16:07:41

worker would be the thing being cancelled

popeye16:07:06

where i can pass my function name

noisesmith16:07:12

@U050ECB92 makes a good point - lambda already has facilities to shut down after a time-out, that's the expensive resource here, it can just shut down the whole vm

noisesmith16:07:53

@U01J3DB39R6 you would need to use future to start it and use the return value of future, but the way @U050ECB92 is suggesting is better since this is running in its own vm on a lambda

popeye16:07:09

oh, let me search on it! all the above conversation really helped

bnstvn09:07:15

what does it depend on exactly whether (future-cancel) works or not? current thread state?

noisesmith17:07:57

I linked the Thread.interrupt docs above (future-cancel uses that method): > If this thread is blocked in an invocation of the wait(), wait(long), or wait(long, int) methods of the Object class, or of the join(), join(long), join(long, int), sleep(long), or sleep(long, int), methods of this class, then its interrupt status will be cleared and it will receive an InterruptedException. > > If this thread is blocked in an I/O operation upon an InterruptibleChannel then the channel will be closed, the thread's interrupt status will be set, and the thread will receive a ClosedByInterruptException. > > If this thread is blocked in a Selector then the thread's interrupt status will be set and it will return immediately from the selection operation, possibly with a non-zero value, just as if the selector's wakeup method were invoked. > > If none of the previous conditions hold then this thread's interrupt status will be set

noisesmith17:07:47

it always "works" in that if the thread in question doesn't invoke any of the specific methods mentioned there, a flag is set and you can check the flag

noisesmith17:07:25

it "doesn't work" in that if you are doing work that never uses any of those methods, and don't check that flag, it has no effect on the thread's execution

noisesmith17:07:19

there's a more powerful method to force stop a thread (which some code shared above uses), but there are no guarantees about safety when you use that, and it can crash the vm or leave objects in inconsistent states or leave locks that can't be unlocked

noisesmith17:07:21

the code shared above is arguably OK, because it's for a IRC bot that's mainly there for entertainment and education and if it crashes it auto restarts

noisesmith17:07:33

your code might deal with something more serious where those risks are not OK

bnstvn07:07:42

thank you! so as plain function doesnt have an periodic interrupt check, if it is busy inside intensive computation interrupt does nothing — while if io is happening an exception will be raised (but still possible it will be swallowed i guess?)

noisesmith15:07:55

well - not just IO, sleep counts too (so you can be interruptable by adding a short sleep to a loop), but yeah that's the basic idea

noisesmith15:07:37

also, it's not just an exception that's raised, there's a flag that the thread's code can check too

Endre Bakken Stovner16:07:37

Is str/index-of the best way to see if a string contains a char (represented as a string)? I love writing if "a" in "casablanca" in Python

noisesmith16:07:16

for a simple character check, index-of is your best bet, unlike python clojure values simplicity of syntax over intuitive / natural language like syntax

Endre Bakken Stovner16:07:09

I felt like contains? sounded like it could have done the job, but it is for associative structures.

noisesmith16:07:47

I guess you could do (contains? (set "casablanca") \a) but I don't think that really addresses your needs

noisesmith16:07:57

and it does a lot of work you don't need to do

Endre Bakken Stovner16:07:33

Agreed. But I felt like checking the value of an int was a bit... C-like?

noisesmith16:07:52

you could also do (first (filter (set "a")) "casablanca") but that still does a lot of work you don't need to do

dpsutton16:07:09

There’s str/includes? That has a name that indicates what it does similar to your expectation for contains?

💯 4
Endre Bakken Stovner16:07:53

Thanks! It did not show in lein rebl for some reason

Endre Bakken Stovner16:07:07

So I assumed it did not exist

noisesmith16:07:18

what clojure version?

dpsutton16:07:10

apropos and find-doc and dir are your friends here

🤯 2
Endre Bakken Stovner16:07:44

[org.clojure/clojure "1.10.3"]

Endre Bakken Stovner16:07:35

Wow! Thanks for the tip about clojure.repl/find-doc

dpsutton16:07:13

Leveraging the queryable nature of clojure with the built in tools will serve you well. I use those constantly. After you’re comfortable with them, learn if your editor has some integrations on top of them. Many do

👍 2
noisesmith16:07:57

I frequently use find-doc and apropos and javadoc to answer questions here - I usually start with a good hunch of what should be there (or a vague memory) and let the built in introspection in the language do the rest of the work

noisesmith16:07:16

and those tools are much more direct and helpful than google

noisesmith16:07:07

looking up the most poetic doc string in the clojure codebase:

(ins)user=> (find-doc #"if possible")
-------------------------
clojure.core.protocols/interface-or-naive-reduce
([coll f val])
  Reduces via IReduceInit if possible, else naively.
-------------------------
clojure.core/future-cancel
([f])
  Cancels the future, if possible.
nil

🤯 6
👍 2
noisesmith16:07:46

@endrebak85 one nitpick: it's not c-like in that (if (str/index-of "casablanca" "a") :yes :no) actually works direcly - it returns nil which if treats as false, on failure

Endre Bakken Stovner16:07:33

True! I just assumed that it would return -1 instead of false. But that would have been extremely unclojuric.

Endre Bakken Stovner16:07:51

Bad habits (assumptions?) die hard. Should have tested in the REPL.

noisesmith16:07:45

in C, simple type consistency is valued over semantic clarity, the jvm's comparitive complexity allows things that are closer to the intent of the code to be expressed

👍 2
Karo19:07:07

Hi team how can I query current timestamp from database in Clojure project using Korma? "select current_timestamp"

Karo19:07:17

(defn foo []
  (select my_table
          (fields (raw "CURRENT_TIMESTAMP"))))

borkdude19:07:11

@vnazaryan Korma isn't very maintained anymore I think, but I could be wrong. Instead I would recommend taking a look at #honeysql which is, if you ask me, one of the the best SQL generation libraries for Clojure,

Karo19:07:22

thanks for suggestion, will take a look.

seancorfield20:07:46

In addition to the #honeysql channel for HoneySQL, there's also a more general #sql channel for general SQL questions -- and that also tends to be where folks ask about next.jdbc which is the replacement for clojure.java.jdbc. Also #hugsql if you like your SQL in separate files, outside of your Clojure code.

👍 2
tschady21:07:30

is there a canonical way to parallel parse/process a large file based on a regex to split up the chunks? instaparse was overkill - all the backtracking it kept blew up the memory. I was thinking iota and reducers but each ‘chunk’ of the file is processed as a whole, not line by line, and marked with clear begin/end blocks. It’s a super regular unambiguous grammar, but not XML or such.

Fredrik22:07:17

Could something like pipeline and partition-by from clojure.core.async be an idea? Read the file line by line onto a channel, partition into chunks which then get further processed in a pipeline.

Fredrik22:07:57

It allows for a nice decoupling between reading, splitting into chunks and parsing.

noisesmith17:07:24

you can use re-seq to get a lazy seq of matches of a charsequence, there's at least one lib out there that implements charsequence over very large files https://github.com/fge/largetext - may or may not work for you

noisesmith17:07:47

but for the use case where it applies it's probably the most straightforward approach

noisesmith17:07:50

(charsequence is the superclass of string, and you can do most "string ops" on a charsequence, they are more abstract eg. unlike string don't need to promise to exist in memory all at once)

Mattias21:07:34

Whats the (better?) Clojure equivalent of a SAX parser + lots of messy Java code? I know there are several ways to parse XML, and a bunch of ways to traverse the result but haven’t tried any so far.

seancorfield21:07:57

Probably zippers.

seancorfield21:07:25

At work, we use clojure.data.xml (`org.clojure/data.xml`) to parse XML and clojure.zip to navigate/modify https://clojure.github.io/clojure/clojure.zip-api.html

greg21:07:20

I noticed there are two libraries: clojure.xml and clojure.data.xml . What's the difference between them?

seancorfield21:07:59

clojure.xml is built-in but it is old/deprecated and should not be used. clojure.data.xml is a Contrib lib that is well-maintained, more modern, more feature-rich.

Mattias07:07:11

Thanks! 👍

bnstvn09:07:06

actually, i recently just wanted to count some elements in a jvm with 512mb heap with a ~30Mb xml. i got an out of memory error (with zippers as well), and went with plain java stax. any idea what could have been a clean clojure solution?

(->> "huge.xml"
            io/resource
            io/input-stream
            xml/parse
            xml-seq
            (filter #(= (:tag %) :Event))
            count)

Execution error (OutOfMemoryError) at clojure.data.xml/seq-tree$fn (xml.clj:183).
Java heap space

pithyless11:07:58

@UGSM2S2CS I wonder if this would work:

(x/count
 (comp
  xml-seq*
  (filter #(= (:tag %) :Event)))
 (-> "huge.xml"
     io/resource
     io/input-stream
     xml/parse))
where 1. xml is clojure.data.xml 2. x/count is from https://github.com/cgrand/xforms 3. transducer version of xml-seq, something like:
(def xml-seq*
  (fn [rf]
    (fn
      ([] (rf))
      ([result] (rf result))
      ([result input]
       (rf result (xml/xml-seq input))))))

pithyless11:07:29

(I have not actually tested this code ;))

Mattias12:07:04

So, speaking of - can anyone point to code, or pseudo-code, or any inspiration on how to idiomatically traverse a tree (SAX-style), where you obviously sometimes need to keep track of the depth or other data to figure out what to do with leaf nodes.

Mattias12:07:46

(Slowly trying to leave my Java/imperative brain behind, but it’s… sticky.)

ghadi13:07:25

SAX style is awful because it presumes mutation and accumulators

ghadi13:07:37

StAX is a bit nicer, IMHO, and lends itself to a more functional handling

Mattias13:07:58

Ah, hadn’t even heard of StAX. Will study and catch up with the times…

ghadi13:07:13

non-lazy xml -> clojure data routine