Fork me on GitHub
#clojure
<
2020-09-22
>
suren00:09:06

Why does lein repl throws Execution error (ClassNotFoundException) at .URLClassLoader/findClass (URLClassLoader.java:466). when I try to run a function in my project?

seancorfield00:09:44

Impossible to answer without knowing more about exactly what you are trying to do and exactly what the error is (i.e., more of a stacktrace etc).

suren00:09:33

Got it working. Previously I used

(ns app.db)
(get-posts)
This threw
Execution error (ClassNotFoundException) at java.net.URLClassLoader/findClass (URLClassLoader.java:466).
But when I use
(use 'app.db)
(get-posts)
It works. Don't exactly know why.

seancorfield01:09:06

You should probably use (require 'app.db) and then (app.db/get-posts)

seancorfield01:09:23

use is discouraged (and isn't available in ClojureScript as I recall).

seancorfield01:09:51

(use 'app.db) is the same as (require '[app.db :refer :all]) which, again, is discouraged.

seancorfield01:09:34

(require '[app.db :as db]) followed by (db/get-posts) would be preferable -- always use an alias rather than referring in names.

seancorfield01:09:12

(ns app.db) will create an empty namespace called app.db so that's almost certainly not what you want in the REPL.

suren01:09:37

Thanks, I will keep that in mind.

seancorfield01:09:59

As I said (ns app.db) creates an empty namespace. Nothing defined in it.

seancorfield01:09:18

It sounds from this that you're fairly new to Clojure?

suren01:09:50

Yup you are right. I am trying to jump to Clojure landscape.

seancorfield01:09:57

I'd recommend asking questions in #beginners -- you'll find people are more patient there and less likely to assume knowledge you might not have yet.

seancorfield01:09:21

The experienced folks in #beginners have opted in to helping new folks in great detail.

suren01:09:22

Hmm sweet. I will jump to that channel.

3
nixin7203:09:24

Is there a Clojure/(Script) library for using Secure Remote Password Protocol? I know there's some Java/(Script) ones, but curious if there's something that exists for Clojure already or if I'm gonna have to define a wrapper around existing ones.

chetchan04:09:18

Hello, Does any use Google BigQuery API with in Clojure? I am trying to query some insert jobs in a non-US non-EU region but when I do a get call to fetch the status of the job I get 404 Job not found This is my job spec which has location set in it

(defn- load-job-spec [table]
  (let [{:keys [project-id dataset-id table-name table-suffix]} table
        table-id (str table-name "$" table-suffix)
        time-string (->> (LocalDateTime/now) (.format (DateTimeFormatter/ofPattern "yyyy-MM-dd_HH-mm-ss")))
        job-id (str table-name "-" table-suffix "-" time-string)
        job-reference (doto (JobReference.) (.setLocation "australia-southeast1") (.setJobId job-id) )
        table-reference (doto (TableReference.)
                          (.setProjectId project-id)
                          (.setDatasetId dataset-id)
                          (.setTableId table-id))
        config-load (doto (JobConfigurationLoad.)
                      (.setDestinationTable table-reference)
                      (.setEncoding "UTF-8")
                      (.setWriteDisposition "WRITE_TRUNCATE")
                      (.setSkipLeadingRows (Integer. 1))    ;; CSV headers
                      (.setMaxBadRecords (Integer. 0))      ;; Failure on garbled records
                      (.setAllowQuotedNewlines (Boolean/TRUE)))
        job-config (doto (JobConfiguration.) (.setLoad config-load))
        job (doto (Job.)
              (.setJobReference job-reference)
              (.setConfiguration job-config))]
    job))
This below code is where first run an insert job to load a CSV file into BQ dataset in Australia region and then try the get status on it with a get call below, but the get fails with 404 job not found. I am not sure how I can pass location in the get call. Any help would be great. Thx
(defn load-file-bigquery [fullpath]
  (let [name (s/replace fullpath #"^.*4244_" "")
        name (s/replace (s/lower-case name) #"_\d{6}.csv$" "")
        name-parts (s/split name #"_")
        table-suffix (last name-parts)
        table-name (s/join "_" (butlast name-parts))
        content (FileContent. "application/octet-stream" (File. fullpath))
        project-id (cfg/config :project-id)
        job-spec (load-job-spec {:project-id   project-id
                                 :dataset-id   (cfg/config :dataset-id)
                                 :table-name   table-name
                                 :table-suffix table-suffix})
        job-id (->> job-spec (.getJobReference) (.getJobId))
        _ (log/info "BigQuery Load Job : " job-id " : " fullpath)
        bq @bigquery-conn                                   ;; Blocks until initialised
        job-obj (->> bq
                    (.jobs)
                    (#(.insert % project-id job-spec content))
                    (.execute)
                    )
        status (->> job-obj (.getStatus) )]

    (log/info "job-obj outside "job-obj)
    (loop [status status]                                   ;; Waiting until successfully processed
      (log/info job-id " : " (->> status (.getState)))
      (if (= "DONE" (->> status (.getState)))
        (do (log/info "Status seems done?")
          (if-let [errors (.getErrors status)]
            (do
              (log/info "seems like we have errors")
              (vec (map #(.getMessage %) errors)))
            nil))
        (do
 
          (Thread/sleep 3000)

          (recur (->> bq
                      (.jobs)
                      (#(.get % project-id job-id))
                      (.execute)
                      (.getStatus))
                 ))))))
I have the equivalent java code but I need this to work in Clojure code above
Job job = ().insert(PROJECT_ID, runJob).execute();
String status = job.getStatus().getState();
while(!status.equalsIgnoreCase("DONE")) {
  status = ().get(PROJECT_ID, job.getId()).execute().getStatus().getState();
  System.out.println("Status: " + status);
  Thread.wait(1000);
}

hiredman05:09:54

= is not the same thing as the equalsignorecase method

chetchan06:09:55

sorry for confusing post, i put the java code just as an equivalent

hiredman05:09:55

My guess is you have some error creating the job, but you never print errors outside before starting to loop

chetchan06:09:19

job runs ok, however I cannot get the status unless I pass the location to get call

(->> bq
                      (.jobs)
                      (#(.get % project-id job-id))
                      (.execute)
                      (.getStatus))
How can I know that if the .get here has a particular method in it that I can use etc? Does it resolve automatically if it exists since I am referring to the java classes here

wombawomba08:09:02

How come (let [foo "bar"] (eval (read-string "foo"))) doesn’t work? How can I get eval to pick up locally scoped bindings?

wombawomba08:09:43

Alright, I found this, which says that I either need to def or declare my bindings/vars: https://stackoverflow.com/a/6221829

wombawomba08:09:51

Unfortunately, defing doesn’t seem to work either… Or, it does when I do it in the REPL, but not in my actual namespace. Any ideas why?

vlaaad08:09:52

You’ll need a macro that captures lexical environment and feeds it into eval

wombawomba10:09:14

makes sense, thanks 🙂

Raziyeh Mohajer09:09:19

Hello everyone, I'm using the jackdaw library for connecting to Kafka I want to use a key.serializer and value.serializer for Kafka producer other than Kafka's own serializers. this is my code for the producer

(def serialize nippy/freeze)

(def Kafka-Edn-Serializer
  (reify Serializer
    (serialize [this topic headers data]
      (serialize data))
    (serialize [this topic data]
      (nippy/freeze data))
    (configure [this _ _])
    (close [this])))

(def producer-config
  {"bootstrap.servers" "localhost:9092"
   "key.serializer"    (-> Kafka-Edn-Serializer
                           .getClass
                           .getName)
   "value.serializer"  (-> Kafka-Edn-Serializer
                           .getClass
                           .getName)
   "acks"              "all"
   "client.id"         "foo"})

(defn -main []
  (jc/producer producer-config)
  )
I can run it in repl but when I use lein run to run this I got the following error
Invalid value producer_example$reify__12915 for configuration key.serializer: Class producer_example$reify__12915 could not be found.
How should I solve this problem?

thumbnail10:09:37

Have you tried using deftype instead of reify?

thumbnail10:09:12

I think reify doesn't create an actual type, unless aot compiled.

Raziyeh Mohajer11:09:39

I will try it thanks

fabrao12:09:19

Hello all, are you using lein to create templates for base project? Is there other way instead of lein to do it?

vlaaad13:09:25

mkdir src && echo '{}' > deps.edn 😁

vlaaad13:09:07

ah, you asked about creating templates, not a particular template to use instead of lein..

fabrao17:09:50

@U47G49KHQ yes, creating templates for use as starting of a new project

Ashwin Bhaskar14:09:58

I am using https://github.com/uswitch/opencensus-clojure tracing library which follows the opencensus standard. Here is my code:

(def routes ["/" [["ping" {:get (constantly {:status 200
                                             :body   "pong"})}]
                  ["foo" {:post (->> foo-handler
                                      (trace/span "foo-file-handler"))}]
                  [true (constantly {:status 404})]]])

(m/defstate server
  :start (jetty/run-jetty (bd/make-handler routes) {:port  8080
                                                    :host  "localhost"
                                                    :join? false})
  :stop (.stop ^Server server))

(m/defstate tracer
  :start (jaegar/report "foo-service")
  :stop (jaegar/shutdown))
From the repl, I start the `server` and `tracer` states. I have `jaegar-all-in-one` docker running in my machine. I can see the `jaegar` dashboard in my browser too. But the traces don’t show up there when I call the API. I tried to narrow the problem down by just calling `(trace/span "some-trace" (println "foo))` from the repl. Even this does not show up on the dashboard. What am I missing here?

Ashwin Bhaskar11:09:37

My bad, the traces are probabilistic as mentioned in the README of the library. I was using the default probability which less than 0.1 (I guess). I changed the probability to 1. Now the traces show up

isak15:09:26

For calls to assoc with multiple key-value pairs, would it make sense to use a transient? (In this code): https://github.com/clojure/clojure/blob/ee3553362de9bc3bfd18d4b0b3381e3483c2a34c/src/clj/clojure/core.clj#L192-L199

isak16:09:31

Maybe not, because not everything passed to assoc will be a IEditableCollection

strsnd17:09:49

Are there idioms in clojure which allow me to partially process a seq and return the rest of the sequence without iterating over it twice (as with split-at)? It is easy to write myself but wondering if I am missing something?

emccue04:09:54

That description isn't specific enough for me to say. If you want to partially process a sequence, it would matter for recommendations what the logical predicate is to stop and whether you want that processing to return a value or perform a side effect

emccue04:09:55

you can always split a sequence by index with the combo of take and drop

Linus Ericsson11:09:05

Maybe you could us reduce in combination with reduced which terminates reduce loops. Then you can add both the processed items results and the rest of the sequence.

strsnd13:09:32

@U3JH98J4R coming out of a an iterator-seq I have three types of objects, which are not interleaved. the first type of objects I need to put in some kind of map, while the rest is being processed using earlier created map and are processed with side effects. Naturally partition-by would be useful but it keeps the head of the first stream alive and thus I run into out of heap. For me it is important to not walk this long stream multiple times while also not keeping any heads alive while processing them. a custom loop recur works for the moment but feels non-idiomatic.

strsnd13:09:48

@UQY3M3F6D I haven't thought about reduced interface, I will play with this idea a bit. Thanks!

emccue13:09:54

The simplest cases of loop/recur do usually map to a call to reduce

emccue13:09:09

(let [long-lazy-seq ...
      data-from-before {...}
      process! (partial do-side-effect! data-from-before)]
  (loop [remaining long-lazy-seq
         collected []]
    (if (empty? remaining)
      collected
      (if (predicate? (first remaining))
        (do (process! (first remaining))
            (recur (rest remaining) collected))
        (recur (rest remaining)
               (conj collected (first remaining)))))))

emccue13:09:24

so this (which is what it sounds like you have)

emccue13:09:03

can map to a separate reducing function (which is kinda hard to name, but c'est la vie)

emccue13:09:57

(let [long-lazy-seq ...
      data-from-before {...}
      process! (partial do-side-effect! data-from-before)
      reducing-function (fn [collected item]
                          (if (predicate? item)
                            (do (process! item)
                                collected)
                            (conj collected item)))]
  (reduce reducing-function [] long-lazy-seq))

emccue13:09:11

so roughly this - reduced isn't super helpful except to terminate early, which it doesn't sound like you want to do

strsnd13:09:47

Hm, thanks. My code indeed looks a bit like your earlier one. I wanted to actually have some kind of split in between when it operated on the first types of objects and the second types, which seems hard to do in reduce (e.g. using a transient to construct the map of the first elements).

strsnd13:09:54

While I generally prefer map/reduce, in this case something like:

(let [[first-elem-map r] (do-while is-first-elem? process r)
      [_ r] (do-while (is-second-elem? side-effect-on-second-fn r)
      _ (do-while (last-element? side-effect-on-thid-fn))]
  ...)
looks nicer to me.

emccue13:09:32

I usually don't like reduce and prefer the explicit loop, but thats me. You can maybe handle the different cases via a dispatch thing

emccue13:09:54

so a multimethod or just a condition on what function to call

emccue13:09:09

but conceptually you need to handle each item in the same "place"

emccue13:09:19

splitting a lazy sequence into logical parts necessitates that at least one will hold on to the head while the one you choose to evaluate first is running

emccue13:09:07

you can maybe "queue up" side effects with filter

emccue13:09:36

(let [long-lazy-seq ...
      data-from-before {...}
      process! (partial do-side-effect! data-from-before)]
  (->> long-lazy-seq
       (filter (fn [item] (if (predicate? item)
                             (do (process! item) false)
                             true)))
       (reduce (fn [collected remaining-item]
                 (conj collected remaining-item)
               [])))

emccue13:09:54

but uhh, beauty is in the eye of the beholder

strsnd14:09:06

I think I stay with my do-while approach, which looks like that:

(defn do-while [pred f s coll]
  (let [c (first coll)]
    (if (pred c)
      (recur pred f (f s c) (rest coll))
      [s coll])))
I might try to use a bit of chunking/batching there depending if that shows up in performance profiles.

👍 3
strsnd14:09:13

I tried the multimethod approch as well, but this loop might process a lot of items and I am unsure if I add just too much overhead in that case.

strsnd14:09:23

The filter stuff looks like a neat idea, but I agree, it feels weird to do anything else in filter than filtering.

wombawomba19:09:57

How can I run a shell command in the foreground (i.e. with stdin/-out/-err attached) in Clojure? Ideally I’d like the command to believe it’s in a TTY as well.

Alex Miller (Clojure team)19:09:30

there is a clojure.java.shell namespace with some support for this (not sure about tty)

Alex Miller (Clojure team)19:09:58

lately though I've actually been using Java interop directly to the newer Java ProcessBuilder api

Alex Miller (Clojure team)19:09:45

that has full support for a variety of options of handling in/out/err

ghadi19:09:49

seconding that ^

ghadi19:09:01

java >=9 give you a completion callback too

wombawomba19:09:06

hmm.. how would I attach stdin/out/err with clojure.java.shell?

Alex Miller (Clojure team)19:09:47

if you (doc clojure.java.shell/sh) you can read all the options

Alex Miller (Clojure team)19:09:56

but I find ProcessBuilder is actually easier to use

wombawomba19:09:39

I did look at that doc, but AFAICT it doesn’t support what I’m trying to do

Alex Miller (Clojure team)19:09:14

if you want more clojure-y flavor, there is stuff like https://github.com/clj-commons/conch

Alex Miller (Clojure team)19:09:58

if you're doing a lot of this stuff, that can be pretty nice too

wombawomba19:09:43

I’m also not sure if there’s an easy way to run in the foreground with the standard streams hooked up using the ProcessBuilder API. I tried Googling for it but the answers I found all involve manually redirecting output from/to the process (e.g. https://stackoverflow.com/a/17393606), which seems a bit cumbersome

wombawomba19:09:46

I also tried using conch, but couldn’t figure out how to get it working. The closest I could get was doing something like:

(require '[me.raynes.conch.low-level :as sh])
(def proc (sh/run "bash"))
(future (sh/stream-to-out proc :out))
(future (binding [*out* *err*] (sh/stream-to-out proc :err)))
(sh/feed-from proc *in*)
However, this doesn’t really work because everything gets buffered (for instance, I need to manually send an EOF via ctrl-D for my input to get sent to the process).

seancorfield19:09:41

Isn't that because *in* and *out* are buffered streams?

seancorfield19:09:13

Doing unbuffered console I/O in Java is kind of a pain, as I recall...

wombawomba19:09:39

Yeah that’s probably it

ghadi19:09:53

look at inheritIO options on process builder

wombawomba19:09:37

@ghadi heh, I actually just tried that 🙂 basically doing this: https://stackoverflow.com/a/49706743 works, except that I can’t see my output

wombawomba19:09:08

e.g. running echo hello in my bash proc works, but I can’t see what I’m typing:

$ (def pb (java.lang.ProcessBuilder. (into-array String ["bash"])))
$ (.inheritIO pb)
$ (.waitFor (.start pb))
bash-3.2$ hello
bash-3.2$

wombawomba19:09:46

perhaps this could be because I’m in a REPL though..

wombawomba19:09:43

Yeah, that was it 🙂 It works when I use lein trampoline run

agigao20:09:24

@alexmiller Hi Alex, any plans about the next edition Clojure Applied and/or Programming Clojure?

Alex Miller (Clojure team)20:09:15

hasn't really been much new since 3ed of Programming Clojure so no plans there

Alex Miller (Clojure team)20:09:47

funny enough, we just passed the 5 yr anniversary for Clojure Applied and I was talking to the publisher about it this week. considering whether to do a new ed

Alex Miller (Clojure team)20:09:14

anything you'd be burning to have included? :)

seancorfield20:09:37

@alexmiller Would a new Ed of C.A. deprioritize the use of records somewhat? I recall you mentioning something about that.

seancorfield20:09:14

(and could it also normalize the use of qualified keywords in hash maps from the get-go? 🙂 )

Alex Miller (Clojure team)20:09:27

yes on maps vs records, dunno on qualified keywords

Alex Miller (Clojure team)20:09:51

I have a list of ideas already, just wondering what others would be interested in :)

Alex Miller (Clojure team)20:09:11

having just gone through the whole book again, I think probably 85% is still pretty solid. I mostly have stuff I'd want to add to it (spec - would replace the schema use for validation and expand generative testing in ch 8), clj stuff, would like to add stuff on macros and java interop

👍 3
seancorfield20:09:51

Yeah, +💯 on using Spec instead of Schema and adding more generative testing stuff. I've been working my way through @U050P0ACR’s Property-Based Testing courses online and finding lots more places to use generative testing than I imagined possible!

dpsutton20:09:38

big plus on generative testing

Lennart Buit20:09:20

(to add to that, what I often struggle with in generative testing is how to define my properties)

Lennart Buit20:09:31

(or, what, ‘good’ properties are)

Alex Miller (Clojure team)20:09:31

I took my best swing at this in the current edition, would be curious if you've read it and what you thought

Lennart Buit20:09:57

Its at the office, I happen to go there tomorrow, I’ll read through it

Lennart Buit21:09:07

So I read through the generative testing section again. I have a hard time pinpointing what my exact issue is when I said that I find it difficult to define properties. I think the section is very clear, and definitely hints to what I need. But, I don’t have the intuition when to choose property based testing except for some set patterns (serialize/deserialize springs to mind). Another problem I experience in practice is that properties I do come op with are very broad (serialize -> deserialize = identity) which I struggle to properly assess how useful those are (in a proof burden kinda sense, this generatively tested property ‘buys’ me the assurance that the system does or does not do X). Or, I have trouble finding properties underlying implementations (say, the textbook example of a sort algorithm guarantees orderness with ‘orderness’ being the property and ‘quick-sort’ being a way to get to that) which leads to a feeling of testing by ‘asserting the implementation’ (if you get what I mean). So I guess that most places I read about property based testing explain me how (if I may use an analogy) a screwdriver works, show me examples of screws, but so far, I don’t feel like I have a grasp on starting to realise that a thing is a screw. (Does that make sense?)

Lennart Buit21:09:08

(Not all about the book, btw, I’m just trying to … also for myself, pinpoint what makes it hard for me to grasp property based testing)

Lennart Buit21:09:04

(I guess you heard this a million times, but the last listing on page 154 is duplicated on 155, or actually the other way around)

Alex Miller (Clojure team)22:09:36

it does make sense - I think it is truly difficult to discern what properties are useful to validate. I spent a lot of time thinking about it when writing that chapter. spec has highlighted an additional way to think about it in terms of the in/out of a function too and that's what I'd like to add (although spec 2 is likely to going to change in practice quite a bit)

Alex Miller (Clojure team)22:09:13

and yes, there are a number of copy edit issues in the book. we've got lots of errata to go through

slipset06:09:27

I would think that we’d need to tease the subject a part a bit. I would imagine that the job of Clojure Applied would be to showcase how property based testing could/should be used in Clojure. Writing extensively on how to do property based testing in the general case would probably be outside the scope of C A? Oscar Wickström has a book on this https://leanpub.com/property-based-testing-in-a-screencast-editor and did a talk on it at flatMap(Oslo) in 2019 https://2019.flatmap.no/talks/wickstrom in which, IIRC, there were som links to other resources on property based testing.

Lennart Buit19:09:18

That sounds interesting, I’ll definitely watch the talk

Alex Miller (Clojure team)20:09:36

the only hard parts are defining properties and defining generators :)

dpsutton20:09:38

https://grep.app/search?q=prop/for-all is an assist. and looking for test.check :as is another good search term

Lennart Buit20:09:39

oh thats also clever!

Lennart Buit20:09:18

Yeah I found a blog from the F# community

Lennart Buit20:09:27

uh no, there was another with pretty pictures

Alex Miller (Clojure team)20:09:54

the existing book covers generative testing with test.check, I'd mostly be looking to augment that by talking about how spec plays into it