Fork me on GitHub
#clojure
<
2019-08-13
>
andy.fingerhut05:08:46

Is there any kind of option to clojure.test to cause every deftest to print a message when it starts executing? I know I can simply add a println anywhere I want to show progress -- just curious if there was something built in already that can be enabled for that.

seancorfield05:08:51

@andy.fingerhut Yes, you can redefine do-report for :begin-test-var I think to log that...?

andy.fingerhut05:08:19

thx I will look for those in the code or other projects.

seancorfield05:08:30

user=> (require '[clojure.test :refer :all])
nil
user=> (deftest foo (is (= 42 13)))
#'user/foo
user=> (defmethod clojure.test/report :begin-test-var [m] (println 'running (:var m)))
#object[clojure.lang.MultiFn 0x30814f43 "[email protected]"]
user=> (run-tests)                                                                                                                                                      
Testing user
running #'user/foo

FAIL in (foo) (NO_SOURCE_FILE:1)
expected: (= 42 13)
  actual: (not (= 42 13))
report not do-report

seancorfield05:08:05

These extension points are "blank" by default but you can report on all of them https://github.com/clojure/clojure/blob/master/src/clj/clojure/test.clj#L407-L410

Saikyun10:08:19

is there any way for a macro to know the line/file information of where the macro is invocated? 🙂 could one hack it with (def temp-var nil) (meta temp-var)?

valerauko10:08:45

i think this might be relevant to you: https://stackoverflow.com/a/30813410

Saikyun10:08:22

oh awesome, thanks! the local docs didn't mention the metadata: https://clojure.org/reference/macros

Saikyun11:08:13

it was exactly what I needed 🙂 thanks again

valerauko10:08:38

is there any way to check if something is a primitive number type?

valerauko10:08:09

(type 3) says java.long.Long while i thought it'd be the primitive long

valerauko10:08:28

from the other side: is there any way to force a number to be primitive?

bronsa11:08:20

3 is a primitive long, but type takes an Object so it gets boxed

bronsa11:08:16

there's no way to "force a number to be primitive", the long function returns a primitive long, but it depends on the callsite where that value is used whether it will be boxed or not

bronsa11:08:52

literals always start as primitives, but may get boxed if needed

sveri13:08:13

Hi. I have a test that connects to a websocket. The connect function takes a callback and I want to test that something is sent back. Now the problem is that clojure.test does not recognize is expressions in the callback similar to this:

(g/connect (str ws-base-url "ws")
               :on-receive #(is (= 400 %))
My idea would be to have an atom that is set in the on-receive callback and checked after a given time while the Thread is sleeping. Does someone have a better idea? Something more idiomatic?

vlaaad13:08:28

promise instead of atom

4
vlaaad13:08:24

deliver in callback, put blocking (is (= 400 @my-promise)) after connect

dpsutton13:08:12

also useful to use a timeout in this situation

dpsutton13:08:18

(deref ref timeout-ms timeout-val)

sveri13:08:01

I totally forgot about promises, thank you.

denik14:08:03

how does one package a deps.edn based clojure app into an uberjar?

noisesmith17:08:39

via some third party tool, deps.edn is not a packager or build tool

hugo17:08:11

you can use lein's uberjar functionality programmatically.

ghadi14:08:06

you can use seancorfield/depstar @denik

dpsutton16:08:27

you recommend that over the one you wrote for healthfinch?

dpsutton16:08:38

(i realize its a fork)

borkdude15:08:56

excel sheets and clojure in 2019: docjure or something else? I've used docjure some years ago and it was nice

valtteri17:08:48

I've used docjure in 2019 and it works. I've been tempted to try out something like this though https://youtu.be/qnJs79W0BDo

denik17:08:30

I've packaged the add-lib feature from tools.deps into an uberjar for an app that simulates a REPL using eval. However, upon running add-lib it errors w/ Context classloader is not a DynamicClassLoader what is that about?

alexmiller17:08:08

add-lib relies on hooking the current thread’s context classloader but it needs to find a Clojure DCL to be able to dynamically inject libs

denik17:08:30

how does one add a dynamic classloader?

alexmiller17:08:18

And it didn’t find that in your environment

denik17:08:02

thanks @alexmiller, is a clojure DCL inside of a uberjar possible?

denik17:08:42

I don't need to uberjar this app but did so as a measure to lower startup time

seancorfield17:08:54

@denik Unless you AOT the entire app, I would not expect to see much difference between clj -m ... running the -main from source and java -cp the.jar -m clojure.main ... running your -main inside the uberjar.

seancorfield17:08:28

What impacts startup time is mostly whether the code needs to be compiled as it is loaded vs it being already compiled to .class files -- i.e., AOT.

denik17:08:58

is one gen-class for the main file enough or does this mean I need gen-classes everywhere?

seancorfield17:08:06

But for any long-running process, the difference is mostly irrelevant, especially for relatively small app.

seancorfield17:08:34

If you want your -main to be a callable entry point, instead of going through clojure.main, then :gen-class is needed and you need to AOT compile that file before you make the uberjar (and the .class files need to be on your path, which some tooling takes care of and some needs a hint).

seancorfield17:08:14

When you AOT compile your -main namespace, it will also transitively compile other namespaces that are directly referenced in that namespace. That behavior is considered a bug by many people.

seancorfield17:08:39

At work we do not AOT compile anything.

seancorfield17:08:16

We build uberjar files from source, and run them via java -cp path/to/the.jar clojure.main -m entry.point (to run entry.point/-main).

seancorfield17:08:54

The new CLI/`deps.edn` machinery has no support for AOT compilation -- it's intended just for running Clojure.

seancorfield17:08:07

You can manually compile Clojure code (from the REPL, from the command line, etc), and if those .class files are on the classpath when you run something like depstar via clj, then they'll be added to the JAR file.

seancorfield17:08:11

My recommendation would be: ignore AOT compilation completely unless start up time of your app is critical. And if that is the case, do AOT as one last step before creating a production deployment artifact -- don't try to work with AOT while you're developing.

denik17:08:31

thanks! I'm actually using your seancorfield/depstar so adding gen-class by itself will and calling clojure -A:depstar -m hf.depstar.uberjar myApp.jar is not sufficient?

seancorfield17:08:23

depstar does no AOT by design. I do not think Clojure's AOT is a good model in general.

denik17:08:27

I'm building a desktop app, electron frontend, clojure backend. startup time is somewhat crucial, since it affects the delay between clicking the app icon and having the frontend ready

denik17:08:59

hmm, I'll be looking to maximally shave off every second of startup time

seancorfield17:08:05

The Clojure backend is going to run locally? So you're starting up two things? An Electron app and a Clojure/Java process?

denik17:08:37

yes but the clojure app (incl. a server) needs to start before the electron app

seancorfield17:08:59

Sounds like a complicated and fragile way to build a desktop app to me...

denik18:08:26

I could also start the electron app simultaneously and have a loading screen. still, currently it's around 10s of startup for the backend. I'd want this to be as fast as possible

denik18:08:49

it's very simple on the surface

denik18:08:02

startup time is really the only problem

seancorfield18:08:30

Ten seconds of startup sounds long to me. Most of our servers spin up a lot faster than that (when run without New Relic instrumentation).

denik18:08:46

a felt 10 seconds

denik18:08:02

the thing is when the user clicks and app an expects it to open, any delay feels like forever

seancorfield18:08:35

Have you considered writing it all as an Electron app instead of client/server?

seancorfield18:08:52

(Electron apps aren't exactly nimble when starting up...)

denik18:08:48

possible to do this later. however the app literally contains a clojure repl so that needs to be startup up at some point

denik18:08:11

1-2 seconds for clj startup would be fine!

seancorfield18:08:34

If you reduce the static dependencies in your -main namespace (and then dynamically require the rest of the app in a background thread), it will load much quicker (still without AOT) but you'd have to optimize the path to the code being able to accept some requests from the front end. Doable but fiddly.

seancorfield18:08:20

If I definitely needed AOT of the whole app, I'd write a small Clojure utility to do that as its own -main and run that via clj before building the uberjar.

seancorfield18:08:38

Or you could just use lein to build the uberjar with :aot :all 🙂

denik18:08:14

sounds good. thanks for the insights!

alexmiller17:08:48

Uberjar is just packaging. It’s totally independent of the classloader hierarchy

denik17:08:18

@alexmiller so something must be different between running the app using clj and java as a uberjar

hiredman17:08:56

the difference is you aren't just uberjaring

hiredman17:08:00

you are aot compiling

alexmiller17:08:03

clj just calls java with the classpath you built

denik17:08:04

right, clj also causes the classbuilder error

hiredman17:08:29

if you stop aot compiling when you uberjar I bet it works just fine

gerred17:08:35

is there a stable DAG library that people like to use? seems like ubergraph is a good all-in-one option.

gerred17:08:54

loom seems good too

andy.fingerhut17:08:06

I have used ubergraph recently, and found it to be a pleasure to use.

andy.fingerhut17:08:22

If you want multiple parallel edges between two nodes as a possibility, ubergraph can do that, loom cannot.

gerred17:08:30

ahh good to know, ok

andy.fingerhut17:08:02

I have a little bit of code I hope to submit back to ubergraph that implements strongly connected components using a faster algorithm than the one ubergraph currently provides, which it inherits from loom.

gerred17:08:10

multiple parallel edges offers an interesting twist that was going to make me have to split out my graph into way more nodes/edges than I necessarily wanted (in this case, a Kubernetes pod spec plus all of it's spec items as separate nodes)

gerred17:08:32

instead I can just have a parallel edge for each parameter I'm tracking.

andy.fingerhut17:08:44

It was very useful for my application to have parallel edges, too, and glad I didn't have to work around that wish, nor write my own lib.

andy.fingerhut17:08:46

ubergraph also allows you to have an arbitrary Clojure map of attributes for every node and/or edge, so if you are wishing for parallel edges because of attributes on edges with different keys, you do not need parallel edges for that.

gerred17:08:57

ohhh nice.

gerred17:08:19

this is going to be 1000x more expressive than the Go version of this controller. excellent.

gerred17:08:24

thank you!

denik17:08:04

right, clj also causes the classbuilder error

denik17:08:41

@hiredman am I AOT compiling when using clj myProc.core?

denik17:08:50

I don't have a gen-class anywhere, yet the app works fine

hiredman17:08:05

no, but you just said with clj it works fine

denik17:08:32

@hiredman I didn't actually and didn't mean that. It works fine in the REPL but the Context classloader is not a DynamicClassLoader appears both during clj and java with the uberjar

noisesmith17:08:38

an uberjar tool should let you decide whether to aot witha switch

hiredman17:08:21

if you look at clojure/main.clj's repl function you can see the first thing it does is set the current thread's context classloader

👌 4
🙏 4
devn17:08:39

Anyone mind tossing their 2¢ in on the following? https://gist.github.com/devn/32652a23a32e0874175a6c52c65b5e5f

devn17:08:04

I've written it a couple ways, but would welcome a fresh perspective.

devn17:08:33

Also, I'd share my code, but it contains a ton of work-related stuff I'd have to clean out. This is the essence of the transform.

noisesmith17:08:35

@devn looks like (->> tree-seq filter map) to me

noisesmith17:08:09

oh, that doesn't address the name_path thing

noisesmith17:08:13

so more like (->> prewalk tree-seq filter map) - where prewalk injects the name-path based on parent, tree-seq gets you branches, filter prunes to the branches that you want, and map removes the :children key

hiredman17:08:07

I would do something like (fn f [m] (for [x m i (fx)] i))

devn18:08:49

@hiredman would you care to sketch just a bit further? I'm not sure I follow. m is the initial map, and then...?

devn18:08:29

@noisesmith sort of a similar ask for you, if you don't mind

denik18:08:56

https://clojurians.slack.com/archives/C03S1KBA2/p1565717490419100?thread_ts=1565716868.407600&amp;cid=C03S1KBA2 the next issue is "Could not locate net/cgrand/enlive_html__init.class, net/cgrand/enlive_html.clj or net/cgrand/enlive_html.cljc on classpath. Please check that namespaces with dashes use underscores in the Clojure file name." when using add-lib with the added classloader

seancorfield18:08:43

You verified that this works OK in a regular REPL? (just to eliminate typos etc)

seancorfield18:08:31

(it works for me)

(! 754)-> clj -A:deps
Cloning: 
Checking out:  at d0b33e0d346736aa985c150145b332f97b92135e
Clojure 1.10.1
user=> (require '[clojure.tools.deps.alpha :as t])
nil
user=> (require '[clojure.tools.deps.alpha.repl :refer [add-lib]])
nil
user=> (add-lib 'enlive {:mvn/version "1.1.6"})
Downloading: org/clojure/clojure/1.2.0/clojure-1.2.0.jar from 
true
user=> (require '[net.cgrand.enlive-html :as h])
nil
user=> ^D

seancorfield18:08:09

Does loading other libs and then requiring them work in your DCL in your JAR, or is it just this one that fails?

denik18:08:29

yes this always works in a repl

denik18:08:05

I've been developing on the same set of test files for weeks

denik18:08:08

just double checked

denik18:08:24

it doesn't work outside the repl

seancorfield18:08:42

OK, just wanted to check. Have you managed to get any library to add/load into the DCL? Or is this just the first one you've tried?

denik18:08:32

Interesting, it works with

(require '[clj-http.client :as client])

(:status (client/get ""))

denik18:08:49

incl. the add-lib part

denik18:08:10

stange, in the repl it works across the board

denik18:08:32

confirmed, it definitely works in the REPL and does not using clj -m myproject.main

seancorfield18:08:41

OK, so some libraries can be added and loaded (in the JAR) but some others cannot?

seancorfield18:08:47

Hmm, not sure what to suggest at this point. Sorry.

denik18:08:53

thanks @U04V70XH6, @alexmiller if you can think of a path forward, I'd appreciate it

alexmiller19:08:51

sorry, not sure. If you wanted to look into it further, my suspicion is that add-lib is not tracking the libs the same way in the two scenarios. the clj version leverages the libs determined by tools.deps.alpha, which is passed via a system property iirc. If it can't find that, then it falls back to parsing the java classpath, but if you have an uberjar, that would hide all the deps from it.

seancorfield20:08:01

@denik Do you have (with-bindings {clojure.lang.Compiler/LOADER slcl} ... in your code?

alexmiller21:08:28

(as an aside, one of the reasons add-lib is still sitting on a branch is working through some of these ergonomics)

denik22:08:54

@U04V70XH6 @U5H74UNSF setting the classloader helped resolving the Context classloader is not a DynamicClassLoader error. However it seems now that some libs don't end up on the classpath as @alexmiller suggested. thanks for the help, everyone. I might move forward with a temporary and naive approach of bundling the libraries into the app that fail through add-lib (since some do work) until add-lib makes it into deps proper.

denik23:08:41

the reason some other deps like clj-http worked is not because of add-lib but because they're transitive deps of other libs in the deps.edn

denik23:08:39

@alexmiller after add-lib enlive made it into (clojure.tools.deps.alpha.libmap/lib-map) however, it doesn't get picked up by require

alexmiller00:08:14

Lib-map is bookkeeping. To be loaded, it has to be accessible via the DCL

denik13:08:19

fixed it by adding all paths from lib-map to each classloader after add-lib

devn18:08:38

@noisesmith i don't really follow how you are creating and tracking and creating IDs and parent IDs with prewalk and tree-seq

hiredman18:08:39

((fn f [path item]
   (cons (assoc item :path path)
         (for [child (:children item)
               i (f (conj path (:name item))
                    child)]
           i)))
 [nil]
 {:name "root",
  :description "root",
  :children
  [{:name "Foo",
    :description "foo",
    :children
    [{:name "Bar",
      :description "bar",}
     {:name "Baz",
      :description "Baz",
      :children [{:name "Qux",
                  :description "qux"}]}]}]})

hiredman18:08:06

which does the path thing, but not the id thing, which is tricky

hiredman18:08:11

if you can live with random ids(uuids, gensyms) instead of numbers it would be a lot easier

ghadi18:08:59

I'm doing this exact xform for a library right now, in a reduced way: just passing down the parent name to qualify the child name

noisesmith18:08:10

with a stateful (id) function you can apply it to the leaves as they are visited - that does make it trickier and I didn't see it at first

ghadi18:08:23

the essence of it is that the traversal path is reified somewhere

ghadi18:08:18

pass the parent info to the child along with an accumulator

ghadi18:08:44

I think I used a queue + loop/recur, but can do it recursively too

noisesmith18:08:02

right - I was doing that in prewalk (more precisely, updating children of each node (if any) as the parent is updated)

noisesmith18:08:38

a queue / loop might be more natural than a prewalk updating children before visited, yeah

ghadi18:08:46

I just looked at the code, and each item in the queue was a tuple of [node, context]

ghadi18:08:52

I think in your case you could use [item parent] directly, beginning with [root nil]

ghadi18:08:27

(in my case the context didn't exactly correspond to parent nodes, but was altered at some parent nodes, but the idea is the same)

ghadi18:08:20

(loop [acc [] q (conj clojure.lang.PersistentQueue/EMPTY [root nil])]
  ;; process, add self to acc
  ;; add children to queue

hiredman18:08:40

the loop, the queue, the prewalk whatever, just like, do a recursive for, it is great

ghadi18:08:00

that's really nice and I would have never thought of that

ghadi18:08:49

Thanks for not psychologically priming us @devn

ghadi18:08:00

So.... what was your approach(s)?

devn18:08:52

I actually had something sort of similar to the (for [child (:children item)] ...), but I was calling f in the body, which ultimately meant I needed to concat the initial processed row with the result of for and flatten, which is always a smell IMO

devn18:08:36

something like:

(defn make-row [parent row]
  (let [{:keys [name path]} row]
    {:name name
     :path path
     :name_path (if parent (str (:name_path parent) (:name row) "::") "::")}))

(defn unroll-hierarchy [m & [parent]]
  (let [curr-path (if parent (str (:path parent) id "::") "::")
        row (assoc (make-row parent m) :id id :path curr-path)
        next-parent (assoc row :id id :path path)]
     (flatten [row (for [child (:children m)] (unroll-hierarchy child next-parent))])))

noisesmith18:08:57

couldn't that flatten just be concat?

noisesmith18:08:17

maybe with the help of an apply and [] around row

devn18:08:07

i've left out the id generation bit, rest assured it is actually worse than what you see there

devn18:08:53

what i had/have is something gross where i pass an argument id-space into unroll-hierarchy, and then remove the bits that i've used, but that is actually broken for cases where there are multiple children under a parent-- their id's are the same

jimmy18:08:02

You could do a zipper based approach.

(defn decorate-element [counter node]
  (zip/edit node
            assoc 
            :id (swap! counter inc)
            :parent_id (when-let [parent (zip/up node)] 
                         (:id (zip/node parent)))
            :path "TODO"
            :name_path "TODO"))

;; removed bad code here

jimmy18:08:19

(See the rest below) Fill in the path and name path and it produces what you are looking for.

jimmy19:08:47

Better version

(defn traverse-zipper [f zipper]
  (loop [node zipper]
    (if (zip/end? node)
      (zip/node node)
      (recur (zip/next (f node))))))

(let [counter (atom 0)]
  (->> (zip/zipper map? :children (fn [r c] (assoc r :children c)) x)
       (traverse-zipper (partial decorate-element counter))
       (tree-seq map? :children)
       (map #(dissoc % :children))))

cgrand19:08:19

I have discovered a truly marvelous solution, which this channel is too narrow to contain.

cgrand19:08:14

Seriously: I’m typing on a phone but I would try a mix of CPS and lazy sequences.

devn19:08:11

@cgrand I am very excited to see it.

ghadi19:08:07

@bbloom gave a very enjoyable talk called "Dendrology" a few years ago, covering different styles of tree walking

ghadi19:08:35

sometimes you have to pass information down the tree, sometimes you pass up the tree, sometimes both directions....

ghadi19:08:40

iterative, recursive, etc.

cgrand20:08:18

@devn

(defn walk-and-chew [root]
  (letfn [(mp [p x] (if p (str p x "::") "::")) ; quadratic string building :-/
          (wn [nodes parent id cont]
            (lazy-seq
              (if-some [[{:keys [children name] :as node}] (seq nodes)]
                (let [more-nodes (rest nodes)
                      node (-> node
                             (dissoc :children)
                             (assoc
                               :id id
                               :parent_id (:id parent)
                               :path (mp (:path parent) id)
                               :name_path (mp (:name_path parent) name)))]
                  (cons node
                    (wn children node (inc id)
                      #(wn more-nodes parent % cont))))
                (cont id))))]
   (wn [root] nil 1 (constantly nil))))

jrychter20:08:22

I don't know if this is possible, but I really wish Clojure fns implemented the java.util.function interfaces.

jrychter20:08:07

As I'm interoperating with CompletableFutures, I realized two things: a lot has changed in Java (I thought futures were the latest and greatest, had no idea there is a whole framework for asynchronous processing!) and — I really wish Rich would appear at a conference and give a talk explaining how to deal with exceptions/errors in async code.

alexmiller21:08:07

we' been talking about the function interfaces off and on for a couple years and we would definitely like to make Clojure a better player in this regard. the question is how to do it in a way that derives full advantage.

jrychter21:08:13

It seems extremely complex: for it to be flawless, we'd need to support all the variants (Consumer, Function, BiFunction, etc).

alexmiller21:08:41

there's only about 4 essential ones

jrychter21:08:12

I'd only need the ones used by CompletionStage, to be honest.

jrychter21:08:07

As to exceptions, I feel like I'm using core.async channels to poorly reimplement some of what CompletableFuture provides. There must be a better way than inventing a protocol for sending exceptions on metadata channels.

schmee21:08:44

there’s also https://github.com/ajoberstar/ike.cljj which can help a bit when dealing with functional interfaces

jrychter21:08:35

Interesting… but I'm writing a library (interfacing with FoundationDB), so I'd rather reduce dependencies to an absolute minimum.

devn21:08:57

@cgrand you're my hero

cgrand08:08:57

I forgot the lazy-seq wrapping the whole body of wn

devn21:08:06

thank you!

ghadi21:08:33

@jrychter one thing to decide is whether to support the interfaces within j.u.function directly (Function, Supplier, etc), or to support a mechanism what works for all Java lambdas

jrychter21:08:41

@alexmiller is it OK if I click "Answer" on http://ask.clojure.org to provide context information on how this integration would be useful to me as a user/developer? I'm not sure what is considered good form there. And I don't have meaningful solutions ("answers") to contribute.

alexmiller21:08:31

these "answers" are a little weird in that they were imported from the related jira comments

alexmiller21:08:38

but go for it

jrychter21:08:52

Thanks. I did. But the site is really unhappy with backquotes in the example macros I provided. (correction: got it to work, I thought markdown is supported with code blocks)

ghadi21:08:27

@jrychter those macros should be functions:

ghadi21:08:58

(defn as-function [f]
  (reify java.util.function.Function
    (apply [this arg]
       (f arg))))

ghadi21:08:22

the macro, as written, would generate a ton of identical functions everywhere

ghadi21:08:46

(But the problem of smooth interop still stands)

jrychter21:08:59

@ghadi: I realize that. I'm prematurely optimizing. And these will be used only in a limited number of places.

jrychter21:08:35

But you are definitely right and I do plan to revisit this and check if it makes any sense to have these as macros.

ghadi21:08:02

what's even more challenging about this interop: It looks like FoundationDB takes a bunch of Function<Transaction> as args:

ghadi21:08:09

if you do

(reify Function
  (apply [_ ^Transaction tx] .....))
as you'd expect, it won't compile

ghadi21:08:17

because Function doesn't take a Transaction, it takes a generic U

ghadi21:08:54

so you must do:

(reify Function
  (apply [_ tx]
    (let [^Transaction t tx]
      .....)))

ghadi21:08:12

to get the type hinting right

ghadi21:08:30

Lots of noise, when compared to Java

jrychter21:08:47

Hmm. I wonder why I'm not seeing that problem.

ghadi21:08:41

(reify java.util.function.Function
  (apply [_ ^String s] (str "foo" s) ))
Syntax error (IllegalArgumentException) compiling reify* at (REPL:1:1).
Can't find matching method: apply, leave off hints for auto match.

jrychter21:08:38

I mean, all my functions supplied to as-function are type-hinted, but the apply parameters are generic.

jrychter21:08:04

Well, I'll just ignore that problem until it appears and bites me. For now I'm trying to figure out how to handle exceptions in async code and report/handle them. It feels like I'm reimplementing some of what CompletableFuture does.