Fork me on GitHub
#clojure
<
2020-10-15
>
deadghost01:10:25

https://clojuredocs.org/clojure.spec.alpha/gen > Note that parent generator (in the spec or overrides map) will supersede those of any subtrees. Is there a better way to have a custom generator in a subtree than doing nested gen overrides?

;; Example

;; We love wasteful packaging so we want to create a ::crate that will always
;; contain a ::big-box. We want each ::big-box to contain one or more
;; ::small-box. In each small box, we want to have one Light Bulb.

(s/def ::crate (s/keys :req-un [::big-box]))
(s/def ::big-box (s/coll-of ::small-box :kind vector?))
(s/def ::small-box (s/coll-of string? :kind vector?))

(let [gen-override
      {::big-box
       ;; Override so ::big-box is not empty
       #(gen/not-empty
         (s/gen ::big-box
                ;; And that each ::small-box in ::big-box only has a Light Bulb
                {::small-box (fn [] (s/gen (s/coll-of #{"Light Bulb"}
                                                      :kind vector?
                                                      :count 1)))}))}]
  ;; Generate a ::crate with our nested overrides
  (gen/generate (s/gen ::crate gen-override)))

;; =>
{:big-box
 [["Light Bulb"]
  ["Light Bulb"]
  ["Light Bulb"]
  ["Light Bulb"]
  ["Light Bulb"]
  ["Light Bulb"]
  ["Light Bulb"]
  ["Light Bulb"]]}

Tom Helmuth01:10:37

Not sure if this is the right place to ask this, but here's an oddity that greatly confuses me.

Tom Helmuth01:10:59

I understand what is going on here:

> ('foo {'bar 5 'foo 3})
3

Tom Helmuth01:10:59

But what is up with this behavior:

> ('* 3 2)
2

Tom Helmuth01:10:43

Is it trying to look up the symbol '* in the "map-like thing" 3, and when it doesn't find it, returns the default value of 2?

andy.fingerhut01:10:18

Yes, that is the explanation for that behavior.

Tom Helmuth01:10:08

Why is it happy to treat 3 as a map-like thing without throwing an error? I guess that's the part that really baffles me.

andy.fingerhut01:10:36

I do not know the reason, but I think it is for the same reason that (get 3 '* 2) is happy to treat 3 as a map-like thing.

andy.fingerhut02:10:17

It has been that way in Clojure I believe since its beginning, and a JIRA bug was files in 2012 suggesting such expression throw an exception. Rich Hickey responded in 2014 that such a change would be a breaking change, and the Clojure core team values backwards compatibility very highly. I know that doesn't explain why it was like that in Clojure's original versions -- again, I do not know the reason, just giving you a little bit of history that this wasn't a recent thing, nor can I find anything in Clojure FAQ nor http://ClojureDocs.org entry for 'get' that gives a reason: https://clojure.atlassian.net/browse/CLJ-1107

seancorfield02:10:48

If it's any consolation, I think pretty much everyone new to Clojure asks this... 🙂

seancorfield02:10:51

Symbols and keywords look themselves up in their argument and it's actually very convenient when writing generic code because you don't have to worry about type checking when you simply get nil from an "inapplicable operation".

seancorfield02:10:55

It feels like this question should be added to https://clojure.org/guides/faq in some form -- @U017SU36TG9 do you think you could formulate your question in a form suitable for that page? I'd be happy to help turn it into a contribution for that page (I just recently contributed the ?/`!` FAQ there).

Tom Helmuth02:10:24

Sure, I'd be happy to.

Tom Helmuth02:10:27

FWIW, I've been using Clojure for 10 years and just discovered that (get 3 5 :default) is happy to treat 3 as a map-like thing. @U0CMVHBL2 I appreciate the historical context!

seancorfield02:10:29

Feel free to DM me to chat about it (tomorrow -- it's a bit late tonight for me to coherently process that).

Tom Helmuth02:10:35

Looks like this will happen with any type of value that isn't a thing that expects to be used for lookup. My student who ran into this issue found the source code, and if none of those conditions are met, it just returns the notFound value: https://github.com/clojure/clojure/blob/master/src/jvm/clojure/lang/RT.java#L789-L820

seancorfield03:10:34

Yup, it's a very deliberate design decision: if the type is something you could "get" a thing from, attempt the lookup (and return notFound if it isn't there), else return notFound.

seancorfield03:10:05

But it surfaces in a number of expressions and that's the part I'm not sure how to formulate as a "frequently asked question".

Tom Helmuth03:10:05

The crazy part is that it makes the one-character typo of ('* 3 2) behave in a totally unexpected way. I totally get the two parts making sense independently (symbols can be used as functions for lookup and get returning notFound if you can't get from that type), but the combination is lethal.

andy.fingerhut03:10:33

As you can see from the JIRA issue, at least some people wish that get being called on a non-associative first arg were considered lethal (i.e. throw an exception instead of returning nil) all by itself, but that is unlikely to ever be changed, given backwards compatibility, plus whatever the original design reason was.

mavbozo06:10:20

every time I see this familiar question, i remember

(get get get get)  
;;=> #function[clojure.core/get]

andy.fingerhut16:10:54

@U04V70XH6 Perhaps the frequently asked question might be "Why do several ways of looking up a key in an associative collection return nil when given something that isn't an associative collection?" starting with examples of both the get and symbol-as-first-form-in-parens expressions?

andy.fingerhut16:10:04

Granted that 'get' and the "symbol lookup as first form within parens" don't have good strings I can think of that people would be Google-searching for when they have this question, but folks like us who know what FAQs exist would know it was there on that page somewhere.

andy.fingerhut16:10:21

I wouldn't be surprised if Rich Hickey himself may have addressed the "why is it designed this way?" question around 2009-2010 in the Clojure Google group, but I didn't find such a thing in 10 mins of searching, again given the difficulty of finding the right search terms.

andy.fingerhut16:10:56

This old thread doesn't answer the question, but Chouser gives his quick rules of thumb for choosing between (key-or-symbol some-map) versus (some-map maybe-a-key) versus (get some-map maybe-a-key) alternatives. https://groups.google.com/g/clojure/c/Cxyz6UiTlVY/m/sm36hUX9L44J

andy.fingerhut16:10:09

Ah, found one where Rich himself addresses the question of get returning nil vs. throwing an exception! https://groups.google.com/g/clojure/c/7nqh1Qc84iQ/m/DriZStY3k68J

seancorfield16:10:18

It's a bit of a "throw away" reference to Common Lisp's behavior tho'...

andy.fingerhut17:10:28

Understood it doesn't explain with a lot of words ... Haven't seen anything more than that yet, though.

andy.fingerhut17:10:59

And I am sure that no matter what quotes I might find, any new FAQ will be reviewed by Alex and perhaps even Rich to avoid giving incorrect reasons/rationale.

andy.fingerhut17:10:00

FYI +10 from me if you do end up creating a new FAQ entry for this. It is slightly above my activation energy threshold to create one, but interested to see how it turns out.

andy.fingerhut17:10:38

And those 10 points are redeemable, you know, in all the usual places 🙂

seancorfield17:10:40

A beer at a conference sometime in the post-plague era? 🙂 🍺

andy.fingerhut17:10:11

Sure. I could probably also arrange to have something from a place like Total Wine & more deliver to your doorstep 🙂

andy.fingerhut17:10:44

So far the only chain of stores in USA that I have found carries Rauchbier from Bamburg, Germany. An acquired taste.

seancorfield17:10:42

I've been ordering a lot from Belching Beaver in So. Cal. Lots of fruity sours, if you're into that (I'm not, but I love their other stuff). Also, I've clipped this entire convo into OneNote so I can use fragments of it to distill down to an FAQ at some point. Thanks everyone!

andy.fingerhut17:10:13

And looking more carefully at the last Clojure Google group thread I linked above, it appears Rich is only addressing the question of get returning nil vs. throwing exception when the key is not found. I don't think it addresses the issue of throwing exception when the collection is not an associative thing.

Tom Helmuth01:10:22

This just caused some massive headaches when accidentally wrote '* instead of *', but luckily stumbled into the problem.

ryan echternacht13:10:06

I’m looking to build a “job queue”. It’d poll a db for work (I’m doing that with http://docs.oracle.com/javase/1.5.0/docs/api/java/util/concurrent/ScheduledExecutorService.html in java), and then fire off work. The work is mostly file imports, that are safe to be run concurrently. What’s the best clojure threading primitive for this type of work?

ryan echternacht13:10:39

I care that jobs finish, and their finishing status. but i’m not sure how much more I really care about them while running.

ryan echternacht13:10:46

I could see using core.async for it, but I don’t think I have that much of a pipelining need. Futures seem like they might work, but almost seem to barebone — I feel like I’ll be polling them for status, whereas I’d rather react to when they’re done.

kennytilton18:10:41

It’s not so much polling so much as iterating over a vector of go-loop channels doing a blocking read from each, which will stop at the first unfinished go-loop, “forgetting” loops as they finish. I use core.async in cases like this even without a pipelining requirement. fwiw.

ryan echternacht00:10:06

I’m working through building it with core.async, and i’m pretty happy. I think I had the mental block that core.async was only for “big” things, but that’s a bad model

🍺 1
borkdude13:10:38

@ryan072 I borrowed this bit of code from CLJS to run an amount of tasks in parallel (to fully utilize CPU cores) and wait until they're done: https://github.com/borkdude/clj-kondo/blob/51db4e172478c369208ed90db4b7c29d94a8cbce/src/clj_kondo/impl/core.clj#L209-L233 Not sure if that's similar to your problem

👍 1
borkdude13:10:23

Each of the parallel tasks pick work from a queue, until there is no more work, then they lower some countdown latch.

ryan echternacht13:10:09

thanks, that does help. Gives me some confidence on rolling my own, and a some good java primitives to look into

chunsen14:10:32

HI, I've got an error saying java.lang.NoClassDefFoundError: manifold/bus/IEventBus when running my backend service using uberjar profile, which configed in project.clj as :profiles {:uberjar {:main hello-ring.main, :aot :all} . Finally I found the reason, the class loader of manifold/bus/IEventBus`` has a hierarchy as below

#object[clojure.lang.DynamicClassLoader 0xc7aac7c [email protected]]
#object[clojure.lang.DynamicClassLoader 0x2186d797 [email protected]]
#object[sun.misc.Launcher$AppClassLoader 0x23fc625e [email protected]]
#object[sun.misc.Launcher$ExtClassLoader 0x50313382 [email protected]]
While the class loader of the AOTed class of my business code is #object[sun.misc.Launcher$AppClassLoader 0x23fc625e "[email protected]"] , so it can not find the class which is loaded by child class loader(`#object[clojure.lang.DynamicClassLoader 0xc7aac7c [email protected]]`). Any one have an idea about this issue?

alexmiller14:10:10

I think you'll need to share more to understand what you're seeing. where do IEventBus and the impl come from? how is this app run? are you using java -jar with the uber jar to launch that main class?

chunsen14:10:06

I got this error either run the program through java -jar or through repl .

Endre Bakken Stovner18:10:33

I'd like to get the line and column number of every element in the below data structure. {:a {:aa 2} :b {:ba 0 :bb 3}} So the first { (the whole map) has coordinates 0 0, the :a has 0, 1 {:aa 2} has 0, 4, 3 has 1, 15. Is there anything in clojure or the ecosystem that will help me do this?

p-himik18:10:05

Not directly, but you could write a macro for that that analyzes &form and its metadata:

user=> (defmacro form-meta [] (meta &form))
#'user/form-meta
user=> (form-meta)
{:line 1, :column 1}

👍 1
phronmophobic18:10:33

My approach would probably be use to use https://clojure.github.io/clojure/clojure.zip-api.html Would you need a need it to return a new data structure or adorn the passed in data?

phronmophobic18:10:00

oh wait, you're looking for line and character positions?

alexmiller18:10:33

if you use a LineNumberingPushbackReader to read it, you can get some of that from the reader (in metadata)

👍 1
alexmiller18:10:55

the maps can carry metadata but keywords and numbers cannot so you won't get those

alexmiller18:10:32

I believe there are other tokenizing parser libs for Clojure that can though, just not sure which is the best recommendation

hiredman19:10:54

But keywords can't have metadata, so any method that relies on it to return position information won't tell you where the keywords are

Endre Bakken Stovner19:10:45

I wonder if any libraries are able to work around that limitation in some way? I will have to investigate tomorrow. https://github.com/Engelberg/instaparse

andy.fingerhut19:10:42

The only way I can think of to get position information for things where Clojure does not support metadata, e.g. keywords, numbers, strings, is to return NOT a Clojure collection that is equal to what clojure.edn/read would return, but instead to return some kind of 'parse tree' representation that had an explicit line/column data for each object of the parse tree.

andy.fingerhut19:10:35

For example, if when reading {:a 2} you tried returning a value x such that (= x {:a 2}) was true, then there is no place "inside" of there to put the line/column info of the number 2. If you instead return a value y such that (= y {:a 2}) was false, you have many choices for how y represents the string that it read and its parts, and many choices for how the line/column info is embedded in there.

andy.fingerhut19:10:40

@U04V15CAJ wrote https://github.com/borkdude/rewrite-edn very recently, so should have fresh on his mind whether it supports line/col info on all sub-parts of the returned data, or not.

borkdude19:10:44

@UT770EY2K rewrite-edn is based on rewrite-cljc and rewrite-cljc can give you that metadata There is also https://github.com/borkdude/edamame which will parse directly to sexprs with adding location metadata when possible. Using a post-process step you can also have metadata on numbers etc when you coerce them in something that can have metadata. See this test: https://github.com/borkdude/edamame/blob/ba93fcfca1a0fff1f68d5137b98606b82797a17a/test/edamame/core_test.cljc#L306

Endre Bakken Stovner06:10:26

Thanks for all the replies so far. Perhaps I should explain my use-case. I want to write a highlighter (like https://github.com/benma/visual-regexp.el) for https://github.com/redplanetlabs/specter so that I can see what result my specter-expression returns in real-time. Then it would be neat if I could annotate every value in the data structure I want to query with line/column info. Then a specter highlighter could merely: 1. annotate every value 2. run specter as usual 3. pick out the line/column from the returned results afterward But if one cannot annotate keywords/numbers this approach won't work. I guess I can use the edamame wrapper-approach, but then I would need to convert all specter-queries for numbers/keywords into a specter-query for wrapper(number/keywords). If you can think of simpler strategies please advise.

Endre Bakken Stovner06:10:48

I was unable to see how rewrite-cljs can help me with the above, but many of the linked to pages in their docs were 404s.

dominicm19:10:57

I'm trying to extend this interface: https://github.com/eclipse/lsp4j/blob/release_0.8.1/org.eclipse.lsp4j/src/main/java/org/eclipse/lsp4j/services/TextDocumentService.java#L91-L94 like so:

(reify
  TextDocumentService
  (completion [position]))
but I'm getting:
Can't define method not in interfaces: completion
And I'm confused why, because it's definitely there!

dominicm20:10:58

Apparently I wanted proxy, my bad!

lread20:10:42

Did you need proxy because of the default method?

dominicm21:10:25

I think Alex got it, it's because I forgot this

lread23:10:55

Ah! Thanks.

alexmiller20:10:19

the completion method in your reify should take 2 args - a "this" arg and then the args on the method

alexmiller20:10:51

(reify
  TextDocumentService
  (completion [this position]))

alexmiller20:10:23

you probably don't actually need that arg, so fine to replace this with _ in that case

dominicm21:10:11

Oh, I didn't know clojure-lsp also used Lsp4j, convenient as a reference!

dominicm21:10:56

I was going to take a stab at a runtime-LSP for Clojure, but LSP4J feels like the worst of Java 😁 Maybe I'll leave that for a more determined Dominic in the future.

Célio22:10:27

Hi all. Does anyone know why clojure.data.json/read-str behaves this way? I have this situation where some keys contain multiple slashes. In the first example below, the resulting map is tagged with #:a which appears to come from the key, and the resulting key is missing the a/ prefix. This only happens when the JSON object contains only one key with multiple slashes. In the second example though the object contains two such keys and the behavior changes, it doesn’t add any tag to the resulting map and preserves the form of the original key.

(clojure.data.json/read-str "{\"a/b/c/d\":1}" :key-fn keyword)
;; => #:a{:b/c/d 1}

(clojure.data.json/read-str "{\"a/b/c/d\":1,\"e/f/g/h\":2}" :key-fn keyword)
;; => {:a/b/c/d 1, :e/f/g/h 2}
The same behavior can be observed with Cheshire.
(cheshire.core/parse-string "{\"a/b/c/d\":1}" true)
;; => #:a{:b/c/d 1}

(cheshire.core/parse-string "{\"a/b/c/d\":1,\"e/f/g/h\":2}" true)
;; => {:a/b/c/d 1, :e/f/g/h 2}
Interesting though is that keyword does what I expect, it doesn’t cut off the key’s prefix, so I suspect the behavior described above is caused by something else.
(keyword "a/b/c/d")
;; => :a/b/c/d
Is there any way to prevent the key from being parsed without the prefix, as in the first example?

Lennart Buit23:10:26

In your first example, it appears to be a keyword with namespace a and name b/c/d

Célio23:10:47

@UDF11HLKC At first I was surprised to see that the parser was taking off the a as a namespace and using it to tag the map. But that doesn’t explain the second example’s results.

phronmophobic23:10:57

this isn't about the parser(s). it's just how the map is being printed out:

user> {:a/b/c/d 1, :e/f/g/h 2}
{:a/b/c/d 1, :e/f/g/h 2}
user> {:a/b/c/d 1, :a/f/g/h 2}
#:a{:b/c/d 1, :f/g/h 2}

phronmophobic23:10:07

the prefix is just a shorthand

Lennart Buit23:10:14

Well the #:a{...} syntax denotes all keys share the namespace :a, in your second example it can’t shorten it because there is no common prefix

👆 1
phronmophobic23:10:36

the key is still the same

user> (get {:a/b/c/d 1, :a/f/g/h 2} :a/b/c/d )
1

Célio23:10:59

Hah! Interesting.

Lennart Buit23:10:33

(map namespace (keys ...)) out of the top of my head will tell you what the namespace parts of the keywords are :)

Célio23:10:35

Indeed:

(let [m (cheshire.core/parse-string "{\"a/b/c/d\":1}" true)]
  (keys m))
;; => (:a/b/c/d)

p-himik23:10:19

I know that you didn't really ask about it but perhaps you might want reconsider keywordizing such maps where keys contain special symbols like /. Otherwise, it increases the potential to make things worse down the road.

👆 1
Célio23:10:41

That’s comforting. Now I need to find out why the key :a/b/c/d is getting stored as b/c/d in mongodb.

p-himik23:10:25

Oh no. That's exactly what I'm talking about. :)

p-himik23:10:31

Because it uses name, I bet.

Célio23:10:41

@U2FRKM4TW Totally agree. Unfortunately in this particular case I cannot avoid that. 😞

Lennart Buit23:10:14

Hmm? Why do you need to keywordize while parsing?

Célio23:10:14

@UDF11HLKC It’s a huge map that comes thru a REST endpoint. 99% of the keys are just your good old normal keys but there’s this little map deep inside which contains those bizarre keys with slashes.

Célio23:10:41

I guess I’ll have to give that little map a special treatment—i.e.: converting those keys to strings—before storing in mongo… ugh!

p-himik23:10:35

Or just don't keywordize the keys, separate the things that need to be keywordized from the things that don't need that, and keywordize the former manually.

👍 1