Fork me on GitHub
#clojure
<
2019-02-28
>
devth00:02:16

i've been seeing obscure errors in my stacktraces like in unnamed module of loader 'app' since upgrading to java 11 and clojure 10. others seeing this? example:

class clojure.lang.PersistentList cannot be cast to class clojure.lang.IFn (clojure.lang.PersistentList and clojure.lang.IFn are in unnamed module of loader 'app')

rutledgepaulv00:02:48

I have seen those - my understanding is that’s just letting you know what jvm module (jigsaw and all) was responsible for the classes listed. but the exception is that you’re using a list as a function somewhere

rutledgepaulv00:02:05

I believe with jigsaw there’s an implicit top level module if you don’t define your own modules

rutledgepaulv00:02:12

presumably that’s what ‘app’ is

hiredman00:02:19

yeah, that looks like a regular error with a little extra java module information appended

hiredman00:02:35

user=> (())
Execution error (ClassCastException) at user/eval602 (REPL:1).
class clojure.lang.PersistentList$EmptyList cannot be cast to class clojure.lang.IFn (clojure.lang.PersistentList$EmptyList and clojure.lang.IFn are in unnamed module of loader 'app')
user=> 

Alex Miller (Clojure team)00:02:46

That’s standard for Java ClassCastExceptions now

yuhan03:02:35

Is there a common name for this function in Clojure?

(defn <*>
  [fs xs]
  (reduce-kv (fn [m k f]
               (assoc m k (f (get xs k))))
    (cond (vector? xs)      []
          (associative? xs) {})
    fs))

yuhan03:02:07

it applies each f in fs to each x in xs while trying to keep the original data structure

yuhan03:02:25

> (<*> [dec inc] [10 20])
[9 21]

> (<*> {:a dec :b inc} {:a 10 :b 20})
{:a 9, :b 21}

potetm04:02:36

so, I’ve seen thinks like map-vals that’s basically (into {} (map (fn [[k v]] [k (f v)])) coll)

potetm04:02:16

it’s very questionable to me that there’s value in not knowing whether you have a vector or a map

potetm04:02:29

like, I kinda see it, but I mostly don’t 😛

yuhan04:02:34

hmm ok, just realised that it's almost equivalent to

(defn <*> [fs xs]
  (reduce-kv update xs fs))

potetm04:02:41

I misinterpreted what you were doing

potetm04:02:55

(a little)

yuhan04:02:04

the concept is roughly equivalent to the Applicative typeclass in Haskell, or like juxt with multiple arguments

yuhan04:02:37

so I was wondering if there was a primitive for it

potetm04:02:04

not something I’ve seen before 🙂

yuhan04:02:28

looks like reduce-kv doesn't work on lazy sequences though

yuhan04:02:15

I wonder what would be the expected behaviour for missing keys :thinking_face:

yuhan04:02:38

or variadic arguments

yuhan04:02:30

real-world example: let's say you have a map of strings, each to be parsed differently

yuhan04:02:38

(def data {:date  "2019-02-28"
           :price "$20.50"
           :id    "0a18abb7-dd4f-4aad-a49a-acb46cf23955"})

(<*> {:date  parse-date
      :price parse-currency
      :id    parse-uuid}
  data)

seancorfield04:02:01

@qythium You can use empty instead of that condition (from 30 minutes ago). (empty xs) will produce [] if xs is a vector and {} if xs is a hash map.

👍 5
yuhan04:02:24

Thanks for the tip!

seancorfield04:02:43

For the above example, I'd probably be tempted to use conforming specs...

(s/def ::date parse-date)
(s/def ::price parse-currency)
(s/def ::id parse-uuid)
(s/conform (s/keys :req-un [::date ::price ::id]) data)
where each of the parse-* functions accepted a string and either returned the parsed value or else s/invalid.

seancorfield04:02:18

(but you'll get howls of protest from the folks who say specs should not conform from string to non-string types 🙂 )

yuhan04:02:21

That's interesting, I've only used spec for instrumenting functions during development - is it common practice to use it in this way for actual application logic?

seancorfield04:02:44

Also, in the above example, what should have to keys that aren't in the parse hash map? Clojure's principles would suggest that such any additional key/value pairs should be passed through unchanged.

seancorfield04:02:37

We use spec very heavily in production code. We use it for API parameter validation (and we use heuristics on top of explain-data to produce user-facing error messages).

dtsiedel13:02:29

Can you elaborate a little on these heuristics? We like spec but have found it very troubling to have to wrap any user-facing validation (since the messages are pretty atrocious for those who don't speak Clojure).

seancorfield17:02:53

We have a hash map for each API that maps symbols (or occasionally sequences of symbols) to error codes, and we walk the explain-data, looking at :path and :pred and build a reductions of partial combinations of those, then walk that list looking for a match in our API hash map.

seancorfield17:02:13

Missing parameters match to a contains? check on the API parameter name. Incorrect types match to a pair of the API parameter name and the spec operator, e.g., int?.

dtsiedel18:02:05

Wow, that seems like a lot of work, but is sort of what I imagined would be required to prettify those. Is that code public? Thinking of building something similar at work in the medium-long term.

seancorfield19:02:04

I'm hoping to extract the generic heuristic code into a library and open sourcing it at some point. Right now, it's a bit tangled up with some of our proprietary code.

dtsiedel17:03:01

I think that would definitely help improve practicality of spec

seancorfield19:03:43

@UFMHWUULB Have you looked at expound?

dtsiedel19:03:04

I have, and I think it's pretty solid. But for a lot of purposes I want to leak even less Clojure/spec stuff in error messages. So if a user sends JSON to my application and gets back this error message, they aren't confused by seeing the actual keys, maps, predicates etc. that I used internally.

dtsiedel19:03:30

For me that's just meant trying my best to wrap any spec checks and give "Clojure-free" messages, but it's tedious and failure-prone

dtsiedel19:03:06

Maybe there isn't really a great solution, since failing a spec inherently means getting "you were missing key :some-key in some-map", and you can't really send that out to the world without revealing internals of Clojure code

seancorfield19:03:15

If you need to produce domain-specific error messages, you pretty much have to have domain-specific heuristic code... but if I can reduce it to generic data walking and lookup + a simple hash map, then it would be worth releasing as an OSS lib...

dtsiedel19:03:25

Yes I think so (on both accounts)

seancorfield04:02:41

We also use spec around our persistence model, to strip hash maps down to just the expected keys (which we extract from the s/keys in the spec via s/form) and that the values conform to what the database expects.

seancorfield04:02:37

We have three layers of spec: API parameters (conforming from string to simple values), domain model (this isn't used a huge amount right now), persistence model.

seancorfield04:02:45

We don't use instrumentation a great deal (we use it around some tests). We use s/exercise to produce "random example" data for unit tests. We have a few st/check tests as well, and some "occasionally run" generative tests.

seancorfield04:02:16

(we started using early releases of spec in production during the early 1.9 alpha cycle)

yuhan04:02:43

Thanks for the explanation 🙂 Sounds like I should start looking into spec more deeply

yuhan04:02:19

In the above case, missing keys are dealt with by passing values through unchanged if there's no corresponding function

yuhan04:02:31

and calling the function with nil if there's no corresponding value

yuhan04:02:05

(defn <*>
  "apply functions in `fs` to values in `xs`"
  [fs xs]
  (reduce-kv update xs fs))

> (<*> [dec inc str] [1 2 3 4 5])
[0 3 "3" 4 5]

> (<*> [dec inc str] [1 2])
[0 3 ""]

seancorfield04:02:34

And in the case of hash maps?

yuhan04:02:37

> (<*> {:a str} {:b 3})
{:b 3, :a ""}

seancorfield04:02:01

Ah, OK. Not what I expected.

seancorfield04:02:30

I wasn't expecting the function to add new keys to the map being processed. But that's a reasonable behavior too.

yuhan04:02:24

I guess that using spec would return s/invalid or throw an exception? Unless using opt keys, which I recall were going to be deprecated in the future

seancorfield04:02:40

I don't think a decision has been made on that yet.

seancorfield04:02:11

Specs should return either a successfully conformed value or s/invalid. Throwing an exception is not useful there.

seancorfield04:02:53

What we do for specs that conform strings, is we wrap their parsing in try/`catch` and in the catch we return s/invalid.

yuhan06:02:12

How do I define a custom return value for s/conform? E.g. I have a spec like

(s/def ::nested
  (s/tuple (s/tuple #{"s"} int?)))
and I'm only interested in the int

yuhan06:02:34

so I would like to have (s/conform ::nested [["s" 123]]) => 123

yuhan06:02:00

(apologies if this is more suited to the beginners channel)

yuhan07:02:48

Also is there any way to recursively conform specs?

> (s/conform ::int-string "3")
3

> (s/conform (s/keys :req-un [::int-string])
    {:int-string "3"})
{:int-string "3"}

yuhan07:02:35

Google searches didn't lead me anywhere 😕

Suren P08:02:30

Hey how do Clojure people run db migrations? In .NET I have my project set to auto-migrate a DB on startup in production if there is things to migrate. So I just have to git push and I'm done deploying a new version of my app. Anything similar in Clojure?

lispyclouds08:02:40

I generally call the migrate function in the main which applies it everytime the server starts in an idempotent manner

borkdude13:02:17

Hi @U7ERLH6JX - nice to meet you at ClojureD 🙂

borkdude13:02:07

Btw, there’s also a GraalVM version of migratus: https://github.com/leafclick/pgmig Should be super fast to run, didn’t try it.

lispyclouds13:02:29

Nice to meet you too @borkdude 😄

mkvlr08:02:37

does somebody here have experience with integrating single sign via OAuth or SAML 2? Wondering which of the many java libraries to choose. Today I only need Azure Active Directory support, but it would be handy if the library handled more…

Vesa Norilo09:02:40

hi guys! is there a canonical way to get a subspecification from a clojure.spec? something like get-in where you would supply a spec path? The use case is to validate the results of ´update-in´ when you have a spec for the root without checking the entire datastruct

Vesa Norilo09:02:02

I guess I could work from spec/form, but maybe there is a better way.

vemv12:02:55

Out of a binding vector with destructuring:

[{::keys [foo bar]
  baz ::else/omg
  {:keys [tomato]} :potato
  :as self}]
...how would you get a mapping of the extracted variables to their paths? Like the following
{foo [::sth/foo]
 bar [::sth/bar]
 baz [::else/omg]
 tomato [:potato :tomato]
 self []}

Alex Miller (Clojure team)13:02:57

Destructuring in macros is backed by a function called destructure which rewrites the binding vector into a binding vector without destructuring. It won’t give you paths but it’s halfway there

Alex Miller (Clojure team)13:02:27

That function is intentionally not doc’ed (as it’s an impl detail) but it takes a binding vector and returns a binding vector so you can invoke it by quoting your vector

Alex Miller (Clojure team)13:02:34

The next step of going from that binding vector to paths is probably not fun, but it’s possible

vemv13:02:03

thanks Alex 🙂 I had played a bit down that path... and yes, didn't look very fun!

igrishaev13:02:57

what would be the best way to log an exception’s ex-data using logging? Say with logback as the logging backend. E.g when if I did

(log/errorf (ex-info "test" {:foo 42}))
and it produced something like
- host date | "test" {:foo 42}

igrishaev13:02:01

Does it mean I have to write my own appender or convertor class?

valerauko14:02:53

what would be your expected output?

igrishaev15:02:32

@vale as I posted above, e.g. - <host> <date> | "test" {:foo 42}

👍 5
craftybones18:02:53

I was teaching a bunch of people Clojure. I asked them to implement the game 2048 and one of the constraints I had was to use as few defn as possible. https://gist.github.com/craftybones/42f180cf753db07b6694de87f4182ece

craftybones18:02:13

I’d appreciate it if people can critique that solution.

craftybones18:02:23

Quite a few people landed at roughly the same solution

noisesmith19:02:00

@srijayanth many of those partials could be transducer arities instead, using comp to combine them, then using a transducing context at the top level

craftybones19:02:43

@noisesmith thanks. I hadn’t yet taught them transducers at that point, but point taken

craftybones19:02:28

Also, I didn’t realise that partition returns a transducer

craftybones19:02:40

Darn, so cool 🙂

noisesmith19:02:50

yeah, I think all your partials are valid transducers

noisesmith19:02:58

your comps would need to reverse order of course

craftybones19:02:46

But ok otherwise isn’t it? Its remarkable how a game such as 2048 can be written almost entirely without a single “variable” and letting a managed stack do all the work

noisesmith19:02:11

for transpose, there are few cases where list is better than vector and I suspect this isn't one of them

noisesmith19:02:04

both are eager, you can easily seq on a vector, you can't easily do indexed lookup on a list

noisesmith19:02:44

and in many cases creating a vector is lower overhead (one contiguous object instead of a linked series of allocations)

craftybones19:02:10

so (def transpose (partial apply map vector)) instead?

👍 5
noisesmith19:02:16

and agreed, it's a good demonstration of how elegant and powerful higher order functions can be

craftybones19:02:13

To be clear, a seq is eager, so seq on a vector and a seq on a list should theoretically be the same, so why the vector?

noisesmith19:02:23

list is eager

craftybones19:02:45

But wouldn’t the seq force both?

noisesmith19:02:47

the difference is that vectors are more efficient to allocate, and they have fast indexed lookup

noisesmith19:02:58

there's no forcing in question, both are eager

craftybones19:02:06

Right. That’s what I assumed.

craftybones19:02:21

the allocation is the performance you are talking about

noisesmith19:02:46

lists have few (no?) features that make them superior to vectors

noisesmith19:02:57

the allocation and the lookup time

craftybones19:02:27

lists make better stacks?

noisesmith19:02:42

vectors are great stacks, you just pile onto the end

Alex Miller (Clojure team)20:02:29

lists are much faster than vectors as stacks btw

noisesmith20:02:47

oh! - good to know

Alex Miller (Clojure team)20:02:25

I mean the data structure modification is a pointer, so that should be the guess (but it's easy to see in a perf test)

noisesmith20:02:40

using criterium and a random series of stack operations I was able to confirm that, though the difference was smaller than the smaller standard deviation

Alex Miller (Clojure team)21:02:09

you'll get more variability with the vector I suspect as it will depend on how deep in the data structure you are updating

Alex Miller (Clojure team)21:02:37

so for an empty vector, would likely be pretty fast, but you might see a different story on one with a few dozen elements

noisesmith19:02:00

peek / pop work on both vectors and lists

craftybones19:02:12

I remember reading somewhere about peek performance.

craftybones19:02:21

or last. I can’t remember which

noisesmith19:02:38

peek is faster than last, and happens to find the last element of a vector

noisesmith19:02:03

vectors are indexed, so all lookups by index are faster

noisesmith19:02:33

(of course you lose this advantage by calling seq, but you are still no worse off than you were with a list)

craftybones19:02:53

I am guessing lists are there primarily for homoiconicity?

craftybones19:02:13

and we always need more parentheses, of course

noisesmith19:02:54

yeah, you need lists from macros (but only weird macros are built out of direct calls to list)

craftybones19:02:51

That’s true. Regular old quoting seems to cover most needs

craftybones19:02:46

Is the idea of LightTable dead? There seems to be a brave soul who every now and again revives it from the dead

craftybones19:02:03

I am more interested in the idea itself even if the project doesn’t rekindle itself

hiredman19:02:29

what is the idea of lighttable?

hiredman19:02:12

(like, I am familiar with the project, and have seen chris talk, etc)

hiredman19:02:19

but what do you think the idea is?

craftybones19:02:30

One of the things that the original video from the kickstarter campaign spoke about was a function as a unit of code as opposed to a file and the ability to live debug as you built stuff

craftybones19:02:49

I thought both those ideas had a lot of merit and would have liked to see it go some distance

hiredman19:02:59

isn't that just a clojure repl?

craftybones19:02:18

yes and no. The repl by itself is certainly powerful, but the repl that he showcased and seemed to be aiming at was far more powerful and was real time, often highlighting values overlaid on the original code etc

hiredman19:02:28

have you seen rebl?

craftybones19:02:03

I haven’t yet seen/used rebl. Is that what I want? I have the talk bookmarked and am meaning to watch it

hiredman19:02:28

it is a kind of gui data explorer

hiredman19:02:39

with repl hooks

hiredman19:02:34

I dunno, in general gui tools tend to pass me by because I do all my work ssh'ed into a vm specifically setup for work programming

seancorfield19:02:57

The original demo for LightTable was a radically different way to edit/work with functions -- but in testing with actual users it proved too alien/confusing so they scaled back to a much more "normal" editor experience. The main "novel" features it included were inline result display and auto-evaluation of code as you typed (which, frankly, I found a bit too dangerous/annoying and quickly turned that off).

hiredman19:02:12

so lighttable wasn't very compelling, rebl itself isn't super compelling, but it is built on some protocols which are usuable elsewhere which is kind of nice

craftybones19:02:31

Ok. There was this fancy node.js repl that doesn’t seem to be maintained anymore which was quite cool as well

craftybones19:02:39

Can’t remember the name now

seancorfield19:02:44

I think several editors provide the inline results and auto-eval options now and LT is virtually abandoned (I think).

craftybones19:02:21

@seancorfield - A lot of tools have this thing where they are born again later and somehow seem to catch on

seancorfield19:02:48

REBL is "different" because it's in-process and it's very focused on data viewing/visualization/exploration right now. It'll be interesting to see where it goes.

craftybones19:02:27

@borkdude - yeah, unlike fish that die and are never born again

craftybones19:02:43

I’ll definitely try out REBL

seancorfield19:02:49

Liquid is also a very interesting project: again, it's an in-process editor, that's fairly vim-like, but written entirely in Clojure and using curses-style terminal UI (so it can run in an ssh session on a remote box, for example).

hiredman19:02:33

ed for clojure running from the repl

borkdude19:02:52

btw, which is also kinda cool if you work in a terminal: https://github.com/denisidoro/floki just spit your edn to some file and then inspect it. I’m not sure if I’ll use it, since I can inspect things with CIDER pretty well, but it’s cool that this kind of thing can be written in CLJS now

craftybones19:02:05

Just hanging out in these channels is a wealth of information. Thank you @borkdude and @seancorfield

seancorfield19:02:05

https://github.com/mogenslund/liquid -- see the demo videos for insight into the workflow. I really like it (and will like it more after it completes its transition to more vim-like key bindings!).

craftybones19:02:20

@seancorfield - the best of vim and emacs but with Clojure instead of elisp.

craftybones19:02:16

This is a ridiculously cool project. @seancorfield