Fork me on GitHub
#clojure
<
2021-03-18
>
Noah Bogart16:03:41

(defn destr [& {:keys [a b] :as opts}]
  [a b opts])

(destr :a 1)
->[1 nil {:a 1}]

(destr {:a 1 :b 2})
->[1 2 {:a 1 :b 2}]

metal 3
clojure-spin 6
Noah Bogart16:03:46

that's cool as hecck

Alex Miller (Clojure team)17:03:47

fyi, having some problems with the actual release process so not yet available

pez17:03:52

I get this when trying to run tests in a fresh clj-new library project:

$ clojure -M:test:runner
Error building classpath. Manifest type not detected when finding deps for com.cognitect/test-runner in coordinate {:git/url "[email protected]:cognitect-labs/test-runner.git", :sha "209b64504cb3bd3b99ecfec7937b358a879f55c1"}
What am I not getting?

p-himik17:03:49

I have a very faint memory of that error. Is your clj up to date?

Alex Miller (Clojure team)17:03:56

you probably have an empty dir in your gitlibs directory

Alex Miller (Clojure team)17:03:35

at ~/.gitlibs/libs/com.cognitect/test.runner/209b64504cb3bd3b99ecfec7937b358a879f55c1

Alex Miller (Clojure team)17:03:21

there was some bugs in the recent prereleases that would cause this

Alex Miller (Clojure team)17:03:49

you can rm -rf that sha dir and clojure -Sforce -M:test:runner to fix up

pez17:03:11

Ah, yes, I am still running the prerelease, probably. Sorry for the noice!

Alex Miller (Clojure team)17:03:06

in that case, you might want to clean your whole ~/.gitlibs when you update

pez17:03:26

Yes, that did it. Thanks!

seancorfield17:03:30

@U0ETXRFEW Also, if you make a fresh clj-new project, the test runner should be an https URL:

(! 885)-> clojure -X:new :name pez/example
Generating a project called example based on the 'app' template.
(! 886)-> cat example/deps.edn 
{:paths ["src" "resources"]
 :deps {org.clojure/clojure {:mvn/version "1.10.2"}}
 :aliases
 {:run-m {:main-opts ["-m" "pez.example"]}
  :run-x {:ns-default pez.example
          :exec-fn greet
          :exec-args {:name "Clojure"}}
  :test {:extra-paths ["test"]
         :extra-deps {org.clojure/test.check {:mvn/version "1.1.0"}}}
  :runner
  {:extra-deps {com.cognitect/test-runner
                {:git/url ""
                 :sha "b6b3193fcc42659d7e46ecd1884a228993441182"}}
...

seancorfield17:03:05

Maybe you have a rewrite rule in your Git config?

pez18:03:23

Nah, I had just been testing some different variants. 😃

pez17:03:37

I need some ideas on how to test the output from tap> a bit more reliably than I do now. I first thought I was not going to use CI for the project, but now I am doing that anyway and the tests fails intermittently. The docs for tap> tells me that I am probably tapping while there is no room in the queue. The tests all look similar to this:

(testing "Taps the binding box"
    (let [tapped (atom nil)
          save-tap (fn [v] (reset! tapped v))]
      (add-tap save-tap)
      (is (= [:foo :bar]
             (sut/let> [foo :foo
                        bar :bar]
                       [foo bar])))
      (is (= '[[foo :foo]
               [bar :bar]]
             @tapped))
      (remove-tap save-tap)))
Currently I have five of them and when I run the tests from the command line about 1 or 2 fail because I pick up nil from the tapped atom. And sometimes all pass. (The macro let> there taps, maybe I should mention.)

seancorfield17:03:48

tap> returns true if it succeeds, else false so you can check the result of calling it.

seancorfield17:03:55

And I expect there’s some concurrency at play: after you tap> something, it may not run the tap watchers “immediately” @alexmiller?

nbardiuk18:03:03

to solve "eventually" called tap watchers try to replace atom with promise . The assertion should use explicit deref with timeout like (deref tapped 100 :timeout)

pez20:03:21

Actually this probably is in the right direction. I couldn’t get it to work, because I didn’t really know how to do it. But adding a short pause between tapping and untapping makes it work.

#?(:cljs (set! *exec-tap-fn* (fn [f] (f))))

(def tapped (a/chan (a/sliding-buffer 1)))
(def save-tap (fn [v] (a/offer! tapped v)))

(deftest let>
  (testing "Evaluates as `let`"
    (is (= [:foo :bar] (sut/let> [foo :foo
                                  bar :bar]
                                 [foo bar]))))

  (testing "Taps the binding box"
    (add-tap save-tap)
    (is (= [:foo :bar]
           (sut/let> [foo :foo
                      bar :bar]
                     [foo bar])))
    #?(:clj (a/<!! (a/timeout 10)))
    (is (= '[[foo :foo]
             [bar :bar]]
           (a/poll! tapped)))
    (remove-tap save-tap))
...

pez20:03:16

(Probably would have worked with the atom as well, but I had already switched to channels and it does make for less boilerplate so I let it be like that.)

pez18:03:29

@seancorfield I don’t know how to test the return value from the tap>. It happens inside the macro I am testing…

3
pez18:03:04

@nbardiuk it seems it is not eventually called. Rather it is called and then eventually that succeeds. Also needs to work with CLJS, even if I guess I could use a reader conditional for that.

borkdude19:03:06

Continuing the thread from #news-and-articles, with clojure 1.11.0 I get:

$ clj -A:clojure-1.11.0
Clojure 1.11.0-alpha1
user=> (let [{:keys [] :as m} {:a 1}] m)
{:a 1}
user=> (let [{:keys [] :as m} [{:a 1}]] m)
[{:a 1}]
user=> (let [{:keys [] :as m} (list {:a 1})] m)
{:a 1}
user=> (let [{:keys [] :as m} (list {:a 1} {:b 2})] m)
{{:a 1} {:b 2}}
Why is there a difference between the second and third example? Because a seq with a map in it, destructured as a map, destructures the first map in the seq? :thinking_face:

seancorfield19:03:27

1, 2, and 4 are the same on 1.10.3. 3 is not legal on 1.10.3:

Clojure 1.10.3
user=> (let [{:keys [] :as m} {:a 1}] m)
{:a 1}
user=> (let [{:keys [] :as m} [{:a 1}]] m)
[{:a 1}]
user=> (let [{:keys [] :as m} (list {:a 1})] m)
Execution error (IllegalArgumentException) at user/eval143 (REPL:1).
No value supplied for key: {:a 1}
user=> (let [{:keys [] :as m} (list {:a 1} {:b 2})] m)
{{:a 1} {:b 2}}

seancorfield19:03:58

So there already was a difference between 2 and 3.

seancorfield19:03:31

@nilern FYI since you thought it was a bug.

nilern19:03:14

Okay more like it is an opportunity to create bugs by writing a map pattern and giving it a seq where one of those was a mistake and then it just works but ignores the rest of the seq

didibus07:03:14

Ok, so this is logical actually I believe. Vectors are associatives as well. So if you say destructure associative as m and use m you get back the associative as-is. But sequence are not associative, and when destructured as an associative they are coerced into one. So #3 used to throw because it doesn't have an even number of key/value pairs for the coercion. While in #4 it does so you get a map where the first element in the seq is the key and the second is the value. And with 1.11, sequence of a single or trailing map destructure as an associative of that map. So now in #3 you get single map back.

didibus07:03:13

Though I'm curious, what happens in 1.11 for:

(let [{a :a}
      (list :a 1 :b {:a 2})]
  a)

seancorfield16:03:41

(! 893)-> clj -Sdeps '{:deps {org.clojure/clojure {:mvn/version "1.11.0-alpha1"}}}'
Clojure 1.11.0-alpha1
user=> (let [{a :a}
      (list :a 1 :b {:a 2})]
  a)
1

borkdude19:03:17

The third example was previously not working and now works, but isn't intuitive. Maybe should be seen as a side-effect of supporting maps as kwargs?

Alex Miller (Clojure team)19:03:37

can you please move questions like this to ask.clojure ?

nilern19:03:20

Side-effects are bad, looks like a bug to me...

borkdude19:03:03

all Clojure-related questions should now be posted to ask.clojure instead of here?

Alex Miller (Clojure team)19:03:28

concrete questions that might be bugs can more usefully be asked there where they can be tracked

Alex Miller (Clojure team)19:03:54

conversational questions are probably better asked here

borkdude19:03:06

This was intended as the latter, just exploring the new patch

borkdude19:03:17

The example came more or less directly from the tests

fogus (Clojure Team)19:03:42

The tests are there to check certain behaviors do what they're supposed to do. In practice no one directly destructures a seq with an associative form in a let.

didibus06:03:27

Does this no longer works in 1.11?

(let [{:keys [a b]}
      (list :a 1 :b 2)]
 [a b])
;;> [1 2]

borkdude19:03:17

Another question about the patch: There was a core function introduced, but never used but anywhere in the tests. However, in destructure this same pattern as the core function was used. Wasn't the new core function used because of performance (inlining), but still exposed for people who want to implement their own kind of destructuring?

fogus (Clojure Team)19:03:04

I think the function is in the tests no? But yes, your instinct about why it was exposed.

borkdude19:03:05

Understood. The test just confused me.

user=> (let [[& {:keys [] :as m}] (list {:a 1})] m)
{:a 1}
would have been more intuitive to me.

borkdude19:03:48

yes, it was never used "anywhere but in the tests" (placed the word /but/ in the wrong place)

borkdude19:03:19

Thanks for clearing that up.

fogus (Clojure Team)19:03:47

Also, it wasn't used in destructure not because of the speed of inlining but because emitting a call to a function that only exists in 1.11 isn't backwards compatible.

borkdude19:03:49

In fact, I am implementing my own destructure, so I will jump on that new core fn now :)

borkdude19:03:58

Good point!

nilern19:03:42

I like it when tools prevent me from shooting myself in the foot but I guess Clojure has always been more in the "power tools" category

fogus (Clojure Team)19:03:36

What errors are you hoping that Clojure helps you with in this case?

emccue20:03:39

Has anyone been able to make lein ring serve over http/2? Specifically I have a part of my app that is loading a bunch of static files at once and the dev time performance is abysmal

seancorfield20:03:47

He is concerned that code which used to blow up if you made a mistake now does something, so if you accidentally try to destructure a sequence against a map you get an (unexpected) result now whereas you used to get an exception. Is that accurate @nilern?

Alex Miller (Clojure team)20:03:28

this is a tradeoff we have repeatedly considered and made in the design of clojure, in favor of giving users more power (see transducer arities for another example)

☝️ 3
borkdude20:03:11

@fogus FYI, I backported the new core fn to 1.10 and it seems to work even without the new Java stuff that got added in createAsIfByAssoc. Using it in sci's destructure:

$ ./bb -e '(defn foo [& {:keys [:a :b] :as m}] [a b m]) (foo {:a 1 :b 2})'
[1 2 {:a 1, :b 2}]

🎉 24
borkdude20:03:24

Ah I see. Only the ((partial addn 100 :a 1) {:b 2}) tests still fail on Clojure 1.10.x with this backport. Will be automatically fixed for whoever uses sci with Clojure 1.11.1.

nilern20:03:09

I would have expected the change to only affect rest params. But now that I think of it that would be inconsistent, which I don't like either 🤷

borkdude20:03:08

Arguably transducers could have lived in separate similarly named functions without losing any power, unless I'm missing something (which is very likely). E.g. mapping, filtering, etc. But that ship has long sailed.

imre20:03:52

On that note, could you imagine a linter that warned on usages of the lazy arities of those core fns? I find myself looking for those in PRs

borkdude20:03:34

you mean, force users to use transducers?

borkdude20:03:45

that doesn't seem a good idea?

nilern20:03:37

Maybe (->> stuff (map inc) (filter even?) (into #{})) type of thing which I too optimize quite often

imre20:03:39

Not forcing, just nudging. But at least in our case laziness is rarely if ever justified, and the overhead is a lot less using transducers

kingcode16:03:48

As a side note, I too enjoy the ’ing qualifier/naming convention for transducers - so expressive and meaningful.

Craig Brozefsky20:03:20

been away from JSON parsing a bit, but is cheshire still the goto?

nilern20:03:53

https://github.com/metosin/jsonista is faster and the API is less crufty

nilern20:03:42

But Cheshire seems still more common and I am actually working on a PR to it as we speak

Craig Brozefsky20:03:28

Much appreciated! thanks.

Alex Miller (Clojure team)20:03:52

data.json is rapidly getting faster thx to @U04V5VAUN ...

slipset21:03:01

Depending on the use-case, (for rest, payloads around 1k with shallow maps) data.json is at the same speed as at least Cheshire now. When payloads get bigger, data.json suffer a bit, but not all that much. I haven’t worked that much on writes yet, but I’ve gotten some fairly nice speedups there as well. Regarding crufty api, was that for cheshire or data.json, and if data.json, what do you find crufty about the api? I’m curious since I’m working in that area now (not saying the api will change, just interested).

borkdude21:03:35

Meanwhile @nilern’s PR to speed up encoding of cheshire with 15% was merged :)

borkdude21:03:51

by replacing doseq with reduce, haha

borkdude21:03:36

On the topic of JSON/jackson, don't the same arguments of trying to avoid jackson apply to transit-java?

Alex Miller (Clojure team)21:03:35

for data.json, having no deps is a primary goal. for transit, performance is a higher priority (but I'd love for it to lose it's deps too)

borkdude21:03:50

Or are there any plans to move transit to data.json eventually?

Alex Miller (Clojure team)21:03:12

well note that transit-java is java and having it depend on data.json (clojure) is maybe weird

slipset21:03:29

For the more significant speedups, all of these libs spend a significant amount of time calling assoc! , so I got some code where I wrap a java map so it looks like a persistent map. I’m currently looking into wrapping java lists so it looks like a persistent vector, but couldn’t find the exact interfaces I needed to implement.

Alex Miller (Clojure team)21:03:02

I'm not convinced any of that is a good idea :)

slipset21:03:04

(but for reference, for payloads above 10b data.json (and cheshire) seem to be spending about 60% of their time calling assoc!)

borkdude21:03:07

true. Recently jackson was bumped for cheshire related to a security thing. Would transit-java also be open to such a bump if there was a security risk (or just for keeping up with the ecosystem)?

Alex Miller (Clojure team)21:03:00

I think the issue with transit-java is that it is on a pretty old version of jackson and there have been some (many?) breaking changes in the intervening time so it's a non-trivial change

Alex Miller (Clojure team)21:03:15

if I remember correctly from the last time I looked at it

nilern21:03:17

I meant Cheshire, which has parse-string parse-string-strict, parse-stream, parse-stream-strict etc. and cannot parse InputStream even though Jackson can... while Jsonista just has read-value with protocol dispatch.

👍 11
Alex Miller (Clojure team)21:03:27

but putting this in a transit-java issue is the best place for it (there may be one)

borkdude21:03:15

@alexmiller FYI I am running cheshire and transit within the same project(s) and never saw an issue so far

Alex Miller (Clojure team)21:03:28

well, that's good to know, put all that in a transit-java issue :)

slipset21:03:50

We have exclusions on jackson in our project-clj (for whatever reason)

borkdude21:03:31

Ah, the Jackson CVE is only relevant to cbor which isn't used in transit-java, so I would not have a good case to bump it in that lib

borkdude21:03:56

But according to their docs, newer minor versions should not break libs using older ones: https://github.com/FasterXML/jackson/wiki/Jackson-Releases#general

borkdude21:03:09

(with a caveat about deprecated methods)

seancorfield21:03:30

@alexmiller FWIW, our entire test suite passes at work on Clojure 1.11.0 Alpha 1 (on a combination of OpenJDK 8 and OpenJDK 15 for various apps).

👍 28
clojure-spin 18
seancorfield18:03:43

And we have 1.11.0 Alpha 1 in production now!

👏 6
seancorfield19:03:25

We also just merged a bunch of changes on dev to actually take advantage of the new feature by simplifying a lot of apply / apply concat / (into [] cat ..) style code that was dealing with named arguments being passed around.

didibus20:03:03

You're not a bit worried that this release is meant for "early feedback" on the feature which could mean breaking changes?

seancorfield20:03:22

We’ve run alpha builds of Clojure in production since 2011. Sometimes we’ve had to revert a few early changes, but it’s very rare.

👌 3
🤯 3
💪 3
seancorfield21:03:32

I expect we’ll take it to production next week…