Fork me on GitHub
#clojure
<
2019-06-04
>
vemv02:06:17

Anyone familiar with getting spurious No such var errors? Might have to do with Clojure 1.10+ error reporting Hard to verify since this project uses 1.10+ fearures, so I can't tentatively downgrade so easily

Alex Miller (Clojure team)02:06:19

doesn't sound familiar to me re 1.10 error reporting

vemv02:06:49

known-good code/project is compiling fine on: 1.10.0 1.10.1-beta2 and failing with [org.clojure/clojure "1.10.1-RC1"], 100% of the times it also is failing occasionally with 1.10.0 / 1.10.1-beta2 with the same error, but I can't find what causes it stacktrace doesn't appear to say much: https://gist.githubusercontent.com/vemv/3fd022882c9b2533fb3d669df8bcfc48/raw/56a711137778b25f6c1710721c2b9b8c4ce016ff/gistfile1.txt project will be open-sourced soon. TODOed to provide a repro

Alex Miller (Clojure team)02:06:59

RC1 is identical to beta2 (other than updating the changelog)

vemv02:06:34

Thanks. Now I have found the true culprit: updating a certain dependency. But still I don't know the cause of the error at all, and the reporting from the compiler side is wrong, that seems for sure.

Alex Miller (Clojure team)02:06:35

this is in the compiler analyzer, when trying to analyze a symbol in a form

Alex Miller (Clojure team)02:06:10

to get here, it's a qualified symbol, the namespace is found (here, sut), but when it looks up the var, there is no such var in the ns

Alex Miller (Clojure team)02:06:49

if it's failing sporadically, you may be fighting a concurrent load issue with the namespace where it is in the process of loading

Alex Miller (Clojure team)02:06:25

I've seen something like this with spec gen where it dynaloads test.check.generators - if you've got two threads potentially loading the same namespace at the same time, you can get spurious errors like this

Alex Miller (Clojure team)02:06:04

why do you think the reporting is wrong?

vemv02:06:49

Thanks for the analysis! FWIW I could find https://github.com/benfb/gorilla-repl/issues/10 which seems similar (:compile-syntax-check + no such var). In my repro I don't use a special Lein, any profiles, etc. It also fails in our CircleCI which is a vanilla one The concurrent loading thing seems possible (although I have 0 dynamic loading) > why do you think the reporting is wrong? First and formost because the var is there, the ns form is correct, etc. It's known-good code, used in various projects. Upgrading an unrelated tiny dependency (which in fact is only used in test code!) triggered the error. And the reported error does not include the name of this dependency at all. That makes me think the reporter is wrong - a correct one would point in the direction of the dependency, rather than suggesting that the main project has an undefined var

Alex Miller (Clojure team)02:06:28

repro? fuller msg? environment?

Nazral07:06:40

is there a way to tell leiningen to have a profile which ignore some files in src and dependencies?

ivana07:06:50

I'm not sure, but :exclusions is not about it?

Nazral07:06:55

I'm not sure but I'll try, thank you!

p4ulcristian10:06:45

Hello guys, anyone familiar with clojure machine-head? Trying to connect to Azure Iot Hub, but getting this error: Execution error (EOFException) at java.io.DataInputStream/readByte (DataInputStream.java:267).

prnc16:06:11

Hi! Say I intentionally throw during testing but don’t want the stacktrace in the test output is that possible somehow (w/ lein test/`clojure.test`)? hope that makes sense?

Olical16:06:41

There's (is (thrown? ArithmeticException (/ 1 0))) and

(is (thrown-with-msg? ArithmeticException #"Divide by zero"
                      (/ 1 0)))
as part of clojure.test. Mentioned here https://clojure.github.io/clojure/clojure.test-api.html

Olical16:06:56

Hope that helps!

prnc16:06:32

ah yes! thanks @olical

πŸ‘ 4
vlaaad20:06:25

Is there a guide or tutorial or some kind of explanation to why we should use seq for non-empty checks? I feel like I'm missing something, because semantics of (not (empty? ...)) are clear, and semantics of seq requires deep knowledge of it's behavior and truthiness semantics... Which I know, but don't understand why it is advertised as preferred approach. I also remember there where some discussions on slack awhile ago where Alex Miller suggested to use nils instead of empty vectors, and just use (when xs ...), and I'm interested to know why is this approach considered better, because without this check you might get surprising behavior when conjing will add to beginning instead of end

πŸ‘ 4
dpsutton20:06:02

I kinda agree. I saw that @ghadi mentioned that empty? might get a counted check in the future in which case (not (empty? will be necessary again. It feels like seq is kinda using the implementation details of empty?

ghadi20:06:17

i don't understand this at all

dpsutton20:06:42

empty? is defined as (not (seq ..)) now so the common convention for "not-empty?" is just seq. My point is that this is true because we know the implementation of empty. In the future, if empty? were to be optimized by checking ICounted the idiom of just using seq would both miss the optimization of empty? and not have an obvious semantic as "not-empty?" any longer

ghadi20:06:41

it would check for counted, then do the existing impl

ghadi20:06:01

what is the optimization of empty? ?

dpsutton20:06:28

presumably a check that if it can be counted its count is 0

ghadi20:06:46

it's not intended as an optimization, but something so that empty? can work on transients

dpsutton20:06:49

that seems to argue in favor of using (not (empty? .. on transients then if the standard idiom seq is not sufficient?

ghadi20:06:04

seq doesn't work on transients

dpsutton20:06:53

sorry had work. But that's why seq as a dual to empty? is a poor fit when empty? is updated to work on transients but its dual would not work, right?

ghadi20:06:58

seq and empty? are not duals

βž• 4
markmarkmark20:06:52

the reason that (seq x) is preferred over (not (empty? x)) is because empty? is implemented as (not (seq x))

markmarkmark20:06:08

so (not (empty? x)) is just (not (not (seq x)))

dpsutton20:06:11

right. which was my point about implementation details leaking out. When empty? gets more complicated looking at ICounted then using implementation details is bad, no?

ghadi20:06:01

implementation details?

vlaaad20:06:06

I don't buy not-not argument, it's implementation detail

πŸ‘ 4
markmarkmark20:06:04

@vlaaad yeah, I don't really either

markmarkmark20:06:36

I just know that's a reason that is often given

hiredman20:06:06

http://clojure-log.n01se.net/date/2009-02-19.html#10:24a has some discussion that rich eventually jumps into

sogaiu22:06:33

thanks for this! haven't heard "druthers" in ages πŸ™‚

Alex Miller (Clojure team)20:06:54

my suggestion to @vlaaad earlier is primarily due to the use case in that context - the stuff he is doing is collection stuff. In that case, lean on nil and polymorphic collection fns and avoid using empty collections at all. If you're in more of a seq context, then lean on seq.

Alex Miller (Clojure team)20:06:36

in general in Clojure avoid making stuff to hold a "place" - if you do that Clojure's design will lead you to code that is simultaneously easier to read, has fewer branches, and is more performant.

Alex Miller (Clojure team)20:06:24

this applies to putting nil values in maps (just omit), making empty collections (conj or assoc into nil instead), etc

Alex Miller (Clojure team)20:06:00

if you are working with collections (not sequences), forcing a seq just to check emptiness is not efficient

Alex Miller (Clojure team)20:06:19

the advice to use (seq x) as a terminating predicate is primarily of use when working with sequences, where you are going to force the sequence anyways to walk it (and sequences cache that so the additive cost is negligible)

Alex Miller (Clojure team)20:06:48

that is all my own internalized "best practice", so not trying to speak for Rich in any way here, fyi

vlaaad20:06:47

I understand what you say about advising to use (seq x) where I am going to force sequence anyway, but can you elaborate more on "avoid making stuff to hold a place"? Is (let [x (blah->possibly-empty-vec blah)] (when-not (empty? x) ...)) somehow a place and (let [x (blah->non-empty-vec-or-nil blah)] (when x ...)) is not?

vlaaad20:06:18

I also worry about conjing into nil: I'd usually prefer to work on vectors, not lists, and conjing into nil produces a list

Alex Miller (Clojure team)20:06:55

ah right, I actually was thinking of (conj) for that one

Alex Miller (Clojure team)20:06:22

of those two examples you give, I definitely prefer the latter

Alex Miller (Clojure team)20:06:11

wouldn't you rather avoid constructing an empty vector (which takes time and consumes memory) and also write (when x ...) vs writing (when-not (empty? x) ...)?

vlaaad20:06:55

but (identical? [] []) returns true, what is taking time here?

ghadi20:06:29

I've seen a lot of code that contorts itself to use empty? because it's perceived as more literate than seq -- but seq is a fundamental function and idiom, not some advanced feature

ghadi20:06:50

identical? does not work on vectors -- it happens that the compiler interns the empty vector

Alex Miller (Clojure team)20:06:58

Clojure rewards you again and again for not making stuff. nils are cool. I like nils. I disagree with Hoare. :)

vlaaad20:06:30

good point

vlaaad20:06:45

I just realized I also prefer blah->non-empty-vec-or-nil πŸ™‚

Alex Miller (Clojure team)20:06:55

if you're going to write stuff like (when x ...) though, you are committing to a philosophy and it requires some amount of discipline to return nil and not []

Alex Miller (Clojure team)20:06:11

but make sure everyone in your team agrees ;)

vlaaad20:06:36

yeah, this approach should apply to all the codes

Alex Miller (Clojure team)20:06:55

same thing on omitting attributes from maps when the value is nil

Alex Miller (Clojure team)20:06:51

if you're taking that approach (again, I'm all in), then you need to be careful to avoid adding those attributes when you create the maps in the first place

Alex Miller (Clojure team)20:06:23

and aware of external libs/systems that might give you maps like that

Alex Miller (Clojure team)20:06:00

generally, I think it's rarely worth stripping stuff out of maps if you get them from someone else

vlaaad20:06:02

@ghadi Agree on seq being fundamental idiom, but empty? is still more literate, and a function to check if coll is empty in clojure core seems pretty fundamental as well, I don't see an argument here in favor of seq

Alex Miller (Clojure team)20:06:37

these are functions that apply in different worlds

Alan Thompson20:06:38

I always prefer an empty collection over a nil. Also, I wrote my own function (not-empty? x)

seancorfield20:06:39

For example, clojure.java.jdbc will give you maps with nil values if the column in the database was null. But in next.jdbc, at least on master, there's next.jdbc.optional/as-maps that will give you maps where nil values are omitted.

Alex Miller (Clojure team)20:06:18

in some sense, jdbc is just surfacing sql here, so it makes total sense

seancorfield20:06:25

(the default as-maps is still nil values from null columns)

Alex Miller (Clojure team)20:06:40

sql nulls are inherently overloaded

vlaaad20:06:40

Yeah, right, different worlds. nil instead of [] for the win

Alex Miller (Clojure team)20:06:31

@vlaaad if you are writing seq-oriented code (using ->>, map, filter, etc), seq makes total sense and I think the language designer has been quite clear that it is the proper idiom for checking whether the seq has at least one element or not

vlaaad20:06:29

I usually use transducers instead of lazy sequences wherever I can for performance reasons

Alex Miller (Clojure team)20:06:50

transducers aren't always faster

Alex Miller (Clojure team)20:06:27

nothing wrong with using lazy seqs where it makes sense

βž• 4
vlaaad20:06:24

hmm, I thought they are, with exception of mapcat fully realizing intermediate steps...

Alex Miller (Clojure team)20:06:43

if you're only going to use 5 values, then it's a lot faster to use lazy seqs

vlaaad21:06:39

it's not?

(criterium.core/quick-bench
  (into []
        (comp
          (map inc)
          (filter odd?))
        [0 1 2 3 4 5 6 7 8 9]))
;; Execution time mean : 329.302467 ns

(criterium.core/quick-bench
  (->> [0 1 2 3 4 5 6 7 8 9]
       (map inc)
       (filter odd?)
       doall))
;; Execution time mean : 893.769899 ns

Alex Miller (Clojure team)21:06:49

but you're forcing the seq realization to compute all values

Alex Miller (Clojure team)21:06:58

my point is, if you don't need all of them, it can be much faster

vlaaad21:06:49

ah, sure, it's faster when I don't need everything

Alex Miller (Clojure team)21:06:39

it also depends how big your colls are and how many transformations you're doing. transducers win more with bigger colls and more transformations. the majority of code like this uses small colls and 1-2 transformations.

Alex Miller (Clojure team)21:06:20

and in those cases, it literally doesn't matter

Alex Miller (Clojure team)21:06:28

so write the version that's easier to read

vlaaad21:06:25

but in example above, with a small coll and 1-2 transformations, transducers are almost 3 times faster, why it doesn't matter?

ghadi21:06:03

micro benchmarks don't matter

ghadi21:06:15

try it with a bigger computation in the xform

vlaaad21:06:46

well, it's a good point that real performance issues are not what you think they are, but isn't it a death by a thousand cuts if through all your application you add extra time here and there on every coll operation?

Alex Miller (Clojure team)21:06:56

"3 times" sounds really different but we're talking nanoseconds

Alex Miller (Clojure team)21:06:45

so you can pay that a million times before it's 1 second different

Alex Miller (Clojure team)21:06:02

so really more like death by 100 million cuts to care

vlaaad21:06:02

hmm you are right, that's true

Alex Miller (Clojure team)21:06:45

could easily be written as a transducer, but the typical size of "merge-deps" is 3 and I much prefer the way this reads

Alex Miller (Clojure team)21:06:43

lazy seqs are not inherently bad, don't be afraid of them :)

vlaaad21:06:37

thank you for explanations!

jumar12:06:00

@vlaaad btw. I've run the same experiment you did (3x each measurement) and got much smaller difference: 570 ns for transducers and 720ns for lazy seqs ` Just another point how misleading microbenchmarks results may be.

πŸ‘ 4
ghadi20:06:44

there may be a misconception that seq and empty? are opposites

vlaaad21:06:39

it's not?

(criterium.core/quick-bench
  (into []
        (comp
          (map inc)
          (filter odd?))
        [0 1 2 3 4 5 6 7 8 9]))
;; Execution time mean : 329.302467 ns

(criterium.core/quick-bench
  (->> [0 1 2 3 4 5 6 7 8 9]
       (map inc)
       (filter odd?)
       doall))
;; Execution time mean : 893.769899 ns

vlaaad21:06:19

maybe I misunderstand you?

Alex Miller (Clojure team)21:06:16

replies in thread

πŸ‘ 4