Fork me on GitHub
#clojure
<
2022-03-03
>
mars0i04:03:41

I understand that Clojure functions can return sequences of different classes in different contexts for the sake of efficiency, but I thought that the semantics would be preserved. But that's not true for range in that it can return either a class on which realized? is defined, or a class on which realized? is not defined:

mars0i04:03:49

expt.core=> (class (range))
clojure.lang.Iterate
expt.core=> (realized? (range))
true
expt.core=> (class (range 100000))
clojure.lang.LongRange
expt.core=> (realized? (range 100000))
Execution error (ClassCastException) at expt.core/eval2272 (form-init18423738793199381558.clj:1).
class clojure.lang.LongRange cannot be cast to class clojure.lang.IPending (clojure.lang.LongRange and clojure.lang.IPending are in unnamed module of loader 'app')

mars0i04:03:50

i.e. (realized? (range)) works, but (realized? (range 100000)) . Yes the second range call is lazy (or would be if the argument were larger). This seems like a bug. Do I have a misunderstanding of Clojure's general philosophy of sequences? Or is realized? not really intended for lazy sequences, but only for e.g. promises, and it just happens to work with some lazy sequences?

hiredman04:03:37

realized? is best pretended to not exist

Alex Miller (Clojure team)04:03:44

I don't think realized? is generally very useful

Alex Miller (Clojure team)04:03:21

We talked about what should happen in this spot when I reimplemented range in 1.7 and the long range is somewhat tricky because it kind of always realized

Alex Miller (Clojure team)04:03:39

I guess the question is why you care about realized?

mars0i04:03:17

Thanks Alex. I use it mostly for debugging to figure out what's going on when I have problems that I think might have to do with laziness.

hiredman04:03:44

Using realized? is going to lead to more problems

mars0i04:03:24

I thought I remembered it being more general in the distant past. I think I started with 1.5.

mars0i04:03:57

@hireman you mean that I might just be confused by what realized? tells me?

hiredman04:03:26

Like, the thing with seqs is there are many kinds of seqs, seq is an interface that doesn't require IPending

hiredman04:03:54

So you'll get custom seq implementations that are handled

hiredman04:03:13

And seqs can change types as you traverse them

mars0i04:03:21

Right. I accept that for many purposes I shouldn't care about what kind of sequence I have.

hiredman04:03:44

And lazy seqs are best viewed as "not strict" instead of lazy

mars0i04:03:47

I have something that's behaving differently depending on how long I make it, and I think it might have to do with having lazy sequences embedded in a lazy sequence, which has gotten me into trouble in the past. It's good to be able to figure out when something does or doesn't become realized.

hiredman04:03:49

Chunking, concating, and other things effect how much and when things are evaluted

mars0i04:03:55

Huh. What do you mean by "not strict"?

hiredman04:03:18

So sometimes a lazy-seq will evaluate more than is asked for

mars0i04:03:52

Yes, right about chunking. I think that might be where my problem is coming in. I understand that chunking can cause more to be realized than I need immediately.

mars0i04:03:37

OK, well, I guess I will just stay away from realized? But it's frustrating not to be able to know, for the sake of debugging.

hiredman04:03:40

I think not strict is actually not what I am thinking of but the term for it escapes me, but I dunno, sort of lazy sometimes, but not guaranteed to not compute more than ask for

mars0i04:03:32

Right, I know what you mean. So if it was let's say "strictly lazy", it would never compute more than you asked for. But in fact it might do that.

mars0i04:03:40

I think it would be helpful to find a good discussion of gotchas with lazy sequences of lazy sequences. I recall that that was one of the ways that concat can get you into trouble. I'm not using concat, but maybe the principle is similar. I know I've seen problems with lazy seqs of lazy seqs, but that was very early when I was learning Clojure, so I just started wrapping everything with doall.

wontheone108:03:58

https://ask.clojure.org/index.php/10573/why-does-partition-by-iterates-over-elements-more-than-once Please, could anyone tell if this should be considered as a Clojure bug? (partition-by)

Ferdinand Beyer09:03:04

I don’t think this is a bug. In general, Clojure prefers pure functions without side effects, and many places will call functions more than once. This could be documented better, though.

wontheone109:03:55

I agree that then it should document f should be free of side-effect. But when you can have versions that doesn't call f twice, at least wouldn't you agree then the version that doesn't call f twice is better?

Ferdinand Beyer10:03:20

I’m not sure. For partition, I would expect f to be quite simple: Given an item, return some key. This should be a pure function. The current implementation works like this: 1. Apply f on the first item 2. Take items from the collection where (f item) is the same as the value from (1) This will apply f twice to the first item of a partition, as you found out. There is a trade-off: Making this implementation more complex vs prevent f from being called twice. I’d go for a simpler implementation, unless we have compelling reasons not to call f twice

Ferdinand Beyer10:03:21

Totally agree that this should be documented, as other places explicitly mention that (e.g. swap!). As a tip: In general Clojure prefers pure functions, so you should always be careful when using non-pure ones.

wontheone110:03:15

Yeah for our use cases, using a stateful transducer made sense so we couldn't use partition-by but had to write hand-written alternative. (we were batching multiple collections so that they are not added to batch once batch-size is exceeded and was checking the quotient to partition them. So we had to swap! atom to keep track of running count of current batch). So in this case partition-by wasn't usable to us (because it counted items twice). Just to make a case of why it might be useful to call f only once.

mars0i15:03:37

Re my discussion a few posts above with @alexmiller and @hiredman, I'm slightly embarrassed to see that I discussed the same the behavior of realized? and range with Alex and others in the Google group several years ago: https://groups.google.com/g/clojure/c/NFwHkZxUFuY Forgot all about that.

andy.fingerhut16:03:04

It can happen to the best of us 🙂

andy.fingerhut16:03:38

I'm pretty sure I've filed nearly the same issue against Clojure core more than once, several years apart.

😀 1
mars0i16:03:49

Sometimes I go to StackOverflow and find a perfect answer to my question, and then realize that I wrote it. 😀

2
andy.fingerhut18:03:46

"Man, this person is RIGHT ON in their answer. Who is this?" 🙂

😆 1
fabrao15:03:49

Hello, is with-redefs works with defrecord?

Alex Miller (Clojure team)15:03:31

what do you mean exactly? I suspect the answer is probably no

fabrao15:03:57

I have a http/request inside and I'm trying to use with-redefs to mock it

fabrao15:03:40

(defrecord HttpClient [http-middlewares config]
  http.api/IHttpClient
  (request [_this req]
    (http/with-middleware
      (concat http/default-middleware http-middlewares (:middlewares req))
      (http/request (merge config req)))))

fabrao15:03:07

(defprotocol IHttpClient
  (request [this req]))

ghadi15:03:10

if you already have an abstraction, pass in a different implementation of that abstraction

fabrao15:03:11

So, if I cound't use, is there any alternative solution for it?

ghadi15:03:20

your abstraction is the protocol

ghadi15:03:31

with-redefs sucks in various ways, you should avoid it

ghadi15:03:55

you already did the Right Thing by making an abstraction, and an instance of the abstraction

☝️ 1
fabrao15:03:35

I understand, I'll try to use in another way, thank you

ghadi15:03:00

make another defrecord that implements your abstraction in the same way that you would have redef-ed it

fabrao16:03:02

(defrecord MockClient [http-middlewares config]
  http.api/IHttpClient
  (request [_this req]
    (http/with-middleware
      (concat http/default-middleware http-middlewares (:middlewares req))
      (merge config req))))

fabrao16:03:46

I only need to see the result of merging config with request, I think it solve my problem, thank you

p-himik16:03:04

@U050ECB92 Is there a comprehensive read on why with-redefs sucks?

ghadi16:03:56

comprehensive no, but for me it's that it is a signal that you're reaching into implementation details, rather than passing a parameter

p-himik16:03:22

Makes sense, thanks.

wevrem18:03:38

This last round of AoC I used with-redefs a lot to switch between testing samples, and testing final input. It was very handy. But I suppose that falls under the “useful for testing” category and not sure I would ever use it in production.

dpsutton18:03:33

input should be a function input. That to me feels like a very poor use of with-redefs.

1
dpsutton19:03:20

x is defed to be 3, with redefs runs at the same time and it restores the wrong “original” binding. It ends up “restored” to 4 and is the wrong value under a redef.

Sam Ritchie16:03:14

It seems that user.clj does not respect :refer-clojure forms:

(ns user
  (:refer-clojure :exclude [+])
  (:require [sicmutils.env :refer [+]]))
Is that a known bug?
$ clj
WARNING: + already refers to: #'clojure.core/+ in namespace: user, being replaced by: #'sicmutils.env/+
Clojure 1.10.3
user=>

hiredman17:03:25

ns is additive only and the user namespace is already created and has all of clojure referred into before loading user.clj

👍 1
GGfpc17:03:12

What is the current go-to testing library for clojure including stubbing. Is clojure-test enough?

respatialized17:03:27

Many things that exist in other programming languages as classes and objects are just data structures in Clojure, eliminating a lot of the need for stubs. While mocking and stubbing libraries exist for Clojure, it might be helpful if you described what you're trying to test and why stubbing is necessary for it so people can point you in the right direction.

💯 1
Joshua Suskalo17:03:52

Easy way to stub things so that they throw exceptions when called is to just use declare, but do you mean to mock existing functions to have alternate behaviors? I've found that clojure.test is enough for all my usecases, but I know some others prefer some other libraries.

borkdude19:03:25

@ggfpc12495 with-redefs is useful for stubbing, but even that I almost never use

GGfpc19:03:28

For example, I might want to stub out publishing an event to kafka, or a DB access

Vipin21:03:10

We use Bond to mock functions in our tests. https://github.com/circleci/bond

hiredman19:03:25

make the publishing to kafka a function, pass the function in, in tests pass in a function that does something else

💯 2
hiredman20:03:23

a variant of this I use a lot is to build systems using component, and in tests swap out certain components for testing versions (replace a redis pubsub component with a core.async Pub, etc)

hiredman20:03:45

but the function pass works great too

dvingo19:03:07

Nathan Marz describes some interesting use cases for with-redefs https://tech.redplanetlabs.com/#redefs

Ben Sless20:03:17

Isn't it obviated by tap> ?

dvingo20:03:10

seems like an alternative-to instead of an obviation

p-himik20:03:14

In a way, if they tap> with a special argument and add their own tap that counts how many times tap> was called with that argument.

isak20:03:20

Hmm, is this part of the blog also true about tap> ? > Best of all, since this à la carte event log approach is based on no-op functions, it adds basically no overhead when the code runs in production.

p-himik20:03:13

With tap>, there will be an overhead of putting one value on top of a queue.

1
henryw37410:03:25

with-redefs needs to be used with caution in my experience: https://widdindustries.com/with-redefs-problems/ Also, as an alternative approach to the no-op fns in tests, I change the log appender so all logs go to a list held in an atom. the logs are maps rather than strings, so easy to assert on

Joshua Suskalo20:03:11

Yeah so the publish to kafka as a function is fine, but this is also what Component and similar dependency injection libraries are designed to help with as well, although some of them have better support for testing than others.

p-himik20:03:53

All roads lead to IoC. Perhaps, it's another corollary to the Greenspun's tenth rule.

hiredman20:03:51

context aware computation needs to be parameterized on context

☝️ 3
hiredman20:03:06

you have a program that mostly does A(when deployed) but sometimes you want to do B(when testing)

hiredman20:03:43

the most straight line down the middle way to do that is a parameter that tells the program which one you want

Joshua Suskalo22:03:06

Languages like https://koka-lang.github.io/koka/doc/index.html have been taking more seriously the idea of having low level effects declared and bound through dynamic extent rather than as explicit arguments, and older paradigms like condition systems (with proper restart support) can be used to implement this, as well as just dynamic vars with functions. What do you think of those sorts of systems when compared to explicit arguments, component architectures, and other options?

ghadi22:03:37

High tech solution to a low tech problem

ghadi22:03:25

(I realize effect systems are powerful)