Fork me on GitHub
#clojure
<
2021-08-13
>
Jim Newton08:08:04

can someone help me understand how this works

(defn seq1 [#^clojure.lang.ISeq s]
  (reify clojure.lang.ISeq
    (first [_] (.first s))
    (more [_] (seq1 (.more s)))
    (next [_] (let [sn (.next s)] (and sn (seq1 sn))))
    (seq [_] (let [ss (.seq s)] (and ss (seq1 ss))))
    (count [_] (.count s))
    (cons [_ o] (.cons s o))
    (empty [_] (.empty s))
    (equiv [_ o] (.equiv s o))))
the code is taken from http://blog.fogus.me/2010/01/22/de-chunkifying-sequences-in-clojure/comment-page-1/?unapproved=1391028&amp;moderation-hash=034bf1cd8870d627d36fd727b298df79#comment-1391028. Which functions can I put in this list? Can I put map, filter, and mapcat, and thus allow my object to work with for comprehensions?

delaguardo08:08:42

look at this definition first - https://github.com/clojure/clojure/blob/master/src/jvm/clojure/lang/ISeq.java this interface is extending https://github.com/clojure/clojure/blob/a29f9b911b569b0a4890f320ec8f946329bbe0fd/src/jvm/clojure/lang/IPersistentCollection.java#L14 which is also an extension of https://github.com/clojure/clojure/blob/a29f9b911b569b0a4890f320ec8f946329bbe0fd/src/jvm/clojure/lang/Seqable.java#L15 so to implement a custom ISeq you need to provide implementation of methods defined for those interfaces only. map , filter, mapcat are just functions

Jim Newton08:08:05

ahh and map, filter, and mapcat are not methods on any of those interfaces, so I can't influence the for comprehension for my object?

delaguardo08:08:53

influence how?

delaguardo08:08:30

(defn seq1 [#^clojure.lang.ISeq s]
    (reify clojure.lang.ISeq
      (first [_] (.first s))
      (more [_] (seq1 (.more s)))
      ; (next [_] (let [sn (.next s)] (and sn (seq1 sn))))
      (seq [_] (let [ss (.seq s)] (and ss (seq1 ss))))
      ; (count [_] (.count s))
      ; (cons [_ o] (.cons s o))
      ; (empty [_] (.empty s))
      ; (equiv [_ o] (.equiv s o))
      ))

  (for [x (seq1 '(1 2 3))]
    x)
after some experiments to influence for behavior you have to implement first, more and seq methods only

Jim Newton08:08:23

what I mean by influence is that I can write my-map which when given a particular type of object, returns a particular type, in particular returns the same type. However, I can't force clojure.core/map to return this particular type. And I can't influence for to expand to a call to my-map.

delaguardo08:08:40

where first, more and seq are methods, not functions

Jim Newton08:08:13

As I understand (correct me if I'm wrong) map always returns a chunked lazy sequence. if I give it an unchunked lazy sequence which I have created, it will still returned a chunked lazy sequence. right?

delaguardo08:08:52

(defn re-chunk [n xs]
    (lazy-seq
      (when-let [s (seq (take n xs))]
        (let [cb (chunk-buffer n)]
          (doseq [x s] (chunk-append cb x))
          (chunk-cons (chunk cb) (re-chunk n (drop n xs)))))))

  (first (map #(doto % (prn "!"))
           (map #(doto % (prn "@"))
             (re-chunk 1 (range 100)))))
quick way to check

Jim Newton08:08:39

I really don't understand what is happening. I was trying to implement a 1-chunked lazy sequence using things I understand. Your example seems to work for map. but not for mapcat.

(first (mapcat #(doto [%] (prn "!"))
           (mapcat #(doto [%] (prn "@"))
             (re-chunk 1 (range 100)))))

Jim Newton08:08:58

returns 0 but prints

[0] "@"
[1] "@"
[2] "@"
[3] "@"
[0] "!"
[1] "!"
[2] "!"
[4] "@"
[3] "!"

Jim Newton08:08:25

So what I was doing, was to implement filter, map, mapcat, and concat in a simple enough way to enforce the type. https://gitlab.lrde.epita.fr/jnewton/clojure-rte/-/blob/6fc3b41204a109bcde0ead0e0a69551e1bc47faa/src/clojure_rte/lazy.clj

Jim Newton09:08:12

whereas my implementation of mapcat does not seem to have a hidden chunkiness

clojure-rte.rte-core> (first (lazy/mapcat #(doto [%] (prn "!"))
                                          (lazy/mapcat #(doto [%] (prn "@"))
                                                       (range 100))))
[0] "@"
[0] "!"
0
clojure-rte.rte-core> 

Jim Newton09:08:56

whoever mine fails sometimes. still debugging.

clojure-rte.rte-core> (first (lazy/map (fn [x] (prn [:outer x]) x)
                                       (lazy/map (fn [x] (prn [:inner x]) x)
                                                 '(1 2 3))))
[:inner 1]
[:outer 1]
[:inner 2]
1
the inner function is called twice. Not yet sure why.

delaguardo10:08:29

out of curiosity - why so complex?

(ns clojure-rte.lazy ...)

(alias 'lazy 'clojure-rte.lazy)

(def lazy/concat ...)
even if you have alias defined you still can define all the functions using (defn concat …

delaguardo10:08:52

and back to the topic - the problem with mapcat is that it internally uses concat which itself is checking input sequences to be chunked-seq? and if it is not chunked (effectively not implementing this interface - https://github.com/clojure/clojure/blob/b1b88dd25373a86e41310a525a21b497799dbbf2/src/jvm/clojure/lang/IChunkedSeq.java) then the sequences will be treated as normal sequence.

Jim Newton10:08:55

that's just a convention I have started using for myself. whenever I define a function whose name is likely to conflict accidentally with another function either in clojure.core or in another one of my own namespaces, I completely avoid using the unqualified name. the danger is that if I use something like grep the output won't be clear which function name is in the output.

Jim Newton10:08:46

of course there are many advantages to mapcat , concat, filter, and map being normal functions. However, this is one disadvantage. If they were methods then when mapcat internally used concat, it would dispatch to the appropriate user defined concat.

delaguardo10:08:22

but defn requires the name to be simple symbol. and I think that’s why you have to have (def lazy/concat (fn ... so other tools like clj-kondo might not identify such symbols as functions anymore.

Jim Newton10:08:50

BTW, I'm not even sure if my final experiment will work. I'm trying to see whether in my particular application 1-chunking is advantageous.

Jim Newton10:08:10

yes indeed. defn requires a simple symbol (I think that is a bug in the spec-based syntax check for defn ), while def allows the prefix. it is annoying that the defn check is overzealous.

Jim Newton10:08:47

I would not be surprised if certain tools get confused. But clj-kondo (the one I'm using) is happy with it.

delaguardo10:08:42

I think main intention was to disallow defining function for some another namespace

delaguardo10:08:16

I remember Alex talking about this in some thread.

delaguardo10:08:05

and giving that allowing qualified symbols for def looks like a bug to me

Jim Newton10:08:38

if that was the intent, then indeed the check is overzealous. it also disallows using the current namespace as prefix.

delaguardo11:08:06

not sure if this is bad thou

Jim Newton11:08:48

it seems strange to me (my own personal opinion, others may freely disagree) that I use a name like gns/foo everywhere in my application except in the file where it is defined, where I must use foo . it is a potential source of errors if I have multiple foos defined and I copy code around during refactoring.

Jim Newton11:08:33

of course 99.99% of the time I don't have multiple identifiers of the same name. but it does happen from time to time.

delaguardo11:08:01

true, but this depends on you personal choice of picking alias name

delaguardo11:08:45

it is completely fine to have different aliases for the same ns in different other namespaces

Jim Newton11:08:10

I recently defined two slightly different extensions of Boolean algebra in an application. So I had 3 different ands. the one from clojure.core and then the two from my application. at the beginning I had a hard bug to find. I was afraid I had misused and somewhere. Turned out the bug was something else, but in the end I renamed all ands to include prefixes during my debugging.

Jim Newton11:08:31

two different aliases for the same namespace? I hadn't considered that. but yes you are right.

Jim Newton11:08:11

I've probably done that in some of my test cases come to think of it. Because I don't always force myself to be consistent in test cases files.

didibus18:08:30

I think the reason it doesn't work for mapcat is because it uses varargs

didibus01:08:21

Because mapcat will use apply and apply always realizes the first 4 elements, nothing to do with chunking.

didibus02:08:53

This is the other problem that not all sequence functions are implemented in a way that is maximally lazy, iregardless of chunking.

noisesmith16:08:28

which is why the "canonical" answer is always: if your code correctness / suitability is changed by chunking vs. not chunking, you shouldn't be using lazy seqs for that code

didibus18:08:57

Well, you shouldn't be using any sequence function at all, and I think that includes apply or anything using apply since it'll call seq on the args. And honestly I think that's a wart of Clojure, like it's unsatisfactory to think you have all this lazy machinery, but can't actually use it to implement lazy behavior.

noisesmith18:08:25

but that isn't a problem until you care about lazy realization, without laziness it's just putting four fully realized objects on the stack which is nearly free

noisesmith18:08:46

the problem isn't apply / seq/ etc/ which cannot be avoided, it's the laziness

didibus18:08:10

The problem is Clojure's lazy implementation not being truly lazy.

didibus18:08:26

But also being pervasive enough that it's hard to avoid.

noisesmith18:08:33

it's plenty lazy, it just isn't an execution timing control mechanism

noisesmith18:08:58

you avoid it by not using lazy functions to create expensive calculations or side effecting operations

didibus18:08:42

The only reason to use lazyness is to avoid expensive calculations though.

didibus18:08:25

Well.. okay I guess there is also to deal with sets larger that fits memory.

noisesmith18:08:19

right, but there's a boundary point, where if the difference between realizing 32 items and 1 item breaks your program, using laziness is probably a waste of your time (no matter how many hacks might be able to make it work...)

didibus18:08:18

It doesn't break his program, it just slows it down. I don't think anything would prevent Clojure to take better care in making sure that chunk-size can be controlled and that all sequence functions take great care in being maximally lazy. Beyond that there's not been the time and effort by the core team to do so.

noisesmith18:08:54

> it's unsatisfactory to think you have all this lazy machinery, but can't actually use it to implement lazy behavior we have LazySeq (and the various sequential functions that generate it), delay, promise, and even fn - all of them are ways of defering evaluation to a later time, none of them give you a lazy language a-la haskell

noisesmith18:08:05

and even haskell is full of gotchas around lazy behavior

didibus18:08:03

It's already almost there, most sequence functions are implemented so if not given a chunked-seq they won't chunk their return either. And that if given a chunked seq of n size chunk, they'll return similar sized chunks. It's just that maybe there's some places where it's not done or done properly to be depended on. And then you have the issue of a few other functions that just are greedier then they could be.

noisesmith18:08:26

I suspect that if the language maintainers made any promises about controlling laziness in a granular way they'd be painting themselves into an error prone and brittle corner

noisesmith18:08:49

that implicit chunking is speeding up a lot of real world code currently

didibus18:08:48

Well, that's possible. I'm not saying that guaranteeing a correct control over the level of lazyness wouldn't come with a maintainance burden. I'm sure there'd be some. They'd promise to fix every function that doesn't respect chunk-size and to re-implement everything that uses lazy-seqs in a not maximally lazy way into a lazier variant. And every time they add something new, they'd need to make sure to take that into account for it as well. But if say they had the man-power and will to do it, it would be really nice as a user.

didibus18:08:49

In the meantime, I think either using transducers if possible, or using delay with sequence functions is probably the simplest way for a user to have more control.

didibus18:08:32

What's interesting though is Fogus's post hints that Rich might have wanted to do so: > Bear in mind, that the code for seq1 is in no way official and should not be used for production code. Clojure will one day provide an official version of the function above, but for now I simply took a rough sketch posted by Rich Hickey and made it work with the β€œmaster” branch as an exercise and to hopefully gain more insight into the whole matter of chunkiness in general. Hopefully, it can serve the same purposes for you. > Maybe never got around to it.

didibus19:08:38

I also think to fix apply, maybe you'd need to add a new function to sequence, something like peek, which can see if there's something without realizing it. Otherwise not sure how it could figure out the correct arity.

noisesmith19:08:35

lazy-seq is literally chaining arbitrary method invocations under the hood, asking "is there a next item" is turing complete

noisesmith19:08:50

the only reliable way to get an answer is to realize an item

didibus19:08:39

It depends, I think you could provide a test for a given seq. Say I have a lazy-seqs getting data from a remote queue, you could provide a test function that checks with the remote system if the queue has info. Say you have a lazy-seqs but know the count of elements, you could provide a test that checks if we're at the end. Etc.

noisesmith19:08:56

why not use a data type that's designed for answering these kinds of questions in the first place?

didibus19:08:24

Like what?

noisesmith19:08:28

also, pretending the state of a remote machine is a piece of data is reinventing the worst parts of OO

noisesmith19:08:18

like for example a queue (you can ask if there are items ready) a channel (you can ask if there's an item ready, and if the input is closed), a vector (it has a precise count known at creation time)

didibus19:08:39

What I'm saying is a lot of lazy-seqs could answer of they have a next element, or even N next elements. But it would be specific to each one. (range 100)knows exactly. (filter even? seq) may not.

didibus19:08:26

Sure, but I need it to work with apply πŸ˜‚

noisesmith19:08:34

code that uses things that might or might not provide relevant information gets complicated, I don't think this improves anything

noisesmith19:08:03

isn't it simpler to just not use lazy code to wrap expensive or time sensitive operations?

didibus19:08:23

Well, it depends what you mean by simpler. An algorithm that does lots of compute and benefit heavily from short-circuiting exactly when it's done and no more are often easier to implement in a lazy way. You pull and pull until you're done.

noisesmith19:08:38

surely the version that uses a queue isn't that much more complex?

noisesmith19:08:07

you have one ugly driver, and the function called on each item inside that driver

noisesmith19:08:10

I'd argue the complexity introduced is intrinsic: the complexity inherent in explicitly controlling when things are evaluated and when you stop, and that's exactly what you want

didibus19:08:02

Well, ya I guess. Say you have a prime number generator, I guess you can make it into a blocking process that pushes 1 at a time to a channel. But even then it's 1 more then needed, you'd need the consumer to signal saying ok I want one more (and that's back to a pull model)

noisesmith19:08:19

and it simplifies things like retries / restarts / cleanup which are all garbage when using laziness

didibus19:08:19

I think you can use iterator maybe... I've heard their Clojure implementation short-circuits properly, but I'm not sure if that's true if used with apply as well

noisesmith19:08:33

well you did say channel instead of queue, and being able to choose between push and pull is one of the advantages channels offer

noisesmith19:08:08

if you don't buffer there's no reason for it to be ahead of the consumer

didibus19:08:23

I said channel because you need a queue that has back-pressure. Like if the consumers have what they need, don't produce more things in the queue.

noisesmith19:08:37

right, we are in agreement there

didibus19:08:03

Hum, is that true? Will a unbuffered channel only produce something after the consumer take from it?

noisesmith19:08:22

one moment, firing up a core.async repl

πŸ‘ 2
noisesmith19:08:30

(cmd)user=> (require '[clojure.core.async :as >])
nil
(cmd)user=> (def c (>/chan))
#'user/c
(cmd)user=> (>/go (>/>! c :a) (println "wrote"))
#object[clojure.core.async.impl.channels.ManyToManyChannel 0x7ef570be "clojure.core.async.impl.channels.ManyToManyChannel@7ef570be"]
(cmd)user=> (>/<!! c)
:a
user=> wrote

noisesmith19:08:52

of course you need more machinery than that to control when ":a" is evaluated

noisesmith19:08:27

but I think that shows clearly that the put does not complete until the value is read

didibus19:08:41

Can you try: (>! (println "produced"))

noisesmith19:08:56

well that blows up because println returns nil

didibus19:08:56

Because arguably you'd have println replaced with (compute-next-prime curr-prime) or something.

noisesmith19:08:08

but yeah, >! doesn't evaluate args lazily

didibus19:08:31

Hum, so I'm not sure you can use it, unless again you wrap it in a delay.

didibus19:08:39

It seems it will be 1 ahead at all times

noisesmith19:08:15

that's what I meant by "needing more machinery" - I think there's a couple of ways to do it, but it can be done with idiomatic and clear core.async code

didibus19:08:09

What are you thinking?

didibus19:08:14

Only thing I can think of would become pull again. Something like the consumer will push a message saying :need-more which the producer will block on, and when it sees one it will produce another one and push it.

didibus19:08:29

This is also an interesting read: https://clojure.org/reference/lazy

noisesmith20:08:24

(cmd)user=> (load-file "/home/justin/clojure-experiments/on-demand-async.clj")
#'user/generate
(cmd)user=> (def c (>/chan))
#'user/c
(cmd)user=> (generate c)
#object[clojure.core.async.impl.channels.ManyToManyChannel 0x1a632663 "clojure.core.async.impl.channels.ManyToManyChannel@1a632663"]
(ins)user=> @(>/<!! c)
0
(cmd)user=> @(>/<!! c)
generated 2
2
(cmd)user=> @(>/<!! c)
generated 3
3
(cmd)user=> @(>/<!! c)
generated 5
5
(cmd)user=> @(>/<!! c)
generated 7
7
(cmd)user=> @(>/<!! c)
generated 11
11
(cmd)user=> @(>/<!! c)
generated 13
13
(cmd)user=> @(>/<!! c)
generated 17
17

noisesmith20:08:52

(require '[clojure.core.async :as >])

(defn big-inc
  [b]
  (.add b (BigInteger. "1")))

(defn next-prime
  [b]
  (let [b' (big-inc b)]
    (if (.isProbablePrime b' 1)
      (do (println "generated" b')
          b')
      (recur b'))))

(defn generate
  [c]
  (>/go (loop [v (delay (BigInteger. "0"))]
          (>/>! c v)
          (recur (delay (next-prime (force v)))))))

noisesmith20:08:42

there's probably a more elegant solution (in terms of a consumer needing to both consume a chan and deref...), but probably not much more concise(?)

noisesmith20:08:02

also that prime generating code is crap but I figure it's a decent placeholder :D

noisesmith20:08:49

it might be more correct to move the force call outside the delay to avoid a stack bomb if a consumer repeatedly reads without dereffing...

didibus20:08:50

Hum, I don't know, not sure there's benefits to core.async if you add delay into it. Cause now you can also do a lazy-seq or an iterate that wraps in a delay as well.

didibus20:08:46

Hum... maybe if you deref inside a transducer on the channel?

noisesmith20:08:18

but using a lazy-seq wouldn't have the fine grained control of execution that a go block / channel combo has

didibus20:08:22

Or in fact, what if you generate the prime as a transducer on the channel?

noisesmith20:08:38

yes, a transducer that does a force or deref would work

didibus20:08:15

You could even do something like put the current prime, transducer on channel takes the current prime and return the next one.

didibus20:08:25

And avoid the need for a delay (I think less GC)

noisesmith20:08:28

I think doing the actual work inside a transducer undoes some of what is being attempted here, where the semantics of execution are explicitly controlled

noisesmith20:08:07

another option would be a "generator" macro that yields calculations as they are consumed from a channel

didibus20:08:32

The idea is to have consumers be able to pull elements one at a time through a chain of transforms. I think the thread of execution doesn't really matter. Like even ideally it's all done from the thread of the consumer doing the pull.

didibus20:08:57

I feel there's also probably a way to implement a more lower level >!that's lazy on its arguments.

didibus20:08:32

But still, with core.async, you're also reinventing the wheel for map, partition, mapcat, filter, remove, etc.

noisesmith20:08:05

what I meant by "semantics of execution are explicitly controlled" is that you'd see which thread did the calculation, or what constraints were around it, by seeing it explicitly in the code in question

noisesmith20:08:29

you could compromise with a transducer, but I think that undermines the goal here

noisesmith20:08:47

and even if you add a transducer to the channel, you introduce a non-negotiable 1 element write ahead buffer

didibus20:08:35

For applying the transducer? I remember asking when the transducer ran and on what thread and being told it's undefined

noisesmith20:08:03

you can't have a buffer of 0 on a channel with a transducer

didibus20:08:25

I didn't know that

didibus20:08:02

So I guess you can't introduce anything between the put and the take

didibus20:08:52

(buffer 0) doesn't work 😝

noisesmith20:08:59

it might be easier to experiment with these things in ztellman's manifold lib, IIRC one of the motivations was having easier and more explicit backpressure controls than core.async https://github.com/clj-commons/manifold

noisesmith20:08:19

but I think this is the right direction for the speculation to be going here - by dropping the pretense that the events are data we get simpler, less confusing or error prone means of controlling execution

noisesmith20:08:25

(to be clear, in most cases it's better to reify events as data, we are talking about the cases where this goes wrong as the initial premise of this particular thread of discussion)

noisesmith20:08:55

and I'd argue there's always an edge case where data fails, so it's good to invest in having a least-bad failure state

didibus20:08:41

(def c
 (chan
  (buffer 1)
  (map
   (let [p (volatile! (BigInteger. "0")]
    (fn [_] (vswap! p next-prime))))))
(go-loop []
 (>! c :next-prime)
 (recur))
What about something like that? Not at a repl currently to try.

didibus20:08:53

Ya, but that's where the Rich text I linked is interesting, cause he explored streams, but says: > In working on streams a few things became evident: > > stream code is ugly and imperative > > even when made safe, still ugly, stateful > > streams support full laziness > > this ends up being extremely nice > > Integrating streams transparently (i.e. not having both map and map-stream) would require a change, to relax the contract of the core sequence functions (map, filter etc) > > If I am going to do that, could I achieve the same full laziness while keeping the beautiful recursive style of Clojure? > > while being substantially compatible with existing code > > Yes! > > but - ideally some names should change

didibus20:08:26

But then... It's like they stopped short, like at some point they were like... Ok well it's lazy enough. And like, it seems so close. Maybe that's what I mean that it's kind of unfortunate. It's not even chunkiness that's the issue. You can have chunks of 1 or unchunked fully lazy even on their first element lazy-seqs. And I've not encountered anything that doesn't respect that yet. Now it's only a matter of a few functions like apply, and I think there might be a few more that are a little greedier then they could be.

noisesmith20:08:23

> Integrating streams transparently (i.e. not having both map and map-stream) to me that's the dividing line - the streams solution is for when data no longer works as your abstraction, so having "map-stream" or worse, having "map" work on streams, is counterproductive, it spreads the problem to the entire language instead of solving it

noisesmith20:08:15

I'm arguing that in practice the cleanest thing is to have a line with process on the one side and data on the other, and treat the attempts to treat process as data as source of bugs, and attempts to treat data as process as a waste of programmer effort / time

noisesmith20:08:00

and in clojure, lazy-seqs, as the thing we currently have, are on the data side

didibus20:08:08

I think I'm not following what you mean by process and data here?

noisesmith20:08:17

by data I mean a value you can retrieve, by process I mean a calculation you can make

noisesmith20:08:00

we gain a lot of power by blurring that distinction, but there are places (however narrow / niche) where you gain a lot by preserving it and building up barriers around it

noisesmith20:08:23

going back to our starting point again, someone wanted more control of lazy values because chunking effectively made their program incorrect. in my description that means they are operating in the area where the data / process distinction is needed and a problem was caused by blurring that distinction

noisesmith20:08:59

one way to fix that is to have more complexity (or less utility) on our lazy types, but another one is to have idioms and conventions for things that need to distinguish process from data (in order to have more fine grained control of process)

didibus21:08:48

Ya, but isn't that just a consequence of the abstraction not being good enough? Like if it was simple to control chunk-size, and everything had a truly lazy mode, then we could answer: oh ya, just go (chunk-resize s 1) and you're done. Now you know that everything will be maximally lazy.

noisesmith21:08:33

only if you do it in the right place and something else in between didn't do some other call increasing the size

didibus21:08:18

Ya, but I think that be a given, Like sure if I hand-off my thing to some third party lib. But if you can leverage Clojure core in a maximally lazy way, well, then that be great. Its pretty easy to just create a generating seq of something and then run a mapcat, or a filter on it. Its a lot more annoying to generate a custom thing using some abstraction I need to invent myself using core.async or manifold, and then re-build my own filter, mapcat and all that over it.

didibus22:08:52

I think you're right in that as it stands, its a more reliable approach. Just it would be cool if lazy seqs could be relied on for this as well.

Jim Newton12:08:14

QUESTION: what precisely does recur do when not within a loop ? I have just discovered that recur can be used outside of loop. I suppose that in a function such as clojure.core/some the meaning is obvious, i.e., to call the obvious function.

(defn some
  "Returns the first logical true value of (pred x) for any x in coll,
  else nil.  One common idiom is to use a set as pred, for example
  this will return :fred if :fred is in the sequence, otherwise nil:
  (some #{:fred} coll)"
  {:added "1.0"
   :static true}
  [pred coll]
    (when-let [s (seq coll)]
      (or (pred (first s)) (recur pred (next s)))))
However, the docstring of recur does not seem to describe this behavior. There is also an https://clojuredocs.org/clojure.core/recur#example-55ff3cd4e4b08e404b6c1c7f this implies this behavior, again without explaining it precisely.

quoll19:08:55

I know this was already resolved, but I’ll add to it anyway… I always think of a loop as being like a Ξ» that gets called immediately. Sort of like:

(defmacro my-loop
  [form & body]
  (let [forms (partition 2 form)
        args (map first forms)
        vs (map second forms)]
    `((fn [~@args] ~@body) ~@vs)))
That way you can think of all calls to recur as going to the nearest function entry point.

quoll19:08:35

Loops really are a different kind of thing (`loop*` is a special form handled by the parser), but thinking of them as a consistent thing has helped me sometimes

quoll19:08:06

And it’s always the nearest loop/fn that it jumps to. That’s similar to how break/`continue` work in Java, except that there’s no label option to jump out to a higher level

borkdude12:08:01

@jimka.issy The recur can also be used to recursively call the function you are in

Jim Newton12:08:25

is "the function you are in" always an unambiguous description?

Jim Newton12:08:34

sounds vague to me.

borkdude12:08:41

in your above example that function is some

Jim Newton12:08:30

yes obviously in that example. but if recur is found within an unnamed (fn ...) does that count or not?

borkdude12:08:32

(defn foo [] (recur)) is the non-stack consuming version of (defn foo [] (foo))

borkdude12:08:03

yes, defn expands into a call to fn

borkdude12:08:27

so if you look at the macroexpansion you will see that the recur applies to fn, defn is just syntactic sugar

Jim Newton12:08:28

if a macro introduces an fn might that change the semantics of a captured recur without loop ?

borkdude12:08:42

yes, e.g. future has this: (future (recur))

Jim Newton12:08:10

OK, where is this precisely defined? only in the source code?

borkdude12:08:12

user=> @(future (recur))
(won't finish)

delaguardo12:08:57

The docstrung of recur suggest to visit https://clojure.org/reference/special_forms where it's explained

3
Jim Newton12:08:32

hmmm. this behavior is different than I imagined. I thought the following would recur to the loop.

(loop [x 1]
  (letfn [(y [z] ... do-something (recur (inc z)))]
     ...))
but apparently recur recurs to y not to loop.

emccue13:08:18

Once you get complicated like that you are into the realm of trampoline

Jim Newton17:08:43

This problem happened to me once in Scala, but I think it has never happened to me in clojure. In Scala, we have tail call optimization, but it is somewhat limited. A function can call itself in tail position, and a local function can call itself in tail position, but if a local function is called in tail position, and that function calls the parent function in tail position, the scala compiler does not consider this a tail call. Why is that important? because if you refactor code into a local function, it might cease being tail-call-optimizable, even though it is 100% equivalent. Same for clojure theoretically, If I have a piece of code calling recur in tail position, and I refactor that code into a local function, then the recur call changes semantics. If the local function happens to have the same arity as the parent function, it won't even be caught as a compiler error--it will just be a bug.

Jim Newton17:08:33

(loop [x 1]
  (something)
  (something-else)
  (recur (inc x)))
ought to be (in my option) the same as the following in terms of referential transparency.
(loop [x 1]
  (something)
  (letfn [(f [x]
            (something-else)
            (recur (inc x)))]
   (f x)))
but lo and behold it is not.

Jim Newton17:08:00

we have the same problem of course with reduce / reduced.

Jim Newton12:08:12

good to know πŸ™‚

danm13:08:18

Gaaaah! Does anyone know how to get debug information out of clj as to how it is trying to access a maven repo that's in an AWS S3 bucket? I've got a repo that I can aws s3 [ls|cp] on no problem, but clj is refusing to get deps from it

Alex Miller (Clojure team)13:08:43

the reason for this is almost always creds related

Alex Miller (Clojure team)13:08:59

the most common thing I run into is when the creds do not have grants for s3 GetObject etc operations (even though they have access to the bucket)

danm13:08:14

I pulled the stuff down using aws s3 cp, so that's fine. Worked it out in the end, but haven't worked out how to fix

danm13:08:35

The maven plugin for deps.edn tries to lookup the location for the bucket using the S3 API

danm13:08:24

The S3 API GetBucketLocation command cannot, via AWS IAM, be granted to a user/role not in the same account as the bucket

danm13:08:18

> If the bucket is owned by a different account, the request will fail with an HTTP 403 (Access Denied) error.

danm13:08:06

So the deps.edn maven stuff has an option to specify the repo as {:url ""}, which works fine

danm13:08:28

Unfortunately we're also doing stuff with lein, and the same option doesn't seem to work there

Alex Miller (Clojure team)13:08:03

that is a feature specific to the Clojure CLI implementation

danm13:08:10

From some more digging, it appears that when I use credentials that are directly for the account in question it works, but when I use credentials from another account, which still have access to the bucket, it fails

danm13:08:21

Even though I can use those same credentials to copy bucket contents with the AWS cli

Nom Nom Mousse13:08:36

Another naming question: I have a function that creates a map from id to state. So a good name for the function is perhaps id->state. However, that also seems like the best name for the map created by the function. Is this a problem?

(let [id->state (id->state args)] ...
Another alternative would be ->id->state since it creates maps of id to state, but that might look weird and not be idiomatic.

Alex Miller (Clojure team)13:08:47

it's not a problem syntactically - that local binding will shadow the var

Nom Nom Mousse13:08:07

Yes, so my question is kinda subjective and about semantics. If no-one here would object to the above let I guess I am good.

Alex Miller (Clojure team)13:08:17

I think would probably rename the function to say what it does (key-by-id or index-state or something)

πŸ™ 3
dpsutton13:08:26

If you have watched Stu halloways debugging videos, you can see why shadowing names can be annoying. For instance, you can just def a value with the same name as the locals and evaluate expressions in your source. But if a local shadows a function you can inadvertently break things

πŸ‘ 3
Nom Nom Mousse13:08:40

I think key is a great and descriptive verb for this: key-state-by-id. I'm glad I asked πŸ™‚

p4ulcristian14:08:02

Hi all! Can I extend the clojure.core functions with own functions, without needing to re-import them in every namespace? For example utils like:

(defn concat-vec [& rest] (vec (apply concat rest)))
I tried to use
(in-ns 'clojure.core)
and define my functions, but how should I compile them after clojure.core?

Alex Miller (Clojure team)14:08:37

you should not want to do that :) just make functions in a namespace and require that namespace when you want to use it.

πŸ‘ 3
isak14:08:24

If you wanted to avoid repeating yourself too much in your requires, I think this would be the way:

(ns foo.bar
  (:require-macros [foo.baz :refer [macro-that-expands-to-require]]))
(macro-that-expands-to-require)
(From the May 24 ClojureScript https://clojurescript.org/news/2021-05-24-release)

πŸ‘ 3
p4ulcristian15:08:12

nice, just what I needed, thank you!

plexus04:08:59

That feature is going to be dropped again, there is no good way to make it work without breaking other things

borkdude15:08:18

@isak unfortunately that won't be supported anymore in the future, if I understood a conversation between @plexus and @dnolen in #cljs-dev correctly

plexus05:08:53

That is correct, there's no good way to make it work without breaking existing functionality

3
dnolen15:08:53

yes it's already been rolled back

dnolen15:08:26

trying to re-sync with Closure Compiler and Library - and there'll be another release and we'll mention it's been removed

dnolen15:08:41

and we'll update the old post as well

dnolen15:08:05

the feature never worked to be clear

isak15:08:05

Ah I see. What was the problem with it?

dnolen15:08:10

just cannot work

3
dnolen15:08:23

and Canary, all our tests, etc. didn't catch it

dnolen15:08:17

short story is that ClojureScript is an AOT variant of Clojure - you must know the whole graph of deps

dnolen15:08:26

hard requirement to feed everything into Closure etc.

dnolen15:08:39

but if you put something behind a macro you cannot figure this out in any sensible way

dnolen15:08:03

there might be something that could work - but the code started getting ugly immediately once the bug reports flowed in

dnolen15:08:10

so just not going to do it

isak15:08:46

good to know, thanks @dnolen @borkdude

Russell Mull15:08:17

@paul931224 To add a little color to "you should not want to do that": many people find that this kind of shortcut is a savings only in the very short term. It is extremely valuable to have the source of each of your functions be obvious by inspecting the source. This is the same reason that explicit :refer is encouraged over :use in ns forms.

πŸ‘ 8
tessocog18:08:09

how do i prevent spec from generating ##NaN, ##Inf and such?

dorab18:08:08

Look at a custom generator. In particular, the double* generator. See https://clojure.github.io/test.check/clojure.test.check.generators.html#var-double*

πŸ™ 2
Alex Miller (Clojure team)18:08:45

the double-in generator does not generate them by default. what spec/generator are you using when you see them?

βœ… 2
Alex Miller (Clojure team)18:08:06

(btw, there is #clojure-spec for future q's)

βž• 2
tessocog19:08:15

(spec/gen float?)

tessocog19:08:49

double-in does it thanks

tessocog19:08:51

double-in generates them by default though

Alex Miller (Clojure team)19:08:26

shoot, I thought it was the opposite. we've talked about this in the past