Fork me on GitHub
#clojure
<
2020-03-31
>
didibus00:03:15

Don't think I'm as enthusiastic about unfold. An iterate which support for a side-effectinf f seems more generally useful. Its easy to combine with a take-while when you care.

didibus00:03:40

But in any case, I guess it would solve my use case

didibus00:03:55

So I'd still be for it

didibus00:03:22

I was actually surprised iterate isn't implemented in terms of lazy-seq

didibus00:03:27

I guess I'm just not seeing the point of the non-caching behavior with reduce. What scenario would that be needed for ?

didibus00:03:46

Oh, seems wrapping the iterate in lazy-seq works 😄

didibus00:03:00

(lazy-seq (iterate f))

didibus00:03:50

I'll just do that for now

didibus00:03:36

Unless anyone else sees an issue with that as well ?

hiredman00:03:04

Yes, it is no good, stop trying to "trick" whatever

didibus00:03:17

(defn fetch
  "A stub fetch that pretends like its getting random data from a url, but instead return a rand char from url."
  [url]
  (println "fetching " url)
  (rand-nth url))

(defn lazy-fetch-until-data-not-seen [fetching-fn]
  "Returns a lazy-seq which on every getting of the next element, will fetch the next element by calling
   fetching-fn. Will stop once it sees an element is returned by fetching-fn that has already been fetched before."
  (->> (iterate
         (fn [[_ page-fetching-fn pages-fetched]]
           (let [page (page-fetching-fn)]
             (if (pages-fetched page)
               [:done nil nil]
               [page page-fetching-fn (conj pages-fetched page)])))
         [nil fetching-fn #{}])
       (lazy-seq)
       (map first)
       (drop 1)
       (take-while #(not= :done %))))

(def data (lazy-fetch-until-data-not-seen #(fetch "")))
(mapv upper-case data)
;> fetching  
;> fetching  
;> fetching  
;> fetching  
;> fetching  
;> fetching  
;> fetching  
;;=> ["G" "W" "." "H" "O" "T"]
(mapv upper-case data)
;;=> ["G" "W" "." "H" "O" "T"]
(mapv upper-case data)
;;=> ["G" "W" "." "H" "O" "T"]
(mapv upper-case data)
;;=> ["G" "W" "." "H" "O" "T"]
(reduce (fn[acc e] (conj acc e)) [] data)
;;=> [\g \w \. \h \o \t]
(reduce (fn[acc e] (conj acc e)) [] data)
;;=> [\g \w \. \h \o \t]
(reduce (fn[acc e] (conj acc e)) [] data)
;;=> [\g \w \. \h \o \t]
(reduce (fn[acc e] (conj acc e)) [] data)
;;=> [\g \w \. \h \o \t]

hiredman00:03:18

Just write yourself unfold

didibus00:03:31

I mean it looks pretty sound to me

didibus00:03:12

lazy-seq properly wraps around Iterate, so reduce is now done over the lazy-seq

hiredman00:03:51

No, you don't actually understand how lazy seqs work

hiredman00:03:17

That would behave the same with or without your lazy-seq wrapping

didibus00:03:26

Well, ya I know that

didibus00:03:56

But just to be sure I add it there, just to remind myself I need to make sure something does, or I expose the Iterate to the caller

hiredman00:03:23

The reason the fetches only happen once is because the lazy seqs you are building on top of iterate (with take, map, etc) are caching

didibus00:03:13

Ya, and non caching reduce is why f has to be side effect free over an Iterate

didibus00:03:31

But, if you wrap it in a lazy-seq, then you're good to go

didibus00:03:49

But, maybe I'm still not seeing all edge cases, I admit

didibus00:03:10

But right now, this actually seems to all make sense to me, and seems safe

didibus00:03:33

You lose the performance benefit of iterate's reduce impl, but it is not what I'm looking for here

ghadi00:03:52

feel free to use the code above from CLJ-2555, it's meant for iterated consumption of APIs

didibus00:03:49

Let me have a look

ghadi01:03:03

the code above using iterate is not clear, and @hiredman is right that the lazy-seq usage is wrong

didibus01:03:21

What's wrong with it?

didibus01:03:24

And what's not clear about it? I'd use a similar pattern for generating fibs or anything else side effect free as well? Am I using it wrong?

ghadi01:03:32

for one, it shouldn't be side-effecting because the docstring says not to be

didibus01:03:02

Haha, well, that

didibus01:03:11

is my whole reason for being here

didibus01:03:33

Having now understood why the doc-string says so, it seems wrapping in lazy-seq solves the issues

ghadi01:03:38

passing a named tuple of state is awkward and hard to understand

potetm01:03:15

I mean, isn’t this kind of ridiculous? It’s not fundamentally different than having a FSM with a terminal state:

{:state :done}

ghadi01:03:38

i meant to say unnamed

ghadi01:03:55

s/named/unnamed

potetm01:03:00

ah, so you’re saying a map is preferable here?

ghadi01:03:29

sure information is better than syntax

facepalm 4
potetm01:03:08

genuinely sounds like something Elements of Fauxjure would spit out

potetm01:03:28

if you’re not familiar

ghadi01:03:22

it predates clojure

ghadi01:03:44

ruby community used to call it "connascence of position"

ghadi01:03:01

which sounds way more Fauxjure than Fauxjure :)

potetm01:03:34

I’m not saying the underlying concept is not real. Positional semantics have been with us forever. I’m saying it’s a ridiculous way to frame the issue (and it’s not always true) 🙂 “Syntax is complecting,” is at least more accurate.

ghadi01:03:16

signaling termination is awkward [:done nil nil]

ghadi01:03:59

there's a whole history of linked stuff from CLJ-2555 and CLJ-1906 (the latter especially) that is worth reading

didibus01:03:18

Right, I can see that. I guess I just mean, that's just how iterates work no? Like even for side effect free, you'll be left with code like I have, so its nothing special, just maybe iterate produces ugly code

ghadi01:03:48

iterate is beautiful, you're using it off-label

didibus01:03:58

This is the most used example of iterate: (def fib (map first (iterate (fn [[a b]] [b (+' a b)]) [0 1])))

didibus01:03:13

You have named tuple for carried state

didibus01:03:24

And then a call to map to remove the state tuple

ghadi01:03:27

(take 10 (iterate #(.plusDays % 1) (java.time.LocalDate/now)))

noisesmith01:03:34

I think part of the problem here is a mismatch between lazy data oriented evaluation via laziness and procedural evaluation where side-effects matter - that's pretty fundamental

ghadi01:03:49

again, fib is pure

didibus01:03:01

Ok, but ignoring the purity, its just as ugly

didibus01:03:06

and unreadable

ghadi01:03:18

not sure why you're saying ignore the purity

ghadi01:03:40

the purity is what enabled us to vastly improve its implementation in clojure 1.7.0

didibus01:03:49

I mean you said: > passing a named tuple of state is awkward and hard to understand > signaling termination is awkward [:done nil nil]

Alex Miller (Clojure team)01:03:01

the "no side effects" in the doc pre-dated the 1.7 impl

didibus01:03:04

Those apply to side effect free iterate implementations as well

Alex Miller (Clojure team)01:03:08

it has always said that

didibus01:03:22

So I mean, the only thing different I do to handle side-effecting f is to make sure I return a lazy-seq and not an instance of Iterate

didibus01:03:41

Which just calling map does, but I wanted to be more explicit

didibus01:03:00

I'm not saying it won't be great to have a different fn with maybe a more ergonomic interface, I'm still trying to wrap my head around the iteration fn from CLJ-2555, so maybe once I get how to use it, I'll be fully sold. But I'm also not seeing the issue with what I'm doing with iterate right now, seems all concerns around f being side effect free are irrelevant if you make sure you don't leak the clojure.lang.Iterate

ghadi01:03:33

the iterate fn arg you have written is complected (scratch off the bingo card) - it's handling production (calling a function) & termination

noisesmith01:03:44

> the only thing different I do to handle side-effecting f is to make sure I return a lazy-seq and not an instance of Iterate if you don't use threads, maybe

didibus01:03:38

Hum, I guess that's a general question related to lazy-seq and their caching being thread safe or not

didibus01:03:45

I think they are

didibus01:03:53

Like what is handled by repeatedly

noisesmith01:03:43

it's specifically not safe to call iterate with side effects in multiple threads, an effect could happen twice

noisesmith01:03:16

the fact that this was already a known limitation is why the implementation of reduce which ignores the iterate cache was implemented

didibus01:03:57

Yes, iterate on its own

didibus01:03:02

But I'm wrapping it in a lazy-seq

didibus01:03:17

And I believe once wrapped, it is now safe, and thus it is fine to have f be side effecting

noisesmith01:03:38

that doesn't change the thread safety aspect, iterate will recalculate in a way that other lazy data sources do not

noisesmith01:03:48

that belief is incorrect

didibus01:03:02

Except it won't be able too, because lazy-seq is now in charge, and it will be caching

didibus01:03:30

So unless lazy-seq is not thread safe, I still see no issue

noisesmith01:03:38

lazy-seq is out of the picture after one value is consumed

noisesmith01:03:29

(ins)user=> (type (rest (lazy-seq (list 1 2 3))))
clojure.lang.PersistentList
(ins)user=> (type (rest (lazy-seq (iterate inc 0))))
clojure.lang.Iterate

noisesmith01:03:43

lazy-seq only wraps the outer collection, it is not recursive

noisesmith01:03:10

that suffices for a hack that tricks reduce, it doesn't help with thread safety

didibus01:03:12

Hum... so why does it work

didibus01:03:31

See my example above, f is never recalculated

noisesmith01:03:32

reduce only checks the type of the outer collection

didibus01:03:40

😛, so good enough then

noisesmith01:03:47

in your use case maybe, but it hasn't been made thread safe

didibus01:03:00

But ok, that's a fair call out

didibus01:03:15

Except, if you look at my function

didibus01:03:32

Witht he call to map? would that not change things?

didibus01:03:01

I think map might be continuously wrapping things in a lazy-seq, as it handles its own call to rest

didibus01:03:11

Maybe thats what makes it work in fact

didibus01:03:11

Anyway, you had a convincing argument. So if map does make it work, it does still become more brittle, and less general. Like always remembering to be sure it is someone properly wrapped even on call to rest becomes tricky

ghadi01:03:46

termination concerns are also leaking to the outside

didibus01:03:40

I mean, leaking outside my own use of iterate

ghadi01:03:31

its primary argument is a function called step! which is presumed to be side-effecting

ghadi01:03:18

step! takes one argument, a "key"

ghadi01:03:26

a good example of step! is calling an API like S3 ListObjects or anything paginated

ghadi01:03:20

step! gets called with an initial key. if you don't override it, the initial key is nil

ghadi01:03:36

this is your initial call to S3 ListObjects

ghadi01:03:53

what step! returns is some HTTP Response

potetm01:03:25

fwiw - I would prefer an explicit loop/recur here.

ghadi01:03:48

yes but those are eager, can't use a terminating transducer

potetm01:03:07

Feed it w/ lazy data?

ghadi01:03:17

one sec and it will be clear

potetm01:03:45

What I’m saying is: I would not want it to be lazy.

potetm01:03:09

Side effects are much easier to reason about when you can control when they happen.

potetm01:03:14

That’s a massive value.

ghadi01:03:22

you can just as well use it eagerly

ghadi01:03:08

bear with me because this is a complicated function

ghadi01:03:30

HTTP Responses from these sorts of paginated APIs contain 2 important bits of information: 1) an argument that you can pass for the next call to step! 2) some data

ghadi01:03:57

(In the terminology of that function's docstring, the HTTP Response is called a "ret", the argument that you pass to the next call to step! is "k" and the data is called values)

didibus01:03:39

Seems its very much designed for typical pagination, can it be used more generically?

ghadi01:03:41

to get from the HTTP Response to the next k for step!, there is an argument :kf, which is a function that gets called on the HTTP Response ("ret")

ghadi01:03:12

to extract some values from the HTTP Response, there's an argument called :vf

ghadi01:03:17

"key function" and "value function"

didibus01:03:44

Ah, I kept wondering why they were called vf and kf

ghadi01:03:56

there's one more piece, but just to recap: step! is a function of "k" and return a "ret"

ghadi01:03:07

kf is a function of ret and extracts the next k

ghadi01:03:20

vf is a function of ret and extract the value

ghadi01:03:35

the last piece is the argument :some?

ghadi01:03:18

which is a function of the "ret", and it tells the iteration process whether this "page" (HTTP Response) had anything in it

ghadi01:03:32

some? allows you to abort early

ghadi01:03:50

kf - if it doesn't produce a key, also aborts early

ghadi01:03:43

so the flow is: get a page by calling step on k call some?, abort if falsey call vf to get some values out call kf to get the next k, abort if no k ----loop to top

didibus01:03:07

So in my case, it seems I would still need to hack around it, and have ret be a tuple of state, and k won't be a key, but carrying over all previously returned piece of data no?

ghadi01:03:35

let me give an actual example from AWS that consumes through pages of results

didibus01:03:50

Since I'm doing, fetch data until you receive data which was previously returned by prior calls to fetch

potetm01:03:05

Honestly: It’s like 5 lines of code 😄 Just use lazy-seq if you actually need lazy?

didibus01:03:09

This is not as simple as AWS pagination

potetm01:03:55

reduce optimizations are nice, but perhaps not necessary?

didibus01:03:04

@potetm I know, but I find lazy-seq to be even less ergonomic 😛, so I was trying to get iterate working, but then it can't handle side effects, and now there is iteration, which might seem like its not generic enough, so maybe I'll just go back to lazy-seq

potetm01:03:23

yeah lazy-seq is the standard tool here

ghadi01:03:25

(defn log-groups
  "returns all log groups from CloudWatch"
  [client]
  (->> (iteration
        (fn [token]
          (aws/invoke client (cond-> {:op :DescribeLogGroups}
                               token (assoc :request {:nextToken token}))))
        :kf :nextToken
        :vf :logGroups)
       (into [] cat)))

didibus01:03:37

Not throwing iteration under the bus, I still need to understand it

didibus01:03:09

So in my case, to know if I'm done, I need to look at all previously fetched data

didibus01:03:21

The API does not tell me I am done

potetm01:03:45

If that’s the case, you can’t use laziness, no?

potetm01:03:16

Like, you might as well loop/recur or atom bash in a doseq if you gotta check the full result set each time.

Alex Miller (Clojure team)01:03:34

my general thumb with laziness is if you care how much of the lazy thing gets read when, you shouldn't use a lazy seq for it

potetm01:03:01

ghadi’s proposal uses it under the hood, or do I misread?

ghadi01:03:37

Rich wrote it, I am stewarding it

Alex Miller (Clojure team)01:03:43

it's kind of a special case

ghadi01:03:55

but iteration has a seq side, and a reduce side

potetm01:03:02

how so? still doesn’t control “how much” you get, does it?

ghadi01:03:11

it reifies both Seqable and IReduceInit

potetm01:03:11

unless you use the reduce side

Alex Miller (Clojure team)01:03:16

but you should always be prepared to read 32 items ahead or whatever

potetm01:03:28

right, always the case

potetm01:03:38

(unless you go no-lazy)

Alex Miller (Clojure team)01:03:47

or be careful in how you consume it, which you probably are with something like this

ghadi01:03:49

take doesn't do the chunking

ghadi01:03:00

(32-width chunking)

didibus01:03:05

Doesn't that still leave open a gap in the language, where there is nothing that can really replace a generator easilly?

Alex Miller (Clojure team)01:03:27

if you want control, use loop/recur

💯 4
didibus01:03:33

I also thought lazy-seq wasn't chunking

ghadi01:03:35

iteration has enough knobs to be a generator

potetm01:03:48

lazy-seq isn’t, but most seq fns are

ghadi01:03:48

for different types of iterated APIs

Alex Miller (Clojure team)01:03:52

if you map or filter over it, you're chunking

ghadi01:03:14

you're only chunking if the source is chunked, no?

didibus01:03:22

Hum... I'm not seeing that behavior at the REPL

Alex Miller (Clojure team)01:03:52

map and filter produce chunked sequences

Alex Miller (Clojure team)01:03:32

if it reads 32 elements ahead from the source, then aren't you chunking?

didibus01:03:48

So if map is called over a non chunked sequence, it will map over it in chunk?

Alex Miller (Clojure team)01:03:19

how else could it work?

potetm01:03:48

source code appears to switch on the type

Alex Miller (Clojure team)01:03:54

oh ghadi, you're talking about the (if (chunked seq? ) ... in there?

didibus01:03:57

I mean, it makes a lot of sense, but I thought map inly did that if the coll was an instance of ChunkedLazySeq or wtv

ghadi01:03:59

what I said is correct

Alex Miller (Clojure team)01:03:39

I guess none of them would then be

potetm01:03:54

Yeah, I feel like this was totally a thing at one point, but I’m not seeing it now.

didibus01:03:58

I was kind of hoping everything would respect the difference between ChunkedSeq and LazySeq, and only chunk once a ChunkedSeq was present in the chain

ghadi01:03:06

(map f (iteration ...)) isn't chunked

Alex Miller (Clojure team)01:03:18

but a greater point is that any seq impl could chunk and you don't have any control over that

4
didibus01:03:28

Hum... I understand could, but should? Like wouldn't the semantics dictate it shouldn't

Alex Miller (Clojure team)01:03:53

the semantics very much don't

didibus01:03:16

Okay, so ya, just considering Clojure to have lazyness is a mistake at this point I guess

Alex Miller (Clojure team)01:03:30

well that's absolutely the wrong takeaway imo

didibus01:03:37

You should treat all that as a black box

✔️ 4
potetm01:03:15

Yeah, I think that’s actually right. You gave up control over the execution model when you decided to use map.

Alex Miller (Clojure team)01:03:15

without lazy seqs, you wouldn't have infinite seqs and partial reading, and many things we find great

potetm01:03:37

For many purposes, it’s exactly what you want.

didibus01:03:43

I meant like, don't think you can leverage lazy-seq to build your own lazy one at a time workflow of especially something side-effectful

didibus01:03:14

It really doesn't seem to be the intent of the whole lazy-seq machinery, so doing so is not advised

Alex Miller (Clojure team)01:03:19

with lazy seqs, you do not have control of consumption

Alex Miller (Clojure team)01:03:53

the control is internal, not external

didibus01:03:02

It does leave open the generator use case though it seems, for maybe a community provided library

Alex Miller (Clojure team)01:03:14

people have written clojure generator libs

potetm01:03:14

Worth considering: using core.async for generating values

potetm01:03:31

so that’s a separate process from the processing

ghadi01:03:40

we have also thought about a concurrent flavor of iteration, where the calls to step! can run ahead by a configurable buffer

didibus01:03:10

Ya, I just see more and more people looking for a "generator" like behvior in Clojure and asking how to do that. Since its all the trend in other languages right now

potetm01:03:31

Absent context, I would say: core.async

didibus01:03:51

Probably easiest to just reify a Java Iterator at this point

didibus01:03:15

core.async is still eager though

Alex Miller (Clojure team)01:03:16

just the first one I see on googling, I know I've seen others

potetm01:03:25

lol that’s a wrapper on core async

didibus01:03:34

Well.. I guess ya, with a buffer of 1, it gets pretty lazy

potetm01:03:34

no, it wouldn’t be eager

potetm01:03:50

you can hold a very small amount in memory at a given time

Alex Miller (Clojure team)01:03:53

on a really old version of core.async :)

didibus01:03:57

Ya, that might be the way I go instead, and tell people to go for

Alex Miller (Clojure team)01:03:04

seems like I just saw a much more extensive one recently

potetm01:03:19

I honestly prefer that model to lazy seqs for side effecting stuff. You get fine-grained control over side-effects, and it’s separated from transformation code.

Alex Miller (Clojure team)01:03:07

I think that's the one I was remembering

Alex Miller (Clojure team)01:03:52

definitely is some overlap with how core.async is impl'ed

didibus01:03:13

Implements the more general coroutine, and does not use core.async IOC for it

didibus01:03:20

That erdos.yield one seems interesting as well, similarly re-implement its own machinery instead of using core.async

didibus01:03:53

To be honest though, in my case, I'm not looking for that imperative style of yielding

didibus02:03:24

Just need to create an iterator that can carry state over from prior iteration, and is truly lazy, one element at a time 😛

didibus02:03:59

And is also caching

potetm02:03:20

caching :wat:

didibus02:03:31

Cause I like that over generator, and its easy to just not retain the head if you don't care about prior "generated" elements

potetm02:03:43

are you sure you don’t want to just eagerly load everything?

didibus02:03:05

Yes, I want to load the next element from a server only when it is needed.

potetm02:03:19

but you want to cache it?

didibus02:03:21

But once loaded, I don't want to need to reload them all

potetm02:03:22

in memory?

didibus02:03:37

So if I want to loop over them twice, I'm not fetching twice

potetm02:03:43

so, what’s the downside to loading them all up-front?

potetm02:03:50

like, why is that a problem

potetm02:03:26

if it’s not to save memory, what’s it for?

didibus02:03:54

Maybe the user won't want to see all thousands, only first hundred, but those first 100 still need to be transformed somewhat before being displayed

didibus02:03:31

Like say elements appear as you scroll down the page

potetm02:03:07

users? what’s that? I write servers! I write for the machine!

potetm02:03:13

yeah okay, that makes sense

potetm02:03:16

although, at that point, I would probably bash it into an atom anyways (a la reagent)

didibus02:03:11

Trying to be pure in my use of unpure functions 😛

didibus01:03:41

It can be, see my big code snippet from above

ghadi01:03:39

@didibus using iteration if you have seen something in the result set already, step! can return a ret that fails the some? check

didibus01:03:01

Ya, but where is the result set available to me?

ghadi01:03:03

step! is presumed to be side-effecting

didibus01:03:22

So like I need external mutable state?

ghadi01:03:39

there are other ways to solve your problem

ghadi01:03:58

if you were using iteration - yes you'd use a ref type

didibus01:03:21

Ya, I was hoping not to have too\

didibus01:03:34

Which a side effect safe iterate would allow me too 😛

didibus01:03:09

Well, I think I might be able to use the same trick on iteration

didibus01:03:41

Have step! return a tuple, and kf can extract the state and k can just become state I carry over instead of a key from the response

ghadi02:03:26

(defn didibus-thing
  "fetch is a side-effecting 0 arg fn"
  [fetch!]
  (let [seen (atom #{})]
    (iteration (fn [_]
                 (let [obj (fetch!)
                       [old new] (swap-vals! seen conj obj)]
                   (when (not= old new)
                     obj))))))

ghadi02:03:03

it aborts when fetch returns something that has been seen before

ghadi02:03:35

notice step!'s "k" argument is ignored, and the values returned by each step is the same as the "ret"

ghadi02:03:57

so the defaults for :some?, :kf and :vf all work fine

ghadi02:03:49

the neat thing about this is that you get to choose whether you use it as a seq or a reducible

didibus02:03:45

Ya, its probably what I'll start using once 1.11 is out. I feel like, if it allowed to carry over something, you could avoid the use of the closure over an atom or some other construct, but I can see the challenge in the API

ghadi02:03:55

user=> (vec (didibus-thing #(rand-int 10)))
[3 4 2 0]

ghadi02:03:14

it might not make it into 1.11

ghadi02:03:22

so feel free to crib it from JIRA

ghadi02:03:42

there is nothing wrong with an occasional atom

💯 4
didibus02:03:00

I know, but you know, Clojure always makes me feel dirty when I do 😛

didibus02:03:33

I still conceptually prefer the iterate model, like in its beauty. The idea is just that I want a sequence of (f x) -> (f (f x)) -> (f (f (f x))) -> etc. I find this model pretty conceptually elegant, just here f can't have side effect.

didibus02:03:04

The next element is a result of the previous (and possibly all prior or anything else I choose to carry over)

ghadi02:03:44

meh I don't like it (no offense)

ghadi02:03:24

the state & the tuple mar the conceptual beauty

ghadi02:03:46

whoops forgot the lazy seq

didibus02:03:18

No offense taken, these things are a bit personal opinion. The atom similarly annoys me 😛

ghadi02:03:26

(defn didibus-lazy-seq
  "fetch is a side-effecting 0-arg fn"
  [fetch!]
  (let [step (fn step
               [seen]
               (lazy-seq
                (let [obj (fetch!)]
                  (when (not= (conj seen obj) seen)
                    (cons obj (step (conj seen obj)))))))]
    (step #{})))

didibus02:03:23

Maybe in a language with support for multiple return it would be cleaner. But I got used to tuples in Clojure being an idiomatic way to do so

didibus02:03:13

It also annoys me that the first result of iterate is x on its own, wish it was (f x). That's why I have the whole drop 1 thingy

didibus02:03:37

And I guess you could have a multi-arity iterate, so f could be one or more arguments, again, to avoid the tuple as arg-vector trick

didibus02:03:13

(defn fibonacci[]
  (->> (iterate (fn [[a b]] [b (+ a b)]) [0 1])
       (map first)))
So in the above @ghadi would you say that one could also choose to use iteration for it with an atom? Would there be any advantage? Or do you envision iteration being used only for side effects?

didibus05:03:34

@ghadi Here's my version using iteration:

(defn lazy-iteration-fetch [fetching-fn]
  (->> (iteration
         (fn [pages-fetched]
           (let [page (fetching-fn)]
             (if (pages-fetched page)
               nil
               [page (conj pages-fetched page)])))
         :vf first
         :kf peek
         :initk #{})
       lazy-seq))
What do you think?

didibus05:03:49

I know you intended k to be more like the key needed to make the API request, but I guess in practice it can be whatever. So I can use a similar trick of making it previous state. It still ends up being much more readable then with iterate, so I like it. Have to wrap in a lazy-seq as well, since I see that iteration is non caching, it assumes the API will be idempotent on k, or that consumers won't consume more than once otherwise.

seancorfield05:03:56

@didibus It only talks about being idempotent on initk

seancorfield05:03:16

"it is presumed that step! with non-initk is unreproducible/non-idempotent"

seancorfield05:03:39

"if step! with initk is unreproducible, it is on the consumer to not consume twice"

seancorfield05:03:58

Ah, but your version is unreproducible on initk as well, right? Since it calls (fetching-fn) regardless of what is passed in.

didibus05:03:56

So wrapping in lazy-seq seems to work to make it cached. Though I think at this point, I'm doing the same trick I did with making iterate safe for side effects

didibus05:03:34

Which @noisesmith pointed out that it might not be super reliable

didibus05:03:06

Because lazy-seq only wraps the outer sequence, once you call rest or next on it, you get back something else

didibus05:03:34

But I don't know, seems to work fine

didibus05:03:12

Will iteration without an initk somehow return a caching sequence instead?

didibus05:03:07

The doc is unclear, I think it just means, without an initk, then the server has no way to be idempotent, since you provide it no key, so it is assumed not to be, in which case it is probably expected the consumer won't consume twice

dpsutton05:03:12

I thought the docstring makes it clear that without an initk it just calls the fn with nil the first iteration

didibus05:03:07

Yea it says that

didibus05:03:09

And then: it is presumed that step! with non-initk is unreproducible/non-idempotent

didibus05:03:16

if step! with initk is unreproducible, it is on the consumer to not consume twice

didibus05:03:02

Which I guess, well, I can see it maybe being a bit confusing, like does that mean it assumes that if you pass an initk it must be reproducible?

didibus05:03:31

And if so why? Does it do something different to handle the non reproducible case of using default initk?

didibus05:03:16

Basically at this point I have three implementations and they all seem to work just as well, one with iterate which I map over, which seems to make it behave properly with side effect when you do that. One with iteration which I wrap in a lazy-seq, and again it then seems to have the expected behavior, including (rest data) returning the cached thing and not calling fetch again. And one with lazy-seq.

seancorfield05:03:46

@didibus If the step is non-reproducible on initk, then you may get different answers if you consume it twice -- which is back to the same boat you were in with iterate and trying to do both seq and reduce.

seancorfield05:03:43

It doesn't need to do anything special for that case, it's just a consequence of the non-reproducibility of that call (since if it gets called twice on initk it may return different results).

didibus05:03:13

Ya, I'm investigating. So for iterate, wrapping in lazy-seq fails if you then reduce on the rest, you will get back an instance of Iterate, and reduce will now again re-consume the fetches.

didibus05:03:23

But, if wrapped in map, it works

didibus05:03:42

Like (map identity (iterate ...))

seancorfield05:03:54

In other words, it's saying that if calling step! with a known fixed value can return different values on repeated calls (such as a random number generator), then you can't guarantee that repeated consumption of the sequence of values will return anything predictable.

seancorfield05:03:24

Right, and you're seeing that because you don't have a sane "seed" for your initial fetch call.

didibus05:03:39

Ya, so like... it feels to suffer from the same "non side effecting" as iterate, just I guess, it assumes the caller understands the risk

seancorfield06:03:00

How do you establish the "first call" in your fetching sequence?

seancorfield06:03:22

It seems to me that if there's no "seed" value, then you have a random sequence with no repeatability?

didibus06:03:39

Only difference is, for iteration, wrapping in lazy-seq seems to work fully, and calling rest on it returns a LazySeq

seancorfield06:03:53

You're not answering the question.

didibus06:03:05

Yes, my fetch is this: `

seancorfield06:03:06

You're focusing on a hack instead of the core problem.

didibus06:03:22

(defn fetch
  "A stub fetch that pretends like its getting random data from a url, but instead return a rand char from url."
  [url]
  (println "fetching " url)
  (rand-nth url))

seancorfield06:03:01

That has no "seed" so the initial call is completely unpredictable -- so it fails the guarantees of iteration.

didibus06:03:07

And I want to fetch until I see a repeated value

didibus06:03:32

So iteration only supports idempotent side effects?

seancorfield06:03:42

No, absolutely not.

seancorfield06:03:48

That's not what it says.

didibus06:03:25

Ok, but kind of. Lets take a step back. iterate says it is not safe for side-effects because if consumed twice it will recompute the side-effect

didibus06:03:51

iteration says you can use it for side-effect, but if consumed twice it will recompute the side-effect

seancorfield06:03:03

Only for initk

didibus06:03:16

Only when the side-effect is not idempotent

seancorfield06:03:57

It says that if the step function, invoked on initk, is not reproducible, it's up to the consumer to not attempt to consume it twice.

seancorfield06:03:35

It doesn't have any caveats about step called on non-initk values -- in fact it specifically says it is presumed to be unreproducible.

seancorfield06:03:09

But that initk value -- the first call -- has to be reproducible if you want any guarantees about repeated consumption.

seancorfield06:03:24

i.e., you can't have both.

didibus06:03:30

I doubt it

didibus06:03:41

Like, maybe, but I'm not really seeing it

seancorfield06:03:50

If you don't care about the guarantee, then the reproducibility of step called on initk doesn't matter.

didibus06:03:37

Like, what does that change? If my first initk is reproducable, but not my next k, how does it guarantee it?

didibus06:03:17

Does it cache sequences by initk ?

seancorfield06:03:44

You have the source code -- you can answer that question.

didibus06:03:36

erm, let me do an experiment first 😛

didibus06:03:48

I think I know what you mean

didibus06:03:03

Because it explains why rest returns a LazySeq

didibus06:03:10

while rest on iterate does not

didibus06:03:37

And that would explain why wrapping iteration in a lazy-seq is enough to add the guarantee even with a non reproducible initk

didibus06:03:12

But I also don't understand why people keep saying me wrapping things in lazy-seq is a hack?

seancorfield06:03:46

Very specifically, calling seq on the result of iteration invokes (step! initk) and calling reduce on that same result invokes (step! initk) again.

seancorfield06:03:57

Yeah, I don't know why you don't understand, when everyone keeps telling you the same thing 🙂 I was hoping to explain it so you'd see it but I clearly haven't done any better job that either Ghadi or Alex at this point. Sorry, I tried 😞

didibus06:03:00

Ok, and later in reduce it calls (step! k)

didibus06:03:14

Which I'm not seeing how it would not return a different result

didibus06:03:19

Even if initk does

seancorfield06:03:45

I give up. Sorry, man, but it's late and I've tried.

didibus06:03:19

Yam thanks

didibus06:03:42

I'm still gonna go ahead and use this, not seeing any failure case with it

didibus06:03:55

No one is able to point at as failure case

seancorfield06:03:59

(let [p (iteration ...)]
  (seq p) ; calls (step! initk) to seed the computation
  (reduce ... p)) ; calls (step! initk) AGAIN to seed the computation
That's the clearest I can make this.

didibus06:03:08

Maybe I'm readin "hack" as it will have bugs

didibus06:03:25

Yea, I understand that

didibus06:03:35

But this assumes step! can be seeded

didibus06:03:41

for all following ks

didibus06:03:07

given an initial seed k, it assumes step! itself will return the same thing, and the same subsequent thing on all next ks it returns

dpsutton06:03:29

i don't think it assumes that. i thought you quoted the docstring in which it tells the caller to beware

didibus06:03:49

To me that sounds like step! must be idempotent, not just on the first call, but on the full series of call

dpsutton06:03:48

> > if step! with initk is unreproducible, it is on the consumer to not consume twice" that's saying if your api calls aren't idempotent then only issue them once

didibus06:03:07

And here's the proof:

user=> (def a (iteration foo :initk 0 :kf inc))
#'user/a
user=> (reduce (fn[acc e] (if (> acc 5) (reduced nil) (do (println e) (inc acc)))) 0 a)
0
0.77166746127496
0.07621011513209086
0.9841327041345114
0.821362141438259
0.21009693818295794
nil
user=> (reduce (fn[acc e] (if (> acc 5) (reduced nil) (do (println e) (inc acc)))) 0 a)
0
0.7984895604320498
0.5150573279942907
0.11688034595267827
0.9450752936010137
0.6539525668192183
nil

seancorfield06:03:35

That's not "proof" of anything.

didibus06:03:00

With foo being:

(defn foo [k]
     (if (= k 0)
       0
       (rand)))

didibus06:03:37

Well, here you have a step! which returns the same thing consistently for the same initk

didibus06:03:48

But not on following ks

didibus06:03:08

And so iteration will call step! over and over

seancorfield06:03:19

I give up. I'm going to bed.

dpsutton06:03:21

i have no idea what you're driving at

didibus06:03:09

Thanks anyway Sean, I appreciate. I know I'm being difficult, but I'm really not seeing it

didibus06:03:31

@dpsutton I don't know how far back you've read.

didibus06:03:09

But it all started with me wondering what issues iterate has when used with a side effecting f, against which its docstring warns not too use

didibus06:03:25

And the answer was that reduce will consume f over and over

didibus06:03:46

So f could be called more than once per element

didibus06:03:17

Ghadi said iteration is a new function, might release in 1.11, which is to handle that kind of scenario

didibus06:03:19

But, its docstring says that it will just keep calling f over and over everytime it is consumed

didibus06:03:41

Well, not exactly, but I read it as that

didibus06:03:52

And looking at its reduce implementation, to me, it seems to be the case

didibus06:03:28

seq will not

didibus06:03:42

Which is a little better than iterate

didibus06:03:38

So in my deep dive, iterate reduce calls the function over and over. And iterate seq will return Iterate, and not lazy-seq. So I get why in Iterate's case, it always calls f over and over

didibus06:03:49

But when looking at iteration, its only slightly better, in that reduce also just calls f over and over

didibus06:03:02

though seq won't, as it wraps things in a lazy-seq

didibus06:03:57

To make iterate safe with side effects, I found a call to map works, because map will never get from the Iterate more than once, as map itself will cache its results.

didibus06:03:11

But people tell me this is a hack, not sure why

didibus06:03:14

And with iteration, to make it that it consumes only once from f as well, I wrap it in a lazy-seq, so that reduce is on the lazy-seq now, which will use the iteration seq under the hood. And I'm told that is a hack as well

didibus06:03:10

Finally, I conclude that iterate and iteration are both pretty much safe with side effect if you are okay with f being consumed more than once possibly, as you access the same elements over and over

didibus06:03:29

And so the only difference is in the doc-string

didibus06:03:51

Well, minus that iteration is more safe I guess, due to the way it implements seq, which makes it reducing on the rest of it still win't call f again

didibus06:03:40

That said, I must be misunderstanding something, since very smart people told me I am wrong

didibus06:03:21

So Sean said I'm not understanding how iteration works. So I'm looking into that

didibus06:03:11

I think for iterate I am correct, just most people find wrapping it in a call to map feels hacky, and even though right now we might not think of an issue, its harder to reason about that it works in all scenario as well

didibus06:03:00

I'm looking at the code for iteration and trying it in the REPL, and I think I'm also correct. I'm not seeing what Sean is talking about. iteration seems to only be caching initk, not subsequent values or ks

didibus06:03:55

Which is fine, iteration basically leaves it up to the consumer to do whatever, if it wants to re-run the side effects multiple time, its up to them, and the function has good support for using it with paging APIs with idempotency keys.

didibus06:03:49

I don't know. I appreciate everyone's help and input. I did learn a lot. I think I understand things now, but I'll sleep on it all, since it seems some people say I still got a few things wrong, and maybe tomorrow it'll click for me

didibus07:03:40

For anyone interested, I was researching this rabbit whole for my answer here: https://clojureverse.org/t/is-this-a-good-way-to-stream-data-to-caller-of-a-function/5689/3?u=didibus which I hope captures all the learnings from this chat session

👍 4
ido08:03:13

hey, I have a clojure library that wraps around a java one. I am trying to diable logging during tests using timbre (which I only require in the :dev profile. I tried calling (timbre/merge-config! {:appenders {:spit {:enabled? false}}}) but I can still see some WARNING logging by the java library. The java code calls:

import java.util.logging.Level;
import java.util.logging.Logger;
...
logger.log(Level.WARNING, "blah", error);
what am I doing wrong?

rutledgepaulv13:03:50

Have you already added the shims listed here to route other java logging frameworks through timbre? https://github.com/fzakaria/slf4j-timbre#slf4j-timbre

ido15:03:44

trying this!

ido03:04:57

still not working: I have in lein deps :tree : • [com.fzakaria/slf4j-timbre “0.3.19” :scope “test”] • [org.slf4j/jul-to-slf4j “1.7.30” :scope “test”] • [org.slf4j/slf4j-api “1.7.30” :scope “test”] the underlying library uses java.util.logging but logs still appear on the screen.

rutledgepaulv04:04:23

the spit appender is for logging to files, not logging to stdout (what's appearing on the screen probably). You might need to do something different to turn stdout off.

rutledgepaulv05:04:04

try (timbre/merge-config! {:appenders {:println {:enabled? false}}})

restenb14:03:58

is there an easy way to convert nested hiccup-like structures like this [:key1 [:key2 "someval" [:key3 "otherval"]]]over to a nested map?

restenb14:03:00

i suppose not, the possible existence of values followed by new nested vectors is an issue

Spaceman14:03:36

Hi guys, I'm using the origami library to do image processing in clojure, but I'm puzzled by how the interop works.

Spaceman14:03:09

Here: https://www.codepasta.com/computer-vision/2019/04/26/background-segmentation-removal-with-opencv-take-2.html, there's a code segment

blurred_float = blurred.astype(np.float32) / 255.0
edgeDetector = cv2.ximgproc.createStructuredEdgeDetection("model.yml")
edges = edgeDetector.detectEdges(blurred_float) * 255.0
cv2.imwrite('edge-raw.jpg', edges)

Spaceman14:03:36

so far I have a blurred image: (-> "resources/public/img.png" (imread) (gaussian-blur! (new-size 17 17) 9 9) (imwrite "resources/blurred.png"))

Spaceman14:03:48

which works, but how to get the edgeDetector on it?

restenb14:03:03

@pshar10 does that help?

restenb14:03:16

assuming you're passing or storing cv2 somewhere, not familiar with what that is, or if ximgprocis a field or a nested class

restenb14:03:38

for nested classes you need to use ClassName$nestedClass as interop notation

Spaceman14:03:59

I don't think cv2 is required. Even in what I have so far above, I use imwrite without cv2, and it works with origami.

Spaceman14:03:33

But I have this now:
(let [blurred-float (-> "resources/public/Logo.png"
    (imread)
    (gaussian-blur! (new-size 17 17) 9 9)
    
)
            edgeDetector (-> ximgproc (.createStructuredEdgeDetection "model.yml"))
            edges (* 255 (.detectEdges edgeDetector blurred_float))]
        (imwrite edges "edge-raw.jpg" ))

Spaceman14:03:42

But it gives an error that ximgproc isn't defined

restenb14:03:43

right, if ximgprocis a class field of whatever cv2is, it needs dot-dash notation as in my snippet with .-ximgproc

restenb14:03:23

dealing with state/side-effect full java/JS interop like this in general assumes you're storing or passing whatever your stateful object is that you need to access fields & member functions on later

restenb14:03:43

judging by the Java example you posted that would be cv2in this case

Spaceman14:03:54

No, like in open cv one usually writes cv2.imwrite to write an image. Similarly, shouldn't it work with just ximgproc and not the cv2?

grounded_sage14:03:21

Is there a simple transducer I could use on a channel where it simply breaks apart a list and feeds the result to the consumer 1 by 1?

didibus17:03:12

You mean that you have a list inside a channel, and you want to like flatten it?

didibus17:03:25

I think you could look into cat and mapcat, they might be able to do what you're looking for

didibus17:03:57

Possibly you could use onto-chan as well, though that's not a transducer

restenb14:03:20

i mean. that Clojure snippet you posted has several errors in it. If you intend to access an instance member named ximgproc, you must use either (.or (.-notation, the latter is preferred for fields

restenb15:03:15

as I'm not familiar with this library I don't know what to expect from a call like (imread "logo.png")for example, what does that return?

restenb15:03:38

something like that should work if (imread "logo.png")already returns the object you need for the rest of the chain

noisesmith15:03:38

@pshar10 in a repl you can use javadoc (from clojure.java.javadoc ns, always present in the initially loaded repl ns) - when you pass it some object it will usually find the documentation for that object in your web browser

noisesmith15:03:50

this can help with code that does so much interop

noisesmith15:03:29

eg. (javadoc (gaussian-blur! (imread "logo.png"))) to figure out what precisely that object is

noisesmith15:03:34

and what you can do with it

Spaceman15:03:02

I found it to work with

(import '[org.opencv.ximgproc Ximgproc])
(let [blurred-float (-> "resources/public/Logo.png"
    (imread)
    (gaussian-blur! (new-size 17 17) 9 9)

)
            edgeDetector (Ximgproc/createStructuredEdgeDetection "resources/model.yml")
            edges (* 255 (.detectEdges edgeDetector blurred-float))]
        (imwrite edges "edge-raw.jpg" ))

noisesmith15:03:27

OK - that looks reasonable :D

Spaceman15:03:33

But I get No matching method detectEdges found taking 1 args for class org.opencv.ximgproc.StructuredEdgeDetection

noisesmith15:03:05

then (javadoc org.opencv.ximgproc.StructuredEdgeDetection) and figure out what args that method takes

noisesmith15:03:43

clojure doesn't try much "clever" stuff about guessing method argument types, you likely need a hint to make it pick the method you need

noisesmith15:03:16

quick google claims that you need both an input array (what you have now) and an output array

noisesmith15:03:30

(at least in the C++ version...)

noisesmith15:03:57

yeah it's true in java too

Spaceman15:03:18

How do I create an output array? (. WhatObject?)

noisesmith15:03:30

you can click on the type in the doc

noisesmith15:03:54

that should mention a constructor or a factory method

noisesmith15:03:14

yeah, it has a zero arg constructor, that's probably enough(?)

Spaceman15:03:28

edges (* 255 (.detectEdges edgeDetector blurred-float (. Mat))), gives: Malformed member expression, expecting (. target member ...)

noisesmith15:03:02

the zero arg constructor for Mat would be (Mat.)

noisesmith15:03:39

you might want to refresh yourself on the interop syntax doc - it's actually simpler and more consistent than java's own syntax :D https://clojure.org/reference/java_interop

Spaceman15:03:33

For some reason, multiplying 255 doesn't work in clojure:

(let [blurred-float (-> "resources/public/Logo.png"
    (imread)
    (gaussian-blur! (new-size 17 17) 9 9)
    (float)
    
)
            edgeDetector (Ximgproc/createStructuredEdgeDetection "resources/model.yml")
            mat (Mat.)
            _ (.detectEdges edgeDetector blurred-float mat)
            edges (* (float 255) mat)
            ]
        (imwrite edges "edge-raw.jpg" ))

Spaceman15:03:45

gives: Execution error (ClassCastException) at user/eval63001 (form-init8488930107953136676.clj:373). org.opencv.core.Mat cannot be cast to java.lang.Number

restenb15:03:27

why would that work anywhere? you're trying to multiply a class by a number

restenb15:03:55

probably you want to multiply the result of the detectEdgescall by 255 instead

noisesmith15:03:20

@pshar10 are you trying to do a matrix multiply? mat is a matrix

noisesmith15:03:44

detectEdges returns nil

noisesmith15:03:16

in clojure and afaik in java as well, * doesn't handle matrix multiplies - you need a method for that

restenb15:03:22

the original example he posted does this edges = edgeDetector.detectEdges(blurred_float) * 255.0

noisesmith15:03:42

OK... that doesn't match the detectEdges method I see

noisesmith15:03:57

it returns nil, and that would blow up

Spaceman15:03:37

My bad, it was due to the cast (float) above.

noisesmith15:03:53

wait, it works without the cast? it shouldn't

Spaceman15:03:07

I removed it but then:

(let [blurred-float (-> "resources/public/Logo.png"
    (imread)
    (gaussian-blur! (new-size 17 17) 9 9)

    
)
            edgeDetector (Ximgproc/createStructuredEdgeDetection "resources/model.yml")
            mat (Mat.)
            _ (.detectEdges edgeDetector blurred-float mat)
            edges (.mul mat 255)
            ]
        (imwrite edges "edge-raw.jpg" )
        )

noisesmith15:03:17

OK .mul loks right

Spaceman15:03:22

(-215:Assertion failed) src.type() == CV32FC3 in function 'detectEdges'

Spaceman15:03:21

This means that the blurred-float matrix isn't the write type

noisesmith15:03:26

you can google that type name to see what it wants - this API looks tedious

noisesmith15:03:44

it's a different kind of matrix it wants, and it's complaining about the src not the dest (unless I'm misreading that cryptic message)

noisesmith15:03:07

also .mul wants a double, so use 255.0 - I don't think clojure's clever enough to do that for you(?)

noisesmith15:03:27

oh, and it wants a second matrix - it's not going to work with just a number

Spaceman15:03:28

Still the same error after giving in a second matrix and 255.0

(let [blurred (-> "resources/public/Logo.png"
                  (imread)
                  (gaussian-blur! (new-size 17 17) 9 9))
      blurred-float (Mat.)
      _ (.convertTo blurred blurred-float 5) ;; five is just the f32 type id
      edgeDetector (Ximgproc/createStructuredEdgeDetection "resources/model.yml")
      mat (Mat.)
      _ (.detectEdges edgeDetector blurred-float mat)
      edges (Mat.)
      _ (.mul mat 255.0 edges)
      ]
  (imwrite edges "edge-raw.jpg" )
  )

Spaceman15:03:34

Execution error (ClassCastException) at user/eval65868 (form-init5691178651358499492.clj:101). java.lang.Long cannot be cast to org.opencv.core.Mat

Spaceman15:03:57

And sometimes the whole repl closes unexpectedly when loading this

noisesmith15:03:15

look at the method signature: the second matrix comes before the double

noisesmith15:03:36

crashing: yeah, c interop will do that

noisesmith15:03:56

also, .mul returns a new matrix, and multiplies two matrixes (and optionally a double as well) - you might want something different, like an imperative loop that scales every value in the array(?)

noisesmith15:03:45

eg. a loop calling the .get and .put methods on each index as I don't see a simple scale operation in that API (I'm no expert on this specific API though, I just know how to read javadoc)

Spaceman15:03:59

.mul doesn't take two matrices, only one

noisesmith15:03:12

it operates on your matrix, plus another

noisesmith15:03:15

you are misreading the doc

Spaceman15:03:15

but it crashes the repl

noisesmith15:03:42

mul is a method on a Mat object, so the one in the arg list is another Mat

noisesmith15:03:47

you need two

bfabry15:03:03

mul takes the instance object (to the left of the method call in java, to the right of it in clojure), and then a second Mat object as an argument

noisesmith15:03:05

unless there is a version of mul without a Mat in the arglist

Spaceman16:03:21

Oh I see, (.mul mat edges 255.0) is like mat.mul(edges, 255.0). But why doesn't it work?

bfabry16:03:06

are you sure it doesn't work? in your snippet above you had the arguments in a different order

noisesmith16:03:25

well, I'd expect mul to do a matrix multiply, one of the two matrixes seems to be uninitialized - which gives you either zeroes or garbage

Spaceman16:03:23

(let [blurred (-> "resources/public/Logo.png"
                  (imread)
                  (gaussian-blur! (new-size 17 17) 9 9))
      blurred-float (Mat.)
      _ (.convertTo blurred blurred-float 5) ;; five is just the f32 type id
      edgeDetector (Ximgproc/createStructuredEdgeDetection "resources/model.yml")
      mat (Mat.)
      _ (.detectEdges edgeDetector blurred-float mat)
      edges (Mat.)
      _ (.mul mat edges 255.0)
      ]
  (imwrite edges "edge-raw.jpg" )
  )

Spaceman16:03:34

cv::Exception: OpenCV(4.2.0) /Users/niko/origami-land/opencv-build/opencv/modules/core/src/arithm.cpp:669: error: (-209:Sizes of input arguments do not match) The operation is neither 'array op array' (where arrays have the same size and the same number of channels), nor 'array op scalar', nor 'scalar op array' in function 'arithm_op'

noisesmith16:03:13

so you probably need to provide dimension args to the Mat constructor to make it match your input

noisesmith16:03:32

at least that's an intelligible error :D

Spaceman16:03:31

Here you go: This gives an output for a test input .png image, which is completely black and doesn't have the edges. And the repl closes if I give it a jpg input. No errors. Just closes:

(let [blurred (-> "resources/public/test.jpg"
                  (imread)
                  (gaussian-blur! (new-size 17 17) 9 9))
      blurred-size (.size blurred)
      blurred-float (Mat. blurred-size 5)
      _ (.convertTo blurred blurred-float 5) ;; five is just the f32 type id
      edgeDetector (Ximgproc/createStructuredEdgeDetection "resources/model.yml")
      mat (Mat. blurred-size 5) ;; five is just the f32 type id
      _ (.detectEdges edgeDetector blurred-float mat)
      edges (Mat. blurred-size 5) ;; five is just the f32 type id
      _ (.mul mat edges 255.0)
      ]
  (prn "blurred size" blurred-size)
  (imwrite edges "edge-raw1.jpg" )
  )

Spaceman16:03:39

Seems like the repl crashes when the image is big regardless of the image type

noisesmith16:03:22

I would expect a multiply by an uninitialized matrix to give zeroes, and I'd expect zero to be black pixels

noisesmith16:03:02

edges probably needs a different content - and maybe there's some trick I couldn't find in the docs to just do a normal multiply

noisesmith16:03:29

eg. edges could be the identity matrix, or the matrix that scales its input by 255

Spaceman16:03:32

Okay, so edges ( mat 255.0) in let should work because is overloaded in c++: https://stackoverflow.com/questions/17892840/opencv-multiply-scalar-and-matrix#17894118

bfabry16:03:56

java and clojure do not support operator overloading

bfabry16:03:42

so while the C++ implementation underneath that works, the java library wrapper you're using it doesn't. hence the existence of mul etc

Spaceman16:03:25

So there's no simple way but to multiply with the identity matrix?

bfabry16:03:39

seems that's a limitation of that library yeah

bfabry16:03:31

there's a Mat#clone function and a Mat#setTo(Scalar) function that should let you create an identity matrix relatively easily

Spaceman16:03:31

There's a static function Mat::eye for that, but how do I use it? For using Mat I was doing (import '[org.opencv.core Mat])

bfabry16:03:03

you call a static function like (Mat/eye rows cols type)

Spaceman16:03:07

I guessed

(let [ 
      img (-> "resources/public/Logo.png"
              (imread))
      size (.size img)
      blurred (Mat. size 5)
      _ (gaussian-blur img blurred (new-size 17 17) 9 9)
      blurred-float (Mat. size 5)
       _ (.convertTo blurred blurred-float 5) ;; five is just the f32 type id
      edgeDetector (Ximgproc/createStructuredEdgeDetection "resources/model.yml")
      mat (Mat. size 5)
      _ (.detectEdges edgeDetector blurred-float mat)
      identity- (Mat/eye size 5)
      edges (.mul mat identity- 255.0)
      ]
  (prn "blurred size" size)
;;  (imwrite edges "edge-raw1.jpg" )
  )

Spaceman16:03:17

But it crashed the repl

bfabry16:03:52

what is the type of (.size img), because it expects an opencv Scalar

bfabry16:03:20

you're trying to do too much at once, execute each of those pieces of code one at a time and capture their output in a def

4
noisesmith17:03:15

yeah - clojure rewards iterative incremental changes

noisesmith17:03:40

keep your design in mind so you don't make a total mess, but do repeated minimal changes

noisesmith17:03:08

and just remove everything after the error, add things back one by one and make the individual things work

Spaceman17:03:07

The error is that eye is a static function in class Mat. I think my syntax for calling static functions is wrong

Spaceman17:03:17

like printing the value just before it works. mat prints: #object[org.opencv.core.Mat 0x26c3d186 "Mat [ 1024*4096*CV_32FC1, isCont=true, isSubmat=false, nativeObj=0x7ff133f34e90, dataAddr=0x1370aa000 ]"]

Spaceman17:03:16

ok (. Mat eye size 5) got rid of the error and gave me an output, but it's completely black again

Spaceman17:03:12

Still, using a large image crashes the repl.

noisesmith17:03:02

pedantic point: it's a static method, functions are Objects with invoke and applyTo methods implementing Callable and Runnable

noisesmith17:03:26

(Mat/eye size 5) should work if it's a static method

noisesmith17:03:36

(the difference between method and function is less important in languages where you only use one or the other, in Clojure we use both and the differences are important)

Spaceman17:03:54

Well, anyway, even for smaller images when it isn't crashing, the result isn't completely black. The result, i.e., edges seems to be identity * 255, from doing (.mul mat identity- 255.0)

Spaceman17:03:29

as if both mat and identity- are identity matrices

Spaceman17:03:15

(.mul mat identity- 255.0)

dpsutton17:03:09

this should probably be in a thread at this point. its getting a bit long

p-himik17:03:51

IMO even just a single reply should go into a thread. Not least because you can never guarantee that it won't start a long discussion.

🤝 4
Spaceman17:03:23

anyway, to debug this, it's probably a good idea to convert the Mat object to a clojure vector

Spaceman17:03:38

how can I do that?

noisesmith17:03:33

you can use one of the .get methods to get an array out from the Mat object, and the use into to put an array into a clojure vector

noisesmith17:03:03

you can also use clojure's get on arrays directly, but vectors are definitely nicer - they print readably for example

Spaceman17:03:12

so (.get mat what?)

noisesmith17:03:34

you could iterate it across the other dimension

noisesmith17:03:09

hmm - I mean this one since it doesn't anchor nicely:

public double[] get​(int[] idx)

noisesmith17:03:34

actually maybe this one

public double[] get​(int row, int col)

Spaceman17:03:23

For

(let [img (-> "resources/public/test-small.jpg"
              (imread))
      size (.size img)
      height (.height size)
      width (.width size)
      blurred (Mat. size 5)
      _ (gaussian-blur img blurred (new-size 17 17) 9 9)
      blurred-float (Mat. size 5)
      _ (.convertTo blurred blurred-float 5) ;; five is just the f32 type id
      edgeDetector (Ximgproc/createStructuredEdgeDetection "resources/model.yml")
      mat (Mat. size 5)

      arr (.get mat width height)
      _ (.detectEdges edgeDetector blurred-float mat)
      identity- (. Mat eye size 5)
      edges (.mul mat identity- 255.0)
      ]
  (prn "blurred size" size)
  (prn "mat is " mat)
  (prn "height is " height)
  (prn "width is " width)
  (imwrite blurred-float "blurred-float.jpg")
  (imwrite mat "mat.jpg")
  (imwrite identity- "identity.jpg")
  (imwrite edges "edge-raw1.jpg" )
  )
I get: No matching method get found taking 2 args for class org.opencv.core.Mat

noisesmith17:03:58

try (int width) (int height) - maybe clojure isn't doing the right auto-coercion

noisesmith17:03:27

also you might find it easier to create individual objects in the repl and experiment, instead of reloading the entire function

💯 4
Spaceman18:03:38

okay

(def img (-> "resources/public/test-small.jpg" (cv/imread)))
(def size (.size img))
(def height (.height size))
(def width (.width size))
(def blurred (Mat. size 5))
(cv/gaussian-blur img blurred (cv/new-size 17 17) 9 9)
(def blurred-float (Mat. size 5))
(.convertTo blurred blurred-float 5) ;; five is just the f32 type id
(def edge-detector (Ximgproc/createStructuredEdgeDetection "resources/model.yml"))
(def mat (Mat. size 5))

(.detectEdges edge-detector blurred-float mat)
(def identity- (. Mat eye size 5))
(def edges (.mul mat identity- 255.0))

(def arr (.get mat (int width) (int height))) ;;gives nil

Spaceman18:03:41

arr just gives nil

Spaceman18:03:31

It's supposed to return an array

Spaceman18:03:55

where as mat gives [ 302*302*CV_32FC1, isCont=true, isSubmat=false, nativeObj=0x7fc9d9eba720, dataAddr=0x12ec1c000 ]

noisesmith18:03:39

perhaps you want (dec width) and (dec height) because I think that it's 0 indexed and you are getting the array of rgb doubles at that index

Spaceman18:03:37

dec gives: No matching method get found taking 2 args for class org.opencv.core.Mat

noisesmith18:03:52

right, you're back to the hinting (int (dec width)) etc.

Spaceman18:03:02

arr is [0.0]

noisesmith18:03:47

OK - you can do this in a loop to get all of these single item vectors of doubles, that would be one approach

Spaceman18:03:32

like a nested loop?

Spaceman18:03:50

but how to do it in clojure?

Spaceman18:03:41

like I can't do (nth mat x)?

Spaceman18:03:43

oh I see I think you mean doing for all x y (.get mat x y)

Spaceman18:03:25

Doing (doseq [x (range width) y (range height)] (prn (.get mat (int x) (int y))) ) prints like a lot of these, my guess is 320x320 many: #object["[D" 0x2a0db4b8 "[[email protected]"]

noisesmith18:03:49

so maybe something like:

(into []
      (for [x (range width)]
        (into []
              (for [y (range height)]
                (into []
                      (.get mat (int x) (int y)))))))

Spaceman18:03:28

How to multiply each item with 255, which I ultimately want to do? I tried this:

(into []
      (for [x (range width)]
        (into []
              (for [y (range height)]
                (into []
                      (* 255.0 (get (.get mat (int x) (int y)) 0)))))))
but it says Don't know how to create ISeq from: java.lang.Double

bfabry19:03:56

you might find dragan's clojure numerical algebra book interesting https://aiprobook.com/numerical-linear-algebra-for-programmers/

Spaceman19:03:20

This works: (into [] (for [x (range width)] (into [] (for [y (range height)] [(* 255.0 (get (.get mat (int x) (int y)) 0))] ))))

Spaceman19:03:18

are there any good libraries to convert clojure vectors into images?

Spaceman19:03:53

I probably can use .put somehow to convert it back into a Mat.

Spaceman20:03:49

I have the following code for it:

(into []
            (for [x (range width)]
              (into []
                    (for [y (range height)]
                      (do
                        (.put edges  (int x) (int y) (* 255.0 (get (.get mat (int x) (int y)) 0)))
                        [(* 255.0 (get (.get mat (int x) (int y)) 0))]
                        )
                      ))))
But I get the error:
No matching method put found taking 3 args for class org.opencv.core.Mat
But Mat.put documentation says that this method exists:
put​(int row, int col, double... data)
Then what am I doing wrong?

p-himik17:03:51

IMO even just a single reply should go into a thread. Not least because you can never guarantee that it won't start a long discussion.

🤝 4
Darin Douglass18:03:06

so it seems metadata is properly handled when reading in EDN files:

Clojure 1.10.1
user=> (clojure.edn/read-string "^{:params {:a 1}} {:jack :jill}")
{:jack :jill}
user=> (meta *1)
{:params {:a 1}}
however, i cannot seem to find any reference to metadata in the edn spec (https://github.com/edn-format/edn). is the documentation lacking or is this just a happy little accident?

👀 4
Darin Douglass18:03:57

this response seems to indicate it's not an accident (at least in clojure): https://github.com/edn-format/edn/issues/52#issuecomment-24337627

Spaceman19:03:55

are there any good libraries to convert clojure vectors into images?

Spaceman19:03:47

Suppose I have a vector like so:

[
[[50][60][70][20][0]...]
[[90][56][67][98][78]...]
...]
Which represents a single channel 8-bit image. Is there a simple way to convert this into a jpg or a png?

Alex Miller (Clojure team)19:03:09

java2d is in the jdk and has support for stuff like this, but it's quite a bit more complicated iirc

Alex Miller (Clojure team)19:03:40

I'm sure simpler Java apis probably exist and that's where I'd look, rather than in Clojure

Alex Miller (Clojure team)19:03:03

here's a really simple example I wrote in Java 13 years ago when I was doing a mandelbrot thing https://puredanger.github.io/tech.puredanger.com/2007/10/12/images-java2d/

Spaceman19:03:24

how about opencv? Is there a way to convert a vector to an opencv Mat?

didibus20:03:46

You can also use JOGL and LWJGL for 3D, and I think they support 3D to 2D projection, which can be faster than Java2D due to leveraging the GPU

Spaceman19:03:46

how would the java interop work in this case?

Alex Miller (Clojure team)19:03:33

you've rapidly crossed my line of knowledge :)

aarkerio19:03:13

Hi! what is the "standard" solution to keep a map ordered?

bfabry19:03:09

sorted-map

andy.fingerhut19:03:08

depends on what order you want. Sorted by keys via a comparison function, or insertion order of keys?

Spaceman20:03:23

I'm trying to read a Mat in open cv, multiply each of the components with 255 and save it as a Mat. I have the following code for it:

(into []
            (for [x (range width)]
              (into []
                    (for [y (range height)]
                      (do
                        (.put edges  (int x) (int y) (* 255.0 (get (.get mat (int x) (int y)) 0)))
                        [(* 255.0 (get (.get mat (int x) (int y)) 0))]
                        )
                      ))))
But I get the error:
No matching method put found taking 3 args for class org.opencv.core.Mat
But Mat.put documentation says that this method exists:
put​(int row, int col, double... data)
Then what am I doing wrong?

dpsutton20:03:56

put is a method on a matrix. in java that means mat.put(row, col, double) and in clojure it means (.put mat row col double)

Spaceman20:03:45

edges is defined as: (def edges (Mat. size 5))

souenzzo20:03:54

TIP: Don't use for to imperative things Prefer doseq or dotimes Use a reduce if you need to return something