Fork me on GitHub
#clojure
<
2020-03-31
>
didibus00:03:15

Don't think I'm as enthusiastic about unfold. An iterate which support for a side-effectinf f seems more generally useful. Its easy to combine with a take-while when you care.

didibus00:03:40

But in any case, I guess it would solve my use case

didibus00:03:55

So I'd still be for it

didibus00:03:22

I was actually surprised iterate isn't implemented in terms of lazy-seq

didibus00:03:27

I guess I'm just not seeing the point of the non-caching behavior with reduce. What scenario would that be needed for ?

didibus00:03:46

Oh, seems wrapping the iterate in lazy-seq works šŸ˜„

didibus00:03:00

(lazy-seq (iterate f))

didibus00:03:50

I'll just do that for now

didibus00:03:36

Unless anyone else sees an issue with that as well ?

hiredman00:03:04

Yes, it is no good, stop trying to "trick" whatever

didibus00:03:17

(defn fetch
  "A stub fetch that pretends like its getting random data from a url, but instead return a rand char from url."
  [url]
  (println "fetching " url)
  (rand-nth url))

(defn lazy-fetch-until-data-not-seen [fetching-fn]
  "Returns a lazy-seq which on every getting of the next element, will fetch the next element by calling
   fetching-fn. Will stop once it sees an element is returned by fetching-fn that has already been fetched before."
  (->> (iterate
         (fn [[_ page-fetching-fn pages-fetched]]
           (let [page (page-fetching-fn)]
             (if (pages-fetched page)
               [:done nil nil]
               [page page-fetching-fn (conj pages-fetched page)])))
         [nil fetching-fn #{}])
       (lazy-seq)
       (map first)
       (drop 1)
       (take-while #(not= :done %))))

(def data (lazy-fetch-until-data-not-seen #(fetch "")))
(mapv upper-case data)
;> fetching  
;> fetching  
;> fetching  
;> fetching  
;> fetching  
;> fetching  
;> fetching  
;;=> ["G" "W" "." "H" "O" "T"]
(mapv upper-case data)
;;=> ["G" "W" "." "H" "O" "T"]
(mapv upper-case data)
;;=> ["G" "W" "." "H" "O" "T"]
(mapv upper-case data)
;;=> ["G" "W" "." "H" "O" "T"]
(reduce (fn[acc e] (conj acc e)) [] data)
;;=> [\g \w \. \h \o \t]
(reduce (fn[acc e] (conj acc e)) [] data)
;;=> [\g \w \. \h \o \t]
(reduce (fn[acc e] (conj acc e)) [] data)
;;=> [\g \w \. \h \o \t]
(reduce (fn[acc e] (conj acc e)) [] data)
;;=> [\g \w \. \h \o \t]

hiredman00:03:18

Just write yourself unfold

didibus00:03:31

I mean it looks pretty sound to me

didibus00:03:12

lazy-seq properly wraps around Iterate, so reduce is now done over the lazy-seq

hiredman00:03:51

No, you don't actually understand how lazy seqs work

hiredman00:03:17

That would behave the same with or without your lazy-seq wrapping

didibus00:03:26

Well, ya I know that

didibus00:03:56

But just to be sure I add it there, just to remind myself I need to make sure something does, or I expose the Iterate to the caller

hiredman00:03:23

The reason the fetches only happen once is because the lazy seqs you are building on top of iterate (with take, map, etc) are caching

didibus00:03:13

Ya, and non caching reduce is why f has to be side effect free over an Iterate

didibus00:03:31

But, if you wrap it in a lazy-seq, then you're good to go

didibus00:03:49

But, maybe I'm still not seeing all edge cases, I admit

didibus00:03:10

But right now, this actually seems to all make sense to me, and seems safe

didibus00:03:33

You lose the performance benefit of iterate's reduce impl, but it is not what I'm looking for here

ghadi00:03:52

feel free to use the code above from CLJ-2555, it's meant for iterated consumption of APIs

didibus00:03:49

Let me have a look

ghadi01:03:03

the code above using iterate is not clear, and @hiredman is right that the lazy-seq usage is wrong

didibus01:03:21

What's wrong with it?

didibus01:03:24

And what's not clear about it? I'd use a similar pattern for generating fibs or anything else side effect free as well? Am I using it wrong?

ghadi01:03:32

for one, it shouldn't be side-effecting because the docstring says not to be

didibus01:03:02

Haha, well, that

didibus01:03:11

is my whole reason for being here

didibus01:03:33

Having now understood why the doc-string says so, it seems wrapping in lazy-seq solves the issues

ghadi01:03:38

passing a named tuple of state is awkward and hard to understand

potetm01:03:15

I mean, isnā€™t this kind of ridiculous? Itā€™s not fundamentally different than having a FSM with a terminal state:

{:state :done}

ghadi01:03:38

i meant to say unnamed

ghadi01:03:55

s/named/unnamed

potetm01:03:00

ah, so youā€™re saying a map is preferable here?

ghadi01:03:29

sure information is better than syntax

facepalm 4
potetm01:03:08

genuinely sounds like something Elements of Fauxjure would spit out

potetm01:03:28

if youā€™re not familiar

ghadi01:03:22

it predates clojure

ghadi01:03:44

ruby community used to call it "connascence of position"

ghadi01:03:01

which sounds way more Fauxjure than Fauxjure :)

potetm01:03:34

Iā€™m not saying the underlying concept is not real. Positional semantics have been with us forever. Iā€™m saying itā€™s a ridiculous way to frame the issue (and itā€™s not always true) šŸ™‚ ā€œSyntax is complecting,ā€ is at least more accurate.

ghadi01:03:16

signaling termination is awkward [:done nil nil]

ghadi01:03:59

there's a whole history of linked stuff from CLJ-2555 and CLJ-1906 (the latter especially) that is worth reading

didibus01:03:18

Right, I can see that. I guess I just mean, that's just how iterates work no? Like even for side effect free, you'll be left with code like I have, so its nothing special, just maybe iterate produces ugly code

ghadi01:03:48

iterate is beautiful, you're using it off-label

didibus01:03:58

This is the most used example of iterate: (def fib (map first (iterate (fn [[a b]] [b (+' a b)]) [0 1])))

didibus01:03:13

You have named tuple for carried state

didibus01:03:24

And then a call to map to remove the state tuple

ghadi01:03:27

(take 10 (iterate #(.plusDays % 1) (java.time.LocalDate/now)))

noisesmith01:03:34

I think part of the problem here is a mismatch between lazy data oriented evaluation via laziness and procedural evaluation where side-effects matter - that's pretty fundamental

ghadi01:03:49

again, fib is pure

didibus01:03:01

Ok, but ignoring the purity, its just as ugly

didibus01:03:06

and unreadable

ghadi01:03:18

not sure why you're saying ignore the purity

ghadi01:03:40

the purity is what enabled us to vastly improve its implementation in clojure 1.7.0

didibus01:03:49

I mean you said: > passing a named tuple of state is awkward and hard to understand > signaling termination is awkward [:done nil nil]

Alex Miller (Clojure team)01:03:01

the "no side effects" in the doc pre-dated the 1.7 impl

didibus01:03:04

Those apply to side effect free iterate implementations as well

Alex Miller (Clojure team)01:03:08

it has always said that

didibus01:03:22

So I mean, the only thing different I do to handle side-effecting f is to make sure I return a lazy-seq and not an instance of Iterate

didibus01:03:41

Which just calling map does, but I wanted to be more explicit

didibus01:03:00

I'm not saying it won't be great to have a different fn with maybe a more ergonomic interface, I'm still trying to wrap my head around the iteration fn from CLJ-2555, so maybe once I get how to use it, I'll be fully sold. But I'm also not seeing the issue with what I'm doing with iterate right now, seems all concerns around f being side effect free are irrelevant if you make sure you don't leak the clojure.lang.Iterate

ghadi01:03:33

the iterate fn arg you have written is complected (scratch off the bingo card) - it's handling production (calling a function) & termination

noisesmith01:03:44

> the only thing different I do to handle side-effecting f is to make sure I return a lazy-seq and not an instance of Iterate if you don't use threads, maybe

didibus01:03:38

Hum, I guess that's a general question related to lazy-seq and their caching being thread safe or not

didibus01:03:45

I think they are

didibus01:03:53

Like what is handled by repeatedly

noisesmith01:03:43

it's specifically not safe to call iterate with side effects in multiple threads, an effect could happen twice

noisesmith01:03:16

the fact that this was already a known limitation is why the implementation of reduce which ignores the iterate cache was implemented

didibus01:03:57

Yes, iterate on its own

didibus01:03:02

But I'm wrapping it in a lazy-seq

didibus01:03:17

And I believe once wrapped, it is now safe, and thus it is fine to have f be side effecting

noisesmith01:03:38

that doesn't change the thread safety aspect, iterate will recalculate in a way that other lazy data sources do not

noisesmith01:03:48

that belief is incorrect

didibus01:03:02

Except it won't be able too, because lazy-seq is now in charge, and it will be caching

didibus01:03:30

So unless lazy-seq is not thread safe, I still see no issue

noisesmith01:03:38

lazy-seq is out of the picture after one value is consumed

noisesmith01:03:29

(ins)user=> (type (rest (lazy-seq (list 1 2 3))))
clojure.lang.PersistentList
(ins)user=> (type (rest (lazy-seq (iterate inc 0))))
clojure.lang.Iterate

noisesmith01:03:43

lazy-seq only wraps the outer collection, it is not recursive

noisesmith01:03:10

that suffices for a hack that tricks reduce, it doesn't help with thread safety

didibus01:03:12

Hum... so why does it work

didibus01:03:31

See my example above, f is never recalculated

noisesmith01:03:32

reduce only checks the type of the outer collection

didibus01:03:40

šŸ˜›, so good enough then

noisesmith01:03:47

in your use case maybe, but it hasn't been made thread safe

didibus01:03:00

But ok, that's a fair call out

didibus01:03:15

Except, if you look at my function

didibus01:03:32

Witht he call to map? would that not change things?

didibus01:03:01

I think map might be continuously wrapping things in a lazy-seq, as it handles its own call to rest

didibus01:03:11

Maybe thats what makes it work in fact

didibus01:03:11

Anyway, you had a convincing argument. So if map does make it work, it does still become more brittle, and less general. Like always remembering to be sure it is someone properly wrapped even on call to rest becomes tricky

ghadi01:03:46

termination concerns are also leaking to the outside

didibus01:03:40

I mean, leaking outside my own use of iterate

ghadi01:03:31

its primary argument is a function called step! which is presumed to be side-effecting

ghadi01:03:18

step! takes one argument, a "key"

ghadi01:03:26

a good example of step! is calling an API like S3 ListObjects or anything paginated

ghadi01:03:20

step! gets called with an initial key. if you don't override it, the initial key is nil

ghadi01:03:36

this is your initial call to S3 ListObjects

ghadi01:03:53

what step! returns is some HTTP Response

potetm01:03:25

fwiw - I would prefer an explicit loop/recur here.

ghadi01:03:48

yes but those are eager, can't use a terminating transducer

potetm01:03:07

Feed it w/ lazy data?

ghadi01:03:17

one sec and it will be clear

potetm01:03:45

What Iā€™m saying is: I would not want it to be lazy.

potetm01:03:09

Side effects are much easier to reason about when you can control when they happen.

potetm01:03:14

Thatā€™s a massive value.

ghadi01:03:22

you can just as well use it eagerly

ghadi01:03:08

bear with me because this is a complicated function

ghadi01:03:30

HTTP Responses from these sorts of paginated APIs contain 2 important bits of information: 1) an argument that you can pass for the next call to step! 2) some data

ghadi01:03:57

(In the terminology of that function's docstring, the HTTP Response is called a "ret", the argument that you pass to the next call to step! is "k" and the data is called values)

didibus01:03:39

Seems its very much designed for typical pagination, can it be used more generically?

ghadi01:03:41

to get from the HTTP Response to the next k for step!, there is an argument :kf, which is a function that gets called on the HTTP Response ("ret")

ghadi01:03:12

to extract some values from the HTTP Response, there's an argument called :vf

ghadi01:03:17

"key function" and "value function"

didibus01:03:44

Ah, I kept wondering why they were called vf and kf

ghadi01:03:56

there's one more piece, but just to recap: step! is a function of "k" and return a "ret"

ghadi01:03:07

kf is a function of ret and extracts the next k

ghadi01:03:20

vf is a function of ret and extract the value

ghadi01:03:35

the last piece is the argument :some?

ghadi01:03:18

which is a function of the "ret", and it tells the iteration process whether this "page" (HTTP Response) had anything in it

ghadi01:03:32

some? allows you to abort early

ghadi01:03:50

kf - if it doesn't produce a key, also aborts early

ghadi01:03:43

so the flow is: get a page by calling step on k call some?, abort if falsey call vf to get some values out call kf to get the next k, abort if no k ----loop to top

didibus01:03:07

So in my case, it seems I would still need to hack around it, and have ret be a tuple of state, and k won't be a key, but carrying over all previously returned piece of data no?

ghadi01:03:35

let me give an actual example from AWS that consumes through pages of results

didibus01:03:50

Since I'm doing, fetch data until you receive data which was previously returned by prior calls to fetch

potetm01:03:05

Honestly: Itā€™s like 5 lines of code šŸ˜„ Just use lazy-seq if you actually need lazy?

didibus01:03:09

This is not as simple as AWS pagination

potetm01:03:55

reduce optimizations are nice, but perhaps not necessary?

didibus01:03:04

@potetm I know, but I find lazy-seq to be even less ergonomic šŸ˜›, so I was trying to get iterate working, but then it can't handle side effects, and now there is iteration, which might seem like its not generic enough, so maybe I'll just go back to lazy-seq

potetm01:03:23

yeah lazy-seq is the standard tool here

ghadi01:03:25

(defn log-groups
  "returns all log groups from CloudWatch"
  [client]
  (->> (iteration
        (fn [token]
          (aws/invoke client (cond-> {:op :DescribeLogGroups}
                               token (assoc :request {:nextToken token}))))
        :kf :nextToken
        :vf :logGroups)
       (into [] cat)))

didibus01:03:37

Not throwing iteration under the bus, I still need to understand it

didibus01:03:09

So in my case, to know if I'm done, I need to look at all previously fetched data

didibus01:03:21

The API does not tell me I am done

potetm01:03:45

If thatā€™s the case, you canā€™t use laziness, no?

potetm01:03:16

Like, you might as well loop/recur or atom bash in a doseq if you gotta check the full result set each time.

Alex Miller (Clojure team)01:03:34

my general thumb with laziness is if you care how much of the lazy thing gets read when, you shouldn't use a lazy seq for it

potetm01:03:01

ghadiā€™s proposal uses it under the hood, or do I misread?

ghadi01:03:37

Rich wrote it, I am stewarding it

Alex Miller (Clojure team)01:03:43

it's kind of a special case

ghadi01:03:55

but iteration has a seq side, and a reduce side

potetm01:03:02

how so? still doesnā€™t control ā€œhow muchā€ you get, does it?

ghadi01:03:11

it reifies both Seqable and IReduceInit

potetm01:03:11

unless you use the reduce side

Alex Miller (Clojure team)01:03:16

but you should always be prepared to read 32 items ahead or whatever

potetm01:03:28

right, always the case

potetm01:03:38

(unless you go no-lazy)

Alex Miller (Clojure team)01:03:47

or be careful in how you consume it, which you probably are with something like this

ghadi01:03:49

take doesn't do the chunking

ghadi01:03:00

(32-width chunking)

didibus01:03:05

Doesn't that still leave open a gap in the language, where there is nothing that can really replace a generator easilly?

Alex Miller (Clojure team)01:03:27

if you want control, use loop/recur

šŸ’Æ 4
didibus01:03:33

I also thought lazy-seq wasn't chunking

ghadi01:03:35

iteration has enough knobs to be a generator

potetm01:03:48

lazy-seq isnā€™t, but most seq fns are

ghadi01:03:48

for different types of iterated APIs

Alex Miller (Clojure team)01:03:52

if you map or filter over it, you're chunking

ghadi01:03:14

you're only chunking if the source is chunked, no?

didibus01:03:22

Hum... I'm not seeing that behavior at the REPL

Alex Miller (Clojure team)01:03:52

map and filter produce chunked sequences

Alex Miller (Clojure team)01:03:32

if it reads 32 elements ahead from the source, then aren't you chunking?

didibus01:03:48

So if map is called over a non chunked sequence, it will map over it in chunk?

Alex Miller (Clojure team)01:03:19

how else could it work?

potetm01:03:48

source code appears to switch on the type

Alex Miller (Clojure team)01:03:54

oh ghadi, you're talking about the (if (chunked seq? ) ... in there?

didibus01:03:57

I mean, it makes a lot of sense, but I thought map inly did that if the coll was an instance of ChunkedLazySeq or wtv

ghadi01:03:59

what I said is correct

Alex Miller (Clojure team)01:03:39

I guess none of them would then be

potetm01:03:54

Yeah, I feel like this was totally a thing at one point, but Iā€™m not seeing it now.

didibus01:03:58

I was kind of hoping everything would respect the difference between ChunkedSeq and LazySeq, and only chunk once a ChunkedSeq was present in the chain

ghadi01:03:06

(map f (iteration ...)) isn't chunked

Alex Miller (Clojure team)01:03:18

but a greater point is that any seq impl could chunk and you don't have any control over that

4
didibus01:03:28

Hum... I understand could, but should? Like wouldn't the semantics dictate it shouldn't

Alex Miller (Clojure team)01:03:53

the semantics very much don't

didibus01:03:16

Okay, so ya, just considering Clojure to have lazyness is a mistake at this point I guess

Alex Miller (Clojure team)01:03:30

well that's absolutely the wrong takeaway imo

didibus01:03:37

You should treat all that as a black box

āœ”ļø 4
potetm01:03:15

Yeah, I think thatā€™s actually right. You gave up control over the execution model when you decided to use map.

Alex Miller (Clojure team)01:03:15

without lazy seqs, you wouldn't have infinite seqs and partial reading, and many things we find great

potetm01:03:37

For many purposes, itā€™s exactly what you want.

didibus01:03:43

I meant like, don't think you can leverage lazy-seq to build your own lazy one at a time workflow of especially something side-effectful

didibus01:03:14

It really doesn't seem to be the intent of the whole lazy-seq machinery, so doing so is not advised

Alex Miller (Clojure team)01:03:19

with lazy seqs, you do not have control of consumption

Alex Miller (Clojure team)01:03:53

the control is internal, not external

didibus01:03:02

It does leave open the generator use case though it seems, for maybe a community provided library

Alex Miller (Clojure team)01:03:14

people have written clojure generator libs

potetm01:03:14

Worth considering: using core.async for generating values

potetm01:03:31

so thatā€™s a separate process from the processing

ghadi01:03:40

we have also thought about a concurrent flavor of iteration, where the calls to step! can run ahead by a configurable buffer

didibus01:03:10

Ya, I just see more and more people looking for a "generator" like behvior in Clojure and asking how to do that. Since its all the trend in other languages right now

potetm01:03:31

Absent context, I would say: core.async

didibus01:03:51

Probably easiest to just reify a Java Iterator at this point

didibus01:03:15

core.async is still eager though

Alex Miller (Clojure team)01:03:16

just the first one I see on googling, I know I've seen others

potetm01:03:25

lol thatā€™s a wrapper on core async

didibus01:03:34

Well.. I guess ya, with a buffer of 1, it gets pretty lazy

potetm01:03:34

no, it wouldnā€™t be eager

potetm01:03:50

you can hold a very small amount in memory at a given time

Alex Miller (Clojure team)01:03:53

on a really old version of core.async :)

didibus01:03:57

Ya, that might be the way I go instead, and tell people to go for

Alex Miller (Clojure team)01:03:04

seems like I just saw a much more extensive one recently

potetm01:03:19

I honestly prefer that model to lazy seqs for side effecting stuff. You get fine-grained control over side-effects, and itā€™s separated from transformation code.

Alex Miller (Clojure team)01:03:07

I think that's the one I was remembering

Alex Miller (Clojure team)01:03:52

definitely is some overlap with how core.async is impl'ed

didibus01:03:13

Implements the more general coroutine, and does not use core.async IOC for it

didibus01:03:20

That erdos.yield one seems interesting as well, similarly re-implement its own machinery instead of using core.async

didibus01:03:53

To be honest though, in my case, I'm not looking for that imperative style of yielding

didibus02:03:24

Just need to create an iterator that can carry state over from prior iteration, and is truly lazy, one element at a time šŸ˜›

didibus02:03:59

And is also caching

potetm02:03:20

caching :wat:

didibus02:03:31

Cause I like that over generator, and its easy to just not retain the head if you don't care about prior "generated" elements

potetm02:03:43

are you sure you donā€™t want to just eagerly load everything?

didibus02:03:05

Yes, I want to load the next element from a server only when it is needed.

potetm02:03:19

but you want to cache it?

didibus02:03:21

But once loaded, I don't want to need to reload them all

potetm02:03:22

in memory?

didibus02:03:37

So if I want to loop over them twice, I'm not fetching twice

potetm02:03:43

so, whatā€™s the downside to loading them all up-front?

potetm02:03:50

like, why is that a problem

potetm02:03:26

if itā€™s not to save memory, whatā€™s it for?

didibus02:03:54

Maybe the user won't want to see all thousands, only first hundred, but those first 100 still need to be transformed somewhat before being displayed

didibus02:03:31

Like say elements appear as you scroll down the page

potetm02:03:07

users? whatā€™s that? I write servers! I write for the machine!

potetm02:03:13

yeah okay, that makes sense

potetm02:03:16

although, at that point, I would probably bash it into an atom anyways (a la reagent)

didibus02:03:11

Trying to be pure in my use of unpure functions šŸ˜›

didibus01:03:41

It can be, see my big code snippet from above

ghadi01:03:39

@didibus using iteration if you have seen something in the result set already, step! can return a ret that fails the some? check

didibus01:03:01

Ya, but where is the result set available to me?

ghadi01:03:03

step! is presumed to be side-effecting

didibus01:03:22

So like I need external mutable state?

ghadi01:03:39

there are other ways to solve your problem

ghadi01:03:58

if you were using iteration - yes you'd use a ref type

didibus01:03:21

Ya, I was hoping not to have too\

didibus01:03:34

Which a side effect safe iterate would allow me too šŸ˜›

didibus01:03:09

Well, I think I might be able to use the same trick on iteration

didibus01:03:41

Have step! return a tuple, and kf can extract the state and k can just become state I carry over instead of a key from the response

ghadi02:03:26

(defn didibus-thing
  "fetch is a side-effecting 0 arg fn"
  [fetch!]
  (let [seen (atom #{})]
    (iteration (fn [_]
                 (let [obj (fetch!)
                       [old new] (swap-vals! seen conj obj)]
                   (when (not= old new)
                     obj))))))

ghadi02:03:03

it aborts when fetch returns something that has been seen before

ghadi02:03:35

notice step!'s "k" argument is ignored, and the values returned by each step is the same as the "ret"

ghadi02:03:57

so the defaults for :some?, :kf and :vf all work fine

ghadi02:03:49

the neat thing about this is that you get to choose whether you use it as a seq or a reducible

didibus02:03:45

Ya, its probably what I'll start using once 1.11 is out. I feel like, if it allowed to carry over something, you could avoid the use of the closure over an atom or some other construct, but I can see the challenge in the API

ghadi02:03:55

user=> (vec (didibus-thing #(rand-int 10)))
[3 4 2 0]

ghadi02:03:14

it might not make it into 1.11

ghadi02:03:22

so feel free to crib it from JIRA

ghadi02:03:42

there is nothing wrong with an occasional atom

šŸ’Æ 4
didibus02:03:00

I know, but you know, Clojure always makes me feel dirty when I do šŸ˜›

didibus02:03:33

I still conceptually prefer the iterate model, like in its beauty. The idea is just that I want a sequence of (f x) -> (f (f x)) -> (f (f (f x))) -> etc. I find this model pretty conceptually elegant, just here f can't have side effect.

didibus02:03:04

The next element is a result of the previous (and possibly all prior or anything else I choose to carry over)

ghadi02:03:44

meh I don't like it (no offense)

ghadi02:03:24

the state & the tuple mar the conceptual beauty

ghadi02:03:46

whoops forgot the lazy seq

didibus02:03:18

No offense taken, these things are a bit personal opinion. The atom similarly annoys me šŸ˜›

ghadi02:03:26

(defn didibus-lazy-seq
  "fetch is a side-effecting 0-arg fn"
  [fetch!]
  (let [step (fn step
               [seen]
               (lazy-seq
                (let [obj (fetch!)]
                  (when (not= (conj seen obj) seen)
                    (cons obj (step (conj seen obj)))))))]
    (step #{})))

didibus02:03:23

Maybe in a language with support for multiple return it would be cleaner. But I got used to tuples in Clojure being an idiomatic way to do so

didibus02:03:13

It also annoys me that the first result of iterate is x on its own, wish it was (f x). That's why I have the whole drop 1 thingy

didibus02:03:37

And I guess you could have a multi-arity iterate, so f could be one or more arguments, again, to avoid the tuple as arg-vector trick

didibus02:03:13

(defn fibonacci[]
  (->> (iterate (fn [[a b]] [b (+ a b)]) [0 1])
       (map first)))
So in the above @ghadi would you say that one could also choose to use iteration for it with an atom? Would there be any advantage? Or do you envision iteration being used only for side effects?

didibus05:03:34

@ghadi Here's my version using iteration:

(defn lazy-iteration-fetch [fetching-fn]
  (->> (iteration
         (fn [pages-fetched]
           (let [page (fetching-fn)]
             (if (pages-fetched page)
               nil
               [page (conj pages-fetched page)])))
         :vf first
         :kf peek
         :initk #{})
       lazy-seq))
What do you think?

didibus05:03:49

I know you intended k to be more like the key needed to make the API request, but I guess in practice it can be whatever. So I can use a similar trick of making it previous state. It still ends up being much more readable then with iterate, so I like it. Have to wrap in a lazy-seq as well, since I see that iteration is non caching, it assumes the API will be idempotent on k, or that consumers won't consume more than once otherwise.

seancorfield05:03:56

@didibus It only talks about being idempotent on initk

seancorfield05:03:16

"it is presumed that step! with non-initk is unreproducible/non-idempotent"

seancorfield05:03:39

"if step! with initk is unreproducible, it is on the consumer to not consume twice"

seancorfield05:03:58

Ah, but your version is unreproducible on initk as well, right? Since it calls (fetching-fn) regardless of what is passed in.

didibus05:03:56

So wrapping in lazy-seq seems to work to make it cached. Though I think at this point, I'm doing the same trick I did with making iterate safe for side effects

didibus05:03:34

Which @noisesmith pointed out that it might not be super reliable

didibus05:03:06

Because lazy-seq only wraps the outer sequence, once you call rest or next on it, you get back something else

didibus05:03:34

But I don't know, seems to work fine

didibus05:03:12

Will iteration without an initk somehow return a caching sequence instead?

didibus05:03:07

The doc is unclear, I think it just means, without an initk, then the server has no way to be idempotent, since you provide it no key, so it is assumed not to be, in which case it is probably expected the consumer won't consume twice

dpsutton05:03:12

I thought the docstring makes it clear that without an initk it just calls the fn with nil the first iteration

didibus05:03:07

Yea it says that

didibus05:03:09

And then: it is presumed that step! with non-initk is unreproducible/non-idempotent

didibus05:03:16

if step! with initk is unreproducible, it is on the consumer to not consume twice

didibus05:03:02

Which I guess, well, I can see it maybe being a bit confusing, like does that mean it assumes that if you pass an initk it must be reproducible?

didibus05:03:31

And if so why? Does it do something different to handle the non reproducible case of using default initk?

didibus05:03:16

Basically at this point I have three implementations and they all seem to work just as well, one with iterate which I map over, which seems to make it behave properly with side effect when you do that. One with iteration which I wrap in a lazy-seq, and again it then seems to have the expected behavior, including (rest data) returning the cached thing and not calling fetch again. And one with lazy-seq.

seancorfield05:03:46

@didibus If the step is non-reproducible on initk, then you may get different answers if you consume it twice -- which is back to the same boat you were in with iterate and trying to do both seq and reduce.

seancorfield05:03:43

It doesn't need to do anything special for that case, it's just a consequence of the non-reproducibility of that call (since if it gets called twice on initk it may return different results).

didibus05:03:13

Ya, I'm investigating. So for iterate, wrapping in lazy-seq fails if you then reduce on the rest, you will get back an instance of Iterate, and reduce will now again re-consume the fetches.

didibus05:03:23

But, if wrapped in map, it works

didibus05:03:42

Like (map identity (iterate ...))

seancorfield05:03:54

In other words, it's saying that if calling step! with a known fixed value can return different values on repeated calls (such as a random number generator), then you can't guarantee that repeated consumption of the sequence of values will return anything predictable.

seancorfield05:03:24

Right, and you're seeing that because you don't have a sane "seed" for your initial fetch call.

didibus05:03:39

Ya, so like... it feels to suffer from the same "non side effecting" as iterate, just I guess, it assumes the caller understands the risk

seancorfield06:03:00

How do you establish the "first call" in your fetching sequence?

seancorfield06:03:22

It seems to me that if there's no "seed" value, then you have a random sequence with no repeatability?

didibus06:03:39

Only difference is, for iteration, wrapping in lazy-seq seems to work fully, and calling rest on it returns a LazySeq

seancorfield06:03:53

You're not answering the question.

didibus06:03:05

Yes, my fetch is this: `

seancorfield06:03:06

You're focusing on a hack instead of the core problem.

didibus06:03:22

(defn fetch
  "A stub fetch that pretends like its getting random data from a url, but instead return a rand char from url."
  [url]
  (println "fetching " url)
  (rand-nth url))

seancorfield06:03:01

That has no "seed" so the initial call is completely unpredictable -- so it fails the guarantees of iteration.

didibus06:03:07

And I want to fetch until I see a repeated value

didibus06:03:32

So iteration only supports idempotent side effects?

seancorfield06:03:42

No, absolutely not.

seancorfield06:03:48

That's not what it says.

didibus06:03:25

Ok, but kind of. Lets take a step back. iterate says it is not safe for side-effects because if consumed twice it will recompute the side-effect

didibus06:03:51

iteration says you can use it for side-effect, but if consumed twice it will recompute the side-effect

seancorfield06:03:03

Only for initk

didibus06:03:16

Only when the side-effect is not idempotent

seancorfield06:03:57

It says that if the step function, invoked on initk, is not reproducible, it's up to the consumer to not attempt to consume it twice.

seancorfield06:03:35

It doesn't have any caveats about step called on non-initk values -- in fact it specifically says it is presumed to be unreproducible.

seancorfield06:03:09

But that initk value -- the first call -- has to be reproducible if you want any guarantees about repeated consumption.

seancorfield06:03:24

i.e., you can't have both.

didibus06:03:30

I doubt it

didibus06:03:41

Like, maybe, but I'm not really seeing it

seancorfield06:03:50

If you don't care about the guarantee, then the reproducibility of step called on initk doesn't matter.

didibus06:03:37

Like, what does that change? If my first initk is reproducable, but not my next k, how does it guarantee it?

didibus06:03:17

Does it cache sequences by initk ?

seancorfield06:03:44

You have the source code -- you can answer that question.

didibus06:03:36

erm, let me do an experiment first šŸ˜›

didibus06:03:48

I think I know what you mean

didibus06:03:03

Because it explains why rest returns a LazySeq

didibus06:03:10

while rest on iterate does not

didibus06:03:37

And that would explain why wrapping iteration in a lazy-seq is enough to add the guarantee even with a non reproducible initk

didibus06:03:12

But I also don't understand why people keep saying me wrapping things in lazy-seq is a hack?

seancorfield06:03:46

Very specifically, calling seq on the result of iteration invokes (step! initk) and calling reduce on that same result invokes (step! initk) again.

seancorfield06:03:57

Yeah, I don't know why you don't understand, when everyone keeps telling you the same thing šŸ™‚ I was hoping to explain it so you'd see it but I clearly haven't done any better job that either Ghadi or Alex at this point. Sorry, I tried šŸ˜ž

didibus06:03:00

Ok, and later in reduce it calls (step! k)

didibus06:03:14

Which I'm not seeing how it would not return a different result

didibus06:03:19

Even if initk does

seancorfield06:03:45

I give up. Sorry, man, but it's late and I've tried.

didibus06:03:19

Yam thanks

didibus06:03:42

I'm still gonna go ahead and use this, not seeing any failure case with it

didibus06:03:55

No one is able to point at as failure case

seancorfield06:03:59

(let [p (iteration ...)]
  (seq p) ; calls (step! initk) to seed the computation
  (reduce ... p)) ; calls (step! initk) AGAIN to seed the computation
That's the clearest I can make this.

didibus06:03:08

Maybe I'm readin "hack" as it will have bugs

didibus06:03:25

Yea, I understand that

didibus06:03:35

But this assumes step! can be seeded

didibus06:03:41

for all following ks

didibus06:03:07

given an initial seed k, it assumes step! itself will return the same thing, and the same subsequent thing on all next ks it returns

dpsutton06:03:29

i don't think it assumes that. i thought you quoted the docstring in which it tells the caller to beware

didibus06:03:49

To me that sounds like step! must be idempotent, not just on the first call, but on the full series of call

dpsutton06:03:48

> > if step! with initk is unreproducible, it is on the consumer to not consume twice" that's saying if your api calls aren't idempotent then only issue them once

didibus06:03:07

And here's the proof:

user=> (def a (iteration foo :initk 0 :kf inc))
#'user/a
user=> (reduce (fn[acc e] (if (> acc 5) (reduced nil) (do (println e) (inc acc)))) 0 a)
0
0.77166746127496
0.07621011513209086
0.9841327041345114
0.821362141438259
0.21009693818295794
nil
user=> (reduce (fn[acc e] (if (> acc 5) (reduced nil) (do (println e) (inc acc)))) 0 a)
0
0.7984895604320498
0.5150573279942907
0.11688034595267827
0.9450752936010137
0.6539525668192183
nil

seancorfield06:03:35

That's not "proof" of anything.

didibus06:03:00

With foo being:

(defn foo [k]
     (if (= k 0)
       0
       (rand)))

didibus06:03:37

Well, here you have a step! which returns the same thing consistently for the same initk

didibus06:03:48

But not on following ks

didibus06:03:08

And so iteration will call step! over and over

seancorfield06:03:19

I give up. I'm going to bed.

dpsutton06:03:21

i have no idea what you're driving at

didibus06:03:09

Thanks anyway Sean, I appreciate. I know I'm being difficult, but I'm really not seeing it

didibus06:03:31

@dpsutton I don't know how far back you've read.

didibus06:03:09

But it all started with me wondering what issues iterate has when used with a side effecting f, against which its docstring warns not too use

didibus06:03:25

And the answer was that reduce will consume f over and over

didibus06:03:46

So f could be called more than once per element

didibus06:03:17

Ghadi said iteration is a new function, might release in 1.11, which is to handle that kind of scenario

didibus06:03:19

But, its docstring says that it will just keep calling f over and over everytime it is consumed

didibus06:03:41

Well, not exactly, but I read it as that

didibus06:03:52

And looking at its reduce implementation, to me, it seems to be the case

didibus06:03:28

seq will not

didibus06:03:42

Which is a little better than iterate

didibus06:03:38

So in my deep dive, iterate reduce calls the function over and over. And iterate seq will return Iterate, and not lazy-seq. So I get why in Iterate's case, it always calls f over and over

didibus06:03:49

But when looking at iteration, its only slightly better, in that reduce also just calls f over and over

didibus06:03:02

though seq won't, as it wraps things in a lazy-seq

didibus06:03:57

To make iterate safe with side effects, I found a call to map works, because map will never get from the Iterate more than once, as map itself will cache its results.

didibus06:03:11

But people tell me this is a hack, not sure why

didibus06:03:14

And with iteration, to make it that it consumes only once from f as well, I wrap it in a lazy-seq, so that reduce is on the lazy-seq now, which will use the iteration seq under the hood. And I'm told that is a hack as well

didibus06:03:10

Finally, I conclude that iterate and iteration are both pretty much safe with side effect if you are okay with f being consumed more than once possibly, as you access the same elements over and over

didibus06:03:29

And so the only difference is in the doc-string

didibus06:03:51

Well, minus that iteration is more safe I guess, due to the way it implements seq, which makes it reducing on the rest of it still win't call f again

didibus06:03:40

That said, I must be misunderstanding something, since very smart people told me I am wrong

didibus06:03:21

So Sean said I'm not understanding how iteration works. So I'm looking into that

didibus06:03:11

I think for iterate I am correct, just most people find wrapping it in a call to map feels hacky, and even though right now we might not think of an issue, its harder to reason about that it works in all scenario as well

didibus06:03:00

I'm looking at the code for iteration and trying it in the REPL, and I think I'm also correct. I'm not seeing what Sean is talking about. iteration seems to only be caching initk, not subsequent values or ks

didibus06:03:55

Which is fine, iteration basically leaves it up to the consumer to do whatever, if it wants to re-run the side effects multiple time, its up to them, and the function has good support for using it with paging APIs with idempotency keys.

didibus06:03:49

I don't know. I appreciate everyone's help and input. I did learn a lot. I think I understand things now, but I'll sleep on it all, since it seems some people say I still got a few things wrong, and maybe tomorrow it'll click for me

didibus07:03:40

For anyone interested, I was researching this rabbit whole for my answer here: https://clojureverse.org/t/is-this-a-good-way-to-stream-data-to-caller-of-a-function/5689/3?u=didibus which I hope captures all the learnings from this chat session

šŸ‘ 4
ido08:03:13

hey, I have a clojure library that wraps around a java one. I am trying to diable logging during tests using timbre (which I only require in the :dev profile. I tried calling (timbre/merge-config! {:appenders {:spit {:enabled? false}}}) but I can still see some WARNING logging by the java library. The java code calls:

import java.util.logging.Level;
import java.util.logging.Logger;
...
logger.log(Level.WARNING, "blah", error);
what am I doing wrong?

rutledgepaulv13:03:50

Have you already added the shims listed here to route other java logging frameworks through timbre? https://github.com/fzakaria/slf4j-timbre#slf4j-timbre

ido15:03:44

trying this!

ido03:04:57

still not working: I have in lein deps :tree : ā€¢ [com.fzakaria/slf4j-timbre ā€œ0.3.19ā€ :scope ā€œtestā€] ā€¢ [org.slf4j/jul-to-slf4j ā€œ1.7.30ā€ :scope ā€œtestā€] ā€¢ [org.slf4j/slf4j-api ā€œ1.7.30ā€ :scope ā€œtestā€] the underlying library uses java.util.logging but logs still appear on the screen.

rutledgepaulv04:04:23

the spit appender is for logging to files, not logging to stdout (what's appearing on the screen probably). You might need to do something different to turn stdout off.

rutledgepaulv05:04:04

try (timbre/merge-config! {:appenders {:println {:enabled? false}}})

restenb14:03:58

is there an easy way to convert nested hiccup-like structures like this [:key1 [:key2 "someval" [:key3 "otherval"]]]over to a nested map?

restenb14:03:00

i suppose not, the possible existence of values followed by new nested vectors is an issue

Spaceman14:03:36

Hi guys, I'm using the origami library to do image processing in clojure, but I'm puzzled by how the interop works.

Spaceman14:03:09

Here: https://www.codepasta.com/computer-vision/2019/04/26/background-segmentation-removal-with-opencv-take-2.html, there's a code segment

blurred_float = blurred.astype(np.float32) / 255.0
edgeDetector = cv2.ximgproc.createStructuredEdgeDetection("model.yml")
edges = edgeDetector.detectEdges(blurred_float) * 255.0
cv2.imwrite('edge-raw.jpg', edges)

Spaceman14:03:36

so far I have a blurred image: (-> "resources/public/img.png" (imread) (gaussian-blur! (new-size 17 17) 9 9) (imwrite "resources/blurred.png"))

Spaceman14:03:48

which works, but how to get the edgeDetector on it?

restenb14:03:03

@pshar10 does that help?

restenb14:03:16

assuming you're passing or storing cv2 somewhere, not familiar with what that is, or if ximgprocis a field or a nested class

restenb14:03:38

for nested classes you need to use ClassName$nestedClass as interop notation

Spaceman14:03:59

I don't think cv2 is required. Even in what I have so far above, I use imwrite without cv2, and it works with origami.

Spaceman14:03:33

But I have this now:
(let [blurred-float (-> "resources/public/Logo.png"
    (imread)
    (gaussian-blur! (new-size 17 17) 9 9)
    
)
            edgeDetector (-> ximgproc (.createStructuredEdgeDetection "model.yml"))
            edges (* 255 (.detectEdges edgeDetector blurred_float))]
        (imwrite edges "edge-raw.jpg" ))

Spaceman14:03:42

But it gives an error that ximgproc isn't defined

restenb14:03:43

right, if ximgprocis a class field of whatever cv2is, it needs dot-dash notation as in my snippet with .-ximgproc

restenb14:03:23

dealing with state/side-effect full java/JS interop like this in general assumes you're storing or passing whatever your stateful object is that you need to access fields & member functions on later

restenb14:03:43

judging by the Java example you posted that would be cv2in this case

Spaceman14:03:54

No, like in open cv one usually writes cv2.imwrite to write an image. Similarly, shouldn't it work with just ximgproc and not the cv2?

grounded_sage14:03:21

Is there a simple transducer I could use on a channel where it simply breaks apart a list and feeds the result to the consumer 1 by 1?

didibus17:03:12

You mean that you have a list inside a channel, and you want to like flatten it?

didibus17:03:25

I think you could look into cat and mapcat, they might be able to do what you're looking for

didibus17:03:57

Possibly you could use onto-chan as well, though that's not a transducer

restenb14:03:20

i mean. that Clojure snippet you posted has several errors in it. If you intend to access an instance member named ximgproc, you must use either (.or (.-notation, the latter is preferred for fields

restenb15:03:15

as I'm not familiar with this library I don't know what to expect from a call like (imread "logo.png")for example, what does that return?

restenb15:03:38

something like that should work if (imread "logo.png")already returns the object you need for the rest of the chain

noisesmith15:03:38

@pshar10 in a repl you can use javadoc (from clojure.java.javadoc ns, always present in the initially loaded repl ns) - when you pass it some object it will usually find the documentation for that object in your web browser

noisesmith15:03:50

this can help with code that does so much interop

noisesmith15:03:29

eg. (javadoc (gaussian-blur! (imread "logo.png"))) to figure out what precisely that object is

noisesmith15:03:34

and what you can do with it

Spaceman15:03:02

I found it to work with

(import '[org.opencv.ximgproc Ximgproc])
(let [blurred-float (-> "resources/public/Logo.png"
    (imread)
    (gaussian-blur! (new-size 17 17) 9 9)

)
            edgeDetector (Ximgproc/createStructuredEdgeDetection "resources/model.yml")
            edges (* 255 (.detectEdges edgeDetector blurred-float))]
        (imwrite edges "edge-raw.jpg" ))

noisesmith15:03:27

OK - that looks reasonable :D

Spaceman15:03:33

But I get No matching method detectEdges found taking 1 args for class org.opencv.ximgproc.StructuredEdgeDetection

noisesmith15:03:05

then (javadoc org.opencv.ximgproc.StructuredEdgeDetection) and figure out what args that method takes

noisesmith15:03:43

clojure doesn't try much "clever" stuff about guessing method argument types, you likely need a hint to make it pick the method you need

noisesmith15:03:16

quick google claims that you need both an input array (what you have now) and an output array

noisesmith15:03:30

(at least in the C++ version...)

noisesmith15:03:57

yeah it's true in java too

Spaceman15:03:18

How do I create an output array? (. WhatObject?)

noisesmith15:03:30

you can click on the type in the doc

noisesmith15:03:54

that should mention a constructor or a factory method

noisesmith15:03:14

yeah, it has a zero arg constructor, that's probably enough(?)

Spaceman15:03:28

edges (* 255 (.detectEdges edgeDetector blurred-float (. Mat))), gives: Malformed member expression, expecting (. target member ...)

noisesmith15:03:02

the zero arg constructor for Mat would be (Mat.)

noisesmith15:03:39

you might want to refresh yourself on the interop syntax doc - it's actually simpler and more consistent than java's own syntax :D https://clojure.org/reference/java_interop

Spaceman15:03:33

For some reason, multiplying 255 doesn't work in clojure:

(let [blurred-float (-> "resources/public/Logo.png"
    (imread)
    (gaussian-blur! (new-size 17 17) 9 9)
    (float)
    
)
            edgeDetector (Ximgproc/createStructuredEdgeDetection "resources/model.yml")
            mat (Mat.)
            _ (.detectEdges edgeDetector blurred-float mat)
            edges (* (float 255) mat)
            ]
        (imwrite edges "edge-raw.jpg" ))

Spaceman15:03:45

gives: Execution error (ClassCastException) at user/eval63001 (form-init8488930107953136676.clj:373). org.opencv.core.Mat cannot be cast to java.lang.Number

restenb15:03:27

why would that work anywhere? you're trying to multiply a class by a number

restenb15:03:55

probably you want to multiply the result of the detectEdgescall by 255 instead

noisesmith15:03:20

@pshar10 are you trying to do a matrix multiply? mat is a matrix

noisesmith15:03:44

detectEdges returns nil

noisesmith15:03:16

in clojure and afaik in java as well, * doesn't handle matrix multiplies - you need a method for that

restenb15:03:22

the original example he posted does this edges = edgeDetector.detectEdges(blurred_float) * 255.0

noisesmith15:03:42

OK... that doesn't match the detectEdges method I see

noisesmith15:03:57

it returns nil, and that would blow up

Spaceman15:03:37

My bad, it was due to the cast (float) above.

noisesmith15:03:53

wait, it works without the cast? it shouldn't

Spaceman15:03:07

I removed it but then:

(let [blurred-float (-> "resources/public/Logo.png"
    (imread)
    (gaussian-blur! (new-size 17 17) 9 9)

    
)
            edgeDetector (Ximgproc/createStructuredEdgeDetection "resources/model.yml")
            mat (Mat.)
            _ (.detectEdges edgeDetector blurred-float mat)
            edges (.mul mat 255)
            ]
        (imwrite edges "edge-raw.jpg" )
        )

noisesmith15:03:17

OK .mul loks right

Spaceman15:03:22

(-215:Assertion failed) src.type() == CV32FC3 in function 'detectEdges'

Spaceman15:03:21

This means that the blurred-float matrix isn't the write type

noisesmith15:03:26

you can google that type name to see what it wants - this API looks tedious

noisesmith15:03:44

it's a different kind of matrix it wants, and it's complaining about the src not the dest (unless I'm misreading that cryptic message)

noisesmith15:03:07

also .mul wants a double, so use 255.0 - I don't think clojure's clever enough to do that for you(?)

noisesmith15:03:27

oh, and it wants a second matrix - it's not going to work with just a number

Spaceman15:03:28

Still the same error after giving in a second matrix and 255.0

(let [blurred (-> "resources/public/Logo.png"
                  (imread)
                  (gaussian-blur! (new-size 17 17) 9 9))
      blurred-float (Mat.)
      _ (.convertTo blurred blurred-float 5) ;; five is just the f32 type id
      edgeDetector (Ximgproc/createStructuredEdgeDetection "resources/model.yml")
      mat (Mat.)
      _ (.detectEdges edgeDetector blurred-float mat)
      edges (Mat.)
      _ (.mul mat 255.0 edges)
      ]
  (imwrite edges "edge-raw.jpg" )
  )

Spaceman15:03:34

Execution error (ClassCastException) at user/eval65868 (form-init5691178651358499492.clj:101). java.lang.Long cannot be cast to org.opencv.core.Mat

Spaceman15:03:57

And sometimes the whole repl closes unexpectedly when loading this

noisesmith15:03:15

look at the method signature: the second matrix comes before the double

noisesmith15:03:36

crashing: yeah, c interop will do that

noisesmith15:03:56

also, .mul returns a new matrix, and multiplies two matrixes (and optionally a double as well) - you might want something different, like an imperative loop that scales every value in the array(?)

noisesmith15:03:45

eg. a loop calling the .get and .put methods on each index as I don't see a simple scale operation in that API (I'm no expert on this specific API though, I just know how to read javadoc)

Spaceman15:03:59

.mul doesn't take two matrices, only one

noisesmith15:03:12

it operates on your matrix, plus another

noisesmith15:03:15

you are misreading the doc

Spaceman15:03:15

but it crashes the repl

noisesmith15:03:42

mul is a method on a Mat object, so the one in the arg list is another Mat

noisesmith15:03:47

you need two

bfabry15:03:03

mul takes the instance object (to the left of the method call in java, to the right of it in clojure), and then a second Mat object as an argument

noisesmith15:03:05

unless there is a version of mul without a Mat in the arglist

Spaceman16:03:21

Oh I see, (.mul mat edges 255.0) is like mat.mul(edges, 255.0). But why doesn't it work?

bfabry16:03:06

are you sure it doesn't work? in your snippet above you had the arguments in a different order

noisesmith16:03:25

well, I'd expect mul to do a matrix multiply, one of the two matrixes seems to be uninitialized - which gives you either zeroes or garbage

Spaceman16:03:23

(let [blurred (-> "resources/public/Logo.png"
                  (imread)
                  (gaussian-blur! (new-size 17 17) 9 9))
      blurred-float (Mat.)
      _ (.convertTo blurred blurred-float 5) ;; five is just the f32 type id
      edgeDetector (Ximgproc/createStructuredEdgeDetection "resources/model.yml")
      mat (Mat.)
      _ (.detectEdges edgeDetector blurred-float mat)
      edges (Mat.)
      _ (.mul mat edges 255.0)
      ]
  (imwrite edges "edge-raw.jpg" )
  )

Spaceman16:03:34

cv::Exception: OpenCV(4.2.0) /Users/niko/origami-land/opencv-build/opencv/modules/core/src/arithm.cpp:669: error: (-209:Sizes of input arguments do not match) The operation is neither 'array op array' (where arrays have the same size and the same number of channels), nor 'array op scalar', nor 'scalar op array' in function 'arithm_op'

noisesmith16:03:13

so you probably need to provide dimension args to the Mat constructor to make it match your input

noisesmith16:03:32

at least that's an intelligible error :D

Spaceman16:03:31

Here you go: This gives an output for a test input .png image, which is completely black and doesn't have the edges. And the repl closes if I give it a jpg input. No errors. Just closes:

(let [blurred (-> "resources/public/test.jpg"
                  (imread)
                  (gaussian-blur! (new-size 17 17) 9 9))
      blurred-size (.size blurred)
      blurred-float (Mat. blurred-size 5)
      _ (.convertTo blurred blurred-float 5) ;; five is just the f32 type id
      edgeDetector (Ximgproc/createStructuredEdgeDetection "resources/model.yml")
      mat (Mat. blurred-size 5) ;; five is just the f32 type id
      _ (.detectEdges edgeDetector blurred-float mat)
      edges (Mat. blurred-size 5) ;; five is just the f32 type id
      _ (.mul mat edges 255.0)
      ]
  (prn "blurred size" blurred-size)
  (imwrite edges "edge-raw1.jpg" )
  )

Spaceman16:03:39

Seems like the repl crashes when the image is big regardless of the image type

noisesmith16:03:22

I would expect a multiply by an uninitialized matrix to give zeroes, and I'd expect zero to be black pixels

noisesmith16:03:02

edges probably needs a different content - and maybe there's some trick I couldn't find in the docs to just do a normal multiply

noisesmith16:03:29

eg. edges could be the identity matrix, or the matrix that scales its input by 255

Spaceman16:03:32

Okay, so edges ( mat 255.0) in let should work because is overloaded in c++: https://stackoverflow.com/questions/17892840/opencv-multiply-scalar-and-matrix#17894118

bfabry16:03:56

java and clojure do not support operator overloading

bfabry16:03:42

so while the C++ implementation underneath that works, the java library wrapper you're using it doesn't. hence the existence of mul etc

Spaceman16:03:25

So there's no simple way but to multiply with the identity matrix?

bfabry16:03:39

seems that's a limitation of that library yeah

bfabry16:03:31

there's a Mat#clone function and a Mat#setTo(Scalar) function that should let you create an identity matrix relatively easily

Spaceman16:03:31

There's a static function Mat::eye for that, but how do I use it? For using Mat I was doing (import '[org.opencv.core Mat])

bfabry16:03:03

you call a static function like (Mat/eye rows cols type)

Spaceman16:03:07

I guessed

(let [ 
      img (-> "resources/public/Logo.png"
              (imread))
      size (.size img)
      blurred (Mat. size 5)
      _ (gaussian-blur img blurred (new-size 17 17) 9 9)
      blurred-float (Mat. size 5)
       _ (.convertTo blurred blurred-float 5) ;; five is just the f32 type id
      edgeDetector (Ximgproc/createStructuredEdgeDetection "resources/model.yml")
      mat (Mat. size 5)
      _ (.detectEdges edgeDetector blurred-float mat)
      identity- (Mat/eye size 5)
      edges (.mul mat identity- 255.0)
      ]
  (prn "blurred size" size)
;;  (imwrite edges "edge-raw1.jpg" )
  )

Spaceman16:03:17

But it crashed the repl

bfabry16:03:52

what is the type of (.size img), because it expects an opencv Scalar

bfabry16:03:20

you're trying to do too much at once, execute each of those pieces of code one at a time and capture their output in a def

āž• 4
noisesmith17:03:15

yeah - clojure rewards iterative incremental changes

noisesmith17:03:40

keep your design in mind so you don't make a total mess, but do repeated minimal changes

noisesmith17:03:08

and just remove everything after the error, add things back one by one and make the individual things work

Spaceman17:03:07

The error is that eye is a static function in class Mat. I think my syntax for calling static functions is wrong

Spaceman17:03:17

like printing the value just before it works. mat prints: #object[org.opencv.core.Mat 0x26c3d186 "Mat [ 1024*4096*CV_32FC1, isCont=true, isSubmat=false, nativeObj=0x7ff133f34e90, dataAddr=0x1370aa000 ]"]

Spaceman17:03:16

ok (. Mat eye size 5) got rid of the error and gave me an output, but it's completely black again

Spaceman17:03:12

Still, using a large image crashes the repl.

noisesmith17:03:02

pedantic point: it's a static method, functions are Objects with invoke and applyTo methods implementing Callable and Runnable

noisesmith17:03:26

(Mat/eye size 5) should work if it's a static method

noisesmith17:03:36

(the difference between method and function is less important in languages where you only use one or the other, in Clojure we use both and the differences are important)

Spaceman17:03:54

Well, anyway, even for smaller images when it isn't crashing, the result isn't completely black. The result, i.e., edges seems to be identity * 255, from doing (.mul mat identity- 255.0)

Spaceman17:03:29

as if both mat and identity- are identity matrices

Spaceman17:03:15

(.mul mat identity- 255.0)

dpsutton17:03:09

this should probably be in a thread at this point. its getting a bit long

p-himik17:03:51

IMO even just a single reply should go into a thread. Not least because you can never guarantee that it won't start a long discussion.

šŸ¤ 4
Spaceman17:03:23

anyway, to debug this, it's probably a good idea to convert the Mat object to a clojure vector

Spaceman17:03:38

how can I do that?

noisesmith17:03:33

you can use one of the .get methods to get an array out from the Mat object, and the use into to put an array into a clojure vector

noisesmith17:03:03

you can also use clojure's get on arrays directly, but vectors are definitely nicer - they print readably for example

Spaceman17:03:12

so (.get mat what?)

noisesmith17:03:34

you could iterate it across the other dimension

noisesmith17:03:09

hmm - I mean this one since it doesn't anchor nicely:

public double[] getā€‹(int[] idx)

noisesmith17:03:34

actually maybe this one

public double[] getā€‹(int row, int col)

Spaceman17:03:23

For

(let [img (-> "resources/public/test-small.jpg"
              (imread))
      size (.size img)
      height (.height size)
      width (.width size)
      blurred (Mat. size 5)
      _ (gaussian-blur img blurred (new-size 17 17) 9 9)
      blurred-float (Mat. size 5)
      _ (.convertTo blurred blurred-float 5) ;; five is just the f32 type id
      edgeDetector (Ximgproc/createStructuredEdgeDetection "resources/model.yml")
      mat (Mat. size 5)

      arr (.get mat width height)
      _ (.detectEdges edgeDetector blurred-float mat)
      identity- (. Mat eye size 5)
      edges (.mul mat identity- 255.0)
      ]
  (prn "blurred size" size)
  (prn "mat is " mat)
  (prn "height is " height)
  (prn "width is " width)
  (imwrite blurred-float "blurred-float.jpg")
  (imwrite mat "mat.jpg")
  (imwrite identity- "identity.jpg")
  (imwrite edges "edge-raw1.jpg" )
  )
I get: No matching method get found taking 2 args for class org.opencv.core.Mat

noisesmith17:03:58

try (int width) (int height) - maybe clojure isn't doing the right auto-coercion

noisesmith17:03:27

also you might find it easier to create individual objects in the repl and experiment, instead of reloading the entire function

šŸ’Æ 4
Spaceman18:03:38

okay

(def img (-> "resources/public/test-small.jpg" (cv/imread)))
(def size (.size img))
(def height (.height size))
(def width (.width size))
(def blurred (Mat. size 5))
(cv/gaussian-blur img blurred (cv/new-size 17 17) 9 9)
(def blurred-float (Mat. size 5))
(.convertTo blurred blurred-float 5) ;; five is just the f32 type id
(def edge-detector (Ximgproc/createStructuredEdgeDetection "resources/model.yml"))
(def mat (Mat. size 5))

(.detectEdges edge-detector blurred-float mat)
(def identity- (. Mat eye size 5))
(def edges (.mul mat identity- 255.0))

(def arr (.get mat (int width) (int height))) ;;gives nil

Spaceman18:03:41

arr just gives nil

Spaceman18:03:31

It's supposed to return an array

Spaceman18:03:55

where as mat gives [ 302*302*CV_32FC1, isCont=true, isSubmat=false, nativeObj=0x7fc9d9eba720, dataAddr=0x12ec1c000 ]

noisesmith18:03:39

perhaps you want (dec width) and (dec height) because I think that it's 0 indexed and you are getting the array of rgb doubles at that index

Spaceman18:03:37

dec gives: No matching method get found taking 2 args for class org.opencv.core.Mat

noisesmith18:03:52

right, you're back to the hinting (int (dec width)) etc.

Spaceman18:03:02

arr is [0.0]

noisesmith18:03:47

OK - you can do this in a loop to get all of these single item vectors of doubles, that would be one approach

Spaceman18:03:32

like a nested loop?

Spaceman18:03:50

but how to do it in clojure?

Spaceman18:03:41

like I can't do (nth mat x)?

Spaceman18:03:43

oh I see I think you mean doing for all x y (.get mat x y)

Spaceman18:03:25

Doing (doseq [x (range width) y (range height)] (prn (.get mat (int x) (int y))) ) prints like a lot of these, my guess is 320x320 many: #object["[D" 0x2a0db4b8 "[D@2a0db4b8"]

noisesmith18:03:49

so maybe something like:

(into []
      (for [x (range width)]
        (into []
              (for [y (range height)]
                (into []
                      (.get mat (int x) (int y)))))))

Spaceman18:03:28

How to multiply each item with 255, which I ultimately want to do? I tried this:

(into []
      (for [x (range width)]
        (into []
              (for [y (range height)]
                (into []
                      (* 255.0 (get (.get mat (int x) (int y)) 0)))))))
but it says Don't know how to create ISeq from: java.lang.Double

bfabry19:03:56

you might find dragan's clojure numerical algebra book interesting https://aiprobook.com/numerical-linear-algebra-for-programmers/

Spaceman19:03:20

This works: (into [] (for [x (range width)] (into [] (for [y (range height)] [(* 255.0 (get (.get mat (int x) (int y)) 0))] ))))

Spaceman19:03:18

are there any good libraries to convert clojure vectors into images?

Spaceman19:03:53

I probably can use .put somehow to convert it back into a Mat.

Spaceman20:03:49

I have the following code for it:

(into []
            (for [x (range width)]
              (into []
                    (for [y (range height)]
                      (do
                        (.put edges  (int x) (int y) (* 255.0 (get (.get mat (int x) (int y)) 0)))
                        [(* 255.0 (get (.get mat (int x) (int y)) 0))]
                        )
                      ))))
But I get the error:
No matching method put found taking 3 args for class org.opencv.core.Mat
But Mat.put documentation says that this method exists:
putā€‹(int row, int col, double... data)
Then what am I doing wrong?

p-himik17:03:51

IMO even just a single reply should go into a thread. Not least because you can never guarantee that it won't start a long discussion.

šŸ¤ 4
Darin Douglass18:03:06

so it seems metadata is properly handled when reading in EDN files:

Clojure 1.10.1
user=> (clojure.edn/read-string "^{:params {:a 1}} {:jack :jill}")
{:jack :jill}
user=> (meta *1)
{:params {:a 1}}
however, i cannot seem to find any reference to metadata in the edn spec (https://github.com/edn-format/edn). is the documentation lacking or is this just a happy little accident?

šŸ‘€ 4
Darin Douglass18:03:57

this response seems to indicate it's not an accident (at least in clojure): https://github.com/edn-format/edn/issues/52#issuecomment-24337627

Spaceman19:03:55

are there any good libraries to convert clojure vectors into images?

Spaceman19:03:47

Suppose I have a vector like so:

[
[[50][60][70][20][0]...]
[[90][56][67][98][78]...]
...]
Which represents a single channel 8-bit image. Is there a simple way to convert this into a jpg or a png?

Alex Miller (Clojure team)19:03:09

java2d is in the jdk and has support for stuff like this, but it's quite a bit more complicated iirc

Alex Miller (Clojure team)19:03:40

I'm sure simpler Java apis probably exist and that's where I'd look, rather than in Clojure

Alex Miller (Clojure team)19:03:03

here's a really simple example I wrote in Java 13 years ago when I was doing a mandelbrot thing https://puredanger.github.io/tech.puredanger.com/2007/10/12/images-java2d/

Spaceman19:03:24

how about opencv? Is there a way to convert a vector to an opencv Mat?

didibus20:03:46

You can also use JOGL and LWJGL for 3D, and I think they support 3D to 2D projection, which can be faster than Java2D due to leveraging the GPU

Spaceman19:03:46

how would the java interop work in this case?

Alex Miller (Clojure team)19:03:33

you've rapidly crossed my line of knowledge :)

aarkerio19:03:13

Hi! what is the "standard" solution to keep a map ordered?

bfabry19:03:09

sorted-map

andy.fingerhut19:03:08

depends on what order you want. Sorted by keys via a comparison function, or insertion order of keys?

Spaceman20:03:23

I'm trying to read a Mat in open cv, multiply each of the components with 255 and save it as a Mat. I have the following code for it:

(into []
            (for [x (range width)]
              (into []
                    (for [y (range height)]
                      (do
                        (.put edges  (int x) (int y) (* 255.0 (get (.get mat (int x) (int y)) 0)))
                        [(* 255.0 (get (.get mat (int x) (int y)) 0))]
                        )
                      ))))
But I get the error:
No matching method put found taking 3 args for class org.opencv.core.Mat
But Mat.put documentation says that this method exists:
putā€‹(int row, int col, double... data)
Then what am I doing wrong?

dpsutton20:03:56

put is a method on a matrix. in java that means mat.put(row, col, double) and in clojure it means (.put mat row col double)

Spaceman20:03:45

edges is defined as: (def edges (Mat. size 5))

souenzzo20:03:54

TIP: Don't use for to imperative things Prefer doseq or dotimes Use a reduce if you need to return something

dpsutton20:03:37

so you want to call the .put method on the edges matrix?

dpsutton20:03:56

then call it on it. (.put edges the args from javadocs)

souenzzo20:03:11

The problem probably is something with "type mathing", once in clojure, everything end up with Object, JAVA can't find if it should use put(int int int) or put(float float float)

Spaceman20:03:11

I want to set edges[x,y] to that long expression (* 255.0 (get (.get mat (int x) (int y)) 0))

souenzzo20:03:43

Can you link a "javadoc" about this method?

dpsutton20:03:50

No matching method put found taking 3 args

dpsutton20:03:19

now you're calling get i feel like what you're trying to do keeps changing

Derek20:03:01

Java varargs is really passing an array behind the scenes. When calling from Clojure, youā€™d have to use double-array

Spaceman20:03:24

@dpsutton I'm doing both, set edges[x, y] := 255 * mat[x, y]

Spaceman20:03:14

since mat[x, y] returns a vector of size 1 like [0.1231], I wrap (get ... 0)

Spaceman20:03:11

@UPEKQ7589 can you please explain how that would work?

souenzzo20:03:16

(doseq [x (range width)
        y (range height)]
  (.put edges
        (int x) (int y)
        (into-array Double [(* 255.0 (get (.get mat (int x) (int y)) 0))])))

Derek20:03:20

If the method you wish to call looks like this in Java putā€‹(int row, int col, double... data) it would look like this in Clojure (.put a-matrix row col (double-array ā€¦ ))

souenzzo20:03:55

(doseq [x (range width)
        y (range height)
        :let [data1 (* 255.0 (get (.get mat (int x) (int y)) 0))]]
  (.put edges
        (int x) (int y)
        (double-array [data1])))

souenzzo20:03:13

Not sure if you need to (double (* 255 ...))

phronmophobic20:03:50

it may be overkill and slow for your use case, but if youā€™re just trying to get something working, you can use https://github.com/phronmophobic/membrane

(ns membrane.example.squares
  (:require [membrane.ui :refer [filled-rectangle]
             :as ui]
            [membrane.skia :refer [draw-to-image!]
             :as skia]))



(def nums (repeatedly 300
                      (fn []
                        (repeatedly 500 #(vector (rand-int 255))))))

(defn render-nums [nums]
  (vec
   (for [[j row] (map-indexed vector nums)
         [i col] (map-indexed vector row)
         :let [color (-> col
                         first
                         (/ 255.0))]]
     (ui/translate i j
                   (filled-rectangle [color color color] 1 1))))
  )

(draw-to-image! "vec-image.png" (render-nums nums))

Spaceman20:03:29

@U2J4FRT2Tā€™s code works, but for some reason it gives a nullpointerexception on bigger images and then crashes the repl, but works fine with smaller images. Why would that be?

didibus20:03:56

@U7RJTCH6J Wow, membrane looks great. Are you just implementing your own layer above each OS native GUI toolkit in C ? Or re-using some existing C/C++ layer for that and just providing a Clojure API above it?

phronmophobic21:03:36

if itā€™s hard crashing the repl, then maybe itā€™s an out of memory issue?

Spaceman21:03:07

can nullpointerexceptions happen if there's a memory issue?

phronmophobic21:03:58

maybe? do you have the stack trace?

phronmophobic21:03:02

thatā€™s a segfault. that means the underlying c code is accessing invalid memory

Spaceman21:03:39

why would the c code have trouble accessing memory for a larger file and not for a smaller one?

phronmophobic21:03:47

it looks like this is happening during edge detection

Spaceman21:03:08

Actually, these images are about the same size the 320x320 one is 86KB and the larger one is 1024x768 and it's about 93KB

phronmophobic21:03:57

if the wrapper library isnā€™t validating arguments, then itā€™s totally possible that not passing the right args will crash the jvm

phronmophobic21:03:27

is there an ā€œoutputā€ matrix youā€™re passing as an arg?

phronmophobic21:03:38

do you have the code thatā€™s calling edge detection?

phronmophobic21:03:57

your output matrix might be the wrong size

Spaceman21:03:11

Here's the whole code:

(do
  (def img (-> "resources/public/test-img.jpg" (cv/imread)))
  (def size (.size img))
  (def height (.height size))
  (def width (.width size))
  (def blurred (Mat. size 5))
  (cv/gaussian-blur img blurred (cv/new-size 5 5) 0)
  (def blurred-float (Mat. size 5))
  (.convertTo blurred blurred-float 5) ;; five is just the f32 type id
  (def edge-detector (Ximgproc/createStructuredEdgeDetection "resources/model.yml"))
  (def mat (Mat. size 5))
  (def edges (Mat. size 5))

  (.detectEdges edge-detector blurred-float mat)

  )

(doseq [x (range width)
        y (range height)
        :let [data1 (* 255.0 (get (.get mat (int x) (int y)) 0))]]
  (.put edges
        (int x) (int y)
        (double-array [data1]))) ;; crashes here for larger image
(cv/imwrite blurred "blurred.jpg")
(cv/imwrite edges "edge-raw1.jpg")

phronmophobic21:03:33

ah, how did you choose size 5?

Spaceman21:03:50

if you're hinting at the filter size going outside the image size then none of the dimensions of either of the images is divisible by 5

phronmophobic21:03:43

does it reliably crash on the same image, or just sometimes?

Spaceman21:03:11

yes, it reliably crashes on the same image.

Spaceman21:03:39

I was wrong before when I'd said that it only sometimes crashes. It has crashed every time ten times or so.

phronmophobic21:03:52

whatā€™s the java library youā€™re using?

phronmophobic21:03:52

my guess is that the size is wrong, but iā€™m not familiar enough with opencv to know for sure

Spaceman21:03:39

Here's how you can replicate the error: library is http://origamidocs.hellonico.info/#/ Then

(require '[opencv4.core :as cv])
(import
   '[org.opencv.core Mat]
   '[org.opencv.ximgproc Ximgproc]
   )
(do
  (def img (-> "resources/public/test-small.jpg" (cv/imread)))
  (def size (.size img))
  (def height (.height size))
  (def width (.width size))
  (def blurred (Mat. size 5)) ;; five is just the f32 type id
  (cv/gaussian-blur img blurred (cv/new-size 5 5) 0)
  (def blurred-float (Mat. size 5)) ;; five is just the f32 type id
  (.convertTo blurred blurred-float 5) ;; five is just the f32 type id
  (def edge-detector (Ximgproc/createStructuredEdgeDetection "resources/model.yml"))
  (def mat (Mat. size 5)) ;; five is just the f32 type id
  (def edges (Mat. size 5)) ;; five is just the f32 type id

  (.detectEdges edge-detector blurred-float mat)

  )

(doseq [x (range width)
        y (range height)
        :let [data1 (* 255.0 (get (.get mat (int x) (int y)) 0))]]
  (.put edges
        (int x) (int y)
        (double-array [data1])))
(cv/imwrite blurred "blurred.jpg")
(cv/imwrite edges "edge-raw1.jpg")

Spaceman21:03:33

It works on this image:

Spaceman21:03:58

But not on this:

phronmophobic21:03:19

instead of 5, what happens when you do (.type img)

phronmophobic21:03:38

thatā€™s what all the examples are doing

Spaceman21:03:59

error: (-215:Assertion failed) src.type() == CV32FC3 in function 'detectEdges'

Spaceman21:03:14

5 is CV_32FC3

phronmophobic21:03:32

and try running the equivalent of img.convertTo(src, CvType.CV_32F, 1.0 / 255.0); on your image

phronmophobic21:03:27

I think itā€™s crashing because youā€™re saying itā€™s type CV_32FC3 when itā€™s actually not

Spaceman21:03:37

The crazy thing is that I got nullpointerexception, but I it worked this time with (/ 1 255). I got the right output

sheepy 4
Spaceman21:03:08

Partial success. There shouldn't be any nullpointerexceptions in my code

Spaceman21:03:18

Here's the trace: Numbers.java: 3845 clojure.lang.Numbers/multiply REPL: 623 vendo.core/eval67142 REPL: 621 vendo.core/eval67142 Compiler.java: 7177 clojure.lang.Compiler/eval Compiler.java: 7132 clojure.lang.Compiler/eval core.clj: 3214 clojure.core/eval core.clj: 3210 clojure.core/eval main.clj: 437 clojure.main/repl/read-eval-print/fn main.clj: 437 clojure.main/repl/read-eval-print main.clj: 458 clojure.main/repl/fn main.clj: 458 clojure.main/repl main.clj: 368 clojure.main/repl RestFn.java: 137 clojure.lang.RestFn/applyTo core.clj: 665 clojure.core/apply core.clj: 660 clojure.core/apply regrow.clj: 18 refactor-nrepl.ns.slam.hound.regrow/wrap-clojure-repl/fn RestFn.java: 1523 clojure.lang.RestFn/invoke interruptible_eval.clj: 79 nrepl.middleware.interruptible-eval/evaluate interruptible_eval.clj: 55 nrepl.middleware.interruptible-eval/evaluate interruptible_eval.clj: 142 nrepl.middleware.interruptible-eval/interruptible-eval/fn/fn AFn.java: 22 clojure.lang.AFn/run session.clj: 171 nrepl.middleware.session/session-exec/main-loop/fn session.clj: 170 nrepl.middleware.session/session-exec/main-loop AFn.java: 22 clojure.lang.AFn/run Thread.java: 748 java.lang.Thread/run

Spaceman21:03:45

What I mean is that I get the right output despite a nullpointerexception. I don't know how that's possible. Should the code just stop running on an exception?

phronmophobic21:03:46

usually. I guess it depends on how youā€™re running it

phronmophobic21:03:07

thereā€™s only one place in your code with multiplication, so I would look there

Spaceman21:03:54

It doesn't give an exception anymore for some reason.

Spaceman21:03:12

so I'd say success.

Spaceman21:03:18

Now there's this python code:

def filterOutSaltPepperNoise(edgeImg):
    # Get rid of salt & pepper noise.
    count = 0
    lastMedian = edgeImg
    median = cv2.medianBlur(edgeImg, 3)
    while not np.array_equal(lastMedian, median):
        # get those pixels that gets zeroed out
        zeroed = np.invert(np.logical_and(median, edgeImg))
        edgeImg[zeroed] = 0

        count = count + 1
        if count > 70:
            break
        lastMedian = median
        median = cv2.medianBlur(edgeImg, 3)

Spaceman21:03:43

with the line np.array_equal(lastMedian, median).

Spaceman21:03:06

Since I don't have np in java how to compare these two Mat's?

Spaceman21:03:28

like do I convert them into a clojure vector first and then compare?

Spaceman22:03:12

with doseq I can iterate through the mat and do something side-effect related on its individual components, but how to return a clojure vector?

phronmophobic22:03:56

i would probably use loop

Spaceman22:03:23

like nested loops?

Spaceman22:03:27

(loop [y 0] (loop [x 0] ... (recur ...) ) ... (recur ....) )

Spaceman22:03:53

Or maybe one can use reduce somehow

phronmophobic22:03:49

I think your intuition of converting the matrix to a clojure vector of vectors is probably the easiest thing to get back into clojure land

phronmophobic22:03:23

to convert a matrix to a vector, I would probably do somthing like

(vec
  (for [i (range width)]
    (vec 
     (for [j (range height)]
      (.get mat i j)))))
fixed*?

phronmophobic22:03:36

or something like that. coding in slack without* paredit is hard

Spaceman22:03:34

(.get mat i j) itself returns a vector, because the image can have multiple color channels

Spaceman22:03:53

so it would be a 3d not a 2d vector.

phronmophobic22:03:01

you may want to wrap (.get mat i j) in a vec as well to make it a clojure vector

Spaceman22:03:36

(defn ->vector [mat]
  (vec
   (for [i (range width)]
     (vec
      (for [j (range height)]
        (vec
         (let [p (.get mat i j)]
           (for [c (range (count p))]
             (nth p c)
             )
           )
          )))))
  )
I think is fine

Spaceman22:03:04

No actually that gives a null pointer exception

phronmophobic22:03:15

(vec (.get mat 0 0)) work?

phronmophobic22:03:43

you may not need a third level

phronmophobic22:03:59

and youā€™ll almost certainly want to get width and height from the passed in mat

phronmophobic22:03:35

I would also check the javadoc for whatever the type mat is, there might already be a more helpful method for accomplishing this

Spaceman22:03:34

doesn't seem to be. What if I wanted to make a vector of arbitrary dimension n?

phronmophobic22:03:42

I was hoping it implemented a java interface that made it seqable

Spaceman22:03:52

yes (vec (.get ...)) works

phronmophobic22:03:38

instead of width and height, youā€™ll want to use (.cols mat) and (.rows mat)

phronmophobic22:03:26

or (.width mat) and (.height mat) ?

phronmophobic22:03:50

not sure if itā€™s the same or depends on if the matrix is irregular

Spaceman22:03:59

Is there something that does logical and of two vectors of arbitrary dimensions in clojure?

Spaceman22:03:30

numpy version

zeroed = np.invert(np.logical_and(median, edgeImg))

Spaceman22:03:25

where median and edgeImg are single color channel image vectors

phronmophobic22:03:26

iā€™m sure there is, but itā€™s fairly easy to write

(mapv (fn [channel1 channel2] (some-compare channel1 channel2)) median edge-img)

phronmophobic22:03:29

something like that

phronmophobic22:03:43

this will truncate to the size of the smaller vector

phronmophobic22:03:26

and some-compare is the logical comparison function you would need to write

Spaceman22:03:50

some-compare is just *

Spaceman22:03:02

But this doesn't work: (logical-and [[[1] [0]] [[1] [1]]] [[[1] [0]] [[0] [0]]])

Spaceman22:03:42

should return [[[1][0]][[0][0]]]

Spaceman22:03:51

but I get clojure.lang.PersistentVector cannot be cast to java.lang.Number

Spaceman22:03:44

Like: [[[1] [0]] [[1] [1]]] and [[[1] [0]] [[0] [0]]]

phronmophobic22:03:52

everything in numpy works on vectors

phronmophobic22:03:18

the default clojure operations donā€™t

Spaceman22:03:19

are two images and we are taking the logical and of them to get [[[1][0]] [[0][0]]]

phronmophobic22:03:25

there are libraries like that, http://incanter.org/

phronmophobic22:03:35

i havenā€™t used them in a while

phronmophobic22:03:42

which will let you write* code in that style

phronmophobic22:03:12

but otherwise, you have to do maps and reduces

Spaceman23:03:48

okay, let's say we have only three channels. What would be the last step?

Spaceman23:03:13

So that each corresponding color channel is and'ed

phronmophobic23:03:32

i have to step out, but it seems like youā€™re at least through the toughest part! you may want to try some clojure exercises that will cover a lot of the tools youā€™ll need

souenzzo23:03:54

Try *e on REPL

Spaceman23:03:39

I tried this, but it didn't work: (defn logical-and [mat1 mat2] (mapv (fn [channel1 channel2] (map (fn [mat1-c mat2-c] (* mat1-c mat2-c)) channel1 channel2)) mat1 mat2) )

souenzzo23:03:47

Side effects with returning usually use reduce

Spaceman23:03:17

There are no side-effects in the function logical-and

souenzzo23:03:44

You can swap doseq with for and use a doall to ensure that there is no lazyness

Spaceman23:03:24

well, how do I fix the logical-and?

(defn logical-and [mat1 mat2]
  (mapv (fn [channel1 channel2] (map (fn [mat1-c mat2-c] (* mat1-c mat2-c)) channel1 channel2)) mat1 mat2)
  )

souenzzo13:04:10

@pshar10 not sure. But take a look at https://neanderthal.uncomplicate.org/ I think that neanderthal implements many of these operations

souenzzo14:04:27

sorry. my slack client fucked up and send many delayed messages šŸ˜ž

Spaceman12:04:22

Hi, suppose I have an array list created like so (def some-array-list (ArrayList.)), and it itself contains array-lists. I want to sort this some-array-list based on the first elements of the constituent array lists. How can I do that?

phronmophobic20:03:17

@didibus, I didnā€™t want to sideline spacemanā€™s thread so Iā€™m replying this separate thread

phronmophobic20:03:57

thereā€™s a platform agnostic that all the ui codes against

phronmophobic20:03:41

then thereā€™s several implementations for the graphical primitives. the main ones right now are skia for desktop, https://skia.org/ and webgl for the web

phronmophobic20:03:20

being able to render the DOM is likely at some point. not sure how likely implementations for cocoa, X, or windows are

phronmophobic20:03:53

not sure if that answered your question

didibus07:04:13

Hum... I guess kind of. So it does sound a bit like you're building your own UI toolkit, though the rendering for different OS seems like it can be handled by skia, which I hadn't heard off before

didibus07:04:19

Anyways, looks pretty cool

phronmophobic17:04:38

the main goal is having as much as possible in clojure itself

phronmophobic17:04:18

my background is in mobile games and web and for both of those, you end up styling your own buttons/textboxes/checkboxes/etc yourself

phronmophobic17:04:54

itā€™s been a long time since Iā€™ve worked on something where someone cared about having ā€œnativeā€ UI widgets

didibus17:04:20

Ah, I wasn't really caring either way, I was more curious šŸ˜„

šŸ‘ 4
didibus17:04:36

I think people care more about the look and feel matching with the rest of their desktop, and just overall looking good. People did not like old Java Swing in my opinion mostly because it was ugly and had weird UX by default compared to native ones

phronmophobic17:04:19

I think the web has shifted that view

didibus17:04:50

Ya, I'd say it did, at least for looking similar to the rest of the desktop, but I think people still expect things to look good and behave intuitively, maybe even more so than before

didibus17:04:38

And most programmers suck at visual design šŸ˜›, so they often expect their UI library to auto-style and make everything beautiful by default

phronmophobic18:04:50

iā€™ve been spending most of my time trying to get the fundamentals right from a programming perspective. the plan for making it look good is a GUI builder targeting graphic designers*. I think itā€™s been a mistake to put programmers in charge of pixels

phronmophobic18:04:36

Iā€™ve made a few prototypes for the GUI builder a la http://worrydream.com/DrawingDynamicVisualizationsTalkAddendum/ with some new ideas, but still havenā€™t had a chance to make it available yet

didibus19:04:06

Ah that's neat. Well I'll keep at eye on it all, seems quite promising.

colinkahn20:03:20

Are there any good libraries for refactoring keywords in a Clojure codebase? Seems like a non-trivial task just because of the different form they can take like {:keys [ā€¦]}

p-himik20:03:12

Cursive seems to be handling it quite well.

colinkahn20:03:10

was hoping for something more scriptable

noisesmith21:03:24

this is easier with namespaced keywords, but unless your keyword is something like :f or :test a global search for the symbol of the keyword will catch literal uses and keys destructures (that works for f or test too, but you get too many false positives along with it...)

šŸ‘ 4
noisesmith21:03:59

here's a regex search of all functions loaded via require

user=> (->> (all-ns) (mapcat (comp keys ns-publics)) (keep clojure.repl/source-fn) (filter #(re-find #"foo" %)) count)
6

noisesmith21:03:16

it doesn't find repl definitions (a limitation of source-fn) but that shouldn't matter for you

noisesmith21:03:47

if your app is already loaded, it probably runs faster than your editor would for the same search

colinkahn23:03:11

@noisesmith thanks, in this case the keys arenā€™t namespaced but I can see how far I can get with find/replace

noisesmith00:04:44

readable / usable version of the source regex search above:

(defn all-qualified-symbols
  [ns]
  (->> ns
       (ns-publics)
       (keys)
       ;; takes the symbol, and turns it into a namespaced symbol of ns
       (map #(symbol (name (.name ns))
                     (name %)))))

(defn source-search
  [re]
  (->> (all-ns)
       (mapcat all-qualified-symbols)
       ;; get a "header" and the source, if findable
       (map (juxt #(str "\n****** " % " ******")
                  clojure.repl/source-fn))
       (filter #(some->> %
                         peek
                         (re-find re)))
       ;; "flatten" by one level
       (into [] cat)
       (run! println)))

Spaceman23:03:11

I want to do a logical and of two vectors: [[[1] [0]] [[1] [1]]] and [[[1] [0]] [[0] [0]]] to get [[[1][0]] [[0][0]]]

rutledgepaulv00:04:42

I use a piecewise function to help with this kind of thing.

(defn piecewise [f & colls]
  (cond
    (empty? colls)
    ()
    (= 1 (bounded-count 2 colls))
    (seq (first colls))
    :else
    (for [cells (apply map vector colls)]
      (reduce f cells))))

(piecewise bit-and [1 0] [0 1])

Spaceman23:03:25

How to do this in clojure? Is there a way to do this for arbitrary dimensions?

OrdoFlammae23:03:01

@pshar10, maybe just write your own utility? It doesn't seem like it would be terrifically hard. This seems like an edge-case that probably isn't covered by the standard library.

Spaceman23:03:42

I have this so far for there dimension case. For now, three all I want:

(defn logical-and [mat1 mat2]
  (mapv (fn [channel1 channel2] (map (fn [mat1-c mat2-c] (* mat1-c mat2-c)) channel1 channel2)) mat1 mat2)
  )

Spaceman23:03:32

but it gives clojure.lang.PersistentVector cannot be cast to java.lang.Number

Eddie23:03:19

For arbitrary dimensions, you could always use recursion until you hit a scalar. Something along these lines.

(defn element-wise-and
  [a b]
  (if (and (integer? a) (integer? b))
    (int (and a b))
    (map element-wise-and a b)))
(^^^ Coded in slack... not sure if that is exact working code)