This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2022-08-27
Channels
- # announcements (8)
- # babashka (2)
- # calva (21)
- # clojure (30)
- # clojure-europe (4)
- # conjure (1)
- # core-logic (5)
- # data-science (1)
- # emacs (10)
- # hyperfiddle (1)
- # introduce-yourself (1)
- # luminus (62)
- # meander (23)
- # nbb (26)
- # off-topic (9)
- # reitit (2)
- # spacemacs (2)
- # squint (65)
- # tools-build (4)
- # tools-deps (17)
- # xtdb (20)
That made me wonder: I think it would be useful if we would have a babel plugin for clava perhaps
@lilactown I went ahead and merged your work on lazy seqs. Since I also made a few improvements, I thought it would be too complex to wait with merging. Now can incrementally move forward. E.g.: partition
and partition-all
should be made lazy now
Hi @U04V15CAJ I can work on this tomorrow if no one has picked it up yet.
Made an issue for that: https://github.com/clavascript/clavascript/issues/146
I guess we should document the caveat that a lazy iterator, when used multiple times, is re-calculated (similar to eduction):
(def seq1 (map (fn [i] (prn i) i) [1 2 3]))
(prn seq1)
(def seq2 (map inc seq1))
(prn seq2)
1
core.js:287 2
core.js:287 3
core.js:287 [1,2,3]
core.js:287 1
core.js:287 2
core.js:287 3
core.js:287 [2,3,4]
since we don't cache them, their underlying (mutable) structure can change and it will reflect those changes
If we would add caching it would behave less surprising for CLJS users maybe. But I don't know what would be the cost of adding that
But that would result into GC problems because even when you don't use the front of the seq anymore you would hold on to the entire array which is bad
Unless you splice the array etc. Not sure how performant this is compared to a linked list
And before you know it weβre re-implementing all of CLJS (which might not be that bad if you get back good treeshaking and improved interop)
@lilactown That seems like a bug to me (with the recent changes):
$ ./node_cli.js -e '(first (rest (range)))'
hangsSeems it does not: https://clojure.atlassian.net/plugins/servlet/mobile?originPath=%2Fbrowse%2FCLJS-705#issue/CLJS-705
Seems it's rare that in-browser people process large data, so holding on the head and consuming more memory then needed might not be a big problem
This post might be interesting: https://blog.fikesfarm.com/posts/2016-01-15-clojurescript-head-holding.html
> Seems it's rare that in-browser people process large data When procedurally generating audio I frequently create gigantic arrays of data. Just one use-case to be aware of. Another place is crypto stuff.
Yep, added these notes: https://github.com/clavascript/clavascript#memory-usage
I think when dealing with these gigantic mutable arrays you would usually not use them in combination with lazy seqs anyway
Yes probably true I don't think I've done that.
The smart thing to do is probably create a fixed Float32Array or similar and write into it but I think I have use vec
before too.
Wow that's going to take me a few days to parse. Will get back to you!
Presumably another bordude tip that flips my coding practice on it's head!
Ok wait I thought matrix was a funciton here. It's getting late. π
I should go to sleep ha ha. π
hehe. This does work right now with clavascript:
$ ./node_cli.js -e '(prn (update-in [[0 0] [0 0]] [0 0] inc))'
[[1,0],[0,0]]
but it creates new arrays, which isn't that performant. This is why we also should have update-in!
Very nice. I'm amazed by how fast you are all moving on this.
maybe rest should be something like:
function* rest(x) { let iter = iterator(x); iter.next(); yield* iter; }
but this has the problem that the first element will always be held on to - or not?well, you have a solution for that, I guess. the thing I was wondering about is how to "let go" of the first
export function rest(coll) {
return new LazyIterable(function* () {
let first = true;
for (const x of iterable(coll)) {
if (first) first = false;
else yield x;
}
});
}
I think might workI think that we should look at LazyIterable
as a whole. this would I imagine have the same behavior as any other iteration
here's how to get GC info in node: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Memory_Management#node.js
@lilactown I've done this test in node:
// run with: node --expose-gc mem.js
import { rest, first } from './core.js';
var x = new Array(10000000).fill(0); // about 85mb
global.gc();
console.log(process.memoryUsage());
var y = [x,1];
y = rest(y);
console.log(first(y));
x = null;
global.gc();
console.log(process.memoryUsage());
Unfortunately, the memory big array doesn't seem to be freedit does work as expected in cherry when I use a cons
so it might be the wrapper array not being garbage collected:
var y = cons(new Array(10000000).fill(0), cons(1, null));
I played with this in cherry: to use immutable.js instead of the clojure data structures to get the same value semantics, while having a smaller bundle (65kb or so vs 300kb). I didn't go through with that since immutable.js probably has some differences to CLJS, but for something like clavascript this difference might be ok (since we don't promise compatibility). If we would adopt that as the standard thing, we could get a lot of things back what we had in CLJS (immutability, value semantics). The major thing that bothers me about that is interop: you can't just pass an immutable map to another API, but you first have to convert it with toJS
etc
The immutableJS lib has a (lazy) seq data structure that we might want to borrow, even if maps will still be mutable in clava. (See GC issue above)
I tried it and the best you can get is all 60kb. Maybe it's best to go back to the state we were before and base stuff on concrete arrays in order to stay as close to JS without surprises (GC/recalculations when you use a result multiple times, etc) and a complex/laborious + big stdlib. I bet most of these lodash-like libs do this. Thoughts, @lilactown?
iterable in -> array out. just need to document why we went back to this approach, if we do, so we remember why we did so
that, or we could go back to the "map returns iterable you can only use once" approach
the only "surprising" (?) thing there is if you mutate the underlying collection, e.g.
let a = [1, 2, 3]
let b = rest(a);
prn(vec(b));
a.push(4)
prn(vec(b));
will print [2, 3]
both times. whereas if you put the .push
before the first realization of it, they would both print [2, 3, 4]