I'm all in on lazy iterables. forget transducers 😄

Now running JSX tests with babel/core :)

That made me wonder: I think it would be useful if we would have a babel plugin for clava perhaps

@lilactown I went ahead and merged your work on lazy seqs. Since I also made a few improvements, I thought it would be too complex to wait with merging. Now can incrementally move forward. E.g.: partition and partition-all should be made lazy now

Made an issue for that: https://github.com/clavascript/clavascript/issues/146

Since all tests still worked, I thought it would be ok to merge and go forward

https://twitter.com/borkdude/status/1563485312592269320

I guess we should document the caveat that a lazy iterator, when used multiple times, is re-calculated (similar to eduction):

(def seq1 (map (fn [i] (prn i) i) [1 2 3]))

(prn seq1)

(def seq2 (map inc seq1))

(prn seq2)

1
core.js:287 2
core.js:287 3
core.js:287 [1,2,3]
core.js:287 1
core.js:287 2
core.js:287 3
core.js:287 [2,3,4]

since we don't cache them, their underlying (mutable) structure can change and it will reflect those changes

we could add caching

If we would add caching it would behave less surprising for CLJS users maybe. But I don't know what would be the cost of adding that

Could add an array which contains the realized elements or so?

And grow that array when a non realized element is requested

right

But that would result into GC problems because even when you don't use the front of the seq anymore you would hold on to the entire array which is bad

Unless you splice the array etc. Not sure how performant this is compared to a linked list

And before you know it we’re re-implementing all of CLJS (which might not be that bad if you get back good treeshaking and improved interop)

But just leaving this as it is now would also be fine maybe

With the current implementation you can decide where to cache yourself by using vec

rest also caches, since destructuring returns an array

let [_, ...rest] = iterable(coll)

@lilactown That seems like a bug to me (with the recent changes):

$ ./node_cli.js -e '(first (rest (range)))'

agreed

If you cache I believe you need locals clearing take it work no?

maybe rest should be something like:

function* rest(x) { let iter = iterator(x); iter.next(); yield* iter; }

that has the problem that it would return an iterator, which would be mutable

well, you have a solution for that, I guess. the thing I was wondering about is how to "let go" of the first

sorry, not trying to nit pick

export function rest(coll) {
  return new LazyIterable(function* () {
    let first = true;
    for (const x of iterable(coll)) {
      if (first) first = false;
      else yield x;
    }
  });
}

it's basically the same as dropping the first

hmm nice. how can we verify that GC works as it should?

there's probably ways to inspect GC in chrome

I think that we should look at LazyIterable as a whole. this would I imagine have the same behavior as any other iteration

here's how to get GC info in node: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Memory_Management#node.js

@lilactown I've done this test in node:

// run with: node --expose-gc mem.js

import { rest, first } from './core.js';

var x = new Array(10000000).fill(0); // about 85mb

global.gc();

console.log(process.memoryUsage());

var y = [x,1];

y = rest(y);

console.log(first(y));

x = null;

global.gc();

console.log(process.memoryUsage());

But so isn't it in cherry, so I could be doing something wrong here

it does work as expected in cherry when I use a cons so it might be the wrapper array not being garbage collected:

var y = cons(new Array(10000000).fill(0), cons(1, null));

GC issue: https://github.com/clavascript/clavascript/issues/160

I played with this in cherry: to use immutable.js instead of the clojure data structures to get the same value semantics, while having a smaller bundle (65kb or so vs 300kb). I didn't go through with that since immutable.js probably has some differences to CLJS, but for something like clavascript this difference might be ok (since we don't promise compatibility). If we would adopt that as the standard thing, we could get a lot of things back what we had in CLJS (immutability, value semantics). The major thing that bothers me about that is interop: you can't just pass an immutable map to another API, but you first have to convert it with toJS etc

The immutableJS lib has a (lazy) seq data structure that we might want to borrow, even if maps will still be mutable in clava. (See GC issue above)

I tried it and the best you can get is all 60kb. Maybe it's best to go back to the state we were before and base stuff on concrete arrays in order to stay as close to JS without surprises (GC/recalculations when you use a result multiple times, etc) and a complex/laborious + big stdlib. I bet most of these lodash-like libs do this. Thoughts, @lilactown?

iterable in -> array out. just need to document why we went back to this approach, if we do, so we remember why we did so

that, or we could go back to the "map returns iterable you can only use once" approach

I've used ImmutableJS in apps and hated it exactly for the whole toJS fromJS dance

I think we should try adding a cache to the LazyIterable

the only "surprising" (?) thing there is if you mutate the underlying collection, e.g.

let a = [1, 2, 3]

let b = rest(a);

prn(vec(b));

a.push(4)

prn(vec(b));

Maybe it would be easier to start with transducer no? Iterables are closer to transducers it seems. With transducer it's all just adding up the set of transforms and when ready the iterables is passed to the chain and collected into a concrete collection of the user's choosing.