Clojurians Log v2

@caleb.macdonaldblack that worked just fine, much appreciated!

I ended up going with stream-buffers

@p-himik thanks for the js-await suggestion!

And many thanks to everyone else who chipped in

Probably because the future returned by .close doesn't get resolved. As to "why" - no clue, that would be a question for the authors of the Parquet library.

Have you tried removing the calls to log/spy? No clue how that would work, but maybe it holds some references inside writer that prevent it from being closed.

could it be that .close is throwing?

@lilactown that does make the most sense to me, but I can't catch anything...

Consider the following:

(defn create-parquet []
  (let [{:keys [stream get-buffer]} (create-memory-stream)
        schema (parquet/ParquetSchema. (clj->js {
                                                 :name {:type "UTF8"}
                                                 :age  {:type "INT64"}
                                                 }))
        opts #js {}]
    (-> (prom/let [writer (parquet/ParquetWriter.openStream schema stream opts)
                   _ (.appendRow writer #js {:name "apples" :age 10})
                   _ (.close writer)
                   ]
          writer)
        (prom/handle (fn [result error]
                       (log/spy result)
                       (log/spy error))))))

This will print nothing at all

if I comment _ (.close writer) , that (spy result) will print the writer object...

So it does seem that .close is blowing up, but I should see something on the error arg of handle , right

this is the source of close btw https://github.com/ZJONSSON/parquetjs/blob/3e5d76b781bc9045c95ccfc087e7792480f85667/lib/writer.js#L114

I can't see anything that would eat up the error. JS is not my forte though...

It accepts a callback. If you pass something, does it ever get called?

Some of the writers i tried don’t call the promise callback. this works though:

(defn create-parquet []
  (async/go
    (let [schema    (new (.-ParquetSchema parquetjs) (clj->js {:name {:type "UTF8"} :age {:type "INT64"}}))
          stream (new (.-PassThrough readable-stream))
          writer    (<p! (.openStream (.-ParquetWriter parquetjs) schema stream (clj->js {})))]
      (spy "A" writer)
      (<p! (.appendRow writer #js {:name "apples" :age 10}))
      (spy "B" writer)
      (<p! (.appendRow writer #js {:name "oranges" :age 11}))
      (spy "C" writer)
      (js/console.log writer)
      (<p! (doto (.close writer)
                 (js/console.log)))
      (spy "D" writer))))

Stuff like (new (.-PassThrough readable-stream)) can be written as (readable-stream/PassThrough.) (assuming readable-stream is an alias coming from :require).

woah thanks!

This one repoduces the problem:

(defn create-parquet []
  (async/go
    (let [schema    (new (.-ParquetSchema parquetjs) (clj->js {:name {:type "UTF8"} :age {:type "INT64"}}))
          ;stream (new (.-PassThrough readable-stream))

          newHandle (<p! (js/window.showSaveFilePicker))
          stream    (<p! (.createWritable newHandle))

          writer    (<p! (.openStream (.-ParquetWriter parquetjs) schema stream (clj->js {})))]
      (spy "A" writer)
      (<p! (.appendRow writer #js {:name "apples" :age 10}))
      (spy "B" writer)
      (<p! (.appendRow writer #js {:name "oranges" :age 11}))
      (spy "C" writer)
      (js/console.log writer)
      (<p! (doto (.close writer)
                 (js/console.log)))
      (spy "D" writer))))

But I can’t see the source code because its internal to the browser

Specifically here

A.write

const oswrite = function (os, buf) {
    return new Promise((resolve, reject) => {
        os.write(buf, (err) => {
            if (err) {
                reject(err);
            }
            else {
                resolve(err);
            }
        });
    });
};

That code is fine. I think there is something wrong with the implementation of .write on some streams

I'd try creating a minimal reproducible example in plain JS and posting it on their issue tracker since at this point it has nothing to do with CLJS at all.

agreed

the clojure here is fine

Just in case you haven't seen me mentioning it times and times again on this server - I'd still ditch core.async here. :) The use case is trivial, plain promise interop or the js-await macro available in shadow-cljs (that you can just copy from there, it's small) would be perfectly fine, with zero overload and no potential debugging or externs inference issues.

yeah I’ve never use core.async outside of experimentation.

js/Promise & futures go pretty far

No. Use the dot syntax for all code, add ^js as necessary. Use goog.object for data. Or unchecked-get.

Can you elaborate on what you mean by code vs data here? By data do we mean something that can only be learned at runtime? As in, we fetch some json from over the wire.

Can be a compile-time constant - #js {...}. Can be something coming from a JS library. Or from js-obj.

Your saying that "code" can be a compiled time constant, right?

No, that data can be.

Code is already most often a compiled-time thing. Very, very rarely does one have to create some actual code at run time.