Fork me on GitHub
Slack Archive

This page is not created by, affiliated with, or supported by Slack Technologies, Inc.

2022-09-02

Channels

#nbb
<
2022-09-02
>
genRaiy16:09:26

I have some sample JS code to read a S3 object that I would like to translate to Clojure but :man-shrugging::skin-tone-3: I'm too dumb so can anyone help me with my homework?

genRaiy16:09:50

export const run = async () => {
  try {
    // Create a helper function to convert a ReadableStream to a string.
    const streamToString = (stream) =>
      new Promise((resolve, reject) => {
        const chunks = [];
        stream.on("data", (chunk) => chunks.push(chunk));
        stream.on("error", reject);
        stream.on("end", () => resolve(Buffer.concat(chunks).toString("utf8")));
      });

    // Get the object} from the Amazon S3 bucket. It is returned as a ReadableStream.
    const data = await s3Client.send(new GetObjectCommand(bucketParams));
      return data; // For unit tests.
    // Convert the ReadableStream to a string.
    const bodyContents = await streamToString(data.Body);
    console.log(bodyContents);
      return bodyContents;
  } catch (err) {
    console.log("Error", err);
  }
};
run();

lilactown16:09:08

which part are you struggling with?

lilactown16:09:31

what have you tried? I'd write it like (.on stream "data" #(.push chunks %)) etc.

1
genRaiy16:09:21

I think I was going code blind

lilactown16:09:54

ur working too hard ray. take a break!

💯 1
1
genRaiy17:09:53

Error: AwaitExpression is not implemented
Couldn't compile JavaScript code into ClojureScript :(

valtteri19:09:08

Here’s one version that could work. 

(ns s3
  (:require ["@aws-sdk/client-s3" :refer [S3Client GetObjectCommand]]))

(def cli (S3Client. #js{:region "eu-west-1"}))

(-> (.send cli (GetObjectCommand. #js{:Bucket "my-bucket" :Key "some-file.txt"}))
    (.then
     (fn [resp]
       (js/Promise.
        (fn [resolve reject]
          (let [stream (.-Body resp)
                arr #js[]]
            (.on stream "data" (fn [chunk] (.push arr chunk)))
            (.on stream "error" reject)
            (.on stream "end" (fn [] (resolve (-> arr
                                                     js/Buffer.concat
                                                     (.toString "utf-8"))))))))))
    (.then println)
    (.catch println))

😍 1
valtteri19:09:37

Dunno how necessary that “streaming” there really is

valtteri19:09:43

Sorry, actually tried that and it had a couple of bugs. 🙂

1
valtteri19:09:01

Now it works

valtteri19:09:43

That :region eu-west-1 was required for my test-bucket because it resides there.

valtteri19:09:02

And if you want to go cool you can also use the node streams api to pipe the response to some other thing that supports streams. Like here I’m piping the body to stdout. 

(ns s3
  (:require ["@aws-sdk/client-s3" :refer [S3Client GetObjectCommand]]
             ["stream/promises" :as sp]))

(def cli (S3Client. #js{:region "eu-west-1"}))

(-> (.send cli (GetObjectCommand. #js{:Bucket "nbb-adapter-test" :Key "concave.json"}))
    (.then (fn [resp] (sp/pipeline (.-Body resp) js/process.stdout))))

valtteri19:09:42

Pipeline can also have transforming steps. I think I wrote something that here a while ago

valtteri19:09:24

I don’t know what you’re going to do with the s3 object but if you for instance want to send it forward you can do that with very low memory footprint using streams.

genRaiy20:09:00

just processing some data but those are great tricks - thanks @U6N4HSMFW