Fork me on GitHub
#babashka
<
2020-10-14
>
bartuka00:10:14

hello guys, I am trying to use the s3-pod inside a bb app.. but I am having trouble to figure out what is the expected format for my :Body object in a PutObject operation. I tried to pass the same format as the aws-api from cognitect expects and got this error.

29: (defn run [opts]
30:   (let [{:keys [filepath partner content dag-name]} opts]
31:     (s3/put-object {:Bucket "captalys-analytics"
        ^--- Cannot JSON encode object of class: class java.io.BufferedInputStream: java.io.BufferedInputStream@7fa4620f4f20
32:                     :Key "testando"
33:                     :Body (io/input-stream (io/file filepath))
34:                     :Tagging "dagName=OK"})))

jumar03:10:05

Does this work in the standard REPL? If not you may want to ask in #aws

bartuka03:10:44

when using the aws-api from cognitect it works without a problem. The issue is the interface between clj and golang sdk managed by https://github.com/tzzh/pod-tzzh-aws

bartuka00:10:46

If I try to pass bytes (as is specified in the go sdk) I then ge tthe following error

29: (defn run [opts]
30:   (let [{:keys [filepath partner content dag-name]} opts]
31:     (s3/put-object {:Bucket "captalys-analytics"
        ^--- json: cannot unmarshal string into Go struct field PutObjectInput.Body of type io.ReadSeeker
32:                     :Key "testando"
33:                     :Body (.getBytes "okokk")
34:                     :Tagging "dagName=OK"})))
seems like the msg is converting my bytes to string and the go structure cannot parse it

borkdude07:10:10

@iagwanderson Have you tried passing in a string? The arguments to a pod are serialized via EDN or JSON, so there should be special handling of bytes in a pod (e.g. serialize to base64 under the hood in bb and deserialize to bytes at the pod side). This can be implemented so the user doesn't even notice, but it does need attention. Maybe @thomas.ormezzano can think about this?

tzzh09:10:13

ah yes good point I am afraid bytes arguments are not supported atm I need to look into this

tzzh09:10:16

also I noticed there are a bunch of helpers functions in the s3manager module that could be helpful for uploading/downloading https://docs.aws.amazon.com/sdk-for-go/api/service/s3/s3manager/ - I mostly use the s3 fns for listing/deleting files so I haven’t looked into this as much but going to have a look when I have a bit of time

borkdude09:10:44

@thomas.ormezzano Should be doable I think using a client side wrapper which coerces the bytes into base64 (or something similar which can go through json)

👍 3
borkdude09:10:21

@thomas.ormezzano Maybe pods should start supporting transit. If there's a transit library for Go which supports bytes, then we could just use that

borkdude09:10:27

@thomas.ormezzano Alternatively you could invent your own representation of bytes, like: {"tzzh/bytes" "base64string"} so if you encounter an arg like that on the pod side, deal with it You could even choose to just use bencode as the byte representation, since bencode supports sending just bytes I think

borkdude09:10:35

This is what I do with database connection objects and terminal objects in pods too. Right now I'm working on a lantera pod: https://github.com/babashka/pod-babashka-lanterna

tzzh09:10:09

ok thanks I will take a look and see what’s easier :thumbsup:

borkdude11:10:07

Found a Go transit lib here: https://github.com/russolsen/transit If you can verify that it's possible to serialize bytes using this lib, I'm willing to add support for transit to babashka.pods

borkdude09:10:33

@thomas.ormezzano I now added transit+json support to babashka.pods and pushed it to bb master. To use this format, specify "format" "transit+json" in the describe message.

borkdude09:10:58

This will allow you to send bytes back and forth, hopefully. I don't know how well the russolsen/transit lib works.

tzzh09:10:14

ok cool thanks I had actually started looking at doing using base64 encoding (and add some client side code to do the decoding transparently) so maybe I’ll finish that first and look at transit after, it’s a bit hard to tell if the lib is good but it’d be cool to try it out either way

bartuka12:10:44

hello guys, thanks for the comments. I tried to pass a plan string but got the same error

29: (defn run [opts]
30:   (let [{:keys [filepath partner content dag-name]} opts]
31:     (s3/put-object {:Bucket "captalys-analytics"
        ^--- json: cannot unmarshal string into Go struct field PutObjectInput.Body of type io.ReadSeeker
32:                     :Key "testando"
33:                     :Body "new string!"
34:                     :Tagging "dagName=OK"})))

tzzh12:10:39

yeah you won’t be able to get this to work at the moment as the underlying go type is

Body io.ReadSeeker `type:"blob"
` 😕

tzzh12:10:09

I’ll try to find have a look later today/this week if I have time

👍 3
borkdude12:10:49

meanwhile I will see if I can support transit in pods. would be nice either way

bartuka12:10:11

ok, got it!

borkdude12:10:28

ok, there's not much you can do then right now. see the above.

Dig17:10:26

How do I check if var is bound in bb?

Babashka v0.2.2 REPL.
Use :repl/quit or :repl/exit to quit the REPL.
Clojure rocks, Bash reaches.

user=> (def a "a")
#'user/a
user=> (bound? #'a)
java.lang.ClassCastException: sci.impl.vars.SciVar cannot be cast to clojure.lang.Var [at <repl>:2:1]

borkdude17:10:02

@i.slack Ah, that needs fixing. For now you can maybe use:

user=> (find-var 'user/a)
#'user/a

borkdude17:10:19

or resolve / ns-resolve

Dig17:10:23

I was trying to see if *file* is bound to distinguish from clj run

Dig17:10:52

is there a better way to know that it is running in bb?

borkdude17:10:09

you can check (System/getProperty "babashka.version")

Dig17:10:31

ahh, thank you! that would be much cleaner.

isak18:10:38

For authoring pods, and this example for error handling:

{"id" "1d17f8fe-4f70-48bf-b6a9-dc004e52d056"
 "ex-message" "Illegal input"
 "ex-data" "{\"input\": 10}
 "status" "[\"done\", \"error\"]"}
What is the "status"? Do we need it?

borkdude19:10:01

yes, error signals whether we should raise an error and done signals that it's the last message for this id

isak19:10:23

oh ok, cool

borkdude19:10:09

Meanwhile: tetris works in pod-babashka-lanterna :)

😎 18
isak19:10:37

For replies from the pod where the messages send a few messages for a request id without being done, how are the messages accumulated/handled on the pod client?

borkdude19:10:42

typically for async functions you have multiple messages coming in

isak20:10:15

:thinking_face: I've been reading that document, but I couldn't see that it was specified.

isak20:10:34

Looked at the source, but not obvious what will happen without knowing how the functions in that namespace will be called. I guess I'll just experiment, will be faster

isak20:10:28

Makes sense if only async calls are involved with the :success, :error and :done callbacks, just not as obvious for the normal calls

borkdude20:10:28

@isak Note that normal functions can also send multiple "out" messages which will be sent to stdout. Also they can send the value and status in separate messages.

borkdude20:10:53

This is more or less the same as with the nREPL protocol

isak20:10:45

Ok makes sense. But only 1 message can deliver the value when not calling the async way, correct? @borkdude

isak20:10:20

Ok, then I'm no longer confused, thanks

isak23:10:54

In a C# pod I'm trying to make, I'm getting an error from babashka when sending this message from the pod:

async Task SendResult(string id, object result) {
    var resp = new BDictionary {
        ["id"] = new BString(id),
        ["value"] = new BString("[]"),
        ["status"] = new BString(JsonSerializer.Serialize(new string[] { "done"}))
    };
    await _writer.WriteAsync(resp.EncodeAsBytes());
}
Error:
#error {
 :cause "java.lang.Byte cannot be cast to byte[]"
 :via
 [{:type java.lang.ClassCastException
   :message "java.lang.Byte cannot be cast to byte[]"
   :at [babashka.pods.impl$bytes__GT_string invokeStatic "impl.clj" 22]}]
 :trace
 [[babashka.pods.impl$bytes__GT_string invokeStatic "impl.clj" 22]
  [babashka.pods.impl$bytes__GT_string invoke "impl.clj" 22]
  [clojure.core$comp$fn__5811 invoke "core.clj" 2571]
  [clojure.core$map$fn__5870 invoke "core.clj" 2757]
  [clojure.lang.LazySeq sval "LazySeq.java" 42]
  [clojure.lang.LazySeq seq "LazySeq.java" 51]
  [clojure.lang.RT seq "RT.java" 535]
  [clojure.core$seq__5405 invokeStatic "core.clj" 137]
  [clojure.core$reduce1 invokeStatic "core.clj" 930]
  [clojure.core$set invokeStatic "core.clj" 4115]
  [babashka.pods.impl$processor invokeStatic "impl.clj" 108]
  [babashka.pods.sci$fn__15492$fn__15525 invoke "sci.clj" 67]
  [sci.impl.vars$binding_conveyor_fn$fn__7722 invoke "vars.cljc" 144]
  [clojure.core$binding_conveyor_fn$fn__5758 invoke "core.clj" 2032]
  [clojure.lang.AFn call "AFn.java" 18]
  [java.util.concurrent.FutureTask run "FutureTask.java" 264]
  [java.util.concurrent.ThreadPoolExecutor runWorker "ThreadPoolExecutor.java" 1128]
  [java.util.concurrent.ThreadPoolExecutor$Worker run "ThreadPoolExecutor.java" 628]

isak23:10:46

This is on windows. (The describe response looks to be correct already, because I'm able to invoke.)