A more appropriate place is #C053AK3F9. Also, impossible to answer such questions when there's no code.

Sorry, I sent the message to channel accidentally while editing it

Pardon me if this is an ignorant question here, but what is “line” where the community feels like a question belongs in #C053AK3F9? And feel free to point me to existing notices where I might have missed that definition. I am now wondering if there is such a thing as “too basic” of a question to belong in this #C03S1KBA2 channel, and so the clarification will help me in the future I’m sure, since I’m also a beginner. 😅

> what is “line” where the community feels like a question belongs in #C053AK3F9? Whenever a person is a total beginner. :) Some questions by beginners would of course be appropriate for #C03S1KBA2 or #C03S1L9DN or some other channels. But most of the time beginners can't judge that themselves, whereas kind people in #C053AK3F9 are always happy to point them in the right direction.

The api docs for the interface you're trying to implement (https://javadoc.io/static/com.amazonaws/aws-lambda-java-core/1.2.1/com/amazonaws/services/lambda/runtime/RequestHandler.html ???) suggest that you're supposed to return a specific event type, but you're returning the result of (.getRecords event) which is probably a LinkedHashMap?

I don’t know much Java, but my assumption is that you inferred that from RequestHandler<I,O>

Which, from my understanding, is a Generic, would the corrective step then be to implement this RequestHandler in Clojure?

O handleRequest(I input, Context context) in the method has 2 generics in it. I which is the input type, and O which is the output type. In the error message you posted, it says it's failing to convert a java.util.LinkedHashMap into a com.amazonaws.services.lambda.runtime.events.models.s3.S3EventNotification (which probably means that your lambda is being triggered by an S3 notification?) so I guessed ... it's also possible that you've got your input types wrong, and there's LinkedHashMap in your input that can't be cast. I'd probably remove all the type hints, and prn your inputs to find out what they are, remove the return value, and check the logs to see what you actually get.

In java Generics only exist at compile time. By the time you're running code, all the generics have been erased. Since clojure doesn`t care about generics, you don't need to worry too much about them when implementing -handleRequest

S3Event inherits S3EventNotification along with its .getRecords method. You're telling Clojure that event is of type S3Event and then calling the .getRecords method that comes from S3EventNotification. The value of the event that you actually receive is, I think, either a Java map or a list. In your case it's a linked hash map.

So that ^S3Event part is wrong. As well as the (.getRecords event) part.

A useful link, even if you don't know Java: https://docs.aws.amazon.com/lambda/latest/dg/java-handler.html#java11-handler-example

I am triggering the Lambda function using an S3 Put event, here is the exact event:

{
  "Records": [
    {
      "eventVersion": "2.0",
      "eventSource": "aws:s3",
      "awsRegion": "us-east-1",
      "eventTime": "1970-01-01T00:00:00.000Z",
      "eventName": "ObjectCreated:Put",
      "userIdentity": {
        "principalId": "EXAMPLE"
      },
      "requestParameters": {
        "sourceIPAddress": "127.0.0.1"
      },
      "responseElements": {
        "x-amz-request-id": "EXAMPLE123456789",
        "x-amz-id-2": "EXAMPLE123/5678abcdefghijklambdaisawesome/mnopqrstuvwxyzABCDEFGH"
      },
      "s3": {
        "s3SchemaVersion": "1.0",
        "configurationId": "testConfigRule",
        "bucket": {
          "name": "example-bucket",
          "ownerIdentity": {
            "principalId": "EXAMPLE"
          },
          "arn": "arn:aws:s3:::example-bucket"
        },
        "object": {
          "key": "test%2Fkey",
          "size": 1024,
          "eTag": "0123456789abcdef0123456789abcdef",
          "sequencer": "0A1B2C3D4E5F678901"
        }
      }
    }
  ]
}

And you'll get a linked hash map that has that data.

So what is the value of having S3Event and the methods that it supports?

sure ... that's a textual representation of some json. But by the time your function is called, it's been deserialised into a java data structure. Read the docs that @U2FRKM4TW posted above. And I'd suggest just printing your inputs to see what's in them.

> So what is the value of having S3Event and the methods that it supports? for people writing java code to have code completion in their ide's? ;)

Yeah, seems like that. You can deserialize that JSON data into an instance of S3Event and then access everything via interop. But I'd just use that JSON data as represented by the map.

Can you recommend a library for working with JSON?

For JSON, you can use cheshire. For AWS Lambda, you can use my library: https://github.com/igrishaev/lambda

See the "JSON Parsers" section here: https://www.clojure-toolbox.com/

it provides Clojure handlers for both naive and HTTP requests.

You also don't need to parse that particular JSON because it's already parsed for you - that's why you get a linked hash map.

@U1WAUKQ3E I don’t understand how it works

@U05MPHVTGPK you write a core module that uses the handler from the library, then compile it with GraalVM and get a single bootstrap binary file. Then you upload it to AWS Lambda and it works.

What do you mean by run AWS Lambdas compiled with native image

Is it an AWS Lambda runtime? Does it allow me to run AWS Lambda functions locally? Is it a competitor to AWS?

A lambda might be of a various runtime. In your example, you use Java runtime. But you can make a binary file which is free from any kind of environment

Which is what your library does?

yeah, it provides handlers that that do most of work for you.

I am not yet proficient enough with Clojure as it comes out of the box, so adding the complexity of this other thing is a daunting task.

I wouldn’t be able to justify on my own why using your library is better in my use case, and how it compares to doing things the way I am doing them now

Well, since your code completely relies on AWS SDK, why not writing it in pure Java?

It would have been easier rather than porting Java code to Clojure

I began learning Clojure to do a programming exercise for a company. That company is unable to hire me because of where I am. I have some free time now and want to do the exercise. Most examples I encountered were implementing the Java interface, so I stuck with that.

Also, Java runtime takes up to 7 seconds for cold start, whereas a binary file takes 50 msec

I'm not pushing my library exactly, just wanted to note that there are other good wrappers, e.g. FieryCod/holy-lambda or babashka + scripts

When I looked into holy-lambda, it didn’t seem applicable to my use case: I need to read data from a file in an S3 bucket Parse data and write it to a file in the S3 bucket

Holy Lambda mentions an event loop. I don’t need a loop, I need the lambda function to exit when done doing the above.

Lambdas always do loop, always

They work like a message queue, and you provide a handler

In your code, the loop is handled by the java sdk

Here’s what I get when I print event

#object[java.util.LinkedHashMap 0x489115ef {Records=[{eventVersion=2.0, eventSource=aws:s3, awsRegion=us-east-1, eventTime=1970-01-01T00:00:00.000Z, eventName=ObjectCreated:Put, userIdentity={principalId=EXAMPLE}, requestParameters={sourceIPAddress=127.0.0.1}, responseElements={x-amz-request-id=EXAMPLE123456789, x-amz-id-2=EXAMPLE123/5678abcdefghijklambdaisawesome/mnopqrstuvwxyzABCDEFGH}, s3={s3SchemaVersion=1.0, configurationId=testConfigRule, bucket={name=example-bucket, ownerIdentity={principalId=EXAMPLE}, arn=arn:aws:s3:::example-bucket}, object={key=test%2Fkey, size=1024, eTag=0123456789abcdef0123456789abcdef, sequencer=0A1B2C3D4E5F678901}}}]}]

That's a Java LinkedHashMap you can convert it to a Clojure map with something like (into {} m)

Clojure maps are printed in EDN, but Java objects generally are not, which is why it prints with the #object style notation, but they are still maps, so you can still do (get "Records" m) to look up keys from the map.

hm that returned nil

Sorry ... typing on my phone ... that should have been (get m "Records")

@U2FRKM4TW, I am confused, how is it parsed, if there are = signs in the map?

That's how the map is printed. It's not a string overall, so it's parsed. Don't rely on printed representation. I don't know what AWS Lambda offers in terms of the ability to connect to a running instance, but it's much, much easier and developer-friendly to develop with a REPL at hand. A somewhat relevant comment - since you're a total beginner with Clojure and not a Java developer as well, starting with AWS Lambda is not a good move IMO. It's like a novice car mechanic starting working on a total revamp of a W8 engine. If your goal is to learn Clojure, you would be much better off writing some backend.

To demonstrate the = printing:

$ jshell
|  Welcome to JShell -- Version 19.0.2
|  For an introduction type: /help intro

jshell> new HashMap<String, Integer>() {{ put("a", 1); put("b", 2); put("c", 3); }}
$1 ==> {a=1, b=2, c=3}

jshell>

Here’s the output of (get event "Records") https://clojurians.slack.com/files/U05MPHVTGPK/F05QT912763/untitled However, I get nil when trying (get-in event "Records" "awsRegion")

Check out the docs for get-in.

(type (get event "Records") returns java.util.ArrayList

Ah, then you can't use get-in since array lists are not associative. But you can combine get with nth.

as in (nth 0 (get event "Records"))

No. Please read the docs of the CLJ functions that you're using. In this case, the docs of nth. The https://clojuredocs.org/ website even offers examples.

The -> threading macro would also be helpful here. Since it seems that your initial question is resolved, I would suggest asking follow-up questions as separate messages, outside this thread. And preferably in the #C053AK3F9 channel since people there have explicitly elected to help out beginners with any sorts of questions or problems they might have.

Not sure about clj-ffi but FWIW there's https://github.com/IGJoshua/coffi.

I saw Coffi but need it to work on a LTS JDK

ie 17 (I think. Pretty new to this)

A bird said a new LTS comes out any day now though

Any other modern FFI libs to look at?

I'll ask about JDK specifics in #C02EEAUHSJJ in the mean time

The README in that repo mentions alternatives: https://github.com/IGJoshua/coffi#alternatives

And I think you can use earlier versions of coffi that do support JDK17. Probably from this branch: https://github.com/IGJoshua/coffi/tree/jdk17-lts

new lts comes out this month and coffi will be updated to it.

Nice! Does Coffi's latest release support 17? (what I'm using now)

The latest release is for jdk 19, the latest release for jdk 17 is 0.4.341 on maven

The internal api changes between jdk versions because it's in preview, so i don't support multiple jdks per coffi release

okie doke. thanks!

my needs are simple so shouldn't be much change to switch to the new upcoming LTS

fwiw, this is rather funny: > ingydotnet: use coffi to have clojure call libfyaml with a yaml stream and return the parser events. > > GitHub Copilot: I apologize, but it seems that there is no coffi library in Clojure. Did you mean clj-ffi instead?

This is a pretty good expression of why I don't find copilot particularly useful, especially in Clojure projects. These generative text AI often have a lot in common in the output they produce to StyleGAN and other style transfer AI. They will take some way they've seen frequently to solve a problem described (and that frequently then biases towards more mainstream languages like JS and C++ and Java), and then they will apply Clojure as a "set dressing" for that.

to be fair... ingydotnet: The Clojure coffi library is here https://clojars.org/search?q=coffi and here https://github.com/IGJoshua/coffi GitHub Copilot: Thank you for the information. I apologize for the confusion earlier. I am not familiar with the coffi library, but it appears to be a Clojure library for calling C functions using a more concise syntax than clj-ffi. Here's an example of how you can use coffi to call libfyaml and parse a YAML stream:

(ns my-app.core
  (:require [coffi.core :refer [defcfun]]
            [coffi.types :refer [string int bool pointer]]))

(defcfun "fyaml_parser_initialize" [] pointer)
(defcfun "fyaml_parser_set_input_string" [pointer string int] bool)
(defcfun "fyaml_parser_parse" [pointer] int)
(defcfun "fyaml_parser_cleanup" [pointer] bool)
(defcfun "fyaml_parser_get_event_type" [pointer] int)
(defcfun "fyaml_parser_get_anchor" [pointer] string)
(defcfun "fyaml_parser_get_tag" [pointer] string)
(defcfun "fyaml_parser_get_scalar_value" [pointer] string)
(defcfun "fyaml_parser_get_scalar_length" [pointer] int)
(defcfun "fyaml_parser_get_plain_scalar" [pointer] bool)
(defcfun "fyaml_parser_get_sequence_start_implicit" [pointer] bool)
(defcfun "fyaml_parser_get_mapping_start_implicit" [pointer] bool)

(defn parse-yaml [yaml-string]
  (let [parser (fyaml_parser_initialize)
        result (fyaml_parser_set_input_string parser yaml-string (count yaml-string))
        events (loop [events []]
                 (let [event-type (fyaml_parser_get_event_type parser)]
                   (cond
                     (= event-type 0) events
                     (= event-type 1) (conj events (parse-scalar-event parser))
                     (= event-type 2) (conj events (parse-sequence-start-event parser))
                     (= event-type 3) (conj events (parse-sequence-end-event parser))
                     (= event-type 4) (conj events (parse-mapping-start-event parser))
                     (= event-type 5) (conj events (parse-mapping-end-event parser))
                     :else (throw (Exception. (str "Unknown event type: " event-type))))))]
    (fyaml_parser_cleanup parser)
    events))

(defn parse-scalar-event [parser]
  {:type :scalar
   :anchor (fyaml_parser_get_anchor parser)
   :tag (fyaml_parser_get_tag parser)
   :value (fyaml_parser_get_scalar_value parser)
   :length (fyaml_parser_get_scalar_length parser)
   :plain-scalar (fyaml_parser_get_plain_scalar parser)})

(defn parse-sequence-start-event [parser]
  {:type :sequence-start
   :anchor (fyaml_parser_get_anchor parser)
   :tag (fyaml_parser_get_tag parser)
   :implicit (fyaml_parser_get_sequence_start_implicit parser)})

(defn parse-sequence-end-event [parser]
  {:type :sequence-end
   :anchor (fyaml_parser_get_anchor parser)})

(defn parse-mapping-start-event [parser]
  {:type :mapping-start
   :anchor (fyaml_parser_get_anchor parser)
   :tag (fyaml_parser_get_tag parser)
   :implicit (fyaml_parser_get_mapping_start_implicit parser)})

(defn parse-mapping-end-event [parser]
  {:type :mapping-end
   :anchor (fyaml_parser_get_anchor parser)})

(parse-yaml "foo: bar\nbaz: qux\n")

I'm sure I'll have to debug it. I never use its code. Just ask it sometimes when I'm blocked to get new ideas, or when trying something totally foreign (npi).

That's fair. There's a couple parts of that which are almost right. It kinda seems like it might be pulling on some CL ffi stuff there, but I'm not certain. The readme for coffi should hopefully be enough to help out a lot with knowing what to do, and if you want to see a sample library wrapped, I have https://github.com/IGJoshua/glfw-clj

This uses a slightly older version of coffi, you can check the changes for that here: https://github.com/IGJoshua/coffi/blob/jdk17-lts/CHANGELOG.md

glfw-clj uses 0.2.259, so the changes since then are relevant.

There's no lein usage in the docs I noticed.

But I'll figure it out.

[org.suskalo/coffi "0.4.341"]

thanks. I know it's dumb. I'll get there...

all good

Dtype-next also has ffi bindings that work today on several lts releases

It’s used for libpython-clj

Examples would be avclj, libpython-clj and tmducken

With avclj being the most approachable

Jdk-8 and forward

And graal native bindings if that is your jam

ok thanks. I'll look. I looked at the dtype-next docs but they felt complicated for what I needed... 1 call. I'll look at avclj.

I'm writing clojure thst compiles to a graalvm shared lib actually.

for binding to other langs

that part is all done

I just want to replace the java yaml parser with a better C one

That's important actually, you can't use coffi and compile to a graal native image. coffi uses runtime bytecode generation in order to handle high performance boxing and unboxing for higher order functions.

aha

That doesn't work on graal, so dtype-next is your best bet, unless you want to use the graal native stuff directly.

well that saves me time likely

nope

my test binding (python ctypes) makes a single high level call to the graal-compiled shared lib

thanks for the tip @U5NCUG8NR!!!

Yeah, no problem! Normally I'd also question why a project is using graal, but compiling to a shared object to allow using from python is a pretty good usecase 😅

well and many many other langs

Yeah

Just as a heads up, you may also want to keep an eye on https://jank-lang.org/ , which is a WIP clojure dialect that compiles to native code and offers direct FFI

The high level abstract is: > a libyamlscript binding for YFL (your favorite language) that lets you write YAML files with functional transformations (in a nice highly stylized YAML DSL syntax). YAMLScript is an alternate syntax reader for Clojure. SCI is bundled into libyamlscript to evaluate the translated Clojure code. The result is returned as JSON and parsed/loaded/evaled by FYL giving you the expected data value. > > Literally: >

(->> ys-file
>   slurp
>   ys/ys-to-clj
>   sci/eval-string
>   json/write-str)

import yamlscript
data = yamlscript.load_file("config.yaml")

I'm aware of Jank but unaware of what it could do for me here...

SCI seems to fit very well, and I already have all the compilation working.

In fact everything works. Replacing the java snakeyaml with the C libfyaml is pretty much an optimization at this point.

But I also work closely with the libfyaml author and it will play a much bigger role later on

I don't think it gets you much right now, but in the future it would allow you to have a lighter shared object that doesn't bundle even a minimal JVM

ah. ok. well the linux shared lib is 52MB, so that's fair

I could certainly have the option to swap sci for jank. once the whole project is further down the road and has tons of tests to validate the jank would work out.

thanks for helping me see that... 🙂

(have I thanked you enough yet?)

Lol, definitely have. It's really cool to see someone doing clojure stuff in a shared object!

The goal for Jank in this case is to be able to do "everything that both clj and cljs can do", and they are working towards pretty robust tests.

Having a test suite and making sure it all works the same would be great, and if it doesn't then it's a good way to generate bug reports for jank, which I'm sure the author would appreciate

Ironically I'm here because I am working on a Clojure Platform for Perl (first, then others) called Lingy and gave a talk about https://www.youtube.com/watch?v=9OcFh-HaCyI in July. See me joining the Clojurians slack: https://clojurians.slack.com/archives/C0218CKTVEK/p1689687145013449 My Clojure journey to date gets more amazing by the day. From the things I've learned from folks like yourself, I will be able to make YAML programmable in at least 10 languages by the end of this year. That's insane from my experience in programming. My needs so far with Clojure are a bit reversed from typical: I'm taking Clojure and its concepts and tooling and using them to spread various goodness to all the other langs. That's pretty much been my MO for the past 20 years. Clojure and friends are the giants' shoulders. I'm a little angry I only learned about Clojure in the last year, but also very happy about where it is technically today (AWESOME). Note: YS also considers any existing YAML file to be valid YS, so you can load any YAML (and thus JSON) file with it (using FP calls or not) and it will work the same in everything language (something YAML struggles with today).

Yeah, with the yaml spec being the way that it is and having as many versions as it does, it's very challenging to get a consistent experience with it across different languages.

Right. (fyi I am an author of that spec...) My opinion is that the spec (even though it is huge) it does not contain all the information needed to allow people to produce fully compatible implementations across languages. One solution (my current one) is to make a top notch, api compatible reference implementation in every modern language. And now I think I can reasonably do that with help from clojure and friends.

that's really cool!

First, why is the C parser better than the java parser? And second, if you are transforming the entire datastructure to clojure function by function then I think that will be somewhat slow. It would be more ideal to get pointers to data structures and do the traversal that way as opposed to using granular function calls. Third ham-fisted has optimized datastructures that are somewhat faster to create than the base clojure datastructures -- this was the part of my https://www.youtube.com/watch?v=ralZ4j_ruVg and lastly we could also add yaml parsing to charred. There is an https://github.com/cnuernber/charred/issues/15 about that which would mean you wouldn't need to link to the C library and pay the cost of transforming the datastructures - if you do indeed plan on doing that.

1. Nothing to do with C or Java. The C libfyaml is a better library by far on many levels. 2. I'm not. The transform is from a string containing a YS/YAML document to a a string of clojure forms. 3. ok 4. The call to the C lib (`libfyaml`) and the Java lib is to "parse" a single yaml document into a sequence of YAML parse events. That's all. I don't see any advantage to avoiding it, and later I'll need libfyaml for other advanced reasons.

The term "parse" in YAML context is only one layer (text to event structs) of a more complex stack called "load" (text to native)

parsing YAML is the very hard part that no parser in the world (afaik) can do yet. libfyaml and the reference parser I wrote and eemeli's java parser pass 100% of the test suite, but I know of many cases where they fail taht have not yet made it into the test suite. All these implementations were written by YAML core team members (5 of us)

We should continue this in #C05HQFMTURF if there is more to go over. 🙂

Nope - this is all interesting especially the high level design and makes sense. Good luck and if you want help with dtype-next's ffi system I am happy to help.

@UDRJMEFSN many thanks, and I'll almost certainly need some help along the way!