Fork me on GitHub
#clojure
<
2021-09-15
>
Joshua Suskalo14:09:17

I'm working on support for project panama in a Clojure library I'm calling coffi. I'm looking for some feedback on the current planned syntax:

Joshua Suskalo14:09:52

This is just for actual ffi calls. I have another portion of the library that's mostly done for serialization and deserialization, here it's often called with ffi/serialize et al. For structures those serialization functions will be automatically defined.

p-himik14:09:10

"strlen" [::ffi/c-string] ::ffi/integer
These are all positional - I would make them a single map with short plain keywords for each value. Also, the first string immediately follows the docstring, making it hard to distinguish. Since defcfun is a macro, you can make that string a symbol - I'm pretty sure C is more strict than Clojure when it comes to names, so you won't encounter something that you cannot use in there.

Joshua Suskalo14:09:06

The fact that compilers and similar can put symbols that begin with numbers and that the rules for what symbols can be linked against isn't dependant on C's rules for symbol names means that I'm not comfortable with it being anything less than a string.

Joshua Suskalo14:09:25

However making it a map instead of positional is something I'll consider for sure.

p-himik14:09:35

> compilers and similar can put symbols that begin with numbers Ah, so what C allows is not a subset of what Clojure allows? Alright, good to know.

Joshua Suskalo14:09:11

I believe C allows only a subset of what Clojure allows, but the standard for symbols to be put into shared objects isn't directly tied to the C standard.

Joshua Suskalo14:09:19

at least to my understanding

p-himik14:09:44

Right, but strings are even less restricting.

Joshua Suskalo14:09:22

That's fair, but since the JVM is already trusting you with matching the symbols up correctly, and the JVM takes a string, I don't want to arbitrarily limit what functions you can link against. What I might do is support both simple symbols and strings, so that normally it's a symbol but strings are supported when it doesn't conform.

p-himik14:09:02

Makes sense!

borkdude15:09:03

@U5NCUG8NR I would argue the opposite perhaps: FFI should be as fast as possible. Having to create maps for an FFI call could create too much overhead

Joshua Suskalo15:09:31

I think the map suggestion was for the syntax, not for what must be passed at runtime. My library currently makes no requirements that you marshal data at every call, it's only a well-supported usecase.

2
borkdude15:09:18

You're probably aware of dtype.next as well right?

Joshua Suskalo15:09:22

I think that would probably compose well with coffi, but I think they don't have much overlap.

borkdude15:09:46

check out this project, he made libffi bindings in clojure using dtype.next ffi: https://github.com/cnuernber/avclj/blob/master/deps.edn

borkdude15:09:19

I'm not too deep into that library but I do think there's overlap, it would be good to study this perhaps

borkdude15:09:25

and talk to the author

Joshua Suskalo15:09:52

Oh, I guess the readme for dtype.next didn't really say that it was about ffi, mostly just operating on native structures.

Joshua Suskalo15:09:30

So I see. Hmm. Definitely a different target.

Joshua Suskalo15:09:36

My intention with coffi is to make it so that it's relatively easy to wrap native functions and make them act as if they were Clojure functions. I don't intend this as a library for screaming performance, and the fact that this library exists pushes me further in that direction.

Joshua Suskalo15:09:31

Basically my target usecase is wrapping opengl or other similar libraries where the stuff getting passed back and forth is usually simple variables and it's relatively simple to wrap them with clojure-native functionality, and it's not meant as a way to move performance-sensitive operations to native code. IMO that's what Java is for in most cases on the JVM.

Joshua Suskalo15:09:45

Especially with the vector api coming down the pipeline.

emccue15:09:26

I'm not the biggest fan of c-string and integer

emccue15:09:13

c-string would be the same as pointer right? Why not [::ffi/pointer ::ffi/char] if you wanted to specify for some reason?

emccue15:09:05

and :integer is int in c, probably, but is that a u32

emccue15:09:34

maybe thats just a general complaint with panama - the names of things are based on C names but the layouts are agnostic

Joshua Suskalo15:09:47

So for clarity, those keywords are extensible. c-string is on top of a pointer and its deserialization will deref the pointer and construct a java string with it when it's done.

Joshua Suskalo15:09:26

Also panama (and java) don't distinguish unsigned vs signed integral types.

phronmophobic16:09:01

I would highly recommend dtype-next. Here's what the strlen example looks like:

(ns ffiexample.clj
  (:require [tech.v3.datatype.ffi :as dt-ffi]))

(dt-ffi/define-library-interface
  {:strlen {:rettype :size-t
            :argtypes [['s :pointer]]}})

(strlen (dt-ffi/string->c "Hello World"))
;; 11

phronmophobic16:09:10

I've wrapped several c libraries from clojure. I usually like having a callable function in clojure that maps as closely as possible to the actual c function. If that's not convenient to use, then I write a separate wrapper. Only providing the wrapper is tricky because whatever DSL that's available for defining the interface must also have a strategy for the memory management of the args and return values.

phronmophobic16:09:55

as another source of inspiration, you can also check out https://github.com/Chouser/clojure-jna/

phronmophobic16:09:34

I was using clojure-jna before I started using dtype-next. The main benefit of dtype-next is that it is graalvm native-image compatible.

Joshua Suskalo16:09:35

That's fair, but I'll admit I don't see the value in my going in the same direction as the existing libraries. I have likely less time to spend on getting them into a good state, and so the way I can provide the most value is to provide an entirely different angle to native interface than from low-level performance-sensitive wrappers.

Joshua Suskalo16:09:31

This is more or less the same reason that I created americano. It's not as powerful as most of the other options for compiling java code, but it provides something that didn't exist yet: simple data-driven java compilation for simple build pipelines.

phronmophobic16:09:14

Well, as someone who might be your target audience, the features I would be looking for are: • a way to deal with lifetimes of args/return values for memory management • easier usage of c structs • better support for callbacks

phronmophobic16:09:52

all of those features could be implemented as a library orthogonally to dtype-next

phronmophobic16:09:51

basically, dype-next, panama, etc do the heavy lifting of the ffi calls and coffi helps make defining the interface easier.

Joshua Suskalo16:09:38

Yup, both of those are targets I have. I'm using the lifetimes of args by way of Panama's own resource scopes, which means I can provide an interface that looks like this:

(with-open [scope (ffi/stack-scope)]
  ;; use the scope
  )
And support for callbacks will be nearly-transparent as long as you specify the callback type in the cfun type constructor. C structs will also be transparent with maps/records mostly, and will have a macro to define them in a way that automatically defines the serialization and deserialization mechanisms.

👍 2
Joshua Suskalo16:09:46

And because I am not mandating the types that serialization and deserialization work with, that means that you can if needed not actually marshal data all the way to clojure-native structures if desired.

Joshua Suskalo16:09:24

I have designs for how all this will work figured out, it'll be pretty simple to implement just on top of panama. The only unanswered question I have for now is what throwing exceptions from callbacks does, since I can't find in panama docs anything that talks about this, and because of the way panama works it could conceivably have defined semantics.

phronmophobic16:09:28

sounds neat. I'm not that familiar with project panama. Any idea if panama will be compatible with graalvm's native-image?

Joshua Suskalo16:09:24

Panama is a core part of the JDK starting in 17, and it'll be taken out of incubator and into the stdlib in the future. If graal continues to support recent versions of java it will support panama.

bananadance 2
Joshua Suskalo16:09:39

(if you hadn't heard, java 17 came out yesterday)

emccue17:09:37

Remember that you have access to Cleaner too - I learned that when writing a java wrapper for Enet with a panama prototype

emccue17:09:56

its a pretty straightforward way to "hand" memory to the GC

Joshua Suskalo18:09:54

That's a good one, and Panama also has cleanup actions associated with its scopes.

Joshua Suskalo18:09:18

And it has less issue with closures accidentally keeping an object reference.

deleted19:09:42

when exactly would you use a provided scope for a dependency?

emccue20:09:51

If that dependency is only used at compile time

emccue20:09:08

A good example in java would be https://immutables.github.io/

emccue20:09:41

if you are using deps.edn then the equivalent is to have a profile for that build step and include the dependency in the profile

Ed20:09:28

In maven, it means that the runtime environment will provide this, so don't include it in the package you build. For example in a web container like jboss or tomcat, you would mark the Java EE deps as provided, so you can omit them from your ear / war file.

👍 2
Ed20:09:50

Pretty much ... You might be compiling your java against one dependency that contains the reference interfaces for example, but when you deploy it to your app server, that'll provide it's own implementation classes and there's no point in every thing you deploy to the reference interfaces. But you'll need the reference classes to actually compile your java, cos it'll have methods that take an http request, for example... Am I making sense?

Ed21:09:36

I think in Clojure land it has a bit less utility

Eddie03:09:32

One example is if you have an https://spark.apache.org/, or https://storm.apache.org/, or some other big-data scale-out-to-a-cluster kind of project. The cluster already has all the Spark/Storm classes running its JVM process and is waiting a user to submit a new uberjar "job" containing an application to run. Therefore, our uberjars should not contain Spark/Storm classes inside them. That said, those classes will be necessary when compiling our code into the uberjar. Thus Spark projects almost always declare Spark as a provided dependency.