Clojurians Log v2
graalvm-mobile
@dotemacs has joined the channel
Just discovered this channel via borkdude’s tweet of the repo that is in the subject of the channel. Very cool! So since Asami https://github.com/threatgrid/asami has local storage capability, and can be graalvm compiled, any idea what would be needed for the following: - Asami graalvm compiled, but accessible from react-native ClojureScript app? I’m guessing that there would need to be some sort of binding so that Asami compiled could be interacted with from ClojureScript. Maybe somebody with a bit more graalvm experience can chime in on this? > Why would it be cool to have this? Just because we could have datalog database on mobile (with localstorage!). Now that I wrote the above, I guess the more succinct question is: what is needed to make graalvm compiled Clojure code accessible to react-native ClojureScript code? Thanks
I am currently using DataScript + React Native; All running in a JS thread, no “native”.
DataScript is alright, but there’s some gotchas around performance.
@dotemacs Another graalvm compatible datalog db is datalevin btw. So moar options :) But isn't Asami Clojure_Script_ compatible as well, could that be used from React Native + CLJS directly?
It could and it does work, but the local storage is not yet implemented on ClojureScript side. Only on Clojure side. This is a reply to @borkdude’s question above.
@borkdude what about DataScript? Has that been tried with GraalVM?
datalevin is based on datascript
datascript doesn't have durable storage, but datalevin added this
Right, I am using DataScript, hence my biased question… I rolled my naive DataScript durable storage.
it uses lmdb, you might have to have that lib available on your phone, not sure
@huahaiy has joined the channel
@huahaiy might be able to give more insights here.
Since we brought up this interesting question and because of some perf issues in DataScript, I am theoretically curious if running it under GraalVM can yield any perf. improvements.
(I realize my question probably has no clear answer, it all needs to be benchmarked 🙂 )
From what I understand, Datalevin doesn’t retain the history of transactions. So it’s slightly different to DataScript & Asami in that sense.
@dotemacs DataScript also does not retain history by design (you have to roll your own implementation)
@raspasov my bad, TIL
No worries at all; I am no expert either; been using it for a few months;
When you say > I rolled my naive DataScript durable storage Do you mean that you read the data into DataScript from a file and the write the data to a file upon the app going into background?
On every transaction save to file;
On app startup, read from file.
Aha, nice
And the overhead is what, in milliseconds?
Currently, it’s very naive; every time it simply dumps all datoms to a file.
So it would linearly slow down; but I’ve had 1000s of datoms and it hasn’t been an issue.
Nice
Good to hear this
Perhaps Asami offers a better story here
if perf becomes an issue
It’s for data that the user generates on one device, so it would take years for it to grow super big.
Yea, that makes sense
I don’t doubt it… I just wanted to stay as close to Datomic as possible, and DataScript more or less offers that.
(not fully)
Are you syncing with Datomic or …?
it's nice to have that affordance for sure on mobile
Not currently, potentially in the future…
The biggest issue with DataScript is querying perf.
Even with a few thousand datoms, complex queries can take 50-100ms (!!!) on iOS
So you have to be careful; I didn’t expect this initially and tripped me up. The DataScript home page oversells/oversimplifies the product, to put it mildly…
It was my own ignorance and lack of benchmarking. Other than that, I have had no issues so far.
I did look at mentat aka datomish which was kind of a re-write of DataScript with a Sqlite backend:
https://github.com/mozilla/mentat/tree/clojure
(note the branch there clojure).
And then a subsequent re-write in Rust… which seems to compile but haven’t got it running on device yet. Not that I tried very hard.
Perhaps a little bit of caching could help mitigate that
@borkdude Apparently it’s complicated; There’s a number of issues on github brining up performance; To do it well, you need to get into the weeds of “database science”; I’ve dabbled in general database design and it gets deep 🙂
Query planning, etc. The datalog queries are very expressive and it’s not trivial to optimize them; the fact that you’re doing everything in one thread (JS) doesn’t help either, I think.
I mean caching on the side of the app
not in DataScript
e.g. when you need to execute the same query over and over while the data hasn't changed
Ah, yes; That’s what I ended up doing;
It’s a bit tedious but it mostly works; Basically only re-query things that you know have changed based on the most recent transaction;
One of the “proper” ways to achieve better perf. would be some version of differential dataflow (not trivial to add, afaik) https://timelydataflow.github.io/differential-dataflow/
So are you using using datascript in native?
React Native
so it's CLJS
Yes
using a graalvm version might already speed up things considerably
You think? That’s probably worth a shot…
I’ve seen that. CLJS is slower, as expected.
And then iOS, afact, is like 10x slower (no idea why)
But how would you go about using Graalvm compiled db, from cljs?
I think the RN runtime of JavaScript core is a bit restricted as opposed to running in Safari… (just speculating as to the reasons)
@dotemacs mobiletest compiles the graalvm binary to a shared library. This shared library is then accessed from the mobile UI code. Not sure how to do this from React Native/CLJS. This probably relies on iOS specific APIs.
@borkdude So when you say “graalvm version”… Is that running in like a real native thread, or something like Sci? I apologize, still not very well versed in all the differences.
@raspasov what mobiletest does: it compiles everything to "real" native and then exposes hooks to SCI, so you can have dynamic behavior when you eval stuff in SCI.
SCI is an interpreter which you can use as glue code between your "real" compiled functions. Not everything is interpreted by SCI, only the glue code, let's say.
Got it… so some Clojure is compiled to “real” native.
so if you have {:namespaces {'foobar {'foobar foobar}}} and then eval (require 'foobar) (foobar/foobar) in SCI, then the foobar call is exactly the same as outside SCI
Some is interpreted, to help provide REPL-like functionality?
yes
I have to go through the mobiletest example again with the latest updates; thank you 🙂
When you compile DataScript/etc database to “real native”, I assume you’d have to communicate with it via async function calls with the React Native JS thread.
sure, but the shared library can be sync. The async stuff is a JS problem
Yes.
similar to how a REST endpoint is usually synchronous
Right… when you say “shared library” that’s basically a regular method/fn call for other native code. Basically a “real” native library?
I don’t think compile Datascript with GraalVM will help much with performance
no, but JVM-based vs JS-based probably does: https://twitter.com/nikitonsky/status/1398080743025356805
The poor query performance of Datascript is mostly due to its lack of query optimization. I did some trivial optimization in Datalevin and the performance increases about 30% to 50% depending on dataset. I also have a plan to do a rewrite of the query engine in Datalevin. In any case, the slow performance of triple-store (compared with relational DB) is a well known problem, but that should not excuse Datomic and Datascript from doing any optimization at all. Currently, Asami does a decent job in query optimization, Crux claims to do that as well, but the effect may not be obvious from what I heard. Of course, this is just my opinion and I am obviously biased.
@dotemacs
> what is needed to make graalvm compiled Clojure code accessible to react-native ClojureScript code?
TLDR: You would need to make a native module, https://reactnative.dev/docs/native-modules-ios
Roughly, the steps required would be:
1. Compile your clojure code to a shared library with native-image
You need to explicitly mark functions that are available as part of the shared library API. Further, the API can only receive and return C data structures like ints, chars, pointers, structs, etc.
2. Create a native module that exposes the shared library API to your js code.
Not sure how it would affect performance or memory usage, but I assume there is some overhead in the bridge between JS and native code.
I think an interesting alternative would be to compile all your clojure code to native and talk to react native code directly. I've been looking at some of the UI options for mobile and I think one of the major complaints against react native is that the JS/native bridge is slow.
I figured that those would be the steps @smith.adriane but what I’m unsure of is this part: > You need to explicitly mark functions that are available as part of the shared library API How would something like that be done for Clojure functions that would then be compiled via GraalVM?
It looks like:
(defn compile-interface-class
([]
(compile-interface-class nil))
([opts]
(with-bindings {#'*compile-path* "library/classes"}
((requiring-resolve 'tech.v3.datatype.ffi.graalvm/expose-clojure-functions)
{
#'clj_print {:rettype :void
:argtypes [['bs :pointer]]}
#'clj_prn {:rettype :void
:argtypes [['bs :int64]]}
#'clj_eval {:rettype :int64
:argtypes [['bs :pointer]]}
#'clj_start_server {:rettype :void
:argtypes []}
#'clj_print_hi {:rettype :void
:argtypes []}
}
'com.phronemophobic.mobiletest.interface nil)))
)clj_print, clj_prn, etc are just normal clojure functions that receive and return the arg types listedOh, I’m with you, that’s from your project. Sorry for being slow on the uptake 🙂