Please move the stacktrace into an attachment. I think Slack even offered you to do that when you were writing the post. Otherwise, it spans multiple screens right in the main channel, making other messages much harder to reach.

Sorry about that

Check your classpath. There should be multiple tools for Lein that let you do that, including showing situations when a particular dependency is being excluded.

I have tried examing the classpath, I don't see it there. The class I'm missing is here https://github.com/cognitect-labs/aws-api/blob/main/src/cognitect/aws/http.clj but I can't figure out how to include it, as far as I can tell it should already be included just by adding [com.cognitect.aws/api "0.8.641"]but obviously I'm missing something.

Oh, right, it's the same lib. I'd also try clearing class caches, if there are any. Apart from that, can't really suggest anything, would have to debug it.

I have tried lein clean didn't work. I eventually just deleted my .m2 directory and let it download everything again, but still it doesn't work. Are there any other methods I should try?

Does it happen only in a specific project or in an empty project as well?

Looks like it is only this specific project. I have seen that one of my dependencies includes the cognitect aws API as well, I have tried excluding it using

:exclusions [com.cognitect.aws/api
                               com.cognitect.aws/endpoints
                               com.cognitect.aws/s3
                               com.cognitect.aws/eventbridge
                               com.cognitect.aws/sqs
                               com.cognitect.aws/dynamodb]

But at least you have a starting point now - given that it's only in that specific project, you can gradually remove its pieces till you're left with nothing but an MRE, a minimal reproducible example. If coming up with that example won't shed any light on what's going on, you can share it here and I or someone else will be able to take a look.

Or you can use datalevin, which reuses LMDB transactions.

Thanks @U0A74MRCJ - I do use datalevin in another project, but this is currently using LMDB directly.

e,g. https://github.com/wardle/hermes is using LMDB directly, but my other work uses datalevin (e.g. https://github.com/wardle/dmd)!

What would be the benefits of using it directly?

LMDBJava is something I personally want to get ride of. It is not actively maintained.

It has bugs in its iterator logic, and it uses an older version of LMDB that has critical bugs.

Historical mainly as I was using mapdb originally and then switched backends - but I would hope faster.... I don't use iterators but just plain standard cursors. I've been bitten by the lack of releases with running on Apple Silicon but have solved that since. I can see a time with modern FFI that we can use LMDB directly.

Using datalevin for my other projects has been a breeze however!

Sure. I mean, if there are other people actively maintain a LMDB binding for me, I would be happy. The FFI story in JVM will not be solved in a short time. Modern FFI of JVM will not support graalvm for a long time, so one needs to maintain two versions of them at least. These are things I wish someone else are doing it for me.

As to your original question, it is simple to use a concurrrent queue, dequeue, or something from Java to do that.

For SNOMED CT, which is quite enormous, I wanted to hand roll serialisation, and persistence - given its size and my need to run very fast. Interestingly, many commercial equivalents take hours to import, and require complex set-ups such as Elasticsearch etc. whereas LMDB has given me an amazing speed-up (ie 4-5 mins or so) and is usable from anything from a personal machine all the way to suite of load-balanced cloud based servers. I'm using cursors and peeking into buffers to get fast responses in a domain-orientated way, which has some trade-offs - in many ways using a more generic key value store would be preferable. Haven't tried graalvm yet. Did you ever look at the LMDB bindings from JGL?

Thanks! I'll have a look at concurrent queues - sound less problematic than using thread locals, which I've always had an aversion to! Thanks for the advice!

LMDBJava’s LMDB binding is the fastest already. I don’t think it can be improved speed wise. It is using Unsafe to directly manipulate buffer, there’s nothing faster than that in the Java world.

PS. Datalevin underpinned a complex analytics pipeline for a European Medicines Agency project in multiple sclerosis - I loaded up the UK dictionary of medicines and devices, and lots of socioeconomic data and combined with clinical data - so thank you for all of your hard work. https://github.com/wardle/dmd handled all of the drugs, and https://github.com/wardle/deprivare handled the socio-economic data. Made it very easy so thank you!

The only problem is that it is not actively worked on, so I had to do some work myself to make it work on apple silicon, to use newer LMDB version, fixing its iterator logic problem, etc. That’s why I think it is better off for you to switch to datalevin.

we reuse transaction, reuse cursors, and generally take care of things that you have to write yourself anyway.

Thanks! I'll do some benchmarking as the less code I have to maintain the better! I do also use Lucene for my fast free text search, and have used it for a long time (back when I was using Java), so some of my choices are simply inertia...

Looking forward to your benchmarking. There are so many places for performance improvement in these things. It is often easy to have 2 or 3x speed up by some trivial tuning. Lucene data ingestiion speed is about twice faster than datalevin search, but it’s query speed is slower than datalevin search.

Anyway, my goal of developing datalevin is to simplify data access for myself. Most of code we wrote daily is just data plumbing, so it save a lot of effort if we have something simple that takes care of a lot of things for us.

With regard to wrapping lmdb, I've often found that c libraries have a more functional interface than their java counterparts. For this reason, I wrote https://github.com/phronmophobic/clong to make it easier to generate clojure ffi wrappers. Currently, clong only supports generating wrappers for JNA, but the idea is that once the API is available as data, you can programmatically generate a wrapper that supports any idioms you need for performance, usability, or resource management. I'm not sure that using clong would be any easier than maintaining a fork of lmdbjava, but I thought I would throw it out there in case it was interesting. I was surprised to see that lmdbjava uses jnr-ffi rather than JNI. JNI bindings should be faster, but they can be a pain to maintain.

The way LMDBJava works, I don’t think it makes much of a difference. At the bottom, it is all JNI. But the performance hit of wrapping LMDB is mainly about how to handle the data input/output, whether you can do zero-copy of data. The function call itself makes minimal difference. LMDBJava use Unsafe to directly work on native buffers and there’s no copy, so there’s really not much more to shave off. One can use a faster buffer implementation, that would be faster., but the difference is also not much. The original maintainer of JNI wrapper of LMDB has abandoned his own JNI library, and joined LMDBJava, if that tells you something.

fwiw, I was curious to see how clong would handle lmdb's API and a minimal example worked out of the box. https://github.com/phronmophobic/clong/blob/main/examples/lmdb/src/clong/lmdb.clj

JNA's https://java-native-access.github.io/jna/5.4.0/javadoc/com/sun/jna/Memory.html class uses malloc under the hood, so you should be able to do zero-copy similar to lmdbjava.

Wow! I am impressed.

Unless there’s some radical advantage, people are not going to switch FFI at this moment, as panama is around the corner. One advantage of JNA is that it seems to have graalvm support, so there’s no need to do anything extra to get native image.

JNA is known to be one of the slowest, so I think it is a hard sell.

My understanding is that, right now, JavaCPP is the fastest. dtype used to be using javacpp, he switched to dtype-next that does his own thing, but his use case is limited to numerics, e.g. he don’t supports function pointers, etc. So, really, we are just waiting for panama to land in the mainline.

clong is decoupled from any particular ffi implementation. I will probably add a generator for coffi at some point which will work with panama. It probably wouldn't be that hard to write a generator to target jnr-ffi.

I know JNA switched to using libffi which is what jnr-ffi uses under the hood. I'm not sure how recent that is and if there are up-to-date benchmarks available.

That’s great. When panama lands, I may use clong to get going with LMDB when you support panama.

> lein new duct duct23 +site +example +ataraxy +cljs +postgres
Active code page: 1250
Generating a new Duct project named duct23...
Run 'lein duct setup' in the project directory to create local config files.

/duct23> lein duct setup
Active code page: 1250
Created profiles.clj
Created .dir-locals.el
Created dev/resources/local.edn
Created dev/src/local.clj

\duct23> lein repl
Active code page: 1250
nREPL server started on port 50205 on host 127.0.0.1 - 
REPL-y 0.5.1, nREPL 0.9.0
Clojure 1.10.3
Java HotSpot(TM) 64-Bit Server VM 18.0.2.1+1-1
    Docs: (doc function-name-here)
          (find-doc "part-of-name-here")
  Source: (source function-name-here)
 Javadoc: (javadoc java-object-or-class-here)
    Exit: Control+D or (exit) or (quit)
 Results: Stored in vars *1, *2, *3, an exception in *e

user=> (dev)
:loaded
dev=> (go)
Execution error (ConnectException) at  (Net.java:-2).
Connection refused: no further information

you should have the full error in *e, are you sure postgres is running ?

I'm not sure about postgres- I just followed their instructions and didn't do any other/ additional steps:

dev=> *e
#error {
 :cause "Connection refused: no further information"
 :via
 [{:type clojure.lang.ExceptionInfo
   :message "Error on key :duct.database.sql/hikaricp when building system"
   :data {:reason :integrant.core/build-threw-exception, :system {:duct.logger.timbre/brief {:enabled? true, :async? false, :min-level :report, :rate-limit nil, :output-fn #object[duct.logger.timbre$brief_output_fn 0x2d725020 "duct.
logger.timbre$brief_output_fn@2d725020"], :fn #object[taoensso.timbre.appenders.core$println_appender$fn__31301 0x4fe39e5e "taoensso.timbre.appenders.core$println_appender$fn__31301@4fe39e5e"]}, :duct.logger.timbre/spit {:enabled? t
rue, :async? false, :min-level nil, :rate-limit nil, :output-fn :inherit, :fn #object[taoensso.timbre.appenders.core$spit_appender$self__31311 0x301a332b "taoensso.timbre.appenders.core$spit_appender$self__31311@301a332b"]}, :duct.l
ogger/timbre #duct.logger.timbre.TimbreLogger{:config {:level :debug, :appenders #:duct.logger.timbre{:spit {:enabled? true, :async? false, :min-level nil, :rate-limit nil, :output-fn :inherit, :fn #object[taoensso.timbre.appenders.
core$spit_appender$self__31311 0x301a332b "taoensso.timbre.appenders.core$spit_appender$self__31311@301a332b"]}, :brief {:enabled? true, :async? false, :min-level :report, :rate-limit nil, :output-fn #object[duct.logger.timbre$brief
_output_fn 0x2d725020 "duct.logger.timbre$brief_output_fn@2d725020"], :fn #object[taoensso.timbre.appenders.core$println_appender$fn__31301 0x4fe39e5e "taoensso.timbre.appenders.core$println_appender$fn__31301@4fe39e5e"]}}}}}, :func
tion #object[clojure.lang.MultiFn 0x66346c8a "clojure.lang.MultiFn@66346c8a"], :key :duct.database.sql/hikaricp, :value {:jdbc-url nil, :logger #duct.logger.timbre.TimbreLogger{:config {:level :debug, :appenders #:duct.logger.timbre
{:spit {:enabled? true, :async? false, :min-level nil, :rate-limit nil, :output-fn :inherit, :fn #object[taoensso.timbre.appenders.core$spit_appender$self__31311 0x301a332b "taoensso.timbre.appenders.core$spit_appender$self__31311@3
01a332b"]}, :brief {:enabled? true, :async? false, :min-level :report, :rate-limit nil, :output-fn #object[duct.logger.timbre$brief_output_fn 0x2d725020 "duct.logger.timbre$brief_output_fn@2d725020"], :fn #object[
enders.core$println_appender$fn__31301 0x4fe39e5e "taoensso.timbre.appenders.core$println_appender$fn__31301@4fe39e5e"]}}}}, :connection-uri "jdbc:"}}
   :at [integrant.core$build_exception invokeStatic "core.cljc" 283]}
  {:type com.zaxxer.hikari.pool.HikariPool$PoolInitializationException
   :message "Failed to initialize pool: Connection to localhost:5432 refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections."
   :at [com.zaxxer.hikari.pool.HikariPool throwPoolInitializationException "HikariPool.java" 576]}
  {:type org.postgresql.util.PSQLException
   :message "Connection to localhost:5432 refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections."
   :at [org.postgresql.core.v3.ConnectionFactoryImpl openConnectionImpl "ConnectionFactoryImpl.java" 303]}
  {:type java.net.ConnectException
   :message "Connection refused: no further information"
   :at [ pollConnect "Net.java" -2]}]
 :trace
 [[ pollConnect "Net.java" -2]
  [ pollConnectNow "Net.java" 672]
  [sun.nio.ch.NioSocketImpl timedFinishConnect "NioSocketImpl.java" 539]
  [sun.nio.ch.NioSocketImpl connect "NioSocketImpl.java" 594]
  [java.net.SocksSocketImpl connect "SocksSocketImpl.java" 327]
  [java.net.Socket connect "Socket.java" 633]
  [org.postgresql.core.PGStream createSocket "PGStream.java" 231]
  [org.postgresql.core.PGStream <init> "PGStream.java" 95]
  [org.postgresql.core.v3.ConnectionFactoryImpl tryConnect "ConnectionFactoryImpl.java" 98]
  [org.postgresql.core.v3.ConnectionFactoryImpl openConnectionImpl "ConnectionFactoryImpl.java" 213]
  [org.postgresql.core.ConnectionFactory openConnection "ConnectionFactory.java" 51]
  [org.postgresql.jdbc.PgConnection <init> "PgConnection.java" 223]
  [org.postgresql.Driver makeConnection "Driver.java" 465]
  [org.postgresql.Driver connect "Driver.java" 264]
  [com.zaxxer.hikari.util.DriverDataSource getConnection "DriverDataSource.java" 119]
  [com.zaxxer.hikari.pool.PoolBase newConnection "PoolBase.java" 369]
  [com.zaxxer.hikari.pool.PoolBase newPoolEntry "PoolBase.java" 198]
  [com.zaxxer.hikari.pool.HikariPool createPoolEntry "HikariPool.java" 467]
  [com.zaxxer.hikari.pool.HikariPool checkFailFast "HikariPool.java" 541]
  [com.zaxxer.hikari.pool.HikariPool <init> "HikariPool.java" 115]
  [com.zaxxer.hikari.HikariDataSource <init> "HikariDataSource.java" 81]
  [hikari_cp.core$make_datasource invokeStatic "core.clj" 251]
  [hikari_cp.core$make_datasource invoke "core.clj" 248]
  [duct.database.sql.hikaricp$eval32269$fn__32271 invoke "hikaricp.clj" 45]
  [clojure.lang.MultiFn invoke "MultiFn.java" 234]
  [integrant.core$try_build_action invokeStatic "core.cljc" 292]
  [integrant.core$try_build_action invoke "core.cljc" 291]
  [integrant.core$build_key invokeStatic "core.cljc" 300]
  [integrant.core$build_key invoke "core.cljc" 296]
  [clojure.core$partial$fn__5859 invoke "core.clj" 2636]
  [clojure.core.protocols$fn__8181 invokeStatic "protocols.clj" 168]
  [clojure.core.protocols$fn__8181 invoke "protocols.clj" 124]
  [clojure.core.protocols$fn__8136$G__8131__8145 invoke "protocols.clj" 19]
  [clojure.core.protocols$seq_reduce invokeStatic "protocols.clj" 31]
  [clojure.core.protocols$fn__8168 invokeStatic "protocols.clj" 75]
  [clojure.core.protocols$fn__8168 invoke "protocols.clj" 75]
  [clojure.core.protocols$fn__8110$G__8105__8123 invoke "protocols.clj" 13]
  [clojure.core$reduce invokeStatic "core.clj" 6830]
  [clojure.core$reduce invoke "core.clj" 6812]
  [integrant.core$build invokeStatic "core.cljc" 321]
  [integrant.core$build invoke "core.cljc" 303]
  [integrant.core$init invokeStatic "core.cljc" 418]
  [integrant.core$init invoke "core.cljc" 410]
  [integrant.core$init invokeStatic "core.cljc" 415]
  [integrant.core$init invoke "core.cljc" 410]
  [integrant.repl$init_system$fn__5793 invoke "repl.clj" 37]
  [integrant.repl$build_system invokeStatic "repl.clj" 24]
  [integrant.repl$build_system invoke "repl.clj" 22]
  [integrant.repl$init_system invokeStatic "repl.clj" 34]
  [integrant.repl$init_system invoke "repl.clj" 33]
  [integrant.repl$init$fn__5805 invoke "repl.clj" 54]
  [clojure.lang.AFn applyToHelper "AFn.java" 154]
  [clojure.lang.AFn applyTo "AFn.java" 144]
  [clojure.lang.Var alterRoot "Var.java" 308]
  [clojure.core$alter_var_root invokeStatic "core.clj" 5499]
  [clojure.core$alter_var_root doInvoke "core.clj" 5494]
  [clojure.lang.RestFn invoke "RestFn.java" 425]
  [integrant.repl$init invokeStatic "repl.clj" 52]
  [integrant.repl$init invoke "repl.clj" 49]
  [integrant.repl$go invokeStatic "repl.clj" 61]
  [integrant.repl$go invoke "repl.clj" 57]
  [integrant.repl$go invokeStatic "repl.clj" 58]
  [integrant.repl$go invoke "repl.clj" 57]
  [dev$eval29184 invokeStatic "form-init17641653157386341884.clj" 1]
  [dev$eval29184 invoke "form-init17641653157386341884.clj" 1]
  [clojure.lang.Compiler eval "Compiler.java" 7181]
  [clojure.lang.Compiler eval "Compiler.java" 7136]
  [clojure.core$eval invokeStatic "core.clj" 3202]
  [clojure.core$eval invoke "core.clj" 3198]
  [nrepl.middleware.interruptible_eval$evaluate$fn__1249$fn__1250 invoke "interruptible_eval.clj" 87]
  [clojure.lang.AFn applyToHelper "AFn.java" 152]
  [clojure.lang.AFn applyTo "AFn.java" 144]
  [clojure.core$apply invokeStatic "core.clj" 667]
  [clojure.core$with_bindings_STAR_ invokeStatic "core.clj" 1977]
  [clojure.core$with_bindings_STAR_ doInvoke "core.clj" 1977]
  [clojure.lang.RestFn invoke "RestFn.java" 425]
  [nrepl.middleware.interruptible_eval$evaluate$fn__1249 invoke "interruptible_eval.clj" 87]
  [clojure.main$repl$read_eval_print__9110$fn__9113 invoke "main.clj" 437]
  [clojure.main$repl$read_eval_print__9110 invoke "main.clj" 437]
  [clojure.main$repl$fn__9119 invoke "main.clj" 458]
  [clojure.main$repl invokeStatic "main.clj" 458]
  [clojure.main$repl doInvoke "main.clj" 368]
  [clojure.lang.RestFn invoke "RestFn.java" 1523]
  [nrepl.middleware.interruptible_eval$evaluate invokeStatic "interruptible_eval.clj" 84]
  [nrepl.middleware.interruptible_eval$evaluate invoke "interruptible_eval.clj" 56]
  [nrepl.middleware.interruptible_eval$interruptible_eval$fn__1282$fn__1286 invoke "interruptible_eval.clj" 152]
  [clojure.lang.AFn run "AFn.java" 22]
  [nrepl.middleware.session$session_exec$main_loop__1352$fn__1356 invoke "session.clj" 218]
  [nrepl.middleware.session$session_exec$main_loop__1352 invoke "session.clj" 217]
  [clojure.lang.AFn run "AFn.java" 22]
  [java.lang.Thread run "Thread.java" 833]]}

Connection to localhost:5432 refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections postgres is an external database, I assume you have to install it and start it manually you can use an embedded database if you don't want to bother with it

Or maybe using :gen-class etc which isn't extremely ergonomic to use

I guess I'll just write some Java

There are also • https://github.com/athos/JiSE • https://github.com/jgpc42/insn Haven't used them myself though. And just in case, there are other projects that help you use Java from Clojure without having to compile stuff.

Yes, in fact, in this project I have used insn already, but it seems a bit overkill for this one

I only needed to do this:

package babashka.impl;

public class URLClassLoader extends java.net.URLClassLoader {

    public URLClassLoader(java.net.URL[] urls) {
        super(urls);
    }

    public URLClassLoader(java.net.URL[] urls, java.net.URLClassLoader parent) {
        super(urls, parent);
    }

    public void addURL(java.net.URL url) {
        super.addURL(url);
    }
}

Can't you use setAccessible instead, without creating a wrapper at all? Can't you use

Yes, probably, but then it would be accessible globally in the runtime right?

which seems like a hack

Oh, and it doesn't seem to be possible: > module java.base does not "opens http://java.net" to unnamed module

those are two literal vector expressions that evaluate to themselves

if you're thinking about [0] as an indexing operator, that is not a syntax in Clojure

right no, i definitely didn't think that it's an indexing operator but I was shocked that it didn't complain and evaluated just fine

so can I bind literal vectors in this way to multiple symbols

no

so does this expression have any type?

the reader reads and evaluates each expression independently (it just doesn't start until you hit enter

there are two expressions here, both are persistent vectors

i see so they are just two completely independant self-evaluating expressions

yes

the REPL just doesn't start reading until you hit enter

but I am getting both of them back after hitting enter meaning that they were simply independantly evaluated

they are not a single expression that evaluates to multiple results

the reader reads [1 2 3 4] it is passed to eval, the result of eval is printed

then the reader reads [0] it is passed to eval, the result of eval is printed

the repl is acutally always reading, just terminals are typically line buffered, meaning they don't send input to a program reading stdin until you hit enter

[1 2 3 4][0] and then hitting enter is the same thing as [1 2 3 4] hitting enter and then [0] and then hitting enter

so the read kind of scans the whole input and as soon as it finds a valid expression that can be immediately evaluated , then it evals and proceeds further for more

This might have been a good candidate for #C053AK3F9 because the explanation given here would probably help others learning Clojure @U04MCS8GJUT

I realized it after posting that I was on the wrong channel, and responses already started coming by then.

I don't know if one can cross-post after the fact

You can share a post into another channel (and then delete the original without affecting the shared version). No worries this time. Just thought it was a very enlightening question (and answers).

(defn foo [] [1 2 3 4][0])

Right that's what I also did, basically kind of like the last expression in body of a let

hey! #C030C4Z2W0Y’s a good channel, datapotato lives under the donut banner. will come back to this soon!