Fork me on GitHub
Pieter Slabbert07:02:59

Hello, I'm using the cognitect aws library to insert items into dynamodb. This worked perfectly on Friday, then when I came back on Monday I started getting this error Caused by java.lang.ClassNotFoundException when trying to evaluate (def dynamodb-client (aws/client {:api :dynamodb})) I have tried including various dependencies in my project.clj, but nothing works, and what really gets me is that I haven't changed any dependencies in this project for weeks prior to this, and suddenly I start getting this.


Please move the stacktrace into an attachment. I think Slack even offered you to do that when you were writing the post. Otherwise, it spans multiple screens right in the main channel, making other messages much harder to reach.

Pieter Slabbert07:02:30

Sorry about that


Check your classpath. There should be multiple tools for Lein that let you do that, including showing situations when a particular dependency is being excluded.

Pieter Slabbert08:02:27

I have tried examing the classpath, I don't see it there. The class I'm missing is here but I can't figure out how to include it, as far as I can tell it should already be included just by adding [ "0.8.641"]but obviously I'm missing something.


Oh, right, it's the same lib. I'd also try clearing class caches, if there are any. Apart from that, can't really suggest anything, would have to debug it.

Pieter Slabbert08:02:34

I have tried lein clean didn't work. I eventually just deleted my .m2 directory and let it download everything again, but still it doesn't work. Are there any other methods I should try?


Does it happen only in a specific project or in an empty project as well?

Pieter Slabbert09:02:13

Looks like it is only this specific project. I have seen that one of my dependencies includes the cognitect aws API as well, I have tried excluding it using

:exclusions [
in my project.clj, but it didn't help. What is even more annoying is that I have tried adding (:import [ HttpClient]) to the namespace and I can evaluate the ns clause just fine, but the moment I tried to actually use the library it complains that the class I just imported isn't there!


But at least you have a starting point now - given that it's only in that specific project, you can gradually remove its pieces till you're left with nothing but an MRE, a minimal reproducible example. If coming up with that example won't shed any light on what's going on, you can share it here and I or someone else will be able to take a look.

Mark Wardle09:02:17

Hi all. I'm benchmarking some code using LMDB. One of the speed-ups suggested by the LMDB author is to re-use transactions. I'm therefore experimenting with how I might create an object pool that can store a transaction and I '`renew`' and '`reset`' instead of open and close. If these experiments don't show a good speed-up, I'll stick with the simple [current] option. But reusing transactions can apparently help when one is using lots of short read-only transactions. I can see I could use ThreadLocal to store a transaction per thread, but this is little used in Clojure, and probably for good reason, and I am not sure how that will work when we have virtual threads in the future. I thought about writing a simple macro (e.g. with-ro-txn) that hides the complexity for handling the state. The alternative is some kind of object pool. Lots of existing libraries look far too generic and heavyweight for my use-case. Is there something simple I could use in the Clojure core to create a per-thread 'instance' in a safe and efficient way? The semantics are per-thread, retain and release, or create if does not exist. I don't think a Clojure var will work, because I couldn't use binding high up up the call stack to get any benefit from per-thread re-use.


Or you can use datalevin, which reuses LMDB transactions.

Mark Wardle16:02:25

Thanks @U0A74MRCJ - I do use datalevin in another project, but this is currently using LMDB directly.

Mark Wardle16:02:21

e,g. is using LMDB directly, but my other work uses datalevin (e.g.!


What would be the benefits of using it directly?


LMDBJava is something I personally want to get ride of. It is not actively maintained.


It has bugs in its iterator logic, and it uses an older version of LMDB that has critical bugs.

Mark Wardle16:02:53

Historical mainly as I was using mapdb originally and then switched backends - but I would hope faster.... I don't use iterators but just plain standard cursors. I've been bitten by the lack of releases with running on Apple Silicon but have solved that since. I can see a time with modern FFI that we can use LMDB directly.

Mark Wardle16:02:18

Using datalevin for my other projects has been a breeze however!


Sure. I mean, if there are other people actively maintain a LMDB binding for me, I would be happy. The FFI story in JVM will not be solved in a short time. Modern FFI of JVM will not support graalvm for a long time, so one needs to maintain two versions of them at least. These are things I wish someone else are doing it for me.


As to your original question, it is simple to use a concurrrent queue, dequeue, or something from Java to do that.

Mark Wardle16:02:18

For SNOMED CT, which is quite enormous, I wanted to hand roll serialisation, and persistence - given its size and my need to run very fast. Interestingly, many commercial equivalents take hours to import, and require complex set-ups such as Elasticsearch etc. whereas LMDB has given me an amazing speed-up (ie 4-5 mins or so) and is usable from anything from a personal machine all the way to suite of load-balanced cloud based servers. I'm using cursors and peeking into buffers to get fast responses in a domain-orientated way, which has some trade-offs - in many ways using a more generic key value store would be preferable. Haven't tried graalvm yet. Did you ever look at the LMDB bindings from JGL?

Mark Wardle16:02:57

Thanks! I'll have a look at concurrent queues - sound less problematic than using thread locals, which I've always had an aversion to! Thanks for the advice!


LMDBJava’s LMDB binding is the fastest already. I don’t think it can be improved speed wise. It is using Unsafe to directly manipulate buffer, there’s nothing faster than that in the Java world.

👍 2
Mark Wardle16:02:51

PS. Datalevin underpinned a complex analytics pipeline for a European Medicines Agency project in multiple sclerosis - I loaded up the UK dictionary of medicines and devices, and lots of socioeconomic data and combined with clinical data - so thank you for all of your hard work. handled all of the drugs, and handled the socio-economic data. Made it very easy so thank you!

👍 2

The only problem is that it is not actively worked on, so I had to do some work myself to make it work on apple silicon, to use newer LMDB version, fixing its iterator logic problem, etc. That’s why I think it is better off for you to switch to datalevin.


we reuse transaction, reuse cursors, and generally take care of things that you have to write yourself anyway.

Mark Wardle16:02:56

Thanks! I'll do some benchmarking as the less code I have to maintain the better! I do also use Lucene for my fast free text search, and have used it for a long time (back when I was using Java), so some of my choices are simply inertia...


Looking forward to your benchmarking. There are so many places for performance improvement in these things. It is often easy to have 2 or 3x speed up by some trivial tuning. Lucene data ingestiion speed is about twice faster than datalevin search, but it’s query speed is slower than datalevin search.


Anyway, my goal of developing datalevin is to simplify data access for myself. Most of code we wrote daily is just data plumbing, so it save a lot of effort if we have something simple that takes care of a lot of things for us.


With regard to wrapping lmdb, I've often found that c libraries have a more functional interface than their java counterparts. For this reason, I wrote to make it easier to generate clojure ffi wrappers. Currently, clong only supports generating wrappers for JNA, but the idea is that once the API is available as data, you can programmatically generate a wrapper that supports any idioms you need for performance, usability, or resource management. I'm not sure that using clong would be any easier than maintaining a fork of lmdbjava, but I thought I would throw it out there in case it was interesting. I was surprised to see that lmdbjava uses jnr-ffi rather than JNI. JNI bindings should be faster, but they can be a pain to maintain.


The way LMDBJava works, I don’t think it makes much of a difference. At the bottom, it is all JNI. But the performance hit of wrapping LMDB is mainly about how to handle the data input/output, whether you can do zero-copy of data. The function call itself makes minimal difference. LMDBJava use Unsafe to directly work on native buffers and there’s no copy, so there’s really not much more to shave off. One can use a faster buffer implementation, that would be faster., but the difference is also not much. The original maintainer of JNI wrapper of LMDB has abandoned his own JNI library, and joined LMDBJava, if that tells you something.

👍 2

fwiw, I was curious to see how clong would handle lmdb's API and a minimal example worked out of the box.

👀 2
👍 2

JNA's class uses malloc under the hood, so you should be able to do zero-copy similar to lmdbjava.

Mark Wardle22:02:24

Wow! I am impressed.


Unless there’s some radical advantage, people are not going to switch FFI at this moment, as panama is around the corner. One advantage of JNA is that it seems to have graalvm support, so there’s no need to do anything extra to get native image.


JNA is known to be one of the slowest, so I think it is a hard sell.


My understanding is that, right now, JavaCPP is the fastest. dtype used to be using javacpp, he switched to dtype-next that does his own thing, but his use case is limited to numerics, e.g. he don’t supports function pointers, etc. So, really, we are just waiting for panama to land in the mainline.


clong is decoupled from any particular ffi implementation. I will probably add a generator for coffi at some point which will work with panama. It probably wouldn't be that hard to write a generator to target jnr-ffi.


I know JNA switched to using libffi which is what jnr-ffi uses under the hood. I'm not sure how recent that is and if there are up-to-date benchmarks available.


That’s great. When panama lands, I may use clong to get going with LMDB when you support panama.

Martin Půda12:02:38

Hi, I'm generating a new Duct project, following, with these hints:

> lein new duct duct23 +site +example +ataraxy +cljs +postgres
But (go) in REPL returns this:
dev=> (go)
Execution error (ConnectException) at  (
Connection refused: no further information
What am I doing wrong? (all steps to reproduce this in 🧵)

Martin Půda12:02:54

> lein new duct duct23 +site +example +ataraxy +cljs +postgres
Active code page: 1250
Generating a new Duct project named duct23...
Run 'lein duct setup' in the project directory to create local config files.

/duct23> lein duct setup
Active code page: 1250
Created profiles.clj
Created .dir-locals.el
Created dev/resources/local.edn
Created dev/src/local.clj

\duct23> lein repl
Active code page: 1250
nREPL server started on port 50205 on host - 
REPL-y 0.5.1, nREPL 0.9.0
Clojure 1.10.3
Java HotSpot(TM) 64-Bit Server VM
    Docs: (doc function-name-here)
          (find-doc "part-of-name-here")
  Source: (source function-name-here)
 Javadoc: (javadoc java-object-or-class-here)
    Exit: Control+D or (exit) or (quit)
 Results: Stored in vars *1, *2, *3, an exception in *e

user=> (dev)
dev=> (go)
Execution error (ConnectException) at  (
Connection refused: no further information


you should have the full error in *e, are you sure postgres is running ?

Martin Půda12:02:22

I'm not sure about postgres- I just followed their instructions and didn't do any other/ additional steps:

dev=> *e
#error {
 :cause "Connection refused: no further information"
 [{:type clojure.lang.ExceptionInfo
   :message "Error on key :duct.database.sql/hikaricp when building system"
   :data {:reason :integrant.core/build-threw-exception, :system {:duct.logger.timbre/brief {:enabled? true, :async? false, :min-level :report, :rate-limit nil, :output-fn #object[duct.logger.timbre$brief_output_fn 0x2d725020 "duct.
logger.timbre$brief_output_fn@2d725020"], :fn #object[taoensso.timbre.appenders.core$println_appender$fn__31301 0x4fe39e5e "taoensso.timbre.appenders.core$println_appender$fn__31301@4fe39e5e"]}, :duct.logger.timbre/spit {:enabled? t
rue, :async? false, :min-level nil, :rate-limit nil, :output-fn :inherit, :fn #object[taoensso.timbre.appenders.core$spit_appender$self__31311 0x301a332b "taoensso.timbre.appenders.core$spit_appender$self__31311@301a332b"]}, :duct.l
ogger/timbre #duct.logger.timbre.TimbreLogger{:config {:level :debug, :appenders #:duct.logger.timbre{:spit {:enabled? true, :async? false, :min-level nil, :rate-limit nil, :output-fn :inherit, :fn #object[taoensso.timbre.appenders.
core$spit_appender$self__31311 0x301a332b "taoensso.timbre.appenders.core$spit_appender$self__31311@301a332b"]}, :brief {:enabled? true, :async? false, :min-level :report, :rate-limit nil, :output-fn #object[duct.logger.timbre$brief
_output_fn 0x2d725020 "duct.logger.timbre$brief_output_fn@2d725020"], :fn #object[taoensso.timbre.appenders.core$println_appender$fn__31301 0x4fe39e5e "taoensso.timbre.appenders.core$println_appender$fn__31301@4fe39e5e"]}}}}}, :func
tion #object[clojure.lang.MultiFn 0x66346c8a "clojure.lang.MultiFn@66346c8a"], :key :duct.database.sql/hikaricp, :value {:jdbc-url nil, :logger #duct.logger.timbre.TimbreLogger{:config {:level :debug, :appenders #:duct.logger.timbre
{:spit {:enabled? true, :async? false, :min-level nil, :rate-limit nil, :output-fn :inherit, :fn #object[taoensso.timbre.appenders.core$spit_appender$self__31311 0x301a332b "taoensso.timbre.appenders.core$spit_appender$self__31311@3
01a332b"]}, :brief {:enabled? true, :async? false, :min-level :report, :rate-limit nil, :output-fn #object[duct.logger.timbre$brief_output_fn 0x2d725020 "duct.logger.timbre$brief_output_fn@2d725020"], :fn #object[
enders.core$println_appender$fn__31301 0x4fe39e5e "taoensso.timbre.appenders.core$println_appender$fn__31301@4fe39e5e"]}}}}, :connection-uri "jdbc:"}}
   :at [integrant.core$build_exception invokeStatic "core.cljc" 283]}
  {:type com.zaxxer.hikari.pool.HikariPool$PoolInitializationException
   :message "Failed to initialize pool: Connection to localhost:5432 refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections."
   :at [com.zaxxer.hikari.pool.HikariPool throwPoolInitializationException "" 576]}
  {:type org.postgresql.util.PSQLException
   :message "Connection to localhost:5432 refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections."
   :at [org.postgresql.core.v3.ConnectionFactoryImpl openConnectionImpl "" 303]}
   :message "Connection refused: no further information"
   :at [ pollConnect "" -2]}]
 [[ pollConnect "" -2]
  [ pollConnectNow "" 672]
  [ timedFinishConnect "" 539]
  [ connect "" 594]
  [ connect "" 327]
  [ connect "" 633]
  [org.postgresql.core.PGStream createSocket "" 231]
  [org.postgresql.core.PGStream <init> "" 95]
  [org.postgresql.core.v3.ConnectionFactoryImpl tryConnect "" 98]
  [org.postgresql.core.v3.ConnectionFactoryImpl openConnectionImpl "" 213]
  [org.postgresql.core.ConnectionFactory openConnection "" 51]
  [org.postgresql.jdbc.PgConnection <init> "" 223]
  [org.postgresql.Driver makeConnection "" 465]
  [org.postgresql.Driver connect "" 264]
  [com.zaxxer.hikari.util.DriverDataSource getConnection "" 119]
  [com.zaxxer.hikari.pool.PoolBase newConnection "" 369]
  [com.zaxxer.hikari.pool.PoolBase newPoolEntry "" 198]
  [com.zaxxer.hikari.pool.HikariPool createPoolEntry "" 467]
  [com.zaxxer.hikari.pool.HikariPool checkFailFast "" 541]
  [com.zaxxer.hikari.pool.HikariPool <init> "" 115]
  [com.zaxxer.hikari.HikariDataSource <init> "" 81]
  [hikari_cp.core$make_datasource invokeStatic "core.clj" 251]
  [hikari_cp.core$make_datasource invoke "core.clj" 248]
  [duct.database.sql.hikaricp$eval32269$fn__32271 invoke "hikaricp.clj" 45]
  [clojure.lang.MultiFn invoke "" 234]
  [integrant.core$try_build_action invokeStatic "core.cljc" 292]
  [integrant.core$try_build_action invoke "core.cljc" 291]
  [integrant.core$build_key invokeStatic "core.cljc" 300]
  [integrant.core$build_key invoke "core.cljc" 296]
  [clojure.core$partial$fn__5859 invoke "core.clj" 2636]
  [clojure.core.protocols$fn__8181 invokeStatic "protocols.clj" 168]
  [clojure.core.protocols$fn__8181 invoke "protocols.clj" 124]
  [clojure.core.protocols$fn__8136$G__8131__8145 invoke "protocols.clj" 19]
  [clojure.core.protocols$seq_reduce invokeStatic "protocols.clj" 31]
  [clojure.core.protocols$fn__8168 invokeStatic "protocols.clj" 75]
  [clojure.core.protocols$fn__8168 invoke "protocols.clj" 75]
  [clojure.core.protocols$fn__8110$G__8105__8123 invoke "protocols.clj" 13]
  [clojure.core$reduce invokeStatic "core.clj" 6830]
  [clojure.core$reduce invoke "core.clj" 6812]
  [integrant.core$build invokeStatic "core.cljc" 321]
  [integrant.core$build invoke "core.cljc" 303]
  [integrant.core$init invokeStatic "core.cljc" 418]
  [integrant.core$init invoke "core.cljc" 410]
  [integrant.core$init invokeStatic "core.cljc" 415]
  [integrant.core$init invoke "core.cljc" 410]
  [integrant.repl$init_system$fn__5793 invoke "repl.clj" 37]
  [integrant.repl$build_system invokeStatic "repl.clj" 24]
  [integrant.repl$build_system invoke "repl.clj" 22]
  [integrant.repl$init_system invokeStatic "repl.clj" 34]
  [integrant.repl$init_system invoke "repl.clj" 33]
  [integrant.repl$init$fn__5805 invoke "repl.clj" 54]
  [clojure.lang.AFn applyToHelper "" 154]
  [clojure.lang.AFn applyTo "" 144]
  [clojure.lang.Var alterRoot "" 308]
  [clojure.core$alter_var_root invokeStatic "core.clj" 5499]
  [clojure.core$alter_var_root doInvoke "core.clj" 5494]
  [clojure.lang.RestFn invoke "" 425]
  [integrant.repl$init invokeStatic "repl.clj" 52]
  [integrant.repl$init invoke "repl.clj" 49]
  [integrant.repl$go invokeStatic "repl.clj" 61]
  [integrant.repl$go invoke "repl.clj" 57]
  [integrant.repl$go invokeStatic "repl.clj" 58]
  [integrant.repl$go invoke "repl.clj" 57]
  [dev$eval29184 invokeStatic "form-init17641653157386341884.clj" 1]
  [dev$eval29184 invoke "form-init17641653157386341884.clj" 1]
  [clojure.lang.Compiler eval "" 7181]
  [clojure.lang.Compiler eval "" 7136]
  [clojure.core$eval invokeStatic "core.clj" 3202]
  [clojure.core$eval invoke "core.clj" 3198]
  [nrepl.middleware.interruptible_eval$evaluate$fn__1249$fn__1250 invoke "interruptible_eval.clj" 87]
  [clojure.lang.AFn applyToHelper "" 152]
  [clojure.lang.AFn applyTo "" 144]
  [clojure.core$apply invokeStatic "core.clj" 667]
  [clojure.core$with_bindings_STAR_ invokeStatic "core.clj" 1977]
  [clojure.core$with_bindings_STAR_ doInvoke "core.clj" 1977]
  [clojure.lang.RestFn invoke "" 425]
  [nrepl.middleware.interruptible_eval$evaluate$fn__1249 invoke "interruptible_eval.clj" 87]
  [clojure.main$repl$read_eval_print__9110$fn__9113 invoke "main.clj" 437]
  [clojure.main$repl$read_eval_print__9110 invoke "main.clj" 437]
  [clojure.main$repl$fn__9119 invoke "main.clj" 458]
  [clojure.main$repl invokeStatic "main.clj" 458]
  [clojure.main$repl doInvoke "main.clj" 368]
  [clojure.lang.RestFn invoke "" 1523]
  [nrepl.middleware.interruptible_eval$evaluate invokeStatic "interruptible_eval.clj" 84]
  [nrepl.middleware.interruptible_eval$evaluate invoke "interruptible_eval.clj" 56]
  [nrepl.middleware.interruptible_eval$interruptible_eval$fn__1282$fn__1286 invoke "interruptible_eval.clj" 152]
  [clojure.lang.AFn run "" 22]
  [nrepl.middleware.session$session_exec$main_loop__1352$fn__1356 invoke "session.clj" 218]
  [nrepl.middleware.session$session_exec$main_loop__1352 invoke "session.clj" 217]
  [clojure.lang.AFn run "" 22]
  [java.lang.Thread run "" 833]]}


Connection to localhost:5432 refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections postgres is an external database, I assume you have to install it and start it manually you can use an embedded database if you don't want to bother with it

👀 2

If I remember correctly it's not possible to implement extends directly in Clojure right?


Or maybe using :gen-class etc which isn't extremely ergonomic to use


I guess I'll just write some Java


There are also • Haven't used them myself though. And just in case, there are other projects that help you use Java from Clojure without having to compile stuff.


Yes, in fact, in this project I have used insn already, but it seems a bit overkill for this one


I only needed to do this:

package babashka.impl;

public class URLClassLoader extends {

    public URLClassLoader([] urls) {

    public URLClassLoader([] urls, parent) {
        super(urls, parent);

    public void addURL( url) {
To make addURL public


Can't you use setAccessible instead, without creating a wrapper at all? Can't you use


Yes, probably, but then it would be accessible globally in the runtime right?


which seems like a hack


Oh, and it doesn't seem to be possible: > module java.base does not "opens" to unnamed module

Chitradeep Dutta Roy16:02:09

When I type [1 2 3 4][0] in a repl it returns it as is, doesn't complain. What does this expression mean in clojure, is it actually evaluated?

Alex Miller (Clojure team)16:02:14

those are two literal vector expressions that evaluate to themselves

Alex Miller (Clojure team)16:02:52

if you're thinking about [0] as an indexing operator, that is not a syntax in Clojure

Chitradeep Dutta Roy16:02:35

right no, i definitely didn't think that it's an indexing operator but I was shocked that it didn't complain and evaluated just fine

Chitradeep Dutta Roy16:02:19

so can I bind literal vectors in this way to multiple symbols

Chitradeep Dutta Roy16:02:14

so does this expression have any type?

Alex Miller (Clojure team)16:02:15

the reader reads and evaluates each expression independently (it just doesn't start until you hit enter

Alex Miller (Clojure team)16:02:49

there are two expressions here, both are persistent vectors

👍 2
Chitradeep Dutta Roy17:02:07

i see so they are just two completely independant self-evaluating expressions

Alex Miller (Clojure team)17:02:48

the REPL just doesn't start reading until you hit enter

Chitradeep Dutta Roy17:02:03

but I am getting both of them back after hitting enter meaning that they were simply independantly evaluated


they are not a single expression that evaluates to multiple results


the reader reads [1 2 3 4] it is passed to eval, the result of eval is printed


then the reader reads [0] it is passed to eval, the result of eval is printed


the repl is acutally always reading, just terminals are typically line buffered, meaning they don't send input to a program reading stdin until you hit enter

👍 4

[1 2 3 4][0] and then hitting enter is the same thing as [1 2 3 4] hitting enter and then [0] and then hitting enter

👍 2
Chitradeep Dutta Roy17:02:46

so the read kind of scans the whole input and as soon as it finds a valid expression that can be immediately evaluated , then it evals and proceeds further for more

yes 8

This might have been a good candidate for #C053AK3F9 because the explanation given here would probably help others learning Clojure @U04MCS8GJUT

Chitradeep Dutta Roy20:02:56

I realized it after posting that I was on the wrong channel, and responses already started coming by then.

Chitradeep Dutta Roy20:02:46

I don't know if one can cross-post after the fact


You can share a post into another channel (and then delete the original without affecting the shared version). No worries this time. Just thought it was a very enlightening question (and answers).

(defn foo [] [1 2 3 4][0])
is valid too and calling (foo) will evaluate both expressions and return the second [0] -- essentially throwing away the value of the first expression.

Chitradeep Dutta Roy20:02:20

Right that's what I also did, basically kind of like the last expression in body of a let

Michael Gardner23:02:33

@nonrecursive hopefully this is an OK way to ask for help with datapotato, since there's no #datapotato (yet?). How would I tell datapotato that two different fields in the same object need to have different values from each other? E.g. src-id and dst-id.


hey! #C030C4Z2W0Y’s a good channel, datapotato lives under the donut banner. will come back to this soon!