Clojurians Log v2

Thanks for filing the issue. Will investigate.

error: java.util.concurrent.ExecutionException: clojure.lang.ExceptionInfo: Fail to operate list val range: #error {
 :cause nil
 :via
 [{:type java.nio.InvalidMarkException
   :message nil
   :at [java.nio.Buffer reset "Buffer.java" 433]}]
 :trace
 [[java.nio.Buffer reset "Buffer.java" 433]
  [java.nio.ByteBuffer reset "ByteBuffer.java" 1593]
  [java.nio.MappedByteBuffer reset "MappedByteBuffer.java" 358]
  [java.nio.MappedByteBuffer reset "MappedByteBuffer.java" 73]
  [datalevin.bits$get_value_STAR_ invokeStatic "bits.clj" 730]
  [datalevin.bits$get_value_STAR_ invoke "bits.clj" 710]
  [datalevin.bits$get_value invokeStatic "bits.clj" 736]
  [datalevin.bits$get_value invoke "bits.clj" 733]
  [datalevin.bits$avg__GT_r invokeStatic "bits.clj" 838]
  [datalevin.bits$avg__GT_r invoke "bits.clj" 832]
  [datalevin.storage$eav_scan_v_single_STAR_ invokeStatic "storage.clj" 524]
  [datalevin.storage$eav_scan_v_single_STAR_ invoke "storage.clj" 507]
  [datalevin.storage.Store$eav_list_single__31272 invoke "storage.clj" 1150]
  [datalevin.scan$operate_list_val_range invokeStatic "scan.clj" 544]
  [datalevin.scan$operate_list_val_range invoke "scan.clj" 540]
  [datalevin.binding.cpp.CppLMDB operate_list_val_range "cpp.clj" 1125]
  [datalevin.storage.Store eav_scan_v_list "storage.clj" 1166]
  [datalevin.db.DB _eav_scan_v_list "db.clj" 193]
  [datalevin.query.InitStep _execute "query.clj" 78]
  [datalevin.query$init_steps$fn__35176 invoke "query.clj" 1554]
  [datalevin.query$init_steps invokeStatic "query.clj" 1552]
  [datalevin.query$init_steps invoke "query.clj" 1530]
  [datalevin.query$base_plan invokeStatic "query.clj" 1647]
  [datalevin.query$base_plan invoke "query.clj" 1640]
  [datalevin.query$base_plan invokeStatic "query.clj" 1642]
  [datalevin.query$base_plan invoke "query.clj" 1640]
  [datalevin.query$build_base_plans$fn__35230 invoke "query.clj" 1665]
  [datalevin.util$map_PLUS_$fn__21164$fn__21165 invoke "util.clj" 631]
  [clojure.lang.AFn call "AFn.java" 18]
  [java.util.concurrent.FutureTask run "FutureTask.java" 317]
  [java.util.concurrent.ThreadPoolExecutor runWorker "ThreadPoolExecutor.java" 1144]
  [java.util.concurrent.ThreadPoolExecutor$Worker run "ThreadPoolExecutor.java" 642]
  [java.lang.Thread run "Thread.java" 1583]]} {:dbi "datalevin/eav", :val-range [:closed #datalevin/Indexable [nil 45 :db.value/sysMin nil "00" 0] #datalevin/Indexable [nil 45 :db.value/sysMax nil "FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF" 0]]} (FutureTask.java:23)

ps. I enabled value-type checks,everyting seems fine. pps: the part of the error changes ...`[:closed #datalevin/Indexable [nil 23 :db.value/sysMin -13 "00" 0]`...attributes 45 also is of valueType db.type/ref

error: java.util.concurrent.ExecutionException: clojure.lang.ExceptionInfo: Fail to visit list range: #error {
 :cause nil
 :via
 [{:type java.nio.BufferUnderflowException
   :message nil
   :at [java.nio.Buffer nextGetIndex "Buffer.java" 721]}]
 :trace
 [[java.nio.Buffer nextGetIndex "Buffer.java" 721]
  [java.nio.DirectByteBuffer getLong "DirectByteBuffer.java" 853]
  [datalevin.bits$get_long invokeStatic "bits.clj" 146]
  [datalevin.bits$get_long invoke "bits.clj" 143]
  [datalevin.bits$get_eg invokeStatic "bits.clj" 844]
  [datalevin.bits$get_eg invoke "bits.clj" 841]
  [datalevin.bits$read_buffer invokeStatic "bits.clj" 984]
  [datalevin.bits$read_buffer invoke "bits.clj" 959]
  [datalevin.storage$retrieve_ave invokeStatic "storage.clj" 395]
  [datalevin.storage$retrieve_ave invoke "storage.clj" 392]
  [datalevin.storage$ave_tuples_scan_need_v$fn__31159 invoke "storage.clj" 459]
  [datalevin.scan$visit_STAR_ invokeStatic "scan.clj" 383]
  [datalevin.scan$visit_STAR_ invoke "scan.clj" 376]
  [datalevin.scan$visit_list_range invokeStatic "scan.clj" 526]
  [datalevin.scan$visit_list_range invoke "scan.clj" 521]
  [datalevin.binding.cpp.CppLMDB visit_list_range "cpp.clj" 1111]
  [datalevin.binding.cpp.CppLMDB visit_list_range "cpp.clj" 1107]
  [datalevin.storage$ave_tuples_scan_STAR_ invokeStatic "storage.clj" 452]
  [datalevin.storage$ave_tuples_scan_STAR_ invoke "storage.clj" 443]
  [datalevin.storage$ave_tuples_scan_need_v invokeStatic "storage.clj" 456]
  [datalevin.storage$ave_tuples_scan_need_v invoke "storage.clj" 454]
  [datalevin.storage.Store ave_tuples "storage.clj" 1067]
  [datalevin.storage.Store ave_tuples_list "storage.clj" 1073]
  [datalevin.db.DB _init_tuples_list "db.clj" 167]
  [datalevin.query.InitStep _execute "query.clj" 82]
  [datalevin.query$init_steps$fn__35176 invoke "query.clj" 1554]
  [datalevin.query$init_steps invokeStatic "query.clj" 1552]
  [datalevin.query$init_steps invoke "query.clj" 1530]
  [datalevin.query$base_plan invokeStatic "query.clj" 1647]
  [datalevin.query$base_plan invoke "query.clj" 1640]
  [datalevin.query$base_plan invokeStatic "query.clj" 1642]
  [datalevin.query$base_plan invoke "query.clj" 1640]
  [datalevin.query$build_base_plans$fn__35230 invoke "query.clj" 1665]
  [datalevin.util$map_PLUS_$fn__21164$fn__21165 invoke "util.clj" 631]
  [clojure.lang.AFn call "AFn.java" 18]
  [java.util.concurrent.FutureTask run "FutureTask.java" 317]
  [java.util.concurrent.ThreadPoolExecutor runWorker "ThreadPoolExecutor.java" 1144]
  [java.util.concurrent.ThreadPoolExecutor$Worker run "ThreadPoolExecutor.java" 642]
  [java.lang.Thread run "Thread.java" 1583]]} {:dbi "datalevin/ave", :key-range [:closed #datalevin/Indexable [nil 23 :db.value/sysMin -13 "00" 0] #datalevin/Indexable [nil 23 :db.value/sysMax -13 "FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF" 0]], :val-range [:all]} (FutureTask.java:23)

What's your query?

at last this is one:

[:find ?seated . 
 :in $ ?game 
 :where 
 [?table :table/seated ?seated] 
 [?game :game/table ?table]]

Have you tried (untuple ?seated)?

no, i can try, but i don't know the number of elements of the tuple at query time

just like to get the whole vector

if you can produce a minimal reproducible case, I can take a look.

thanks, the strange thing is, sometimes queries just hangs.... (i run tests from a fresh db everytime). this time it hung for ...`[:find ?trick . :in $ ?game :where [?game :game/trick ?trick]]` , no tuple attributes involved there...I am new to datalog and must be messing up something in transactions, interestingly it produces all these different error messages, I'll try to investigate further

yes, the errors you are see is not about the data types, it's about the state of the DB values, they are incorrect.

What's the environment?

ubuntu with deps.edn alias:

:datalevin {:extra-deps {datalevin/datalevin {:mvn/version "0.9.22"}
                         com.cognitect/transit-clj {:mvn/version "1.0.333"}
                         io.airlift/aircompressor {:mvn/version "2.0.2"}}
             :jvm-opts ["--add-opens=java.base/java.nio=ALL-UNNAMED"
                                  "--add-opens=java.base/sun.nio.ch=ALL-UNNAMED"]}

The errors are mostly due to concurrency problems. Which Java is this, which version of ubuntu.

what are the machine spec?

ah ok , yes when i turned on logging, i needed more tries to produce errors...so maybe the timings changed

openjdk 21.0.7 2025-04-15
OpenJDK Runtime Environment (build 21.0.7+6-Ubuntu-0ubuntu124.04)
OpenJDK 64-Bit Server VM (build 21.0.7+6-Ubuntu-0ubuntu124.04, mixed mode, sharing)

Ubuntu 24.04.2 LTS on a hetzner vps

I would clone datalevin and see if I can run all the tests. Then so, I would investigate my code to narrow down to a reproducible case.

👍 ok, i'll try, it is just a toy project to learn datalog, so no time pressure there

This code is developed and tested on some beefy machines, even the CI/CD machines are not small instances.

(mainly was curious to ask, before rewriting code, if tuple of various sizes are ok, in datomic they seem not to)

it should be ok, if the sizes are too big, you will get errors

<= 12 elements mostly

that should be fine, as refs are 8 bytes, 12 of that is only 196 bytes, way less than the limit. So that's not the problem. The problem is concurrency related, so you see transaction reset error, buffer underflow etc, they are all errors related to doing things in wrong order, which should not be the case. So something is seriously wrong.

ok, i'll run the datalevin tests and report back, thanks for investigating!

that's why I ask you to run datalevin tests in your environment

tests are hanging at lein test datalevin.async-test with almost no cpu usage, probably not a good sign

What’s the spec of your machine?

CPU, number of cores, memory

not much, a shared virtual cpu (intel xeon gold) 2 cores, 4GB ram. what are the minimum specs? i'm running the datalevin babashka pod version with a key value store on it without problems.

concurrency test runs without failures, but i killed the async test after 3 hours

part of me still thinks i just did a transaction wrong, mixed up tempids or so...i'll have to look again when i am more awake

Well, the tests didn’t pass. I don’t think I tested on a machine with only two cores.

So the async mechanism probably needs some work. File a GitHub issue if you can

Datalog features do rely on asynchronous features. KV features do not

ok, i'll do tomorrow, wanted to try it on my laptop (4 cores), but thats on WSL which i dont trust very much

i was too curious, no problems with the async test on my laptop with 4 cores, i filed an issue, let me know when I can be of any help. https://github.com/juji-io/datalevin/issues/329

You can run side effects inside rules, but you shouldn't expect the order to be something you expect.

found this one https://github.com/juji-io/datalevin/blob/rule/test/datalevin/rules_test.clj

Is it possible to run side effects from those rules?

The current rule engine is the same as that of datascript, which is not very efficient. Rewrite of the rule engine is one of the major milestones to reach 1.0

If you don’t have a lot of recursive rules, it is okay to use. However, if you do, it can run out of memory easily.

The plan for the new engine will also expand the syntax to allow much more powerful rules.

By definition, rules are logical programs, so they are side effect free.

Thanks!