Iterators are already implemented.

you mean get-range?

all these functions use an iterator

right, get-range

As I said last time, all these functionalities are already there

that is statefull though and might not always work. I don't always control who calls entrySet

no need to implement your own

I am trying the implement the java.util.Map interface over datalevin API

iterator is stateful, that’s the contract

by definition

next, hasNext, that’s state

sure, but range-seq opens a transaction that should be closed

how can I make sure clients will close transaction ?

that’s why you don’t use range-seq, entrySet will return the whole thing in one go,that’s the contract

ok, but that won't work for large data sets - which is my case

it is already spilling to disk

so that’s not a problem

in term of respecting the API

if I have functions to get first key from db, and get next key after a key than this could be solved in a cleaner way IMO

that’s an iterator, which by defintion, is stateful

you are talking about map API, which is not stateful. The stateful thing are already implemented, what you do will not be different, you are not buying anything by doing your own

get a key, then get next, that’s what iterator is

which is already implemented, and it is complicated

I know get range can get keys using some filters > range-type can be one of :all, :at-least, :at-most, :closed, :closed-open, :greater-than, :less-than

don’t think that’s somethong you should be concerned about, you either use it, or not, there’s no point trying to implement it

I will see if I can get something that for a given key "a" I can get the next key in DB after it.

I think lmdb is the state. I can keep the current key in an atom inside iterator

You are not doing anything useful by doing that. If you want an iterator, it is in there

what you describe is an iterator, which itself requires state that is independent from data

not just LMDB, iterator requires its own state

(defn entry-set-iterator
  "Implemnts ^java.util.Iterator for use in entrySet implementation.
   
   "
  [db dbi & _opts]
  (let [state (atom {:current-key nil})]
    (reify java.util.Iterator
      (hasNext [_this]
        (throw (UnsupportedOperationException. "Not implemented")))
      (next [_this]
        (throw (UnsupportedOperationException. "Not implemented")))
      (remove [_this]
        (throw (UnsupportedOperationException. "Not implemented"))))))

I can keep iterator state in atom

you don’t need these

you are doing unnecessary work, because we already have an iterator, you can just use it

i don’t see how this is helping

Of course, you can layer things on top of things, but you gain nothing but lose performance. I don’t see why we you want to do that

ok, for some context. I pass my map implementation to a function that uses this code:

(doseq [x coll]
      (.write wr (print* x))
      (.write wr (int \newline)))

I understand fro you that I can use: • get-range - loads everything in memory and spills to disk if OOM • range-seq - opens a transaction that needs to be closed

iterate-kv is a part of IDB protocol, it returns an iterator

Really, I don’t say how you can do better than just use get-range

I mean, you say you have big data, how implement your own iterator can avoid this? You still got big data.

yes, but I don't need to keep a transaction open if I open a transaction only when next() is called

if I keep the current key

java.util.map API entrySet requires you return the whole set, you cannot escape that

and call functions to get next key from lmdb then I should be ok

> Returns a Set view of the mappings contained in this map.

a set VIEW - does not need to be realized

that is how I understand it anyway

you are still going to keep the data somewhere? right, what would that achieve?

yeah but I need to keep the current key in memory

get-range reads all

while range-seq needs an open transaction

in my case it should open a transaction whenever it's doing work

(e.g. call next() / hasNext() , etc)

sure, you can do that. it is doable, the performance will suffer, but it’s a trade off

I can put it in the notes

I guess all versions have some rade-off

as long as you are willing to open a transaction to read every key, sure, why not

we do renew read transaction, so it may not be too bad

you can use get-range and get-some to implement this

get-some will try to get the next one

so it is doable

ok, thanks for the pointers

and for the talk.

I feel it was a bit intense at some point

I am fine. I hope you are too

no worries, don’t be alarmed, that’s how I talk with people,

ok

i tend to talk straight, it’s about ideas, not about people

I saw both get-* return k-v . I guesss it's no way to avoid getting the value back

i am not a people person

but I get a buffer so I can avoid reading it - and it should be ok ?!

get-kv is when you know the key, but you don’t in an iterator

it should be ok, you don’t have to read the buffer, when you do, you don’t have to read the whole buffer

you can do whatever with the buffer in fact. you can write the buffer as one data type, but read it as another, or partially. I do these in writing the search engine, for example.

thanks. I will see what I can make out.

there is also with-transaction-kv that I forgot about.

I can make so processing is using a single transaction

i would still not deal with these low level things for this purpose though, I would use get-some, etc.

i don’t think that’s possible

with-transaction-kv is for write

not for read

read transacton and write transaction are two different thing

I mean use it like this:

(with-transaction-kv ....

(.entrySet my-mao)
....
) 

iterators can remove elements

and I think the entrySet implementation can also support remove operation

the purpose of with-transaction-kv is to make sure reads during write see what’s just being written

otherwise, writes are only visible after commit

so with-transaction-kv is for implement things like CAS

ok

when you use with-transaction-kv, you are blocking write, which is not user would expect

there can be only one writer at a time

yeah, unless it's documented 🙂

then it's a feature 🙂

btw, I did find guava-testlib that has unit tests for collection implementation. Planning to use those to check the implementation. Added a comment on issue with links

I wouldn’t think it is user friendly for a java.util.map to be blocking writes to the whole DB.

yeah, it wouldn't be

btw, for > 0.11.0 A new Datalog query engine with improved performance. I have an idea to implement datalog over Apache Calcite relational algebra. I think that might be a better query engine.

I did a small lib that allows you to use calcite from Clojure https://github.com/ieugen/calcite-clj

and do SQL over clojure collections

I believe it is possible to have datalog over relational algebra : https://calcite.apache.org/docs/algebra.html

and calcite has a powerfull query engine (or that is my impression)

Well, I tend to not believe marketing words. I think these things do need innovation from research, which is what I intend to do.

These things are not easily improved without doing innovative work, which most open source projects do not

the reason I do datalevin is because I think SQL is a bad language

even its inventors admit that

I came from IBM Research - Almaden, I worked in the same floor as people who invented SQL

I don’t think they like SQL themselves. LOL

its’ a historical accident that sql gets to where it is now. it’s not good

my intention is to bring datalog query processing performance to be on par with RDBMS, which is doable.

clojure is good for experimentation. If the experiment is successful, it is possible to write it in a lower level language like rust, zig, or whatever, that’s my long term plan

the benefit of triple store is the unrolled storage, which makes the most difficult part of query optimization easy, that is cardinality estimation. This is what I see as the breakthrough.

ok, calcite has an SQL parser. it translates the SQL query to relational algebra that it uses to run the query. there could be a Datalog -> relational algebra conversion

relation algebra with recursion, that’s datalog. they are the same

final FrameworkConfig config;
final RelBuilder builder = RelBuilder.create(config);
final RelNode node = builder
  .scan("EMP")
  .build();
System.out.println(RelOptUtil.toString(node));

SQL is not a good relational algebra implmentation. That’s the part where it fails.

anyway, that’s why i am doing this, so database is so central to programming. a litle improvement will have huge impact.

not sure, but I don't think calcite's relational algebra is SQL only.

thanks for the personal info

I don't have such a high vision

just playing around for now 🙂

I would like to expore datalog on top of calcite relational algebra when I get to that part

somebody has to have.

another vision is to bring most of data access into an integrated story for application development.

since most of code is just transforming data from one db to another, such a waste of human efforts.

these are all laid out in README, if one reads it carefully.

sounds like a long term plan.

good luck!

thanks again for the conversation.

signing off soon

cheers, thanks for the interest.

I agree that adding a third way of iterate over the data would be a good addition.