On the webapp served at localhost:3000. My query:

{:find [e]
 :where [[e :crux.db/id _]]}

The page becomes white with a footer, there are React errors in the console.

The same bug happens when in the UI I manually search for an entity from its crux.db/id . In my case, it is in the form:

{:product-life/id 1234567, :product-life/product-type :foobar}

I am not 100% sure that I am running the right UI, in this video the port used was 8080, and the UI style looked better. https://www.youtube.com/watch?v=ykbYNBE-V3k&feature=youtu.be&t=1655

The console is still WIP. Do you mind sharing the react error so it’s very straight forward for us to fix the issue? Thanks :)

not sure how to share it efficiently, but here is a screenshot

I am using

juxt/crux-core {:mvn/version "20.07-1.9.2-beta"}
juxt/crux-rocksdb {:mvn/version "20.07-1.9.2-beta"}
juxt/crux-http-server {:mvn/version "20.07-1.9.2-alpha"}

Thanks will reproduce locally and keep you updated

@U8MJBRSR5 hey, the UI seen in the Strange Loop video is a previous re-frame implementation that did everything fully client-side (it used an unsupported cljs client lib + CORS). The feedback on the styling is appreciated, we've been more focussed on the functionality whilst we get ready for an official release 🙂

I don't know what to say about the styling, for now I can only see half of the UI, the other is crashing. Will provide feedback once I see the whole picture.

Well it works without JavaScript too, you should be able to get almost-the-same experience

After a restart of my repl, the crash disappeared. However, the list of entities became a list of .. white space, and the search for an entity from its id show an error telling that the entity was not found.

The white space is only for the column crux.db/id

Can you see the entities are in the db via a normal query at the repl?

yes, I can see them

right now, the entities are loaded, but the id part is rendered in blank, and not clickable.

Hmm, can you check what happens with javascript disabled also?

What kind of IDs are you using? Keywords? UUIDs? Maps?

with JS disabled, it displays the ids in the table correctly.

I am using maps. https://clojurians.slack.com/archives/CG3AM2F7V/p1594625434260200?thread_ts=1594623855.259500&cid=CG3AM2F7V

in the repl, when I look for the entity using crux.api/entity , it returns nil

.. but the entity is found when I use crux.api/q

it would make sense that the UI doesn't understand that IDs can be maps (sounds like it will be easy to reproduce @UE35Y835W!) but what you're describing now seems very strange. Are you definitely passing the same db value for entity?

I had the same bug last week, after getting some errors in a transaction function (crux.api/entity returning nil). I am no longer using transaction functions and the problem is there again.

When I restart the (local) repl, the problem persists.

(let [db (crux/db crux-node)]
  (crux/entity db {:product-life/id 1234567, :product-life/product-type :foobar}))

did you start with a clean tx-log for 1.9.2? Or did you upgrade from 1.9.0/1 ?

{:crux.node/topology [crux.standalone/topology
                                     crux.kv.rocksdb/kv-store
                                     crux.http-server/module]
                :crux.kv/db-dir "crux-store"}

last week, I was using 1.9.1-beta

ah, so you don't have an event-log-dir for standalone persistence? you're re-populating every time you restart the repl?

The UI definitely understands maps. I am still trying to reproduce

@UE35Y835W the problem might be on the server, the DB reached a state where any call to crux.api/entity returned nil for any id.

I am deleting the DB and try to reproduce.

The weird state happened while I was doing queries only, but the problem persisted after a restart of the REPL.

[wild guess] I suspect that some invalid queries are kept by the DB and retried, and it leads to failure.

Hello vincent - I believe map ids (and a few other types of ids) may still be a problem with the 1.9.2 UI - there was recently some work on it on master to sort out the general handling entity ids, but we mostly brought in bug fixes to the 1.9.2 release so I don't believe they were released in it.

... could it be the way I stop the repl ? (without closing the crux-node)

Ok, I got the problem repeated. Now looking for the minimal problematic scenario.

I think the crux.api/entity problem is related to me not closing the crux-node correctly.

no .. it has nothing to do with closing, I do it correctly

How I reproduce the bug: 1. I delete the db's directory. 2. I start a repl, start a crux node. 3. I insert documents in the db. 4. I query 1 entity from its id, works fine. Multiple times, still works fine. 5. I close the crux-node, recreate one. 6. I query the same entity, nil is returned.

@U8MJBRSR5 do you fancy sharing your screen and pairing on it briefly?

let's continue the conversation in PM

I don't think there's docs, but it's very easy. There's a protocol you can extend. Reading the source of existing implementations would be enlightening.

Yep, we don't have docs on how to write a module currently but looking at how we implemented rocks/lmdb is a good idea. Which backend are you considering?

I was considering MongoDB or PostgreSQL JSONB. But mostly, I just want to study a little bit 🙂 Can you link me the source files I could use as reference? Thanks a lot!

One thing to consider is that the KV indexes should as physically close as possible to the Datalog engine (ideally in-process like Rocks/LMDB) as the algorithm is fairly chatty, but you might be able to get good performance with remote stores like Mongo/Postgres, it would certainly be an interesting experiment. You can see the 3 levels of KV protocols mentioned here: https://github.com/juxt/crux/blob/master/crux-rocksdb/src/crux/kv/rocksdb.clj#L82

This is no longer up-to-date with how Crux modules work, but you can see how the Avisi team managed to get Xodus working as a KV index backend last year (without any help/input from JUXT, I might add!): https://github.com/avisi-apps/crux-xodus/blob/master/src/avisi/crux/kv/xodus.clj