Fork me on GitHub
#asami
<
2023-01-30
>
Jakub Holý (HolyJak)09:01:14

Hi! Asami doesn’t expose a way to access the metadata of its transactions, right? I can get the current version number w/ as-of-t or timestamp w/ as-of-time of the db and I can get a historical snapshot of the db based either on the version number or a date time, And I can get the list of versions (i.e. 0 … the current one). But I cannot find out what transaction/version/timestamp a datom was transacted/retracted at. So e.g. if I want to find out when a datom was transacted I’d need to search through the past versions of the db to find the first one that has it. Or am I overlooking something?

quoll11:01:52

I’m slow, but I will be exposing this. It’s currently not accessible.

👍 2
quoll11:01:55

Every statement is given an ID when it is inserted. This is a monotonically increasing long.

quoll12:01:03

Transactions occur at particular ID points. So you know which transaction a statement was inserted in by its ID

quoll12:01:38

But, I haven’t implemented the access to it yet 😳

Jakub Holý (HolyJak)13:01:24

That is great to hear. Thank you

quoll13:01:29

The plan is for queries to allow patterns of: [?e ?a ?v ?t] where ?t will be bound to the transaction ID, and: [?e ?a ?v :as ?s] where ?s will be bound to the statement ID Patterns can be truncated, which is to say, the ?t is optional. Also, it can still be included in a full pattern; [?e ?a ?v ?t :as ?s]

quoll13:01:36

Well… it will be when I implement it 😬

quoll13:01:05

All the data structures are there. I need to get the query API updated to do this

👍 2
quoll13:01:22

To explain why Asami progress has been slow lately: • I had used clojure.core/format in error messages, but this is not supported in ClojureScript. So I wrote https://github.com/quoll/clormat • I wanted to load work data into Asami, so I wrote https://github.com/quoll/raphael • I want to write that back out (to turtle format), so I wrote a separate piece of code (not yet complete) which I’m calling https://github.com/quoll/raphael • I am still writing my SPARQL parser, which I’ve named Twylyte (not much pushed yet) I need to integrate all of these, but I’m trying to make them as modular as I can, and putting them in separate packages helps enforce that.

🐢 4
Jakub Holý (HolyJak)21:01:52

I wonder, are you fan of ninja turtles or Italian artists. Sounds like the former... :-)

quoll23:01:41

I was just trying to think of turtle names 🙂

quoll23:01:31

One of my previous projects had already been called Yertle :rolling_on_the_floor_laughing:

quoll13:01:09

Meanwhile, I’ve been asked to write a Clojure workshop on data structures, which I’ve spent my last few weeks on. That’s nearly done.

quoll13:01:35

Anyway… except for the workshop, I’ve continued work on this. I just haven’t released it yet. Oh, and I’m partway through integrating core to implement with. It works differently with in-memory and on-disk, and I need to ensure the behavior is the same, so it’s taking time

🙏 2
Jakub Holý (HolyJak)15:01:59

Wow, that is lot of work you have done (and still have ahead of you)! Good luck!

quoll15:01:35

Thank you. It definitely keeps me busy when I have teenagers to wrangle 🙂

quoll15:01:55

In terms of schedule, I have to finish this workshop, but I’m wrapping up on it now. Then I’ll be looking to finish the with implementation, and the integration of Raphael and Donatello (i.e. importing and emitting TTL). Then I’ll be working on finishing SPARQL. I don’t need everything there, but I want the basics in place. Then I’ll be expanding the query patterns to include transaction and statement IDs.

❤️ 4
quoll15:01:03

The next items on the list are: • Adding metadata to results to indicate if they are “local” or “global”. Local results are groups of numbers. Global results are the same data converted to Strings/keywords/URIs/etc. Right now, everything is global, but keeping everything local until final projection will make for much faster querying. • Writing a B-Tree for the Data Pool. I still like the AVL-tree for the triple indices, but the data pool has different access characteristics. • Building an in-block Patricia trie implementation. This is needed for full-text indexing. Yes, I know that everyone uses Lucene (including AWS with ElasticSearch/OpenSearch), but this doesn’t work for ClojureScript, nor does it give me storage flexibility. • Implementing block storage in Redis, and not just in memory-mapped file. • Implementing block storage in Postgres and MySQL

quoll15:01:34

I need to win the lottery so I can focus on this stuff full time 😄

Jakub Holý (HolyJak)20:01:23

:rolling_on_the_floor_laughing:

mkvlr15:01:40

@quoll if you’d also be open to use an existing lib for SPARQL https://github.com/yetanalytics/flint might be worth a look

mkvlr16:01:48

oh, I now see it mentions Asami in the README: Flint borrows certain syntactic conventions from the Datomic and Asami query and update languages.

Jakub Holý (HolyJak)22:01:23

Is there a way to update a value atomicaly in Asami? In PostgreSQL I could do set column = column + 1 but for Asami I either have to hope nothing else changed the DB between I read the current value and compute the next one or use something like transact-update instead of transact-data, right?