Fork me on GitHub
#asami
<
2022-02-16
>
lilactown15:02:04

I have a dumb q: would asami support storing its on disk data in a single file?

quoll15:02:42

It relies on blocks being a consistent size, and those sizes vary between use

quoll15:02:02

It’s possible to pack a committed database into a one file, but it would need to change the architecture to update a bit to use file offsets rather than block numbers.

quoll15:02:32

It might be possible to build all the required structures in memory, and then pack/flush to disk when it’s done. That’s an interesting idea. It would be quite a bit of work over where things are right now, but now that I think about it, it’s actually a bit like what I was considering for cloud storage

lilactown15:02:07

I suppose that would limit the size of the db to whatever can fit in memory

lilactown15:02:23

(which is certainly more than enough for my use cases)

lilactown15:02:13

I'm reading your white paper about storage rn

quoll15:02:46

No… it would limit the size of the transaction to whatever can fit in memory

quoll15:02:37

Though, it would be loading up existing blocks into memory, and doing the transaction into those. Then when the transaction is over, it would take any modified blocks and write them to the end of the file

lilactown15:02:34

another, somewhat separate question I've been thinking about is storing data on disk with some sort of structural sharing

lilactown15:02:25

e.g. if I could store the history of modifying a nested clojure structure without storing copies of each one

lilactown15:02:06

I don't have a complete mental model of asami's storage architecture yet but I'm staring at the image of tree blocks -> data blocks thinking about it...

quoll15:02:34

There are 2 options…

quoll16:02:00

• Breaking the data up into a graph, and letting Asami handle it. That basically means duplicating how Vectors and TreeMaps work, which may be a lot of work. • Repurposing Asami’s data structures to what you want, and forgetting about Asami

lilactown16:02:04

would asami's data structures be suitable for such a thing?

quoll16:02:32

trees and blocks. You can build anything with that

quoll16:02:39

And I did 😜

quoll16:02:27

You also have raw access to the memory mapped files

quoll16:02:49

which is easy enough to do for yourself… unless you want files larger than 2GB

quoll16:02:10

In which case, you could just use the Asami block files code

lilactown16:02:12

if/when I have time, you have given me inspiration to try and use asami's block files code

lilactown16:02:01

stepping into a new role at work as a temporary manager... we'll see if I have any energy for coding again 😛

lilactown16:02:13

but I still have so many ideas!

quoll16:02:37

Same here… unrelated job, but trying to keep up the energy for coding.

quoll16:02:13

This is actually why I’ve been quiet with Asami recently… I’ve been trying to recharge by doing “fun” stuff on the side

lilactown16:02:12

I'm very much of the idea that we ought to follow the fun

lilactown16:02:25

"Play Driven Development"

lilactown16:02:07

asami is already so cool 😎 lots of other things to explore until it's fun again

quoll16:02:06

Oh, it’s fun! I just have SO MUCH TO DO on it

quoll16:02:43

Right now, I’m implementing 2 features: with operations, and background transaction writing (so that transactions return quickly)

Max18:02:48

Hi! I’m enjoying Paula’s talk at reClojure. Is there any documentation on Asami’s assertion metadata capabilities (https://youtu.be/Ug__63h_qm4?t=339)? That’s something I’d like to learn more about. I took a look through the Asami repo’s wiki and didn’t find much. (I posted a similar question way back in December, but figured I’d try again since the message may have gotten lost in the shuffle)

quoll18:02:34

Oh! I haven’t put an API up for that yet 😳

quoll18:02:01

I need to extend the query language and the update API

🙏 1
1