Fork me on GitHub
#asami
<
2020-12-29
>
alandipert20:12:31

oh, hello 👋

alandipert20:12:19

just took a cruise through the asami query engine, very nice 👏

alandipert20:12:36

i like the subgraph discovery piece that avoids unnecessary cross joins

quoll20:12:47

Aww, thank you.

quoll20:12:19

Do you mean the analytics namespace, or are you referring to the aggregates part? (I’m unhappy with aggregate queries right now)

alandipert20:12:43

oh, i was referring to paths i think the function was called

alandipert20:12:58

finding the route thruogh the tuples for the least amount of tuple-scooping

quoll20:12:02

That’s the planner

alandipert20:12:03

err, the rules rather

quoll20:12:17

Yup. It helps a lot!

quoll20:12:48

I can’t say that I did a GREAT job of it, but I did it. 😊

quoll20:12:07

Working code beats perfect proposals

alandipert20:12:16

i like it and it's small

alandipert20:12:36

i have been on a personal quest lately to write the smallest viable tuple store/datalog in javascript

alandipert20:12:46

eschewing optimizations when possible etc

alandipert20:12:14

i got kind of lost in the literature about bottom-up vs top-down evaluation, magic sets, etc

😂 3
quoll20:12:31

Oh... going back to Naga in late 2016 should provide a good Cljs example. Easily portable to JS, I think

quoll20:12:16

The “paths” function wasn’t as good, but the query engine was definitely minimum viable!

quoll20:12:30

On my phone, but let me look...

alandipert20:12:11

https://gist.github.com/alandipert/4e390808b5dcee9020b6622c0a24023e#file-tinylog-js-L54-L60 is my vision for the DSL, tho the engine on that gist is too naive/broken

quoll21:12:48

The only supported operation is inner joins, but it works. The index namespace is tiny as well

alandipert21:12:09

cool. yeah for indexes i'll just be doing JS arrays + binary search i think

quoll21:12:08

That version of the planner used dynamic programming, so definitely don’t touch that old planner

quoll21:12:37

Yes, that will pack in far better, and give you other benefits as well

quoll21:12:56

Hashmaps are expensive

quoll21:12:46

The durable storage does arrays, though they’re broken into chunks and accessed via a tree

quoll21:12:11

But I’m familiar with using arrays 🙂

alandipert21:12:27

i prefer linked lists of course, but i'll deal 😄

alandipert21:12:07

i'd be curious to hear your reaction to my general ideas around using stuff like this for UI maintenance

quoll21:12:39

I think it’s extremely flexible

quoll21:12:22

I was storing lots of things in this sort of format without a query engine, because of the flexibility it provides

quoll21:12:04

And it can be used to represent shallow maps trivially. Deeper structures take more effort, but are possible

quoll21:12:40

Do you have specific ideas in terms of UI?

alandipert21:12:53

a general feeling, punctuated with concrete ideas

alandipert21:12:49

the general UI problem is continuously/efficiently synchronizing/managing UI primitives as business data underneath them changes

alandipert21:12:18

since most business data is hierarchical, most UI toolkits and UI data models propose hierarchical definitions of equality (and thus, change)

alandipert21:12:44

but in practice state used "internally" (for composition of UI primitives) gets muddled with business data

alandipert21:12:25

anyway, i have a good feeling about little datalog databases wired up to one another, each with independent ideas of what constitutes a "change", based on which queries are installed

alandipert21:12:31

my current direction is, make minimum viable database and query, and then build "subscriptions" on that. just run subscribed queries every time a new fact goes in, and if the result is different, fire a callback with the differences

quoll21:12:45

When the data is quickly accessible in memory, then even without a query engine, I think it offers a lot. It’s easy to build any kind of structure in there. And you can store the universe in there without messing things up. You can also link things (like business data and UI elements) arbitrarily. It’s easy to get anything out of it. And it’s efficient.

alandipert21:12:56

the difference set can be interpreted at the DOM node level to inform adding/removing/changing DOM nodes etc

quoll21:12:49

It’s relatively easy to find out if a fact can affect a subscribed query too. Naga does this

alandipert21:12:53

ah cool, i've noodled about that. i was thinking about how you could transform a query into a function that returned whether or not a fact could possibly affect it... and then only actually run the query again if there's a chance it would

alandipert21:12:59

that line of thinking got me back into rete

quoll21:12:31

eg. If your query has: [?e :type ?t] [?e :name ?n] Then the output isn’t affected if you insert: [:entity124 :value 13]

quoll21:12:58

Did you ever see my 2016 talk at the Conj?

quoll21:12:11

Because I was discussing exactly this

alandipert21:12:33

no, but a watching is imminent

quoll21:12:50

And pointing you at that is much easier than typing on my phone :rolling_on_the_floor_laughing:

👍 3
quoll21:12:26

It takes a shortcut. It assumes that data is only added, rather than allowing modifications. This is how Description Logic systems usually work. And you can emulate modification detection by having a system with time stamped add/delete statements. Or you can just save query results and compare them. It depends on your scalability needs

quoll21:12:35

If it’s in pure JS, then comparing results (as opposed to result-counts) is probably ideal

alandipert21:12:00

that's what i'm thinking... data inherently small... and timestamp for handling deletions not required

quoll21:12:02

Naga’s engine would be ideal, I think

quoll21:12:26

It’s in 2 phases. The first finds dependencies between rules (this is run once, but you would run it after each insert) and then the engine looks for changes via counts, though you would look for data changes instead

alandipert21:12:31

i will watch the talk, study naga, and emit JS codes. thanks much for the info and pointers!

quoll21:12:07

The code that does what I just said is all in here: https://github.com/threatgrid/naga/blob/main/src/naga/engine.cljc

pizzaspin 3
3