Clojurians Log v2

@micha has joined the channel

hello aatree world

is there an example application that uses aatree? or rather calf etc

Yeah, look in the test files.

You want to look at yearling. this is the database that builds over virtual aatree structures. An OODBMS for EDN. simple_smile

@micha Thanks for asking.

this is something i have been looking for for a long time

Using the Datomic analogy, I want the hoplon clients to be the peers with their own caches and 80% of the database code running in the client. Unfortunately, it is the 80% that has not yet been converted from java.

dude yes

so can i ask a sort of general kind of question?

maybe off topic

of course

it seems to me that we use relational databases to store our application state only because storing only facts in sorted sets achieves the laziness, like you don't need to fetch an entire tree or sequence of things from storage to perform a query

but in all other ways relational databases are terrible, because you need to build your trees and maps and whatnot via joins

Even my sequences are indexed! simple_smile

and the semantic of the join cannot exist in the database itself

it's just something you have in your head

so people make orms as a way to form abstractions that encompass the joins

My db is very close to datomic, with the same indexes and everything. (my java db, not what I have in clojure so far)

My starting point was to build a db for semantic relationships.

A knowledge db, rather than a data db.

what's the distinction?

everything is about entities, attributes, values and time.

knowledge exists in time.

attributes are dynamic

attributes are largely used to define relationships between entities.

So it can be viewed as a graph database too.

do you do extra indexing of the graph edges?

Dealing with knowledge always seems to require multiple perspectives.

nothing more in that area than datomic.

but I keep a lot more metadata!

alan and i experimented with building a thing on top of datomic

to store clojure data

i.e. relationships are also entities!

we ran into the need to run our own indexes of which refs point where

and stopped

Datomic makes it difficult to deal with changes over time, though I think it can still be done.

I'm doing the reverse. First implement virtual clojure data. Then add support for changes over time. Then build the indexes on top of that.

This gives me a rich mix. I can have some things that are not tracked over time, for example.

we had a specific problem at hand

and we had a datomic license so we thought why not

sounds like a lot of fun

the problem was this huge php application that we needed to debug in production

OUCH!

legacy code, 250K lines of it

so we wanted to log data to datomic

wild

so we could get a historical view of the data

and automate as much as possible

so we made entities for all the types

and had a system for interning them

and making refs to build the things

relationships should also be entities.

ah interesting!

we couldn't avoid following refs in code

gives you the edges. simple_smile

this would allow you to avoid that

Now I am very excited about doing the om next thing, but with haplon and my own db. And put the application code in the browser!

do you think you'll be able to write a "peer" that can run in a browser?

why not?

i know datomic can't do it

it just needs to keep a cache.

browsers don't have a big cache

but my transactor/storage IS the web server.

you can count on maybe 5M

And it has the big cache.

the client just needs a little.

this would be the final piece of the puzzle

a mobile client is very focused so it does not need a lot of cache

--small working set

right now the thing that takes our dev time is incidental complexity involved with shuffling things in and out of memory in the client, making ajax requests etc

it's like a virus infecting all your abstractions

becaue you can't keep a reference to anything

I am a systems dev, not an app dev.

so my approach is a tad different

you can keep entity names, attribute names, times.

in the client?

yes. some of them would be constants for a given app.

I did access control. So users can have a private part of the database for managing their apps.

And it would all exist virtually in the mobile. 😄

or an application on the server that prunes the tree, making a view basically

why?

like a proxy between the client and db

Put the app logic in the mobile, except for updates!

interesting

this is the cqrs approach

which is A+

And even updates on the server would be generic and access controlled. So it is really all on the mobile. the logic anyway. most of the logic is data that can be moved anywhere.

have you seen castra by the way?

Needless to say, I need to build a team to do all this. I am just happy that clojure is denser than java.

I updated castra chat today and put it in my github repository.

oh right, derp

--rebuilt it on top of the new templates.

I dug into the default app. Tomorrow I do chat.

so is the aatree interface very different than that?

i mean just the interface between server and client

i.e. you have opaque functions you can call to update the state

aatree is missing a layer--the entitys/attributes/values and indexes are all built on top.

and a repository of state that may change from time to time

But at least what I have is persistent.

yeah I have that much

but I still need to add logging among other things.

i mean you know, in your vision

always been there, yes

i will clone the repo and see how far i can get with the tests

i would enjoy learning about how databases work

With hoplon methinks I need to add subscriptions to notifications soon. To allow clients to maintain a cache. 😄

here is a thing i ran into in the app i'm working on now

it's a real puzzle

at adzerk we have a pretty good api, and people can use it to create a lot of a thing

like if someone wants to make 100k ads that's ok with us

so the ui has to deal with that

which is a really interesting problem

ok

consider a simple dropdown menu

so you had to introduce some abstractions, hmm?

we have these things called channels, and those have many priorities

I've always had forward and reverse pagination for all the query results in my ui.

yeah but you can't paginate a tree

i mean not in a nice way

simple_smile

you have trees that get large?

huge yes

our solution was to split it into two parts

the channel and the priority as separate typeahead widgets

but one is dependent on the other

so something of a matrix view

well it's a tree

or matrix selection actually

but you can't keep the tree in memory

a partitioned tree?

so if you want to validate that the priority really belongs to the channel, you can't

i mean you can, but you need to make a call to the backend to fetch the part of the tree where it lives

sounds like you really want vm

vm?

virtual memory

oh right

yes

or virtual tree structures

😄

it seems like aatree aims to provide this?

already does in yearling

it just needs a whole lot of love

but as it stands, it ONLY supports trees!

graphs come in the next layer

haha that's all we need in lispland

well enjoy then

If you like it, I can provide support.

--I am not expensive when it comes to working on my own stuff!!!

Or I can just help you learn it.

--no charge

I am full time on this

i would enjoy helping out wherever i can

maybe i could help with the hoplon stuff simple_smile

fer sure!

is it a goal of the project to eventually support many peers?

and large datasets?

i mean reasonably large

i guess i meant to ask about the intended use

I've added you as a collaborator

it depends on the interests of the team we need to put together.

I want to restrict initial use to 1TB on a single host.

cool, that's reasonably large simple_smile

Once we have a reasonable team, we can fork out in different directions. With developers contributing to multiple forks. One size, after all, does not fit all. I think that is necessary to achieve real scale.

My basic design for the dbs is composits of bags. keeps things flexible. A VERY open architecture!

a lot of documentation here

I hate docstrings. Of all the code, docstrings are the most difficult to refactor.

There are at least tools for updating variable names!

hahah

Documents, on the other hand, are allowed to be a bit more abstract so they do not need to be constantly updated.

i'm going to make a thing tomorrow that will let me generate markdown api docs

for boot

Super!

docstrings are nice there because you can attach them to vars

ok

and it is a good time. things are getting more stable

boot has a ton of functions in the api

I'm sure

so it seems like it's finally necessary to generate api docs

but don't let it get like lein--which has more issues than the next 100 most popular projects combined.

i don't like the things that generate html because while it might look fancy, it's harder to version with the code

yeah i mean boot is different than lein, because it's just a library really

a collection of functiona and macros you can use to make a build program

with javadocs I always turned off the timestamps so versioning would be more reasonable.

yeah i was thinking i'd generate the api docs as a precommit hook

so you can go to a branch or a tag in github and see api docs for that commit

wild

the docs would be committed in the repo of course

i guess then you could even git diff the api docs and see a changelog

heh

I wasted a lot of space in the past as I included a copy of the javadocs in each release.

clojure is much better for docs

with java you really need all the hyperlinks and frames and whatnot

yeah 😞

because you have hundreds of classes in various packages

tell me about it!

I got so tired of it all

yeah it makes me angry

like honestly angry

I am happy with clojure

clojure is great!

and excited about all the doors hoplon can open for me

i've been obsessed with making web apps for a while now

Using pure clojure script would be much harder than .hl.

because it seems so simple, like too simple to even be computer science

but look at any real project, most of the work is wasted on trying to get a ui working

so lets close the loop and give the client in the browser access to the virtual data structures.

yeah that's the missing part

also i have been trying to design a sort of "state machine zoo"

a way to form abstractions around distributed systems problems you encounter in the client

I am impressed with no-virtual-dom reactive web--so light weight!

I think we will have a lot of fun

complimentary strengths and all that

by expressing them as a combination of datastructures stored on some persisted storage and the state machines that mediate access and modification of them

like submitting a form in a web page

that's really part of a state machine that is manipulating some data structure

and the semantics of retries, failure modes, etc

I did some distributed stuff. Once you get the basics down, it is pretty simple. Just a tad subtle. Quarm logic on the bottom is key.

i have this feeling that there are a relatively small number of state machines that can describe any kind of distributed system interaction

And google has done some good work in this area.

what i'm specifically interested in is developing a set of abstractions that clearly define the tradeoffs involved

better to say that there are a relatively small number of state machines that can be used to describe any kind of distributed interaction. I.E. You only need a few.

the key thing is to clearly delinieate the tradeoffs and ensure correctness within the constraints of those

it would be great if anyone could use these in a simple and easy to understand way

But for the foreseeable future I plan to focus on plain old client server. Too much stuff left to migrate from Java.

Yes. Constraints are what it is all about.

clojure is the inspiration, with the stm

Most developers chaff against them, but really they are enablers.

not a big fan of stm personally.

rich devised a set of state machines

and the tradeoffs and semantics are clearly defined

but perhaps I just haven't gotten there yet.

cool

i mean the architecture of it

I agree with 90% of what the guy says.

everything in clojure is immutable, but some things can be altered

😄

but anything that can be altered can only be altered via a well defined state machine

i.e. anything alterable can have watches added to it

yes, that is beauty!

this is what i envision for distributed systems sort of

I want to add watches to yearling.

subscription would be key vectors a la assoc-in.

Which then get tied to haplon cells.

events would be deleted and updated.

deleted event would also unsubscribe

this sounds like exactly what i want to work on also

interesting

i had a wacky idea for subscriptions with hoplon and datomic

a pull-based subscription

because reads in datomic are almost free, you can spin on the server

my guess is that most of what I need to learn is in the castra-chat demo

i'll add docs

it's time simple_smile

I think there is a place in my virtual aatree structure where I can generate change notifications.

I'm gona call it a night. I must say, these discussions keep getting more interesting!

see you later!