Hello all, I have some ambiguous questions about how to model state that might be a bit outside the scope of Hoplon/Javelin, if you'll indulge me..

I am writing a todo/note-taking app and I want to be able to reference any piece of data from any other piece. This seems to me like a graph. But am I correct in assuming that such graphs or sub-graphs (Dags?) can still be modeled in Clojure associative data structures (nested maps, etc). I am client/UI oriented and external database models are foreign to me. For standard graph-like analysis and manipulation, is it recommended to look beyond standard clojure data to something like Datomic/Datascript or Ubergraph (https://github.com/Engelberg/ubergraph) or Odin (https://github.com/halgari/odin)?

I believe in Javelin and want to improve my intuitions about it's power and idiomatic usage. That said, I am exploring Specter (new intro article http://nathanmarz.com/blog/clojures-missing-piece.html) and am encouraged by it's declarative "Navigators" DSL that works like a query language for associative Clojure data. With Specter, you can wrangle arbitrarily nested data quite easily. This power lends itself to the notion of referencing and transforming a single-atom that holds all the complex state in a single map. But that contrasts with the Javelin notion of building up complex state out of small stem cells and formula cells of arbitrary complexity. Top-Down vs Bottom-up i guess...

Of course I can use both, calling Specter functions inside formula cells too. But I don't yet appreciate the tradeoffs of this. If Specter is like (update-in) on steroids, but Javelin lets you keep root data-structures shallow, maybe I don't really need "update-in on steroids"?

I will need to persist my client state to external storage. With the single-map approach, it is trivial to write that to a firebase reference, which stores it as JSON. But I am a foggy as to how that state map gets boiled down to something the Javelin cell graph can use for reactive UI. Also, there are concerns about how large that map can be considering client memory constraints and read/write performance.

With the more atomic Javelin cell approach, I think I would need to persist the client state as a flatter catalog of cell contents, that could be synced (rehydrated) easily, rather than arbitrarily nested JSON that would have to be deconstructed to be usable from the bottom-up Javelin cell graph. And if I wanted to use Specter for sophisticated queries or transforms, I would have to operate on the aggregated data structure (formula cell), then explicitly deconstruct that back into the flatter cell index for local Javelin reactiveness and Firebase persistence.

The Specter author speaks of using Specter a lot to analyze and transform directed acyclic graphs. Is the Javelin cell graph a DAG? If I already have a graph in the Javelin formula hierarchy, would it be imprudent to be looking towards some other graph data structure (datascript, etc). I'm not sure how reading and writing to a graph-like data structure from the Javelin dataflow space would look

I know Javelin has lenses, which separate the "read" function from the "write" function. The "read" is distributed across the cell graph. But i think the lens-write might need to do some gymnastics to match the derived state of the query. Specter's version of lenses has both the "read" and "write" based of the same succint declarative query.

I apologize for winding around in circles. I'm just a little uncertain on how to mesh these approaches (Javelin vs Specter), or if that's adding too much confusion and I should just stick to one or the other. Or if what I want to do is outside the common scope of these libraries, Or if I should really be focusing on some in-memory database model like Datascript to get the most out of my ramp-up. I don't want to re-invent a common client-state-database architecture, just cause I'm unfamiliar with databases in general, server-side IO, and the tradeoffs of in-memory data. I figured if I can stick to standard Clojure data (EDN) and operate with it intuitively with powerful but simple libraries like Javelin and Specter, that might be the strongest start. Just don't want to run into any obvious walls. Of course I will keep digging and and developing my own insight, but any feedback is much appreciated. And sorry again about the long-windedness 🙂

ps. To throw another wrinkle: If I have some nested EDN. I can convert that to something query-able with datalog, via Intension (https://github.com/alandipert/intension). Thx @alandipert . I also found some Specter functions which I believe do something similar. So that seems to lend weight to keeping my canonical data in nested/associative clojure data. Agree?

pss. Specter includes an implementation of Zippers, which Alan uses on the Hoplon Orcus (theLarch) project (https://github.com/tailrecursion/orcus/blob/master/src/frontend/orcus/zip.cljs.hl) So in my head: [Javelin + Zippers = good], so [Javelin + Specter = also good!]. It is obvious to me how zippers would help manage an in-memory tree data-structure to render and edit an outline in the UI. But how does that factor into a durable data structure that can be persisted, (re)made reactive, queried, etc? Is it the same data model or different data models for different concerns?

@chromalchemy so a bunch of things, I would implement this as a state namespace (kinda like a big atom) which have state cells that correspond directly to paths in your firebase db

@chromalchemy https://github.com/tonsky/datascript

@mobileink have you used datascript with firebase before?

nope! 😉

I see this being an issue due to how firebase prefers small bits of state transactions, for it to work with something like an in memory db, you would need to generate queries that transform the data both ways

sorry, tl;dr. i thought @chromalchemy was going for in-mem stuff

in fact i have not used datascript at all, but it looks really good.

@mobileink i think you are on the right mark, but I dont think in-memory db systems work well with firebase as is, would probably want a firebase-specific in-memory db due to how the firebase paths work, or at least a specific interop layer, trying to do this at the app layer is where I think many people have issues with firebase

If you are using firebase I recommend local state via javelin cells and hoplon.firebase 😉

oy. it's unbelievable how fast stuff moves. firebase is on my list, along with about 10k other bits. and next month my list will be outdated. :(

Hey @flyboarder I am indeed using your hoplon.firebase lib. I dont' know how to deploy a proper Clojure (Java) app. And your Firebase integration got me up and running 🙂!!! I like the idea of a client that can just deploy to javascript. But if that's to much of a constraint to get a rich data model and editor going, I will figure out the necessary devops stuff. If something like Datomic makes all this hella easier than inventing some messy app-level interop-laden middleware, I can steer in that direction.

@flyboarder Since Specter can operate on arbitrary nested data, doesn't that kind of function like an in-memory DB? How does that square with the Javelin+Firebase approach?

@chromalchemy glad it's useful to someone 😛 im not sure I understand exactly what you are looking for tho, I like to think of firebase+javelin as a 1:1 relationship, that is javelin cells correspond to paths in my db, I then have small bits of local state which update against those root cells

I was imagining a (fb-cell) can contian a relatively large map. Do stuff in the client, then simply (reset) the cell. Although it might be more logical and performant to break up the state into a set of fb-cells (firebase refs) with somewhat nested data in each one.

nested data in fb is frowned upon

fb always wants you to denormalize the data

that is break it up into smaller bits of state

which is actually good as it makes you think more about how you are storing your state

for me I give everything a "type" in my db, at the root level, everything in there get's a unique ID (index) that I use everywhere else to reference the data, I then have a few app level javelin cells which normalizes the data across the root cells

so my root cells are a 1:1 with the db data, my app has local state that merges the "real" data

@chromalchemy does any of this make sense?

I think so. Parsing your language.... type = unique ID = index value? denormalized data = db data => Firebase ref + root cells ? app level javelin cells = "merged" root cells = "real" data = normal data?

So the denormalized data is like the contents of an app cell + indexes, redundant keys, etc?

correct, so in my db I have something like this {:some-type {:some-index {:another-type :another-index}} :another-type {:another-index {:data "value"}}}

and then I merge those cells in my app

there are 2 points when you can mount paths in firebase client app, 1. prior to login 2. at login, all my public (read=true) firebase paths are mounted when the app loads, the secured paths are mounted at login once there is a user id, I have a few javelin cells which merge the app state once either of these things happen

Cool. I haven't tried the auth stuff yet.

Where does complex nested data typically come from?. Seems like clojure best practices promote shallow data structures and small functions. I like the idea of Specter as a query DSL that can mine the relationships implied by nesting. But where would you get such nested data in the first place. And how would you write it, persist it, rehydrate it from db, etc?

I would say usually nested data comes from scoped state

like having a tenant account with multiple user accounts underneath it, ok thats fine now what about a many-to-many relationship, where my users can exists under many tenants, bam complex data

usually any time you have a many-many relationship you are creating complex data

@chromalchemy what kind of data are you working on? clojure is a data-language I doubt you really need a specific DSL for dealing with your data structures

I think my first data is like an outline with aliases, so not exactly a rigid tree. Then to be able to filter on item metadata and parent/child relationships (even if the child is under 2 different parents)

seems like specter would be used within the javelin cells once you normalized the data

ah so the first part would be to define your data in terms of a rigid tree, this is how firebase is designed to store data

not just a rigid tree, but as flat a tree as possible, preferable no more than 2 or 3 levels deep

Once you have (complex) normalized data that is aggregated from smaller (root) cells. How do you denormalize it and get it back into the flattened Firebase structure?

javelin lenses

which write back to the fb database after denormalizing the new cell value and picking back out the parts of the new state I want to store

which allows me to do transformations prior to storing the data

such as converting clj data types to js

Ok I definitely don't want to forsake Javelins full powers. Specter is also Lenses. For reference to its use cases, take a peek at (http://nathanmarz.com/blog/clojures-missing-piece.html). It was also recently debated on Reddit https://www.reddit.com/r/Clojure/comments/5wy188/clojures_missing_piece_specter_10_release/ Some of the examples suggest elegant ways of teasing out meta-relationships of the complex data and doing transformations based on that, that would probably otherwise take some recursive clojure that is a bit over my head. Seems like Firbase requires the Javelin approach, with more atomic reads/writes. I was just concerned that after aggregating some normal data, I might not have the manual clojure wizardry to do more sophisticated queries on it, and that Javelin lenses are more surgical, and it would be difficult to do transformations on larger, more complex aggregates of normal data, or if you did the transforms, denormalizing it would be a pain.

I likely need to just dig in with what I have and feel it out to quash this anxiety. Good to know that Javelin and Firebase are such a nice fit.

yeah if you are using hoplon.firebase I wrote that specifically for javelin, it works well, I think you could use specter for sure, but im not sure it's needed depending on how complex your data gets, It's something I would optimize for later

@chromalchemy if your application is always consuming clojure data from a javelin cell, you can swap the underlying root cells at any time, javelin is great like that

I make my root cells dynamic so that I can rebind them to another data source if needed, this separates my application from my data-store implementation, my app only knows about javelin cells and clojure data, nothing about the underlying system

I will pay attention to that. I followed your basic pattern on the Blaze repo when setting up the FB cells. https://github.com/flyboarder/blaze/blob/master/src/blaze/firebase.cljs.hl

yeah thats the best example I have public right now

Given your example

{:type1 {:index1 {:type2 :index2}} 
 :type3 {:index3 {:data "value"}}}

so using your example, my app would merge the data at :type2 :index2 into :type1 :index1

so if you had this:

{:type1 {:index1 {:type2 :index2}}
 :type2 {:index2 {:somedata "someval"}} 
 :type3 {:index3 {:data "value"}}}

{:type1 {:index1 {:type2 {:somedata "someval"}}}}

in the blog example it would have a post with an author id which merges in the author data

sorry updated^^^

I see, the nested type/index is like a template var for the root type/index of the same name

exactly!

I think the in-memory db should be a layer that normalizes this and provides a dsl for query over the data, then spit the correct parts back to firebase

but it's not a requirement

Not a requirement cause you can often just use the atomic cell data directly in components?

right, and since clojure is all about data manipulation i dont ever need a query dsl

i just navigate maps

relatively shallow maps?

If firebase frowns on nesting. How deep can you practically go?

nevermind. The javelin cells can build up the arbitrary complexity 😉 and javelin lenses can break it down for re-writes

yep...

firebase complains past 32 levels I think? might be 16, but it gets slow past a few

Do you ever write nested maps to Firbase in your own practice?

yes, but still keep them shallow, and scope my lookups

so :posts is a type. :content is an index (can hold multiple blocks of data with the same schema)? What is :blarg?

:blarg is just a random index for users I have firebase generate it automatically

so that is an example where i isolated the content to speed up lookups when I want a list of posts, I can list the meta data without fetching the post content

I usually also have a "default" index that is always merged with whatever data I lookup, so I can pre-populate empty-state

in firebase if you set something to false or nil it removes the entry so you may want to keep paths around that have "default" state

Ok . I'm still having trouble understanding the difference between "type" and "index" with this model. I understand {:key data} at the leaves of the tree.

so "type" is like a sudo-scheme

more of a group of related data, ex. "posts"

This is illuminating though. I see now that you are seeding redundant copies with contexualized meta-structure as necessary. Instead of inferring it all from a nested data structure with no copies.

so a post is made up of both content and meta data, so I split those apart and associate them with the index value

if I wanted the data together it would just be under the post index

like the user data

@chromalchemy I hope this helped more than confused you 😛

It's helping hugely, Thx!

So in the example :users is an index. And :posts is a type. But if :posts did not have sub-indexes, than it would be an index too?

nope, :users is also a type, so posts, is :type :sub-type :index

:users :index

does that clarify it?

Mostly, can you give a simple definition of "index"?

any unique ID

when you reuse a UID it should mean the data is either merged (reused as index) or inlined (reused as value)

Ok, I think I got it now. Will play around with a schema + javelin and see where it goes 🙂 Thanks hugely for the library and the insight. As far as databases go, Firebase seems simple enough to me. And that is very encouraging.

On another note. I've had trouble hydrating a javelin cell from Firebase on page-load. I need to use a long (with-timeout). (defonce) doesnt seem to work. Any hints?

also, when do you convert clj data to js?

I convert clj->js within firebase-cljs which is used by hoplon.firebase

So that is auto-handled in the (fb-cell)?

^correct you should be able to pass clj data directly and it will be converted for you

regarding page-load you may want to create all your refs during namespace load, so they are available by page-load time, which should populate at that point

ie. via a (def some-ref (fb-cell...))

best to look at blaze again, there is a bunch we load after the page is ready

(def ...) is part of the namespace load?

the value of def is calculated when the namespace is loaded

so in that snip the reference *blaze* is created when the namespace is loaded, posts returns the fb ref to posts, posts= returns a javelin cell containing the posts, I set a variable to (posts=) which is then fetched when the page loads https://github.com/flyboarder/blaze/blob/master/src/index.cljs.hl#L26

I could change the def to defonce and it should work the same

How many dynamic fb refs would one use? One for each type? or do you base them all off one root dynamic fb ref?

https://github.com/flyboarder/blaze/blob/master/src/index.cljs.hl#L21-L29

I usually use a few "application domain types"

like posts or categories for the blog

@chromalchemy https://github.com/flyboarder/blaze/blob/master/src/blaze/core.cljs.hl here is a public API of all the dynamic states I use in blaze

as long I reference them from the core namespace there, I can rebind the cells to something else when my implementation changes, like if I decide to move away from firebase

Ok. I will review Blaze again and poke around with my own denormalized schema. Thanks again for walking through the concepts w me.

In your experience with Firebase + Javelin, would you ever need optimize for something like Datascript? What would be the trigger?

@chromalchemy I have never needed something that complex, javelin makes the local state approach much easier

But I imagine there are large-app related reasons one may want to, but in that case we probably need a generic library that merges javelin+data script before trying to implement that in the app layer

That context is helpful so I dont worry about yak shaving too much. I have enough to chew on. Sounds like merging Specter + Javelin should be more approachable if I need such complexity (I probably wont 😅)

Yeah it's very doable but I don't see why anyone would need it? How big is your client app's data? Should you have a full backend and server resources for working with the data? At that complexity probably.

Unless you are serverless like pouchdb or something