Fork me on GitHub
#datomic
<
2022-07-08
>
Athan14:07:25

How do you visualize your datomic schema ? Hi I made a quick search in this channel to find tools for visualizing Datomic schema. The most promising tool I found was https://github.com/hodur-org/hodur-datomic-schema. This is nice because you can create entity types and connect them through attributes. I would like to share with you an example I made. Here is a simple https://gist.github.com/athanhat/bb581d77cfe224c144a19ea4940f406f and here is my https://gist.github.com/athanhat/7aa53c54110cc3258ba52e00128a065c to run and output the result. There is a better https://github.com/hodur-org/hodur-visualizer-schema using GoJS library. OK that said, I am aware that in Clojure you are trying to break away from entity types, object types in general and work with attribute oriented modeling. But the way you are modeling schema you cannot escape from namespace prefixes and these imply entity types. In any case it is standard practice in any significant large IT project that users want to see and work easily on a white-board with some kind of graph data model that represents an abstract data model. That was the scope of Hodur I guess. In the RDBMs world and more recently in Graph DBMS there are great tools to visualize both the schema and data. So how do you visualize the data modeling process and it's evolution in large projects using Datomic ?

JohnJ16:07:21

maybe malli can help you here https://github.com/metosin/malli#dot

✔️ 1
Athan15:07:55

I had a quick look at schema-cartographer. 1. It creates schema-cartographer dependent edn files when you download schema transactions and schema files from their Web APP UI (see schema.edn) and (schema-txs.edn) 2. You have to work everything on the UI. But I think you cannot download and run this independently on your premises. 3. There is this Create a schema file for an existing annotated ON PREM database from a REPL but again you have to load the resulting output file in the UI that resides at https://schema-cartographer.com 4. It is not flexible enough and it will result in poor visualization when you have a big schema and/or you want to view part of the schema 5. The schema-cartographer UI says GoJS 2.0 evaluation (c) 1998-2019 Northwoods Software Not for distribution or production use http://gojs.net So what tool Cognitect people use to visualize Datomic schema. Is that the tool ?

Jarrod Taylor (Clojure team)17:07:47

1. If you create a schema in the UI you can download the schema transactions as an edn file you can directly use to create the database you have designed. As you pointed out you can also download a representation of the schema that can be loaded back into the UI. If you like you can annotate your schema yourself without using the UI https://github.com/JarrodCTaylor/schema-cartographer/wiki/Learn-More-About-Conventions-&amp;-Schema-Annotations 2 & 3. What is it you want to run “independently on your premises”? the UI is just a static application there is no server component. I am already paying to host the static files and offing that for anyone to use, but if you want to host it yourself it is also open source https://github.com/JarrodCTaylor/schema-cartographer-ui 4. What is not flexible enough and what is your concern with the visualizations? I developed the application specifically to work with large complex database schema. You can start at any namespace and navigate up to entities that reference it or down to entities that are referenced by it and the UI updates displaying the path you are traversing and maintains breadcrumbs of how you got to the current visualizations. This has been very helpful to me in both support of existing databases and creation of new schema. 5. The developer has stated in multiple places that the evaluation version is usable indefinitely for open source projects as long as the watermark is displayed https://github.com/naver/pinpoint/issues/1308#issuecomment-454047375 The library is not included with the repo but sourced from the cdn linked from the official project page https://gojs.net/latest/download.html https://opensource.stackexchange.com/questions/7424/can-gojs-be-included-into-an-open-source-project/7788#7788 I cannot speak for anyone else but I wrote this tool to visualize schema and use it regularly.

👍 1
JohnJ17:07:12

Is it possible in cartographer to have an entity with different namespaces?

Athan17:07:15

Hi @jarrodctaylor thanks for your reply, it makes clear a lot of things. 1. I was confused with the format of the files downloaded from schema-cartographer and I thought Datomic cannot parse them directly. Then I found a complete example of a schema-cartographer you wrote https://github.com/JarrodCTaylor/schema-cartographer/blob/master/resources/complete_example_schema.clj and realized that in fact you enriched Datomic schema with your annotations. Eventually...

(comment
  (d/transact conn {:tx-data annotation-schema-tx})
  (d/transact conn {:tx-data ice-cream-shop-schema}))
2 & 3. OK thanks, that is great, I did not know that you have also made available the UI and as you realize there are worries of making a large production database schema of a company visible at some remote address/tool. 4. Well, I am sure you understand that schema-cartographer is definitely not the state of the art in schema - domain model visualization, but who cares ☺️. What I find extremely useful is a basic feature to visualize part of a big schema by selecting related entity types you want or at least starting at a node (entity type) and getting all links to that node (related entity types). I am not sure if that is what you write in your (4) reply. Next comes the ability to grab groups of nodes, shaping the line any way you like, etc... 5. OK thanks that explains perfectly well why we see that message appearing on your application It's also great that you share this tool with all existing features with the rest of us.

Jarrod Taylor (Clojure team)18:07:51

4. This is a project developed by myself over many evenings and weekends and I have received no compensation for it. I have provided it for free and hosted portions of it at my own expense for the convenience of others. It has been useful to me and makes no claims of being state of the art. With that said for the price it is a hell of a deal 😏 and does exactly what you are describing. When you navigate to an entity grouped in the UI by a namespace the visual will be of that entity, its attrs and all next level referenced entities. The NS Details tab displays links to all other entities that reference the one selected and you can navigate up through them as needed. You can also drill down through referenced attrs as well. Both directions preserve previously visited entities and keep breadcrumbs showing the navigation path.

Jarrod Taylor (Clojure team)18:07:25

@U01KZDMJ411 the current implementation groups “entities” by namespace. This is of course not a full Datomic definition of entity where an entity can be comprised of any number of attrs having any namespace. That is an enhancement that I would like to implement someday but that day hasn’t yet arrived.

👍 1
Athan11:07:47

For anyone reading this thread the best available tool for processing native datomic schemata at the time of reading is schema-cartographer and it is free including the UI web app tool (thanks again @jarrodctaylor). I have played both with the UI and the serialization of schema into edn files. It takes some time to become adapted to the schema language/format and absorb the differences from the Entity-Relationship diagram and modeling but it definitely worth it. Newcomers into Datomic DBMS should pay attention that schema-cartographer utilizes Datomic Client-peer server connection, i.e. it requires atomic.client.api but it is not hard to read the code and change it to work with in-memory database and other kinds of connectivity.

Drew Verlee17:07:44

Is there a way to list all datomic https://docs.datomic.com/on-prem/reference/database-functions.html? [edit] this likely isn't related to my problem.

Drew Verlee17:07:30

Given a lazy-seq abstraction over a HTTP api as described https://www.juxt.pro/blog/new-clojure-iteration how would you then create batches of rows to transact into datomic? I'm imagining something like: (->> {:marker 0 :page-size 10} fetch lazy-concat (partition 10) db-transact!) Is partition going to correctly "fetch" in lazy chunks of 10 to match the page-size? Or might partition try to, behind the scenes, grab more then 10 and cause extra api calls/fetches?

Alex Miller (Clojure team)17:07:00

partition. and lazy sequence functions in general, give no guarantees about how lazy they are

👍 1
Alex Miller (Clojure team)18:07:06

even if you know the behavior of a particular function, it is hard to predict what the actual behavior might be from a mix of chunked and non-chunked lazy seq ops. so if you care, use iteration, or loop/recur, or something else

👀 1
👍 1