This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2017-09-10
Channels
- # beginners (191)
- # boot (3)
- # cljs-dev (2)
- # clojure (46)
- # clojure-austin (1)
- # clojure-spec (4)
- # clojure-uk (32)
- # clojurescript (10)
- # clojurewerkz (3)
- # cursive (14)
- # datomic (88)
- # defnpodcast (1)
- # editors (2)
- # fulcro (2)
- # hoplon (3)
- # jobs (4)
- # jobs-discuss (1)
- # luminus (5)
- # lumo (6)
- # off-topic (13)
- # random (1)
- # re-frame (50)
- # reagent (245)
- # remote-jobs (1)
- # spacemacs (3)
- # specter (16)
- # uncomplicate (4)
is it a good idea to use datomic as a graph database, or is it better to use neo4j? i know datomic doesn't have the same level of graph capabilities as neo4j, but it can still do most graph database operations, right?
i like the capabilities of neo4j's cypher, but i would much prefer to use data. the biggest thing is i can't think of how to add properties to links in datomic...
@cjhowe so, regarding the first question: I am new to Datomic/Clojure but yes, Datomic seems like a great fit for exploring graphs
it’s essentially a triple-store (well, 5-tuple store really if you account for the transaction id and operation)
It really depends on what you want to do; neo4j’s cypher is very high-level, you might have to do a lot more work to run the same queries on top of Datomic
and if you have more complex query needs you can build them on top of Datomic’s primitives
As fa as I aware, a standard approach would be to reify the link as a entity, which can then have attributes
e.g. imagine you had a link between friends, e.g. :person/friends
, an attribute with cardinality many and value-type ref
now say you would like to store an attribute on that link, e.g. an integer “weight” which indicates how good a friend the person is
With Datomic you can’t do that, but you could “reify” that link as an entity, and have an attribute like :person/friendships
which would point to Friendship
entities, which would themselves have a :friendship/target
pointing to the friend Person
entity
since Frienship
is now an entity, you can add :friendship/weight
, or any attribute you want
i think the power of using plain data structures for queries makes up for some additional query complexity
@cjhowe yeah, I think it really depends on the type of queries you are doing to do. If you have very complex graph queries then something like Neo4j might do a lot of the optmization grunt-work for you
@cjhowe yes the peer keeps a cache. I’m not sure how it determines when to clear a portion of the cache and what to clear though (if it’s full)
it seems like running the graph query mostly on the client would help make up for neo4j's extra optimizations
It depends on your dataset and the type of queries but for most use-cases I suspect Datomic will work just fine
@cjhowe to be honest, it might just be a lack of understanding/experience on my side, but I kind of wish there were a few higher-level abstractions built on top of Datomic
For example, if your use-case is manipulating graphs, I wish there was library built on top of Datomic which offered Cypher-like querying capabilities, amongst other things
@cjhowe yes I’m sure you could write a function for that which uses Datomic’s low-level API (direct access to the index) to walk the graph
idk, i'm making a study group app, so however many people use it i guess. it's kind of like tinder, and each one of those tinder-like matches has to find the path with the shortest total weight that starts at and ends with two different users who haven't matched before
and then in addition to that, it's trying to choose matches that will cause complete subgraphs of 3+ nodes to appear
@cjhowe I see, so I guess you won’t have such a large dataset that you would need to start writing fancy, optimized versions of your pathfinding algorithm
My (beginner, ignorant) approach would be: use Datomic, see how far it can get you. If you ever run into a case where Datomic isn’t enough and/or it’s too much work to build the feature you want on top of it, construct a read-only neo4j (or else) replica of your Datomic DB
which should be doable with the direct access to the transaction log and the tx-report-queue API
theoretically you can move to neo4j later, should you want to (ignoring the fact that it would be a pain to rewrite your app)
the right term would probably be “use neo4j as a materialized view of your Datomic database”
thanks for the help! i'll probably do that, since this isn't really a problem right now and i want datomic
There is also another benefit of using Datomic: since it’s lower-level, you’ll learn a lot more about how your graph traversals actually run (since you’ll likely implement them yourself).
ah, yeah, that's great! i just took my last math class for my CS degree so i'm ready to deep dive into graph theory
how do people transact their datomic schemas? is it best to use a boot/leiningen plugin, or should i transact it every time i start my app server with https://github.com/rkneufeld/conformity ?
@cjhowe https://vvvvalvalval.github.io/posts/2016-07-24-datomic-web-app-a-practical-guide.html#schema_/_model_declaration
@val_waeselynck did some nice work writing about this
i guess i just need to know how i should actually run the conformity code at deployment
@cjhowe you may want to read this as well: https://github.com/vvvvalvalval/datofu#managing-data-schema-evolutions
The basic idea is to re-transact your schema (idempotent) and run transactions (non-idempotent) prior to executing new code.
Depending on the write semantics of your app, this protocol may be too naive and present some race conditions (for instance, when migrating the data from an attribute to a new attribute, some Peer may continue to write to the old attribute between the time the migration is run and the time the new code is deployed to the Peer).
In which case you may want to either: 1. if that works, run the migration after new code is deployed to all online Peers, or in 2 phases 2. Temporarily prevent all Peers from writing 3. Move the write semantics from the Peer code to storage (e.g in the form of a transaction function), so that the switch can be atomic
i mean, if i use conformity every time my api server launches, it's a bit of overhead, but it shouldn't do anything if the schema is already there
the JVM’s time to startup is likely an order of magnitude larger than the time to run a transaction for your schema
@cjhowe oh also, regarding your earlier question on Heroku: I think the recommended memory for the transactor (and peers) is pretty high; in the order of 1gb or more
I considered using heroku on a project but realised that would be prohibitively expensive
you might be able to get away with lower memory on small projects / with the right configuration though; I haven’t investigated
haven’t encountered significant issues thusfar, but I have only experimented over short periods of time, not under load and/or in production
oh no, you can run the transactor in dev mode (it will store stuff on the filesystem)
I also managed to run a transactor on http://hyper.sh
ah, well, in my case, i'm just worried about what i can get for very cheap/free since i'm a student
The cheapest you can get for small projects is probably to run your transactor and storage (and maybe also Peers) on one box e.g using Digital Ocean, maybe using the dev storage to save even more memory