This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2019-11-11
Channels
- # adventofcode (1)
- # aleph (1)
- # announcements (3)
- # babashka (39)
- # beginners (84)
- # calva (1)
- # cider (17)
- # clj-kondo (15)
- # cljs-dev (43)
- # clojure (132)
- # clojure-dev (1)
- # clojure-europe (4)
- # clojure-nl (7)
- # clojure-norway (4)
- # clojure-uk (22)
- # clojurescript (56)
- # clojurex (24)
- # cursive (11)
- # data-science (2)
- # datascript (33)
- # datomic (7)
- # docker (2)
- # figwheel-main (11)
- # fulcro (2)
- # jobs (3)
- # joker (29)
- # leiningen (3)
- # nrepl (4)
- # off-topic (11)
- # planck (4)
- # reitit (5)
- # ring (4)
- # shadow-cljs (205)
- # spacemacs (5)
- # xtdb (9)
heya, sorry if this is a silly question (I am not very familiar with datascrip), but are there good examples of how to use datascript and firebase together? I can't find any promising google results for it so I imagine it's likely an unadvisable thing to do
Hey I work with @conaw, we use datascript and firebase together in production at http://roamresearch.com. It certainly has its limitations, but it works well for a small/medium size datascript database. We store the datascript database serialized in firebase, load that into memory and listen/push transactions to a log in firebase so it will sync between connected clients. Can post a modified version of our code later tonight.
Oh wow that's a lot more than I was expecting when I asked!
If you can that'd be great! Thank you!
Do you use real-time database or forestore btw?
real-time database. The values are limited to 10mb though, so we are going to switch to using storage for the database snapshot and realtime database for the tx log. It's been really nice to prototype our app with this setup, but we are running into problems with the databases getting too big (taking too long to download and read string) so we'll have to switch to datomic/datahike/crux soon. Also if you are planning on having multiple clients editing the same datascript database you'll have to transact a uid with each entity because transactions aren't guaranteed to come in order and db/ids can get out of sync between the clients.
awesome, thank you
I see how the 10mb limit is a problem though...
firestore has a 1mb document limit even so it wouldn't be better for that setup
did you consider/try doing something more granular than a full db serialization?
like a conversion layer into firebase documents
We've considered doing something like making each entity a key in firebase and caching the entities on the client so we only have to load the entities that have changed but we would still have to deserialize the entities from local storage and transact them into a db, which I haven't tested but I think takes longer than deserializing a whole db and calling conn-from-db. Maybe this approach would be faster on slow connections, but I'm not sure. In other words I haven't figured out an architecture that would 100% work better than storing the whole db. And if the goal is to just have a db larger than 10mb, we can use firebase storage api to store the db, since we don't actually need the functionality of realtime database for the datascript db store.
gotcha
thanks for being so responsive and providing such a great sample
I'll see if I can get my head around it and maybe use something similar
btw I find roamresearch quite interesting! I've had something in the same problem space in my head for a while
so it's really cool to see how roamresearch and other tools are trying to improve note taking
worth mentioning that the Pixel phones (maybe others, not sure) have two cool things that help with the data input: you can talk to the recorder app and it'll transcribe, and you can take pictures of notes and it'll try to OCR the text, handwriting included
both of these work offline
Glad to see other people interested in this space as well! thats pretty cool, we've thought of integrating with http://otter.ai (really great transcribing app), didn't know phones come with things like that (I have an old galaxy phone haha)
similar thing as you guys have, but instead of focusing on the research field I was focusing on software engineers that wanted to do deep work
was looking at supplementing deep work with graph-based note taking with full-text search
a big driver was to have a persistent context between work sessions
and I wanted something firebase/datascript based because of the implicit transaction history
I think allowing users to replay how the last work session actually happened in the system would help get people back in the right context and mindset
It's awesome to meet someone with such a cross section of interests as ours 🙂. We originally wanted to use the transaction history to replay actions on a database (both to replicate actions that led to a bug and to show the user), but it hasn't really worked in practice with our setup because the db/ids can get out of sync between clients (we mostly have this solved now by using a unique identifier for every entity instead of the db/id, but now the logs are all messed up and can't be replayed)
yeah log changes often break replays in a lot of systems 😞
I see https://github.com/mrmcc3/datascript-firebase-sync but as the commit says it mostly looks like a code dump, so I'm not sure how well done it is
@filipematossilva ^ I believe@conaw did something like this.
Jbrown just followed up on that, super helpful!