Fork me on GitHub
#datascript
<
2019-11-11
>
Filipe Silva20:11:49

heya, sorry if this is a silly question (I am not very familiar with datascrip), but are there good examples of how to use datascript and firebase together? I can't find any promising google results for it so I imagine it's likely an unadvisable thing to do

jbrown21:11:38

Hey I work with @conaw, we use datascript and firebase together in production at http://roamresearch.com. It certainly has its limitations, but it works well for a small/medium size datascript database. We store the datascript database serialized in firebase, load that into memory and listen/push transactions to a log in firebase so it will sync between connected clients. Can post a modified version of our code later tonight.

Filipe Silva21:11:03

Oh wow that's a lot more than I was expecting when I asked!

Filipe Silva21:11:24

If you can that'd be great! Thank you!

Filipe Silva21:11:00

Do you use real-time database or forestore btw?

jbrown22:11:47

real-time database. The values are limited to 10mb though, so we are going to switch to using storage for the database snapshot and realtime database for the tx log. It's been really nice to prototype our app with this setup, but we are running into problems with the databases getting too big (taking too long to download and read string) so we'll have to switch to datomic/datahike/crux soon. Also if you are planning on having multiple clients editing the same datascript database you'll have to transact a uid with each entity because transactions aren't guaranteed to come in order and db/ids can get out of sync between the clients.

Filipe Silva09:11:35

awesome, thank you

Filipe Silva09:11:47

I see how the 10mb limit is a problem though...

Filipe Silva09:11:37

firestore has a 1mb document limit even so it wouldn't be better for that setup

Filipe Silva09:11:02

did you consider/try doing something more granular than a full db serialization?

Filipe Silva09:11:21

like a conversion layer into firebase documents

jbrown19:11:10

We've considered doing something like making each entity a key in firebase and caching the entities on the client so we only have to load the entities that have changed but we would still have to deserialize the entities from local storage and transact them into a db, which I haven't tested but I think takes longer than deserializing a whole db and calling conn-from-db. Maybe this approach would be faster on slow connections, but I'm not sure. In other words I haven't figured out an architecture that would 100% work better than storing the whole db. And if the goal is to just have a db larger than 10mb, we can use firebase storage api to store the db, since we don't actually need the functionality of realtime database for the datascript db store.

Filipe Silva19:11:35

thanks for being so responsive and providing such a great sample

jbrown19:11:56

No problem, I've been wanting to write this up for awhile now 🙂

Filipe Silva19:11:02

I'll see if I can get my head around it and maybe use something similar

Filipe Silva19:11:46

btw I find roamresearch quite interesting! I've had something in the same problem space in my head for a while

Filipe Silva19:11:05

so it's really cool to see how roamresearch and other tools are trying to improve note taking

Filipe Silva19:11:23

worth mentioning that the Pixel phones (maybe others, not sure) have two cool things that help with the data input: you can talk to the recorder app and it'll transcribe, and you can take pictures of notes and it'll try to OCR the text, handwriting included

Filipe Silva19:11:30

both of these work offline

jbrown19:11:04

Glad to see other people interested in this space as well! thats pretty cool, we've thought of integrating with http://otter.ai (really great transcribing app), didn't know phones come with things like that (I have an old galaxy phone haha)

jbrown19:11:40

I'm curious what your looking to use datascript for

Filipe Silva20:11:01

similar thing as you guys have, but instead of focusing on the research field I was focusing on software engineers that wanted to do deep work

Filipe Silva20:11:14

was looking at supplementing deep work with graph-based note taking with full-text search

Filipe Silva20:11:48

a big driver was to have a persistent context between work sessions

Filipe Silva20:11:11

and I wanted something firebase/datascript based because of the implicit transaction history

Filipe Silva20:11:25

I think allowing users to replay how the last work session actually happened in the system would help get people back in the right context and mindset

jbrown05:11:34

It's awesome to meet someone with such a cross section of interests as ours 🙂. We originally wanted to use the transaction history to replay actions on a database (both to replicate actions that led to a bug and to show the user), but it hasn't really worked in practice with our setup because the db/ids can get out of sync between clients (we mostly have this solved now by using a unique identifier for every entity instead of the db/id, but now the logs are all messed up and can't be replayed)

❤️ 4
Filipe Silva10:11:58

yeah log changes often break replays in a lot of systems 😞

Filipe Silva20:11:24

I see https://github.com/mrmcc3/datascript-firebase-sync but as the commit says it mostly looks like a code dump, so I'm not sure how well done it is

metasoarous20:11:18

@filipematossilva ^ I believe@conaw did something like this.

Filipe Silva21:11:52

Jbrown just followed up on that, super helpful!