This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2018-02-27
Channels
- # aleph (7)
- # beginners (80)
- # boot (1)
- # cider (3)
- # cljs-dev (277)
- # cljsjs (52)
- # cljsrn (1)
- # clojure (69)
- # clojure-gamedev (4)
- # clojure-italy (1)
- # clojure-losangeles (2)
- # clojure-russia (89)
- # clojure-spec (92)
- # clojure-uk (196)
- # clojured (1)
- # clojurescript (70)
- # cursive (5)
- # data-science (1)
- # datascript (84)
- # datomic (9)
- # defnpodcast (12)
- # docker (1)
- # emacs (4)
- # events (1)
- # fulcro (112)
- # graphql (1)
- # jobs (1)
- # lumo (1)
- # nrepl (21)
- # off-topic (2)
- # onyx (3)
- # protorepl (10)
- # re-frame (23)
- # reagent (66)
- # reitit (2)
- # rum (13)
- # shadow-cljs (144)
- # spacemacs (14)
- # sql (4)
- # unrepl (29)
- # vim (16)
datascript doesn't let you query old versions anyway, because it's all in memory
why can't you just transact new maps into entities as they become available?
@thedavidmeister the data is basically a dump of a couple database tables. right now, it is just a snapshot of the tables at a point in time. the difference between the current version and the previous version might include updated entities, and without just having the delta of updated and created entities, we'd need to look at every map and create-or-update, no?
if you have no sense of "identity" for each map
every list of maps is a new list
there is no meaningful concept of "update" 😕
ah cool
well then can you map the IDs to :db/id in datascript?
try it
i mean, i don't know if datascript supports your ids
but if they're numbers i don't see why it wouldn't work
ints yeah
although it's probably technically not supported
again...
whatever your definition of identity is
if your definition of identity is the ID, and the IDs collide, then they are the same entity, by definition
i suppose i could get clever and do something like use an offset for each table to prevent collision
datascript does something like that internally to differentiate between eids and txn ids
but really, this is pretty contextual at this point
there's probably no perfect answer
though i do have 100s of thousands of some of these maps, so it might need to be more clever than that
that should still be fine
yes, basically im looking to try and fit a whole lot of batched data into a tight, queryable package
js max int is something like 10^15
can it be two different dbs?
this is getting to where i'm not 100% sure
i have a feeling it's possible to stick multiple dbs into a query
but i haven't done it
not sure if datascript follows that
right now im just doing a (d/with-db (d/empty-db) (concat stuff more-stuff))
in a background thread, and swapping the value
so you're building a new db from scratch every time
why not just (transact! conn [...])
provided you have the db/id bit working, it should ignore anything that is the same
i have no idea which is faster
but transact! certainly seems more idiomatic
it's probably doing similar things under the hood
what about incremental updates?
is it any faster than the initial load?
it took 42 seconds to add 1 record?
oh, no, what im saying is i had a collection, and i changed a record in it but left it's ID the same, and then ran transact! over the whole collection again
i was curious to see if it'd be able to quickly tell it didn't need to do anything with most of the maps
i wonder how long it takes to compare those maps outside datascript
maybe you can do the comparison yourself and just transact the diff
i also wonder if the types on the value side of a map have anything to do with performance
like, clojure.set/difference
or something
maybe, i'd imagine iso8601 strings are faster to compare, but i don't know
yeah i mean, at some point you just run into the limits of comparing lots of data
From the datascript tutorial in reading: > Datom has (-hash) and (-eqiv) methods overloaded so only [e a v] take part in comparisons. It helps to guarantee that each datom can be added to the database only once. Does this mean that a retracted datom cannot be added after retraction?
> When datom is removed from a DB, there’s no trace of it anywhere. Retracted means gone.
That makes it sounds like excision and retraction are the same, but perhaps I’m just confused by the role of :added in light of the comment about e a v comparison operations.
datascript has no history
i think a lot of the datascript API is to line up with datomic API rather than achieve a specific goal for datascript itself
if you wanted to sync datascript with datomic via transaction reports you'd need :added i think
@devn There is also conn-from-datoms
in case you want to manually import. It's pretty fast. I use it to bootstrap my DB on CLJS.
Datomic maintains history; you could rewind and look at the database at a given time, meaning all datoms retracted between that previous time and now would be visible again. Retraction is basically adding new information: "This datom no longer holds true; it used to up 'til now, but now it does not." Excision is much more dangerous, because it removes any trace that the datom was ever there, both currently and historically. Datascript maintains no history, as thedavidmeister said, so it doesn't necessarily make that distinction