This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2016-12-09
Channels
- # adventofcode (187)
- # aws (1)
- # aws-lambda (1)
- # beginners (162)
- # boot (64)
- # cljs-dev (6)
- # cljsjs (2)
- # cljsrn (32)
- # clojure (357)
- # clojure-greece (1)
- # clojure-korea (4)
- # clojure-russia (63)
- # clojure-sanfrancisco (3)
- # clojure-spec (91)
- # clojure-uk (63)
- # clojurescript (74)
- # clojurex (10)
- # code-reviews (55)
- # core-async (4)
- # core-typed (1)
- # cursive (17)
- # datascript (36)
- # datomic (43)
- # devcards (4)
- # dirac (3)
- # emacs (59)
- # hoplon (286)
- # jobs-discuss (399)
- # luminus (4)
- # mount (9)
- # off-topic (30)
- # onyx (53)
- # protorepl (3)
- # re-frame (88)
- # reagent (4)
- # spacemacs (1)
- # specter (14)
- # untangled (1)
- # vim (42)
With datascript, how to handle data sets that are larger? For instance there are ~5 million records in a particular table which I'd like to search, far too many to load into memory at once. How are others dealing with larger data sets and general application structure?
That's the question, and of course, I could never load it into the browser... how are others dealing with larger sets of data that you can't fit in browser memory.
If you need to search really large dataset, you have to do that on a server, I guess. In a real database
I love the idea of using datascript, just trying to make it work across larger sets is the only problem.
Here's a thought, run datascript on top of nodejs server side, then stream results back to browser. Just a thought. As you say datomic is probably the best answer in this case.
So it can grow really large but small queries will still be able to run relatively quick and in constrained memory
@tonsky is it coincidence or an intent that {:db/cardinality :db.cardinality/many}
values are returned as []
, not as #{}
?
second question is (I still need more input by redoing the same in datomic) is this - am I correct that in datomic db.fn/addEntity and db.fn/retractEntity are responsible for all the entity upsertion logic?
also I wonder if tx-report can contain things like [1 2 3 4 :db/retract] [1 2 3 4 :db/assert]
or it autocollapses such things
Semantically :db.cardinality/many attribute's value is a set.
Vectors imply order.
Every time I look at the pulled entity somewhere in the state, I need to double check whether looking-like-sorted :db.cardinality/many attrs are actually sorted or are just a coincidence.
Also it feels weird to (into #{})
:db.cardinality/many attr every time I need it as an actual set.
Of course, in js we need all the performance we can get, so returning #{}
instead of []
by sacrificing performance just for purity's sake would be silly.