This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2019-09-25
Channels
- # announcements (2)
- # babashka (22)
- # beginners (31)
- # calva (4)
- # cider (26)
- # clj-kondo (10)
- # clojure (32)
- # clojure-europe (1)
- # clojure-italy (3)
- # clojure-nl (3)
- # clojure-spec (16)
- # clojure-switzerland (5)
- # clojure-uk (25)
- # clojurescript (108)
- # clojutre (15)
- # code-reviews (3)
- # data-science (1)
- # datomic (92)
- # emacs (1)
- # fulcro (8)
- # graalvm (8)
- # jackdaw (8)
- # jobs (1)
- # jobs-discuss (1)
- # leiningen (6)
- # off-topic (56)
- # pathom (6)
- # pedestal (5)
- # re-frame (11)
- # remote-jobs (1)
- # shadow-cljs (4)
- # spacemacs (2)
- # sql (41)
- # tools-deps (7)
- # xtdb (25)
best practices page states that you should annotate transactions with the what, who, when information… but it seems to me that that makes it less convenient to pull because the tx is a separate entity. Are there some good tricks for that or simply do 2 queries?
vs. adding modification timestamp and modifying user info as direct attributes of the entity
Imagine your tx is a git commit. Would the data be part of the commit message or part of the commit body itself?
for example I have comments, where the when and who are metadata imo… but it would be convenient to pull it at the same time as the comment itself
comment author and time sound like data not metadata to me. What if you have to backdate a comment? or import them?
better to err on the side of data, it seems… it is more flexible precisely if we need to import
that doesn’t mean you still can’t record e.g. the system and user that transacted the comment
So maybe a distinction could be made that is “domain metadata” versus “operational metadata”. Tx entity seems to be for the latter.
would you model the domain metadata as a single attribute type that all entities have, or separate (like :comment/author
and :file/author
etc)
it seems a single one would allow answering questions like “give me all entities authored by this user”
but: make sure you never would need both of them in the same map (that’s a clue there really is some semantic difference between :comment/author and :file/author) and the range (possible legal values) is the same in all contexts
other downsides: it’s less index friendly, and it’s less intrinsically obvious what attributes are expected together on an entity “type” (spec can help here)
in rdbms you often have the same created_by, created_at etc fields in all main tables… as a newcomer to datomic I don’t have an intuition about what is best here
correct, but 1) those are often really tx metadata in disguise 2) they’re in different tables (usually), so they are different fields. they’re equivalent to :TABLENAME/FIELDNAME
if you would use “joined table polymorphism” (I think it’s called?) in sql, that would be like using one attribute
i.e. you have one table with all “common” fields, and other tables use those fields by joining (in either direction)
Given the (abbreviated) schema:
{:db/ident :customer/id
:db/unique :db.unique/identity}
and the entity
{:db/id 1
:customer/id "XYZ"}
What terms are given to the following:
- 1
- "XYZ"
- [:customer/id "XYZ"]
- 1 || [:customer/id "XYZ"]
(i.e. things you can pass to pull
)
The vocabulary in the wild seems to be somewhat inconsistent. I’ve seen words like:
- eid
- lookup
- ident
- entity
Maybe there are specs available?I call them:
1
: an entity id or eid
"XYZ"
: a customer id
[:customer/id "XYZ"]
: a lookup ref
“entity-identifier” = entity-id (eid) OR ident (keyword value of :db/ident attr) OR lookup-ref
Thanks @U09R86PA4, this link I found appears to agree: https://docs.datomic.com/cloud/schema/schema-reference.html#orgee3fac1
Which would suggest:
1
: eid
"XYZ"
: customer id
[:customer/id "XYZ"]
: lookup-ref
:customer/id
: ident
1 || [:customer/id "XYZ"]
: identifier (entity identifier)
However the datomic API itself seems inconsistent. pull
for example uses the arg name eid
, even tho it really can take an identifier
…
Does there happen to be a public spec available for these? Seems like that would be very useful for libraries/frameworks.
This is perhaps the closest: https://github.com/edn-query-language/eql
no afaik. I made one but it’s proprietary. (it’s also harder than it looks to get very narrowly defined types!)
Hm yeah…library authors lament 😅 Public, standalone specs would be amazing for the community imo. Without them I’m afraid the semantics will drift wildly from lib-to-lib.
I don’t have a datomic connection handy at the moment, but is [:db/id 1]
a valid lookup-ref?
I want to start using Datomic, at first in parallel with my existing SQL database. Is it possible (and safe) to put Datomic on top of your existing SQL database to take advantage of existing backup infrastructure, etc?
After all, doesn't Datomic just use a single backing table in SQL? I would assume that it wouldn't break anything in other tables it didn't care about.
I’m getting lots of errors of “Insufficient memory to complete operation” when I run a case-insensitive string matching query against 110K entities on a Datomic Solo instance. I’m running the code on my laptop, running through the datomic proxy. Am I doing something obviously wrong in the query? Or do I need to upgrade the compute instance type? (Ugh. Hoping that’s note the case!!!) THANK YOU in advance! Query is as follows: (annotated with Ghostwheel types, which I freaking love):
(>defn get-id-by-screen-name
"case insensitive search "
[nm] [string? => any?]
;
(let [retval (d/q '[:find ?id
:in $ ?lowercasename
:where
[?e :user/id ?id]
[?e :user/screen-name ?screenname]
[(.toLowerCase ^String ?screenname) ?lowercasename]
[(= ?lowercasename ?screenname)]]
(d/db (get-conn))
(.toLowerCase nm))]
(ffirst retval)))
THX for reply!!! ohh… I will try reordering the clauses — as soon as the Datomic instance accepts queries again! 🙂
Is there something I should do differently with ?lowercasename
?
@genekim That is creating a TON of garbage in the JVM because you're creating 110K new strings. try using https://docs.oracle.com/javase/7/docs/api/java/lang/String.html#equalsIgnoreCase(java.lang.String)
your query as-is is actually looking for cases where the input ?lowercasename, ?screenname, and to-lower ?screenname are all equal
Also you could use https://docs.oracle.com/javase/7/docs/api/java/lang/String.html#compareToIgnoreCase(java.lang.String)
[(.toLowerCase ?screenname) ?lowercasename]
should work without the = clause (but I feel like I’ve gotten burned by “clever” unifications like this before)
but also take @U0CJ19XAM’s advice about using better string methods here
Ah! AWESOME! I’ll give that a try… I think I need to wait 15m while my Datomic instance recovers. @U0CJ19XAM In the meantime, I will study that query, @U09R86PA4 — it does return correct answers, but most of the time, it runs out of memory. I’ll let you know how it goes!!! THX AGAIN!
@genekim You may need to bounce the box. I've run out of mem on those little guys before too and it's annoying 🙂
Wow, that’s amazing! It worked!!! Thanks @U0CJ19XAM and @U09R86PA4!!!!!! You made my morning!!!
Yeah, if you can afford the space, I completely agree with @U050ECB92. Then the datomic =
can ( I believe ) leverage index navigation which is very fast!
@U050ECB92 Oh, that is super smart idea. I can definitely store :screenname-lowercase! Thx!!
Say, any chance of resolving this dependency conflict in the datomic peer library?
[com.datomic/datomic-pro "0.9.5951"] -> [com.google.guava/guava "18.0"]
overrides
[com.datomic/datomic-pro "0.9.5951"] -> [org.apache.activemq/artemis-core-client "1.5.6" :exclusions [org.jgroups/jgroups commons-logging]] -> [org.apache.activemq/artemis-commons "1.5.6"] -> [com.google.guava/guava "19.0"]
@genekim side note: pass the db as an argument -- don't reach out from the inside of the function
Ah… I can see how that might make testing easier, and allow for better determinism… What are other benefits? And what do you typically name the function that gets the db and wraps the actual query? (Looking for some examples and conventions.) (I’ll go look in the musicbrainz example, too…) Thx!
Taking the db as an argument allows you to correctly ask higher-level or larger questions, and have all parts of the process use the same basis for query / calculation
So happy to have solved the datomic memory issue (thank you, all!). I just noticed that the Datoms Cloudwatch graph is totally blank — is there an easy remedy for this? (Like rebooting something? 🙂
@genekim I believe Datoms is only reported when an indexing job occurs; are you actively transacting against the system?