This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2016-01-02
Channels
- # admin-announcements (1)
- # alda (1)
- # aws (4)
- # boot (276)
- # cljs-dev (3)
- # cljsjs (1)
- # cljsrn (3)
- # clojars (22)
- # clojure (174)
- # clojure-austin (1)
- # clojure-italy (2)
- # clojure-russia (20)
- # clojurecup (1)
- # clojurescript (233)
- # cursive (4)
- # datavis (97)
- # datomic (122)
- # hoplon (80)
- # ldnclj (8)
- # leiningen (6)
- # om (82)
- # reagent (10)
- # spacemacs (8)
- # specter (5)
Hello, I’m having an issue trying to add a transaction function; when it tries to add, I get “Can’t embed object in code, maybe print-dup not defined: clojure.lang.Delay"
fn looks like
{:db/ident :add-user
:db/id #db/id [:db.part/user]
:db/fn #db/fn {:lang "clojure"
:params [db params]
:code (pr-str
'(if-let [e (datomic.api/entity db [:user/email (:user/email params)])]
(throw (Exception. "User already exists with email"))
[params]))}}
alternatively to fixing that, is there a way to tell datomic that I don’t want to upsert? I’m using this function because I want to ensure that users have unique email addresses, but if I use a tempid when I insert the new user, duplicates result in updating the user with the existing email, instead of throwing an error
oh, never mind the fn question; apparently d/function works instead of the reader macro
I would still be interested to know if there’s a better way to do this and let datomic’s unique checking do the job for me
@jamesnvc: use unique/value instead of identity. Unique/value throws, unique/identity upserts http://docs.datomic.com/identity.html#unique-values
@bkamphaus: oh, thanks! I think I was misunderstanding what unique/value meant
Anybody know of a converter from JSON schema to Datomic schema? Grasping at straws here, Google turned up nothing.
@dave.dixon: haven't seen one, but I'd be interested in a JSON-to-Datomic data converter if you've seen one. 😉
Is there some reason data.json doesn't work? https://github.com/clojure/data.json
That's what I was planning to use to read the JSON schema into Clojure maps etc.
i dabbled in putting arbitrary graphs into datomic with https://github.com/tailrecursion/monocopy, concept was flawed though
I'm just in the process of rethinking my planned architecture... I wanted datomic on AWS but I think it'll just be too expensive.
Instead now I'm considering dumping the raw data into DynamoDB using Lambda, then extracting it into Datomic for the real work.
if you want to query maps in datomic datalog (not store them) you can circumvent the structural identity problem with an approach like https://twitter.com/alandipert/status/682597011141558273
(run-time is mostly data collection with some lookups; "real" work happens periodically -- as in monthly/annually and is offline)
@curtosis: what kind of data? would you characterize your work as "analytic"?
The data is essentially scores submitted by judges. The core entity is an event: {:ballots [{:judge judgeid :category score}]}
(simplifying greatly .. it's actually 4-5 layers deep)
it's less "analytic" than just "tabulation". There are a bunch of rules for how scores get averaged/dropped/qualified etc.
(thus the workaround... I'd prefer to stay in Datomic from the outset. Audit trail is, as you might expect, rather important here.)
interesting
do already have a SQL db where you store the collated results?
hmmm... I could also flip it around and use DynamoDB directly for the live lookup stuff (name regularization etc) and put the scores on SQS, fire up Datomic once a day or so to consume the queue and update the names list....
the grand impetus for all of this is that a) the inputs look like documents more than tables, b) rails stack consistency over time is a tire fire, and c) I really like Datomic.
i'm in the ad business and we store "events" like clicks and impressions in in S3, then EMR them periodically and put the aggregates in a combination dynamo and redshift
seems like in your case, if your real-time query requirements are light, the most economical thing would be to aggregate somewhere cheap and "wake up" periodically to process
newline-delimited json maps
yeah, that's what I'm thinking. And the free tier of Dynamo is probably plenty sufficient for that "cheap" aggregation.
mostly flat but decorated with geo and other info, maybe 3-4 things deep in spots
S3 would be simpler (no Dynamo schema to care about) but there's no long-term free tier
the only gotcha is the name-lookup service backing... easy to do in Dynamo, harder to do in S3.
but regardless, I think this discussion is super helpful... I think I've reduced the problem now to a simple recurring load process, with a feedback to the name lookup stuff.
hmm.. for that matter, just dumping the records onto SQS for the instance to pick up when it wakes up might work too.
@curtosis: dynamo
well, there are a few "stages"
the first stage is gathering up files from S3 into batches and EMRing them... that's where we use dynamo, to keep track of the files/batches
when EMR is done it puts a batch id on an SQS queue... where a thing that specializes in loading aggregated data into Redshift picks it up
that's one way to do it... another is to attach a lambda function to S3 events
but we do neither, we have a convention for storing in S3 and make a date segment out of the key
eg 2015/10/2/3/0 for "the 0-30 minutes of 3am on 10-20-2015"
so when the EMR wakes up it figures out what the previous segment path was, and scoops the files up there
fair enough. I kind of like the idea of a lambda on S3 events putting something on SQS for my Datomic "core" to pick up.
but that may be more complicated than necessary... I can just get my last-pulled from Datomic and go from there.
and, since I control the front-end, I can just dump EDN into S3 for Datomic to load.
and I think if it's just EDN I can use #db/id[db.part/user]
to defer tempid generation until it gets picked up
sounds pretty sweet
the only hard part (for some values of hard) is the "I don't know this name; create a new one".
oh you mean like make a new name entity?
yeah... I need to create it with a tempid on the client, process it into Datomic, and then update the front-end lookup service.
hmm... I guess I need to decide whether to build my own all-in-one transactor + processor AMI, or use the default Datomic AWS deploy template plus my application-code AMI.
apart from the obvious cost difference, are there any advantages to running one way or the other?
So I am trying to programmatically generate a query, or at least some of one, but I am coming up short. I am wondering if anyone can help me.
I want to make a function to which the db-ident to join on is being passed, so the called can chose the entity that is being joined. I have something like this:
(defn get-all-entities-with-tag [tag-title attr]
(let [db (d/db @conn)
eids (map first (d/q '[:find ?e ?log-title
:in $ ?tag-title
:where
[?tag-e :tag/title ?tag-title]
[?e ~attr ?tag-e]
[?e :log/title ?log-title]]
db
tag-title))]
eids))
This results in an exception though: IllegalArgumentExceptionInfo :db.error/not-an-entity Unable to resolve entity: clojure.core/unquote
So the unquote isn’t doing the trick and it doesn’t work without it either. Any idea what will do the trick?
the problem is you're using ~
inside a regular quote, not a syntax quote '
vs \
`
err `
Hmm ok, so should I use a syntax quote? I am trying to figure out the right way to generate a query like this programmatically
unfortunately clojure's native syntax quote isn't a great fit either because it will try to resolve in namespaces things like ?log-title
i recommend checking out the template
macro in https://github.com/brandonbloom/backtick
user=> (let [a 1 b `(~a 2 ?log-title)] b)
(1 2 user/?log-title)
your other option is to leverage the fact that the query is a vector... so you can use update
on it
but that would be kind of brittle, as you'd need to maintain the index of what inside you want to change
user=> (assoc [1 2 3] 1 "hi")
[1 "hi" 3]
true, that's maybe the best
iirc q
doesn't care if you use vectors
personally i prefer the template
route, i find it the data analog to string interpolation
Yeah it seems better. Just wierd that there is no good way to do it built into Clojure or Datomic
especially since datalog is supposed to be easier to generate programmatically than string based queries like SQL
yeah, i don't really buy that 😉
i dunno tho, sql is filled with weird syntax that needs to be satisfied
fortunately datalog requirements are few in comparison
I did, until I tried it just now and getting fucked by having to quote the vector to avoid symbols being resolved and subsequently can’t really do stuff with it 😬
hehe, sure
you can also always using strings again if you want
(let [x 1] (read-string (str "[" x " 2 3]")))
🍷
Yeah, that is sort of what they suggest in the docs (http://docs.datomic.com/data-structure-literals.html), but I just feel like there should be a better way in Clojure
oh wait, can you not pass in ~attr as a parameter? like the way you pass in tag-title
(d/q '[:find ?e ?log-title
:in $ ?attr ?tag-title
:where
[?tag-e :tag/title ?tag-title]
[?e ?attr ?tag-e]
[?e :log/title ?log-title]]
db
attr
tag-title)
phew 😅
yup, that's datalog
So maybe I am getting hung up on pointless little things, but I also want it to look for any ident via _ (underscore) when the attribute ident isn’t passed to the function
(defn get-all-entities-with-tag [tag-title & [attr]]
(let [db (d/db @conn)
join-attr (or attr '_)
eids (map first (d/q '[:find ?e ?log-title
:in $ ?tag-title ?attr
:where
[?tag-e :tag/title ?tag-title]
[?e ?attr ?tag-e]
[?e :log/title ?log-title]]
db
tag-title
join-attr))]
eids))
Giving this error: IllegalArgumentExceptionInfo :db.error/not-an-entity Unable to resolve entity: _
if you don't have an attr... you could omit the whole [?e ?attr ?tag-e]
clause, right?
True, it isn’t needed in that case. But I still don’t know how to operate on the query programmatically, so I don’t know how to remove it
yeah - i think template
comes back
btw you may consider keeping your queries in functions that take db as an argument
this gives you more control as the code evolves, since you don't have to coordinate conn access
I guess you would generally make a db at beginning of a route (if exposing a service) and operate on that throughout the call
yeah, i think anywhere you want to do a bunch of queries and get consistent results