Fork me on GitHub
#datomic
<
2019-03-04
>
Nolan08:03:32

is there a way to directly pull the :db/ident of a :db.type/ref? e.g. (pull ?e [:the/ref]) results in {:the/ref #:db{:id 123 :ident :ident-value}}, and i’m looking for a way to just get {:the/ref :ident-value}

favila14:03:13

There is no way without postprocessing (see clojure.walk/walk) but please upvote this feature request if it strikes your fancy: https://receptive.io/app/#/case/49752

👍 5
Nolan08:03:25

easy to do with scissors afterwards, just wondering if theres syntax in pull to remove the extra step. separately: datomic is nuts

Piotr13:03:13

What is the idiomatic way to query entities for dates? I want to get all the posts with created-at inst that fall in range of dates. Would that be database asof and since filters or something else?

favila14:03:48

That article is important for getting clear what datomic's time features can and cannot do

favila14:03:07

I would say this: if you want "created-at" to be a business domain concept, make an indexed create-at attribute and query it

favila14:03:41

signs that it is a business domain concept: 1) you want to be able to change it ("oops, I got the created-at time on this thing wrong, it's supposed to be last week")

favila14:03:20

2) It's value comes from another system ("I need to copy this record's created-at time from the inventory to the accounting db")

favila14:03:37

If created-at is really more like a git "commit" time, then you can use some immutable attribute on the entity, and use its TX as the "created at" time

favila14:03:00

note that it will be a bit trickier to get that value, because TX entities are not reachable via pull expressions

benoit13:03:14

@piotr.kurnik One option is to index the created-at attribute and use a query.

benoit13:03:19

If you're interested in transaction time, you have the log API Datomic on-prem.

oz16:03:57

We are seeing a intermittent timeouts from our client trying to connect to datomic cloud.

2019-03-04 08:56:49 ERROR middleware:286 - Exception encountered processing request
clojure.lang.ExceptionInfo: Datomic Client Timeout {:cognitect.anomalies/category :cognitect.anomalies/interrupted, :cognitect.anomalies/message "Datomic Client Timeout"}

oz16:03:23

Anyone else seen this before and can advise on a course of action?

oz19:03:18

If you run into this error is appears you'll need to restart the datomic cloud compute instances.

johanatan19:03:35

hi, is it possible to connect to the localhost transactor running on port 4334 for the ddb database type without going to dynamo to lookup the IP? (i want to use ssh port forwarding from remote 4334 to local 4334 and then connect using the ddb protocol to the local 4334 port). is that possible?

johanatan19:03:38

i tried using datomic:dev://... but got Connection refused (likely due to either ddb v dev protocol mismatch or missing authentication or ??? )

johanatan19:03:16

oh, oops. i see the mention of ddb-local in the transactor.properties file

johanatan19:03:18

i will try that

johanatan19:03:03

oh, nope. i bet that refers to an actual dynamo running locally

marshall19:03:05

@johanatan no; datomic writes the address of the transactor into storage

marshall19:03:14

and the peer looks it up before connecting to the transactor

johanatan19:03:33

so even if i know the transactor is running on localhost, there's no way to connect to it directly?

marshall19:03:54

the database URI is a storage address

marshall19:03:13

the peer will connect to storage (whatever it is, in this case ddb), to look up the transactor endpoint

johanatan19:03:20

yes i know all of that. i know how it "normally" works. but what i want to do here is abnormal.

johanatan19:03:31

my dev is not in any VPC

johanatan19:03:40

so cannot connect to dynamo to get the ip

johanatan19:03:46

and even if it could, that ip would not be reachable

johanatan19:03:56

would rather not have to set up a VPN for this use case

marshall19:03:10

the address of your transactor is configured via the transactor properties file

marshall19:03:19

the peer must be able to read storage

marshall19:03:23

it can’t work otherwise

johanatan19:03:29

yes, that is all true

johanatan19:03:42

my transactor and peer are running in ec2

johanatan19:03:47

my local laptop is on my desk

johanatan19:03:52

there is no vpn between them

marshall19:03:10

if you’re trying to connect to your cloud database from your laptop, you’re running a peer (or client)

johanatan19:03:21

ah, sure. it's in the repl i suppose

marshall19:03:27

if you’re using peer it must be able to read storage

johanatan19:03:34

technically i'm using datomic.api/connect

johanatan19:03:59

with a datomic:... db-uri

marshall19:03:00

which is the peer library

johanatan20:03:35

so what's the easiest way to accomplish the end goal i have here?

johanatan20:03:46

is setting up a vpc/vpn my only option?

marshall20:03:16

is your transactor running in a private vpc?

johanatan20:03:27

it's running on a lightsail instance

marshall20:03:28

or something otherwise inaccessable remotely?

johanatan20:03:32

i doubt there is any vpc

johanatan20:03:35

but i can ssh into it

johanatan20:03:38

and port forward ports

marshall20:03:19

if you can ssh to it, it must have a publicly accessible IP address (presumably the same address your cloud peer uses to connect)

marshall20:03:26

which is already written to storage

marshall20:03:39

so all you need to do is export AWS credentials locally that allow you to access dynamo

johanatan20:03:16

hmm, it would seem that it wrote a "private"/internal IP to storage. i would also rather not open my database port to the world (seems like asking for trouble)

marshall20:03:17

you may need to use HOST & ALT-HOST to accomplish this if you cant resolve the address it is using from your laptop; alternatively you might be able to do something like a bastion server that handles port forwarding and remote dns resolution, but you’d have to have an instance in your VPC (or whatever) to do that with

johanatan20:03:50

mm, ok. alright. thanks so much for your input. some things for me to ponder.

okocim20:03:28

is it possible to run a query that returns multiple different counts in a single query, where each count is for the specific attribute value, and not for the set of attribute values together, sort of like frequencies does in clojure.core?

okocim21:03:20

so I ended up doing this:

(d/q '[:find ?attr ?value (count ?i)
       :where
       [?i :inventory/store ?s]
       [?s :store/short-code "demo-store"]
       (or
        (and
         [(ground :model) ?attr] 
         [?i :item/model ?value])

        (and
         [(ground :size) ?attr]
         [?i :item/size ?value]))]
     (d/db (db/get-conn)))
Not sure if i’d be better off using rules in such a situation, or something else, so I’d appreciate any feedback if anyone has done the same sort of thing before. Ultimately, this code is going to move to a search engine, but I need it for the time being.

favila21:03:55

That is how you would do it. Rules would encapsulate the logic better and make the intent clearer but it's no different from this ultimately. (or, or-join, and, and-join are ultimately just sugar for inlining anonymous rules)

okocim02:03:28

Thanks for replying/validating my logic. I ended up switching to rules since it’s a bit clearer, as you mentioned. Sometimes it takes a bit of thinking to get to what you want, but I’m finding datalog queries to be incredibly expressive.

shaun-mahood21:03:02

Are there any recommendations for batch size when doing a one-off data import to Datomic Cloud? I'm looking at about 100K entities that are logically part of the same group.

johanatan23:03:33

@marshall i think the following will do the trick (with my ssh tunnel in place) without having to open my db port to the world: http://www.vincecutting.co.uk/web-development/redirect-ip-address-back-to-localhost-on-mac-osx/

johanatan23:03:33

i tried it with the AWS internal IP first and it worked like a charm.

johanatan23:03:39

interesting that reads don't require the transactor

johanatan23:03:47

only when attempting a write was this necessary