Fork me on GitHub
#datomic
<
2019-03-30
>
Nolan04:03:50

Curious if anyone has any recommendations on managing datomic connections in an aws lambda. Currently I essentially do this:

(def client (delay (d/client ...)))
(def conn (delay (d/connect @client {:db-name ...})))
(def q '[:find ...])
(defn somefn [db]
  (let [data (d/q q db)]
    ...))
(defn -handleRequest [_ is os _]
  (somefn (d/db @conn))
  ...)
It works most of the time, but occasionally a lambda will spin up and only ever encounter an anomaly on every invocation: Unable to execute HTTP request: Connect to <storage bucket>:443 failed: connect timed out. It’s as if the connection was never made from the get-go, and until that lambda dies, it will only fail. Do i need to be handling any sort of expiration or refreshing of either the client or the connection? Are there any artificial or hard limits on number of connections in either solo or prod? Would be interested in anyones experience with using datomic in lambda, and how they managed making and maintaining the connection.

Daniel Hines13:03:58

I have a database of square and edge entities. Each square has 4 refs to an edge, and squares may share the same edge. Given a square’s ident A, how can I query for every other square B that shares an edge with the A, or every square C that shares an edge with B, or every square D that shares an edge with C… etc. until there are no more connected squares? To make it slightly more concrete, given the database:

[{:db/ident :A :edge/right :e1}
 {:db/ident :e1}
 {:db/ident :B :edge/left :e1 :edge/right :e2}
 {:db/ident :e2}
 {:db/ident :C :edge/right :e2}
 ...]
How can I recursively query for the set of entities who’s values for the attributes :edge#{top bottom left right} are the same (in this example db, the result should be #{:A :B :C})

mg14:03:16

@d4hines Datalog rules can do that. You might do something like,

[[(connected-square ?a ?b)
  [?a :edge/right ?e]
  [?b :edge/left ?e]]
 [(connected-square ?a ?b)
  [?a :edge/left ?e]
  [?b :edge/right ?e]]
 [(connected-square ?a ?b)
  [?a :edge/top ?e]
  [?b :edge/bottom ?e]]
 [(connected-square ?a ?b)
  [?a :edge/bottom ?e]
  [?b :edge/top ?e]]
 [(connected-square ?a ?b)
  (connected-square ?a ?s)
  (connected-square ?s ?b)]]]

Daniel Hines14:03:06

Thanks @michael.gaare ! I’ll try that out.

mg14:03:24

You need to pass that rule into the query, and then you can find all the connected squares with something like:

[:find ?connected :in $ % ?square :where [(connected-square ?square ?connected)]]

benoit14:03:41

@michael.gaare’s solution miss :C I think. It seems that squares can be included in other squares. :B and :C share the same right edge.

mg14:03:28

it doesn't handle squares that have identical edges, no. Given that they're squares, if they share one edge that's the same side, aren't they by definition the same square?

benoit14:03:39

Maybe 🙂

mg14:03:33

if you needed to extend to encompass that idea, could write another rule that does edge comparison

Daniel Hines14:03:37

@me1740 is correct - squares that share the same exact edge on the same attribute are not necessarily the same.

Daniel Hines14:03:55

The trick is that edges don’t have length - they’re lines, in the mathematical (infinitely extended) sense.

mg14:03:02

like maybe,

[[(shares-edge ?e ?square]
  [?square :edge/right ?e]]
 [(shares-edge ?e ?square]
  [?square :edge/left ?e]]
 ;; ... etc
]

mg14:03:29

then connected-square rule clauses instead look like, [(connected-square ?a ?b) [?a :edge/left ?e] [(shares-edge ?e ?b)]]

Daniel Hines14:03:32

Thanks, let me try that out.

mg14:03:23

these sound more rectanglish to me then 😄

Daniel Hines14:03:54

They are. I didn’t think the geometry would matter for the Datalog 😅

mg14:03:21

I wanted to make simplifying assumptions to enable my own laziness, see

mg14:03:43

I guess you could write a function to output these

Daniel Hines14:03:19

Yeah, we have a recursive function that uses db/entity to do this, but I wanted to see if it was possible to do it in pure datalog.

mg14:03:42

A function to write the rules I mean

mg14:03:59

cuz it's super tedoius

Daniel Hines14:03:41

What’s the most effective way to do that? Do I need to use splicing and things like in macro’s?

mg14:03:45

this should output what you want for shares-edge:

(let [edges #{:edge/right :edge/left :edge/top :edge/bottom}
      edge-sym (symbol "?e")
      square-sym (symbol "?s")]
  (for [e edges]
    [(list 'shares-edge edge-sym square-sym)
     [square-sym e edge-sym]]))

mg14:03:56

those sym bindings probably not necessary either

mg14:03:50

here, even smaller:

(for [e #{:edge/right :edge/left :edge/top :edge/bottom}]
  [(list 'shares-edge '?e '?s)
   ['?s e '?e]])

mg14:03:36

then for my own sense of completeness, the connected-square rules can be built like this I think:

(cons
 [(list 'connected-square '?a '?b)
  [(list 'connected-square '?a '?s)]
  [(list 'connected-square '?s '?b)]]
 (for [e #{:edge/right :edge/left :edge/top :edge/bottom}]
   [(list 'connected-square '?a '?b)
    ['?a e '?e]
    [(list 'shares-edge '?e '?b)]]))

mg14:03:25

putting the recursion first might be bad for performance, though, so look out for that when you're playing with this

benoit14:03:50

Not tested and I don't know how efficient it is but something like this might work:

'[
  ;; define what an edge is
  [(edge ?s ?e)
   (or [?s :edge/top ?e]
       [?s :edge/right ?e]
       [?s :edge/bottom ?e]
       [?s :edge/middle ?e])]

  ;; two squares are directly connected if they share an edge
  [(directly-connected-square ?s1 ?s2)
   (edge ?s1 e)
   (edge ?s2 e)]

  ;; the recursion
  [(connected-square ?s1 ?s2)
   (directly-connected-square ?s1 ?s)
   (connected-square ?s ?s2)]]

Daniel Hines19:03:46

How do I query for the value of an attribute that may or may not exist? I suppose I could do an or clause on two queries where one had the attribute and the other didn’t, but is there a short-hand for that?

val_waeselynck20:03:15

I think there's a get-else function

Daniel Hines20:03:37

Yeah, I eventually found taht.

Daniel Hines19:03:27

Oh, maybe I just have to put the one potentially non-existent attribute in the or

Daniel Hines19:03:02

That didn’t quite work.

mg20:03:17

The attribute isn't even in the schema you mean?

Daniel Hines20:03:07

No, it’s in the schema.

Daniel Hines20:03:55

The get-else function did the trick 👌

Daniel Hines20:03:03

@michael.gaare I’m usuing your for expression and it’s working beautifully. What’s the easiest way to compose that into a larger set of rules? I’m getting tripped up with quoting.

mg20:03:02

Just concat them all together, pass it as the rules

Daniel Hines20:03:17

I guess I’m maybe shaky on some Clojure basics here… why do the queries start off in quoted vector? Does it have to be quoted?

mg20:03:41

It's quoted because the symbols will be evaluated otherwise

mg20:03:20

Also the list forms

Daniel Hines20:03:22

Ok. This also seems to be doing what I expect:

(let [big-rule (for ...)]
  `[~big-rule
     [?e ?a ?v]
     ;; other rules...
    ])

Daniel Hines20:03:26

Is that typical?

Daniel Hines20:03:53

What would your way look like?

mg20:03:12

Generally you construct rules as one thing, pass them as a query argument, and call it % in the inputs

mg20:03:33

I'm not sure what you were doing with that macro, give me a second and I'll show you how I would do the shared edge thing we talked about earlier

mg20:03:17

Something like that

Daniel Hines20:03:48

(def rules
  (let [connected (vec (for [[a1 a2] opposite-edges]
                          [(list 'connected '?panel1 '?panel2)
                           ['?panel1 a1 '?edge]
                           ['?panel2 a2 '?edge]]))]
    `[~connected
      [(connected-recursive ?p1 ?p2)
       (connected ?p1 ?p)
       (connected-recursive ?p ?p2)]]))

Daniel Hines20:03:53

That’s where I’m at so far.

Daniel Hines20:03:06

(squares got renamed to panels)

Daniel Hines20:03:48

(This isn’t working, btw.

mg20:03:51

Probably don't want to use syntax quote (`) here

mg20:03:01

That's gonna mess up all the symbols

mg20:03:37

Just use concat there

mg20:03:53

so you could make that work at least syntactically by doing:

(let [connected ... ] ;; what you're doing here already seems fine
  (concat
   connected
   '[[(connected-recursive ?p1 ?p2)
      (connected ?p1 ?p)
      (connected-recursive ?p ?2)]])

Daniel Hines20:03:24

That works! Thanks. Much better than messing with quote/unquote 😛

mg21:03:10

you could also selectively quote symbols and construct lists if you want

mg21:03:52

quoting the whole form is doing two things for you: 1. without quoting, if the clojure compiler sees a symbol like connected-recursive or ?p1 it will try to resolve that symbol to its value in the current namespace, and throw an exception most likely because it's not going to be there 2. if the clojure compiler sees an unquoted list (like (connected ?p1 ?p)) it will try to turn that into a function call, which will also fail

mg21:03:14

You can achieve the same result by individually quoting the datalog symbols, (eg '?p1 rather than ?p1, and constructing lists by using the list function or by quoting the whole list

Daniel Hines21:03:45

Ok, that makes sense. I think where I got tripped up is I just assumed '[] meant something different than [].