Fork me on GitHub
#datomic
<
2020-07-08
>
craftybones04:07:08

@jaret - my formative years were in the Bay Area, so the Kings and Warriors would be the teams I root for, but I enjoy basketball too much to really care who wins.

craftybones04:07:22

but I am not a fan of the Knicks, that’s for sure 🙂

craftybones04:07:32

I have a question around bulk data entry. Is it better to transact one large transaction with temp-ids to associate refs or, transact entities which form references first and then transact the rest of the data?

craftybones04:07:43

The second option sounds tedious, the first one sounds bulky

alex-dixon13:07:27

Hi. I’m wondering how you would model “friendship” in Datomic (assuming “friendship” is mutual)

marshall13:07:28

all references in datomic are bidirectional

alex-dixon14:07:12

@marshall Thanks. So an “add friendship” transaction would need to add two facts: “a is friends with b” and “b is friends with a”?

marshall14:07:53

interesting - i see that the example does do that

marshall14:07:00

It is done that way for the recursive pull

marshall14:07:10

you can navigate the ‘friends’ relationship “backwards”

marshall14:07:21

in a query you just reverse the clause

marshall14:07:42

i.e. : if you have asserted:

{:user/name "a"
 :friend [:user/name "b"]}

marshall14:07:57

you can navigate that relationship from a to b in a query with:

marshall14:07:31

[[?e :user/name "a"]
 [?e :friend ?x]]

marshall14:07:40

and you can do the “reverse” as well:

marshall14:07:59

[[?e :user/name "b"]
 [?x :friend ?e]]

marshall14:07:16

the first one says find friends of “a”

marshall14:07:25

the second says “who are friends with b”

favila14:07:13

I think the problem to solve is deduping friendship relationships. You can transact a :friend b and b :friend a and have your read patterns understand that both are equivalent, but when you add/update/remove friend relationships, you want to make sure those updates are consistent and not vulnerable to transaction races

marshall14:07:44

Agreed @favila ^ generally I would leverage the bidirectional ref so you don’t hit that issue

favila14:07:57

I think using a bidirectional ref may still be vulnerable to read skew unless there’s some canonicalization? I can’t think of a case offhand though

marshall14:07:23

If the relationship is always considered bidirectional, it should be ok. I.e. i cant be friends with you if youre not friends with me

marshall14:07:49

So the ref as a single relationship either exists or doesnt

favila14:07:14

I mean during updates, some ordering of friendship add or remove transactions may end up by failing to remove one of the directions

marshall14:07:34

Not if you only model it from one side

favila14:07:48

but which side?

marshall14:07:55

Does it matter?

marshall14:07:05

Write the query to handle either

marshall14:07:26

VAET makes it efficient either way

alex-dixon14:07:20

I sort of have a feature request in mind…which would the addition of tuple value types that are effectively sets…they don’t bring ordereding into their equality semantics. For things like this that aren’t really directional

favila14:07:08

I don’t think that would help @alex-dixon? where would that assertion live?

favila14:07:16

unless you mean a tuple of sets?

favila14:07:25

rather a set of sets

alex-dixon14:07:32

;; [0 :user/name "foo"]
;; [1 :user/name "bar"]
;; [2 :friendship/between [0 1]] ; I'd like [0 1] and [1 0] to be equal, but they're treated as different.
;; [2 :friendship/since 1949] ; :friendship exists as a fact independent of :user. There can be facts about it.

favila14:07:51

ok, I have wanted something like that

alex-dixon14:07:22

Something like that. Part of what I’m looking for also is a way to add information about the “friendship”

favila14:07:35

I’ve solved it by canonicalizing the A and B parts, or by serializing the relationship consistently

favila14:07:35

e.g. canonicalization: {:friendship/left (always the entity with the lower identity id) :friendship/right (always the other one)}

👍 3
alex-dixon14:07:12

Was thinking the same with a tuple type. Do a sort. Make ordering consistent or not matter

favila14:07:31

just don’t use raw entity ids for this

favila14:07:36

use some public unchanging id you control

favila14:07:26

nowdays I think the best is to combine this approach with a :friendship/left+right composite tuple--best of both worlds

favila14:07:15

but if you make that unique, watch out for deletions. when retracting a member of a friendship, you need to remove the friendships they’re involved in first

favila14:07:30

otherwise you’ll end up with :friendship/left+right [nil something] and get constraint violations

favila14:07:02

I kinda wish there was a composite tuple mode that would retract the tuple instead of allowing nil in it

favila14:07:24

actually on balance it’s probably less trouble to use a value tuple with :friendship/left-id+right-id that you control, as long as reads check friendship/left and friendship/right are nonnull

favila14:07:52

that allows more flexibility about when precisely you schedule the friendship deletions when deleting users

favila14:07:03

which may be necessary to control transaction size

dpsutton14:07:03

been a bit since i've done datomic but could a rule easily add the reflexive checking?

favila14:07:15

e.g. serializing {:friendship/id “lower-identity-id+higher-identity-id”}

favila14:07:24

(this was before tuples--I’d use tuples now)

favila14:07:03

the issue is still that you have an entity that you want to be unique by a value which is a set

favila14:07:44

with the right read patterns and using transaction fns or :db/ensure, you could keep the constraint that only one of these exists, but you can’t model it directly with a unique index without doing tricks like this

alex-dixon14:07:38

Ok. Thanks. I was thinking along those lines but didn’t know if I was missing something. I’m still curious about the possibility of something like a tuple type that would ignore ordering for its equality semantics. It seems that might open up a fact representation that doesn’t require encoding directedness, something like :friendship/between #{a b}, and annotate those with other facts

marshall14:07:38

you could reify the “friendship”

marshall14:07:43

as an entity

marshall14:07:01

which then has a card-many :members attribute

marshall14:07:15

presumably limited (in your app) to 2

marshall14:07:29

but the indirection will have some performance considerations

marshall14:07:39

it adds a step to each friend lookup

Joe Lane15:07:25

An alternative approach could be to model friendship as the hash of two hashes, then keep track of that as a identifier of "friendship" between two people.

Joe Lane15:07:41

Think Merkle Trees

bamarco15:07:43

I am trying to get websockets working with datomic ions. My connection function currently looks like:

clj
(defn add-connection! [id]
  (faraday/put-item nil sockets-table
    {:connection-id id}))

(defn connect [{:as req :keys [input context]}]
  (let [event (json/parse-string input true)
        id (-> event :requestContext :connectionId)]
    (add-connection! id)
    ;(str {:status 200})
    ))
This guide on aws websocket https://aws.amazon.com/blogs/compute/announcing-websocket-apis-in-amazon-api-gateway/ suggests
js
exports.handler = function(event, context, callback) {
  var putParams = {
    TableName: process.env.TABLE_NAME,
    Item: {
      connectionId: { S: event.requestContext.connectionId }
    }
  };

  DDB.putItem(putParams, function(err, data) {
    callback(null, {
      statusCode: err ? 500 : 200,
      body: err ? "Failed to connect: " + JSON.stringify(err) : "Connected"
    });
  });
};
There doesn't seem to be access to this callback function with a datomic ion. This thread https://forum.datomic.com/t/websockets-from-ions/1255 claims to have gotten this working.

bamarco15:07:27

Maybe they are not using clojure for the connect function?

Joe Lane15:07:03

@mail524 I don't understand what problem you're having, could you describe what you expect to happen that isn't happening?

Sam Adams15:07:47

Hey all, question re: the Ions tutorial: once I’ve deployed my solo-topology, lambda-proxied ion and wired it up to an API gateway, why must I append “/datomic” to the invocation URL (or else get a 403 response)? https://docs.datomic.com/cloud/ions/ions-tutorial.html#deploy-apigw

Sam Adams16:07:38

It seems that any appended path will do. But I want to wire up my own domain, and serve my homepage from the empty path.

Sam Adams16:07:51

:face_palm: resolved - it’s an [API gateway quirk](https://stackoverflow.com/questions/52909329/aws-api-gateway-missing-authentication-token), nothing to do with Ions - apologies for the spam

bamarco15:07:07

@joe.lane when I call wscat -c wss://<api-key>. I get error: Unexpected server response: 502 which is describe here https://stackoverflow.com/questions/57438756/error-unexpected-server-response-502-on-trying-to-connect-to-a-lambda-function because I am not calling the callback function the connection does not stay alive. I do successfully get the connection id logged into dynamodb.

Joe Lane15:07:19

@mail524 First, are you sure that you need wss://<api-key>... and not the <api-id>? How do you know that URL is correct? Second, it appears you aren't returning the right thing from your ion. It has nothing to do with callbacks.

Joe Lane16:07:00

Third, are you sure that your call to Faraday is returning with success? Maybe add some calls to cast/event so you can confirm they are being added?

bamarco16:07:37

1. it is the app-id I just called it the wrong thing. I know because I get logging for it in cloudwatch when I connect.

bamarco16:07:16

3. I check the dynamodb directly and ids are being logged into the table

bamarco16:07:23

2. I am not sure what to return that may be the problem. I have tried {:status 200} and (str {:status 200}).

bamarco16:07:12

I have not json-ified it

Joe Lane16:07:24

So, Per the stack overflow article, the body needs to be a string

Joe Lane16:07:42

You should follow the "Set up API Logging Using the API Gateway Console" in the S.O. post and look at what the logs say (different logs than the datomic logs)

bamarco17:07:27

It seems to work now by returning (chesire/generate-string {:statusCode 200}). Got the api gateway logs setup too. Hopefully I can get sockets working entirely without further help. Thanks so much!

👍 3
Joe Lane17:07:23

Great to hear!

bmaddy21:07:26

Is it possible to query maps with datomic? If so, does anyone know where I could find a simple example? (I see the nested vector example here: https://docs.datomic.com/on-prem/query.html)

favila22:07:55

Not directly. Datomic datalog can query datasources, which are fundamentally sets of tuples

favila22:07:23

however, you can transform maps to tuple sets by assigning ids to each map

favila22:07:21

(It uses datascript’s query engine, but the same technique could be used with datomic)

bmaddy22:07:05

Oh, neat. I didn't realize that's how datamaps worked. Thanks @favila!