Fork me on GitHub
#datomic
<
2021-03-09
>
helios08:03:04

Hey folks, I want your opinion. I need to put in my datomic schema an ordered list of refs to other entities. Since datomic doesn't support that natively, how would you do that? The difficult part is probably how to make sure the ordering stays consistent. Are composite tuples a good way of modeling that? :thinking_face:

helios08:03:44

What I had in mind was something like suggested here: https://forum.datomic.com/t/handling-ordered-lists/305/5

Braden Shepherdson18:03:22

I think that discussion is pretty sound. you end up with a "table" of [order, ref] pairs. there's not really a better way to do it and keep it queryable. abusing tuples for a dynamically-sized list is not likely to end well.

thumbnail09:03:52

Hey! Our team is using Datomic Analytics for some dashboards. One metric is based on the depth of trees in our system. I noticed the query sometimes breaks, I think in datomic-land. The result of the query is:

java.sql.SQLException: Query failed (#20210309_090647_00001_tab9g): bytes is negative

thumbnail09:03:32

WITH RECURSIVE t(lvl, parent, db__id) AS (
  SELECT 1, parent, db__id
  FROM mydb.node
  WHERE parent IS NULL
  UNION ALL
  SELECT lvl + 1, t.parent, t.db__id
  FROM mydb.node
  JOIN t
    ON mydb.node.parent = t.db__id
)
SELECT * FROM t LIMIT 1;
This is the query

thumbnail09:03:50

The node table is simply db__id, parent, and name. where parent refers to another node (or nil)

thumbnail09:03:52

It sometimes happens. So running the query multiple times yields different results. We're currently on datomic 1.0.6222.

Joe Lane16:03:35

@UHJH8MG6S how consistently can you reproduce? Once you can consistently, can you upgrade to the latest release and then try to repro again?

thumbnail16:03:37

I could not repro consistently. But I noticed an error in my query. Its recurring until OOM. Sometimes returning "out of resource " (expected). Sometimes returning the bytes-error. I'll upgrade the cluster later this week anyway and see if I can reproduce.

Joe Lane17:03:37

What is the size of your analytics gateway and is it pointed at a query group?

thumbnail17:03:58

I'm using on prem, right now it's a test setup with 10GB ram per query and 16GB max object heap. As a reference; when I fixed the query it consumed 1.5GB, the entire datomic DB is under a gig

Joe Lane17:03:26

Ahh, I was thinking you were on cloud.

thumbnail10:03:56

Spotted another odity; binding the same attribute multiple times doesn't work (in datomic analytics):

java.sql.SQLException: Query failed (#20210309_101720_00072_tab9g): Symbol name already has assignment "name_20", while adding "name_24"

thumbnail10:03:28

WITH RECURSIVE t(lvl, parent, db__id, name, root_name) AS (
  SELECT 1, r.parent, r.db__id, r.name, r.name
  FROM mydb.node AS r
  WHERE r.parent IS NULL
  UNION ALL
  SELECT lvl + 1, c.parent, c.db__id, c.name, root_name
  FROM mydb.node as c
  JOIN t
    ON c.parent = t.db__id
)

SELECT * FROM t ORDER BY t.lvl
Given this query. The need for 2 name-bindings is so the leaf and the root are shown

Joe Lane16:03:34

This ball is entirely in prestosql’s court.

thumbnail16:03:21

Fair enough! It was hard to judge this one. I simply wrapped the value in a array, which did the trick.

danm10:03:42

https://docs.datomic.com/cloud/client/client-api.html#timeouts says that timeouts are normally returned as ::anom/unavailable, but we seem to get a lot of ::anom/interrupted exceptions where the text content is Datomic Client Timeout. Is there some bad way of interacting we're likely to be doing that would cause us to get interrupted rather than unavailable?

ghadi22:03:11

Both are retriable...

pmooser11:03:54

If I have a to-many ref attribute, is there an easy way to query for that being empty?

pmooser11:03:08

I suppose maybe missing? is the easiest way.

1
pmooser11:03:44

I'm not quite sure how to use missing? in an or clause, though, because of the unification requirement. Hmm.

jaret14:03:17

Thanks! I forgot to update that link to reflect the new doc org

👍 1
Ben Hammond22:03:55

Hi. I'm trying to follow the Datomic Ions tutorial, but am stuck on the https://docs.datomic.com/cloud/ions/ions-tutorial.html#configure-connection I'm not sure how to determine the :endpoint : I have hacked up

:endpoint ""
to look more like my system, but I assume I must have got it wrong Can I find the correct endpoint in the CloudFormation logs?

Ben Hammond22:03:26

datomic -r eu-west-1 client access 
seems to have started the SSH tunnel without error

Ben Hammond22:03:53

Ah found it. It was in the Outputs of the Compute Stack

👍 1
1
Ben Hammond23:03:39

hmmm how do I specify an aws region in

clojure -A:ion-dev '{:op :push}'

Ben Hammond23:03:35

{:command-failed "{:op :push}",
 :causes
 ({:message
   "Unable to find a region via the region provider chain. Must provide an explicit region in the builder or setup environment to supply a region.",
   :class SdkClientException})}

Ben Hammond23:03:12

a somewhat low-tech

export AWS_REGION=eu-west-1
does the trick... sure there must be a nicer way though

kenny23:03:40

Add :region "my-region-1"

👍 1
Ben Hammond23:03:47

this seems an unfortunate error message upon the deploy

{:command-failed
 "{:op :deploy, :group vorpal-ion-starter-Compute-A2UL4PZOZHGU, :uname \"benjy-1\"}",
 :causes
 ({:message
   "A Lambda function name, which is the concatenation of the query group and the function name specified in the ion-config.edn file must be <= 64 characters.\nThe following names are too long for Lambda: vorpal-ion-starter-Compute-A2UL4PZOZHGU-get-items-by-type-lambda-proxy",
   :class RuntimeException})}

Ben Hammond23:03:53

I have control of the first 18 chars of that compute group name but not the last 20

kenny23:03:26

Perhaps they should add a warning note to the tutorial on this point.

Ben Hammond23:03:38

so the CodeDeployApplicationName should be no more than 11 characters long

Ben Hammond23:03:55

do I have to tear the whole stack down and recreate it?

Ben Hammond23:03:21

that really ought to be in the tutorial if so

kenny23:03:40

I don't believe you can change the name without recreating it.

kenny23:03:59

In https://docs.datomic.com/cloud/operation/planning.html#naming, they recommend keeping names under 24 characters.

Ben Hammond23:03:22

yeah just saw that. Ah well, thats a sign to pack it in for the evening

Ben Hammond23:03:26

thanks for your help

kenny23:03:30

Sure thing. Sorry the result isn’t ideal 😞

Ben Hammond23:03:51

its all about the journey

jaret13:03:25

@U793EL04V sorry about the frustrating start. Our hands are a bit tied on the master stack creation adding UUIDs for the nested stacks. We need to overhaul the tutorial as well, but I would like to make a recommendation for your next stack. Now that you are subscribed from Marketplace start your new system with split stacks. (launch storage and then compute). You can get the tempaltes from our https://docs.datomic.com/cloud/releases.html#current. And you can follow the split stack instructions here: https://docs.datomic.com/cloud/operation/split-stacks.html#howto

👍 1
Ben Hammond17:03:42

ok will try that. thanks

Ben Hammond21:03:18

Hi @U1QJACBUM. I followed the split stack instructions, and can access datomic from the REPL. However I am not able to successfully deploy; the deploy status returns as

{:deploy-status "FAILED", :code-deploy-status "FAILED"}
When I look into the CodeDeploy events I see that is is complaining DownloadBundle
Error code
UnknownError
Script name
Message
Access Denied
I can successfully download the pushed zip file from S3...

Ben Hammond21:03:49

am wondering if this is some side-effect of the split-stack deletion/recreations?

Ben Hammond21:03:43

oh, the datomic-code-eu-west-1 was permitting ListCodeBucket/ReadCodeBucket to a previous s3 bucket. thats strange

Ben Hammond21:03:57

well, updated it and looks like its working