Fork me on GitHub
Slack Archive

This page is not created by, affiliated with, or supported by Slack Technologies, Inc.

2023-09-07

Channels

#datomic
<
2023-09-07
>
Karol Wójcik10:09:37

I just stepped upon the following bug: https://forum.datomic.com/t/troubles-with-upsert-on-composite-tuples/1355/7. 1. I’m trying to understand whether manually inserting the composite tuples to trigger upsert is okay? 2. Is automatic ref resolving and automatic upsert in composite tuples in the Datomic team roadmap or status is „won’t fix”?

jaret01:09:45

We have an open story to review ref resolution in tuples, but for now you should resolve refs to achieve upsert behavior.

👍 2
Karol Wójcik01:09:58

Thank you for the answer! I will resolve refs manually then 🙂

Karol Wójcik21:09:06

Seeing really strange message for Datomic 1.0.6735. What is the reason datomic transactor tries to connect to some AWS internal url? I need healthchecks to achieve zero downtime deployments for dokku deployment on Hetzner. 

java.lang.RuntimeException: Unable to start ping endpoint localhost:9999
        at datomic.transactor_ext$start_ping_endpoint.invokeStatic(transactor_ext.clj:58)
        at datomic.transactor_ext$start_ping_endpoint.invoke(transactor_ext.clj:39)
        at datomic.transactor$run_STAR_.invokeStatic(transactor.clj:334)
        at datomic.transactor$run_STAR_.invoke(transactor.clj:246)
        at datomic.transactor$run$fn__25113.invoke(transactor.clj:388)
        at clojure.core$binding_conveyor_fn$fn__5823.invoke(core.clj:2047)
        at clojure.lang.AFn.call(AFn.java:18)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.io.FileNotFoundException:

jaret01:09:41

I have not tried running Datomic on Hetzner, but that error is caused by the AWS CLI I think being unable to find other nodes. On Hetzner what is your underlying storage? What does your transactor properties file look like?

Hendrik20:10:07

I get the same error. I am using Hetzner and Datomic 1.0.6735. It only happens if I specify an healtcheck endpoint in the transactor properties file. @karol.wojcik Where you able to solve this issue?

Hendrik21:10:37

On a Hetzner node this works: curl ,(withouth the “placement” path segment), but I have no idea, why this is called and how I could configure it.

Karol Wójcik20:10:26

I didn’t figure it out unfortunately.

Karol Wójcik20:10:23

@U023LKF2PQV do you have some more info you can pass to @U1QJACBUM in this 🧵. I’m not actively working on my hobby project anymore.

jaret20:10:20

Yeah, if I can get some more information specifically transactor properties file, underlying storage etc. I am happy to try to setup Hetzner and explore if we need to do something to support working their

Hendrik06:10:22

I am using postgres backend. My dockerfile is straightforward. Download the transactor. Adjust the properties file with sed command. Create datomic_kv table if it does not exist and run the transactor. This is the transactor file: 

protocol=sql
host=resolvable.transactor.hostname
port=4334


sql-url=jdbc:
sql-user=postgres
sql-password=mysecretpassword


sql-driver-class=org.postgresql.Driver


memory-index-threshold=32m
memory-index-max=256m
object-cache-max=128m

ping-concurrency=6
I start the transactor with bin/transactor -Ddatomic.pingHost=localhost -Ddatomic.pingPort=9999 -Ddatomic.printConnectionInfo=true config/transactor.properties I run it on a CAX11 arm vcpu in a k3s based k8 cluster. Running the transactor without the healtcheck endpoint bin/transactor config/transactor.properties on Hetzner works. Running it with the healtcheck endpoint on Hetzner results in the error that Karol posted. Running the dockerfile locally on my mac with docker and healthcheck endpoint works. I sshed into one of the Hetzner nodes and ran : 
curl
That failed. Then I tried : 
curl
That worked. The former url is AWS specific, the latter Hetzner specific. But I do not know, why the transactor sends a request to the endpoint in the first place. I do not see the point, why this information is needed to spin up a little endpoint that returns 200 ok.

Karol Wójcik17:10:12

@U1QJACBUM I have exactly the same config. Is this info provided by Henrik sufficient for you to reproduce?

jaret17:10:10

Yes @karol.wojcik I am hoping to get to this later this week, but I am working through a Pro Release and a Datomic Local release QA process first.

Karol Wójcik21:10:48

Thank you so much!

Karol Wójcik13:01:41

@U1QJACBUM hey 👋 , sorry for pinging you directly but do you have some additional info about this issue?

jaret13:01:36

@karol.wojcik are you sure you are on a version > 1.0.6735 on both peer and transactor? I suspect this issue is related to our regression, noted here: https://docs.datomic.com/pro/changes.html#1.0.6735. Ideally, I would like to know if you can reproduce on the latest version of Datomic Pro 1.0.7075. Would it be possible for you to upgrade peers and transactors to that version and retest health check?

👍 1
jaret13:01:02

My next intuition is that healthcheck is not starting on these machines and I would need more detail on the specific machines used (cores, etc). Or the port is not actually open on the machines.

jaret13:01:21

Apologies for not getting to this sooner and I appreciate the reminder.

Karol Wójcik13:01:14

Thank you so much. Upgrading to 1.0.7075 and checking if the port is open. Will post my findings soon ;)

Gustavo A.23:09:29

Hi, is it possible to search using partial tuples?, say that I have a tuple with three elements and I only know 2 of them and I would like to do something like this: 

(d/q {:query '{:find  [(count ?n)]
                   :in    [$]
                   :where [[?a :note-attribute/name+value+type ["tenant-id" "7777777" _]]
                           [?n :system-note/attributes ?a]]}
      :args  [db]})
Or, is the only way to have another tuple but with two elements? *Notice that I put an underscore for the unknown value.

favila01:09:38

Comparison operators

favila01:09:15

Nil sorts lower than any value

favila01:09:37

There’s unfortunately no highest value but you can do something data dependent

favila17:09:54

Sorry I have time to expand on this now. You can do something like this 

'{:find  [(count ?n)]
  :in    [$]
  ;; nil sorts before any other value, so you can use this as a start position
  :where [[(ground ["tenant-id" "7777777" nil]) ?start]
          ;; It's making the "end of range" value that is hard
          ;; and will depend on your data structure.
          ;; <= Long/MAX_LONG works well for numbers,
          ;; but I don't know what "type" is.
          [(ground ["tenant-id" "7777778" nil]) ?end]
          [?a :note-attribute/name+value+type ?value]
          [(<= ?start ?value)]
          [(< ?value ?end)]
          [?n :system-note/attributes ?a]]}

Gustavo A.19:09:14

🤯 Thank you! I was trying to do something else without the ground function it was not working btw

favila19:09:15

ground isn’t strictly necessary, you can supply ?start and ?end as inputs, or destructure from something else

favila19:09:05

It’s the [?x :attr ?v][(<= ?start ?v)][(< ?v ?end)] pattern that is essential for performance

Gustavo A.22:09:36

kk, it worked pretty fast, even without an specific end,

Gustavo A.22:09:04

I’m doing a search like count all the notes that have attributes with name "tenant-id" and value "7777777" so the start is the only thing I needed to establish,