Fork me on GitHub
#datomic
<
2022-04-05
>
Shuky Badeer06:04:58

Hi guys, I wanna restrict the query based on "<" predicate. Submitted at is a date in epoch format so it's an integer. I know that i can use ?nps based on trasnaction time but in this specific case i can't use it because this data was transferred from a different db. How do i restrict based on an attribute?

favila07:04:48

[??? :strive_form_data/submitted_at ?some-value][(< ?some-value 1)] will work. But only you know what the ??? should be.

❤️ 1
Shuky Badeer07:04:57

@U09R86PA4 amazing thank you so much! Can i ask, if i wanted some-value to be between 1 and 100 for example, how would i go about doing that? Using clojure syntax inside the query keeps leading to an error

favila07:04:54

[(<= 1 ?some-value)][(<= ?some-value 100)]

favila07:04:52

<= < > >= = != are special in queries. They’re not the normal clojure comparators

Shuky Badeer07:04:49

Cool that's how i did it. But i was afraid it would be inefficient since this basically does an implicit join?

favila07:04:09

all binding does an implicit join

favila07:04:08

how do you get the ?some-value we’ve been talking about in your query?

favila07:04:33

is it just [?nps :strive_form_data/submitted_at ?some-value]?

favila07:04:45

or is it on some other entity?

Shuky Badeer07:04:44

Ohh ok. Yes exactly how you just said

favila07:04:52

so the entity id is already known and bound; this is just retrieving the value and filtering

favila07:04:02

i.e. applying two predicates

favila07:04:55

this query is already scanning all answers/belongs_to

favila07:04:08

are either the second or third clause indexed?

Shuky Badeer12:04:14

@U09R86PA4 thank you very much! I wanted to ask, is there a way to do computation at the datomic query level? For example in sql we can use functions that divide numbers and uses the result as input of a child query. Is that possible in datalog as well?

favila12:04:11

Query rules are one way. You can also (on on-prem) call any function

👍 1
favila12:04:07

Roughly.

👍 1
Daniel Jomphe14:04:30

With Datomic Cloud, does it ever happen to you that you can't deploy anymore because of no more space left on the disk attached to the instance? Automatic rollbacks also then fail for the same reason. Did we do something we shouldn't?? Cannot allocate memory

Daniel Jomphe15:04:39

Obviously, those zip files aren't in cause - they'less than 1 MB. And the dependencies in .m2 and gitlibs that must be copied from the S3 bucket don't weigh more than 150 MB.

icemanmelting16:04:32

Ok, so regarding the problem I mentioned yesterday, I have more input, I get this on the transactor side:

o.a.activemq.artemis.core.client - AMQ212037: Connection failure to /172.20.0.4:57298 has been detected: AMQ229014: Did not receive data from /172.20.0.4:57298 within the 10,000ms connection TTL. The connection will now be closed. [code=CONNECTION_TIMEDOUT]
2022-04-05 16:29:14.005 WARN  default    o.a.activemq.artemis.core.server - AMQ222061: Client connection failed, clearing up resources for session 9d3f8e8c-b4f8-11ec-bb6d-0242ac140004
2022-04-05 16:29:14.014 INFO  default    datomic.update - {:task :reader, :event :update/loop, :msec 2110000.0, :phase :end, :pid 1, :tid 34}
2022-04-05 16:29:14.015 WARN  default    o.a.activemq.artemis.core.server - AMQ222107: Cleared up resources for session 9d3f8e8c-b4f8-11ec-bb6d-0242ac140004
2022-04-05 16:29:14.198 WARN  default    o.a.activemq.artemis.core.server - AMQ222061: Client connection failed, clearing up resources for session 9d435f1d-b4f8-11ec-bb6d-0242ac140004
2022-04-05 16:29:14.199 WARN  default    o.a.activemq.artemis.core.server - AMQ222107: Cleared up resources for session 9d435f1d-b4f8-11ec-bb6d-0242ac140004
2022-04-05 16:29:14.560 INFO  default    o.a.activemq.artemis.core.server - AMQ221002: Apache ActiveMQ Artemis Message Broker version 2.17.0 [9a4ab8f7-b4f8-11ec-85e0-0242ac140003] stopped, uptime 35 minutes
And this on the peer side:
clojure.lang.ExceptionInfo: Error communicating with HOST 0.0.0.0 or ALT_HOST 172.20.0.3 on PORT 4334
	at datomic.connector$endpoint_error.invokeStatic(connector.clj:53)
	at datomic.connector$endpoint_error.invoke(connector.clj:50)
	at datomic.connector$create_hornet_factory.invokeStatic(connector.clj:134)
	at datomic.connector$create_hornet_factory.invoke(connector.clj:118)
	at datomic.connector$create_transactor_hornet_connector.invokeStatic(connector.clj:308)
	at datomic.connector$create_transactor_hornet_connector.invoke(connector.clj:303)
	at datomic.connector$create_transactor_hornet_connector.invokeStatic(connector.clj:306)
	at datomic.connector$create_transactor_hornet_connector.invoke(connector.clj:303)
	at datomic.peer.Connection$fn__12046.invoke(peer.clj:217)
	at datomic.peer.Connection.create_connection_state(peer.clj:205)
	at datomic.peer$create_connection$reconnect_fn__12124.invoke(peer.clj:469)
	at clojure.core$partial$fn__5857.invoke(core.clj:2627)
	at datomic.common$retry_fn$fn__827.invoke(common.clj:543)
	at datomic.common$retry_fn.invokeStatic(common.clj:543)
	at datomic.common$retry_fn.doInvoke(common.clj:526)
	at clojure.lang.RestFn.invoke(RestFn.java:713)
	at datomic.peer$create_connection$fn__12126.invoke(peer.clj:473)
	at datomic.reconnector2.Reconnector$fn__11300.invoke(reconnector2.clj:57)
	at clojure.core$binding_conveyor_fn$fn__5772.invoke(core.clj:2034)
	at clojure.lang.AFn.call(AFn.java:18)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:750)
The system works fine for over half an hour, but then basically just dies with those 2 errors in the logs. On the transactor console output I just get Heartbeat failedAny ideas?

favila17:04:56

172.20.0.4 != 172.20.0.3

icemanmelting17:04:52

yes, but one is the peer, and the other is the transactor

icemanmelting17:04:55

different machines

icemanmelting17:04:30

it works for like 30 mins or so, and then dies. I am using prod level jvm args, 4gb for heap

favila17:04:48

Is the system quiet in that time? Eg no transactions?

favila17:04:47

On google cloud I remember some issue where their networking stack just drops idle tcp connections. I had to add keepalives into the kernel options somehow. IIRC it manifested like this, the peer looked like it went away and it only happened when the system was quiet

favila17:04:19

This was more than 5 years ago, my memory is hazy

icemanmelting17:04:24

no, i am developing a data processing framework with clojure, and currently i am stress testing it, and I am writting the results to datomic

icemanmelting17:04:36

so every second it has 2 tx of 100 datoms to save

icemanmelting17:04:50

so basically 200 a second, splitted between 2 transactions

icemanmelting17:04:29

I should also mention that these are dockerized, and I have also modified the keep alive values, for both containers, to make sure this isn’t a tcp connection issue

favila17:04:13

is it possible that the peer really did just go away for 10 secs, e.g. a long gc pause?

icemanmelting17:04:35

well, on the transactor i have set the max gc pause to be 50ms

icemanmelting17:04:41

on the peer, I haven’t modified that

icemanmelting17:04:49

is that something one should do?

favila17:04:03

those targets don’t apply when there’s a full gc and memory pressure. I’m really just suggesting that if you know the peer is busy or could have memory pressure on it, rule out that the timeout isn’t due to a GC pause

favila17:04:11

there are jvm startup flags that will log GC pause activity

favila17:04:22

to console or to a file

icemanmelting20:04:41

oh, ok, thanks for the pointers, much appreciated 🙂

icemanmelting20:04:47

I will check that out

dazld21:04:39

in the same transaction, is it possible to both retract an older entity, and create a new one, where an identity attribute is shared between both?

dazld21:04:40

I’m guessing no, as there’s no way to disambiguate which entity is being referred to via attribute identities.. ?

dazld21:04:41

ie:

[[:db.fn/retractEntity [:a/id1 "a"]]
 [:db.fn/retractEntity [:a/id2 "b"]]
 {:thing/id "thing"
  :thing/stuff {:a/id1 "a"
                :a/id2 "b"}}]
 
this can’t work, even with db/ids?

favila21:04:59

You should run this to know for sure. I would expect all lookups to happen before retraction, so this will cause {:a/id1 "a" :a/id2 "b"} to expand to [:db/add id1a-entity-id :a/id1 "a"] etc. Since those same assertions are being retracted in the same transaction via the retractEntity, the transaction will fail with a conflict.

👍 1
favila22:04:42

In general all lookups happen on the “before” db and all operations in a transaction are applied atomically. (Exceptions are composite tuples, which do read an intermediate state to know what to update the new values to; and entity predicates, which read an “after” db right before commit.) So it’s not ambiguous at all what a lookup in a transaction will do.

favila22:04:10

there’s no way, even in a transaction function, to see a “in-progress” or “partially-applied” database value

dazld08:04:25

it does indeed fail with a conflict - will try with concrete IDs too (doesn’t work, sadly)

favila12:04:11

concrete ids will work IFF the entity ids for the retracts and the new assertions are different