Fork me on GitHub
#datomic
<
2020-06-16
>
craftybones06:06:42

What am I missing here?

user=> (d/q '[:find ?genre
  #_=>        :where [_ :movie/genre ?genre]] db)
[["Drama, Action"] ["Drama"] ["Sci Fi"]]
user=> (d/q '[:find ?e ?a
  #_=>        :where [(fulltext $ :movie/genre "Drama") [[?e ?a _ _]]]] db)

craftybones06:06:36

Based on what the manual says, this ought to work.

craftybones06:06:58

I’ve even tried a parameterised variety and didn’t get it to work. I am sure I am doing something stupid, just don’t know what it is

favila11:06:25

does :movie/genre have a fulltext index?

favila11:06:20

(`fulltext` passes it straight down to Lucene)

craftybones12:06:09

😄 That was it. Thanks

raspasov09:06:16

@srijayanth Try “Drama*” maybe?

dmarjenburgh10:06:25

We have a lambda ion handler that gets invoked daily with a CloudWatch event. It never gave problems, but since this last night it throws this exception:

No implementation of method: :->bbuf of protocol: #'datomic.ion.lambda.dispatcher/ToBbuf found for class: clojure.lang.PersistentArrayMap: datomic.ion.lambda.handler.exceptions.Incorrect
clojure.lang.ExceptionInfo: No implementation of method: :->bbuf of protocol: #'datomic.ion.lambda.dispatcher/ToBbuf found for class: clojure.lang.PersistentArrayMap {:cognitect.anomalies/category :cognitect.anomalies/incorrect, :cognitect.anomalies/message "No implementation of method: :->bbuf of protocol: #'datomic.ion.lambda.dispatcher/ToBbuf found for class: clojure.lang.PersistentArrayMap"}
	at datomic.ion.lambda.handler$throw_anomaly.invokeStatic(handler.clj:24)
	at datomic.ion.lambda.handler$throw_anomaly.invoke(handler.clj:20)
	at datomic.ion.lambda.handler.Handler.on_anomaly(handler.clj:171)
	at datomic.ion.lambda.handler.Handler.handle_request(handler.clj:196)
	at datomic.ion.lambda.handler$fn__3841$G__3766__3846.invoke(handler.clj:67)
	at datomic.ion.lambda.handler$fn__3841$G__3765__3852.invoke(handler.clj:67)
	at clojure.lang.Var.invoke(Var.java:399)
	at datomic.ion.lambda.handler.Thunk.handleRequest(Thunk.java:35)

dmarjenburgh10:06:55

Nvm, I found that it's the return value from the server handler

craftybones10:06:51

@raspasov - that didn’t work! 😞

Lone Ranger11:06:57

does anyone have any experience running a dockerized peer server or peer application? There seems to be some networking requirement that I'm not fully understanding

favila11:06:50

ports 4334 and 4335 must be open

Lone Ranger11:06:07

outbound or inbound?

favila11:06:35

the host= or alt-host= in the transactor properties file must name the transactor and be resolveable by peers

favila11:06:51

(actually 4335 is only for dev storage)

favila11:06:06

datomic peer connections work like this:

favila11:06:29

transactor writes its own hostname to storage and sets up an artemismq cluster

Lone Ranger11:06:32

gotcha. So it needs to be able to hit 172.17.0.1:4334, in my case?

favila11:06:50

then peers connect to storage, lookup the transactor name, and connect to the transactor on 4334

Lone Ranger11:06:51

ah ok, keep going

favila11:06:19

so they need whatever the txor writes to storage to be resolveable to the transctor in whatever network they are in

favila11:06:42

(4334 is for artemis)

Lone Ranger11:06:41

interesting ... digesting

Lone Ranger11:06:17

and it's a one way connection?

favila11:06:22

looks like from your logs that the peer can find storage, but either 172.17.0.1 doesn’t resolve to the txor, or it’s not allowed to connect to it, or the destination port isn’t open

Lone Ranger11:06:23

there's no inbound from AMQ?

favila11:06:41

there’s inbound data, but the txor doesn’t actively connect to peers

Lone Ranger11:06:13

gotcha. hmm okay thank you. This gives me what I need to work on the puzzle

favila11:06:45

btw why is “localhost” an option?

favila11:06:05

is that for connecting from outside docker?

Lone Ranger12:06:14

I developed it locally and now I'm attempting to dockerize it

Lone Ranger12:06:43

I was considering converting the app code to client process but I'd be back in the same boat with the peer server needing to be dockerized

Lone Ranger12:06:31

interesting -- found this and I'm not seeing any extra exposed ports: https://github.com/frericksm/docker-datomic-peer-server

favila12:06:24

it exposes 9001

favila12:06:36

this is just the peer server

favila12:06:40

no transactor

favila12:06:04

you confirmed that 4334 is exposed on the transactor?

Lone Ranger12:06:15

I'm trying to dockerize my peer application code, not the transactor

Lone Ranger12:06:29

so using this peer server as inspiration -- sorry for the confusion

Lone Ranger12:06:36

transactor is on my host machine right now

Lone Ranger12:06:50

peer application code is on the docker

Lone Ranger12:06:25

OH IT DOES EXPOSE 9001 !!! great catch

Lone Ranger12:06:13

Also I notice it is going with "0.0.0.0" instead of the docker bridge network, interesting

Lone Ranger12:06:21

gives me some stuff to play around with, great eye.

favila12:06:51

9001 is the client api port

favila12:06:11

that is the service the peer-server is exposing

favila12:06:26

but your problem is your peer can’t find or can’t talk to your transactor

favila12:06:35

switching to client-server won’t fix that

Lone Ranger13:06:27

well worst case scenario I can rewrite the code with client API and try to talk to the peer server instead

favila13:06:24

but, the peer-server is a peer

favila13:06:58

it is a peer, which implements the server half of the client api

favila13:06:46

have you tried this peer-server docker image and gotten it to connect to your transactor?

favila13:06:29

wait a sec, I think I know your problem

favila13:06:41

your transactor is binding to localhost

favila13:06:30

it needs to bind to something the docker network layer can route to

favila13:06:58

try host=0.0.0.0 as a first step

favila13:06:37

it may not let you do that; if not, use the container’s IP

Lone Ranger13:06:28

good thinking

Lone Ranger13:06:36

I'll give that a shot

Lone Ranger13:06:15

interesting, new error anyway

Lone Ranger13:06:55

sorry about formatting 😕

Lone Ranger13:06:07

now it's clearly psql that's pissed

Lone Ranger14:06:34

yeah this is interesting, postgres wants a DIFFERENT host identifier than the transactor does. Transactor is connecting on 0.0.0.0, but postgres wants the docker bridge:

favila14:06:20

transactor’s host and storage host are different concepts

favila14:06:22

I think you are misunderstanding something. this error shows you got even less far along

favila14:06:36

you didn’t even manage to connnect to postgres this time

favila14:06:09

you are setting up three things: 1) postgres, exposes 5432, needs routable hostname 2) transactor, exposes 4334, needs routable hostname, needs to connect to postgres. 3) peer; needs to connect to postgres, needs to connect to transactor

favila14:06:06

transactor.properties host= is what the transactor binds to for port 4334 (for peers to connect to it). Both host= and alt-host= are written to postgres for peers to discover

favila14:06:45

alt-host= is an alternative host/ip in case there’s some networking topology where what the transactor binds to isn’t the same thing other peers should connect to

tvaughan13:06:29

If you're running on a single machine, all you need to do is 1) name the running containers, 2) use this name as the host name when connecting to a running container, and 3) run all containers on the same bridge network. Ports don't need to be exposed explicitly if they're only accessed by other containers on the same bridge network. All services should bind to 0.0.0.0 , not localhost. For example, we start the peer server like $DATOMIC_RELEASE/bin/run -m datomic.peer-server -h 0.0.0.0 -p 8998 -a "$DATOMIC_ACCESS_KEY_ID","$DATOMIC_SECRET_ACCESS_KEY" -d $DATOMIC_DATABASE_NAME,datomic:mem://$DATOMIC_DATABASE_NAME

Lone Ranger17:06:31

Awesome, thank you! I'll be sure to use this when we transition to client API

Lone Ranger12:06:29

What is the significance of port 9001 with regards to the peer server? Would this also apply to peer application code?

favila12:06:25

9001 is the port that client api clients connect to.

Lone Ranger12:06:03

interesting. So this should not apply to peer application code? :thinking_face:

favila12:06:46

no. BTW that port is configurable. 9001 is just the one that docker image you were looking at happened to use

favila12:06:57

look at line 25

Lone Ranger14:06:48

okay creeping closer. .properties file modifications:

alt-host=172.17.0.1             
host=0.0.0.0
protocol=sql
#host=localhost
port=4334
peer connection string from docker: datomic:
[main] ERROR org.apache.activemq.artemis.core.client  - AMQ214016: Failed to create netty connection
java.net.UnknownHostException: 172.17.0.1
        at java.base/java.net.InetAddress$CachedAddresses.get(InetAddress.java:797)

favila14:06:58

just to verify--that is indeed the ip address of the transactor?

favila14:06:16

I notice it’s the same as postgres.

Lone Ranger14:06:01

hmm good question. well this is what the transactor says:

Lone Ranger14:06:42

$ ./run-transactor.sh
Launching with Java options -server -Xms1g -Xmx1g -XX:+UseG1GC -XX:MaxGCPauseMillis=50
Starting datomic:sql://<DB-NAME>?jdbc:, you may need to change the user and password parameters to work with your jdbc driver ...
System started datomic:sql://<DB-NAME>?jdbc:, you may need to change the user and password parameters to work with your jdbc driver

Lone Ranger14:06:07

datomic:sql://<DB-NAME>?jdbc:

Lone Ranger14:06:34

(postgres and the transactor are on the same host right now)

Lone Ranger14:06:45

(but in the future they could be on different hosts)

favila14:06:44

are they inside or outside your docker network?

Lone Ranger14:06:18

hmm the transactor is outside, the postgres is also running in a docker probably on the default network

favila14:06:35

so, your peer can connect to postgres, but not to the transactor

favila14:06:26

I don’t understand this part though .UnknownHostException: 172.17.0.1

favila14:06:23

if the peer could connect to postgres on 172.17.0.1 to get the transactor IP, how is that an unknown host?

favila14:06:57

are you absolutely sure this is what the peer used? (d/connect "datomic:<sql://search?jdbc:postgresql://172.17.0.1:5432/datomic?user=datomic&password=datomic>")` ?

Lone Ranger14:06:25

good question

Lone Ranger14:06:36

I'll hardcode it just to be sure

Lone Ranger14:06:26

(ns search.config
  (:require [clojure.tools.logging :as log]))

;; todo -- parameterize
(def dockerized? (System/getenv "DOCKERIZED"))
(log/info (str "Dockerization detected:" dockerized?))
(def db-host (if dockerized?
               "172.17.0.1"
               "0.0.0.0"))
(log/info (str "Using host: " db-host))
(def db-uri (str "datomic:" db-host  ":5432/datomic?user=datomic&password=datomic"))
(log/info (str db-uri))
(def db-version "0.6")
is technically what it's doing

Lone Ranger14:06:41

ah, no wait, this is the same error facepalm

favila14:06:08

again, this indicates that the peer could talk to postgres but not the transactor

favila14:06:42

absolutely baffled how the same IP could be fine and also unknown host

favila15:06:08

it just used that IP to talk to postgres, and got the host and alt-host info from there

Lone Ranger15:06:01

yeah, it's nuts. It's def a networking thing, when I use the --network=host option on the Docker everything works fine

Lone Ranger15:06:16

ah I see so the observation here indicates that the peer process looked up the location of the transactor in storage (the host/alt-host settings) and then attempted to use those to connect to the transactor?

Lone Ranger15:06:02

is there a way to "ping" the transactor?

Lone Ranger15:06:19

because then I could test what the host is supposed to be from the Docker perspective

Lone Ranger15:06:05

ah okay, I have a new hypthesis.

docker_gwbridge: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.17.0.1  netmask 255.255.0.0  broadcast 172.17.255.255
        ether 02:42:b5:8e:77:df  txqueuelen 0  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 2302  bytes 377328 (377.3 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
that's the docker bridge (as seen from host)

Lone Ranger15:06:21

postgres is dockerized, the peer service is dockerized ... the transactor is not

Lone Ranger15:06:08

in my mind it makes sense then what you said about the baffling behavior of the transactor failing when it just talked to postgres to get the information

Lone Ranger15:06:38

so perhaps if I dockerize the transactor it will alleviate this issue

favila17:06:24

that’s not my understanding of “unknownhosterror”

favila17:06:30

but maybe it is just a communication thing

Lone Ranger17:06:46

I'm writing up the current state of affaris

favila17:06:04

re “ping” if you just want to test reachability from various machines use nc -z hostname port

favila17:06:20

it will print if it could establish a tcp connection then terminate

Lone Ranger17:06:44

Either there is a missing piece or I just need to learn more about networking/docker. Current setup: • transactor on docker0 • postgres on docker1 • peer application on docker2 observations: tranasctor can connect to psql

root@docker0:/opt/datomic-pro-1.0.6165#  psql -h 172.17.0.1  -p 5432 -U datomic -W 
Password: 
psql (11.7 (Debian 11.7-0+deb10u1), server 12.3 (Debian 12.3-1.pgdg100+1))
WARNING: psql major version 11, server major version 12.
         Some psql features might not work.
Type "help" for help.

datomic=> 
peer application can connect to psql:
root@c4c329971ae7:/app# psql -h 172.17.0.1  -p 5432 -U datomic -W
Password: 
psql (11.7 (Debian 11.7-0+deb10u1), server 12.3 (Debian 12.3-1.pgdg100+1))
WARNING: psql major version 11, server major version 12.
         Some psql features might not work.
Type "help" for help.

datomic=> 
current error message from peer application:
[main] INFO  search.config  - Dockerization detected:true
[main] INFO  search.config  - Using host: 172.17.0.1
[main] INFO  search.config  - datomic:
[main] INFO  datomic.domain  - {:event :cache/create, :cache-bytes 2086666240, :pid 6575, :tid 1}
[main] INFO  datomic.process-monitor  - {:event :metrics/initializing, :metricsCallback clojure.core/identity, :phase :begin, :pid 6575, :tid 1}
[main] INFO  datomic.process-monitor  - {:event :metrics/initializing, :metricsCallback clojure.core/identity, :msec 0.47, :phase :end, :pid 6575, :tid 1}
[main] INFO  datomic.process-monitor  - {:metrics/started clojure.core/identity, :pid 6575, :tid 1}
[clojure-agent-send-off-pool-0] INFO  datomic.process-monitor  - {:AvailableMB 3880.0, :ObjectCacheCount 0, :event :metrics, :pid 6575, :tid 13}
[clojure-agent-send-off-pool-0] INFO  datomic.kv-cluster  - {:event :kv-cluster/get-pod, :pod-key "pod-catalog", :phase :begin, :pid 6575, :tid 13}
[clojure-agent-send-off-pool-0] INFO  datomic.kv-cluster  - {:event :kv-cluster/get-pod, :pod-key "pod-catalog", :msec 8.89, :phase :end, :pid 6575, :tid 13}
[main] INFO  datomic.peer  - {:event :peer/connect-transactor, :host "172.17.0.1", :alt-host nil, :port 4334, :version "1.0.6165", :pid 6575, :tid 1}
Execution error (ActiveMQNotConnectedException) at org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl/createSessionFactory (ServerLocatorImpl.java:787).
AMQ119007: Cannot connect to server(s). Tried with all available servers.
transactor config:
host=172.17.0.1
#host=0.0.0.0                                                                                                                                                                                         
protocol=sql
#host=localhost                                                                                                                                                                                       
port=4334


sql-url=jdbc:                                                                                                                                                     
sql-user=datomic
sql-password=datomic

Lone Ranger17:06:07

As I write that up I think the host might be wrong. host is supposed to be host of transactor, not sql, yeah?

Lone Ranger17:06:06

that was it 🙂 @favila you are THE MAN!! thank you!!!

Lone Ranger17:06:33

I should probably do a blog thing about this for the benefit of others cause this was a little tricky

Lone Ranger17:06:49

I should probably do anything productive with my time at all 😓

Lone Ranger14:06:08

so that's a new error, the UnknownHostException

Lone Ranger18:06:09

@favila if there was an equivalent of upvotes or reddit gold for this slack channel I'd be throwing them at you, thank you

6
jackson22:06:03

Is qseq not available in datomic.api as suggested here? https://docs.datomic.com/on-prem/clojure/index.html#datomic.api/qseq

jackson22:06:14

Specifically the peer api for 0.9.6045. And I have the same issue with index-pull.

favila23:06:33

This fn is brand new and only available on the latest version (yours is not) https://docs.datomic.com/on-prem/changes.html

👍 3
jackson00:06:21

Not sure how I missed that release, thanks!

zhuxun222:06:20

Earlier I asked a question about EQL in #fulcro but I figured it might concerns datomic users as well so I felt compelled to drop it here as well. Apologize for the long post. 🙏 --- original post --- I have a question about the design of EQL which I'm not sure this is the right place to discuss. However, I feel EQL is something fulcro users deal with heavily so I think it might be valuable for me to hear your opinions: There are couple of quirks that made me wonder why EQL was designed like it is. Let's take a look at a typical EQL -- the kind of which probably show up in your app hundreds of times

[:user/id
     :user/name
     :user/email
     {:user/projects [:project/id
                      :project/name
                      :project/start-date
                      :project/end-date
                      :project/completed?]}
     {:user/contacts [:user/id]}]
EQL claims to have "the same" shape as the returned data. That's awesome! However, why don't we go a step further? Consider the return value of the above query:
{:user/id #uuid "..."
     :user/name "Fred Mertz"
     :user/email ""
     :user/projects [{:project/id #uuid "..."
                      :project/name "Mertz and Kurtz"
                      :project/start-date #inst "..."
                      :project/end-date #inst "..."
                      :project/completed? true}]
     :user/contacts [{:user/id #uuid "..."}
                     {:user/id #uuid "..."}]}
Why wasn't EQL designed to completely mimic that structure:
{:user/id _
     :user/name _
     :user/email _
     :user/projects [{:project/id _
                      :project/name _
                      :project/start-date _
                      :project/end-date _
                      :project/completed? _}]
     :user/contacts [{:user/id _}]}
Or, if the explicitness of pluarity is not desired here:
{:user/id _
     :user/name _
     :user/email _
     :user/projects {:project/id _
                     :project/name _
                     :project/start-date _
                     :project/end-date _
                     :project/completed? _}
     :user/contacts {:user/id _}}
The immediate benefit is that now I can use the map namespace syntax to make it much more succinct and DRY (and easy on the eye):
#:user{:id _
           :name _
           :email _
           :projects #:project{:id _
                               :name _
                               :start-date _
                               :end-date _
                               :comleted? _}
           :contacts #:user{:id _}}

zhuxun222:06:20

IMHO many important semantics are much better aligned this way. For example, in a return value, the order of the keys in a level of map should not matter, and there should not be duplicated keys. However, EQL uses an ordered collection (vector) to denote the keys, the semantics of which has a sense of order while ensures no uniqueness. Also, it feels like in EQL maps are used in place of pairs. I understand that Clojure doesn't have a built-in literal for pairs so it makes sense to use maps, but maps seem to be a poor fit for this role -- here they are only allowed to have one key-value pair, and pushes the key into the next level when it should really belong to the outer level. I feel that the ad-hoc-ish design not only misses mathematical simplicity but also make everything unnecessarily complex. If I were to write a function to retrive all root level keys given a EQL (which should have been trivial), the implementation would be a few lines unnecessarily longer since I need to consider those ref-type attributes. If I am using Emacs to manually write a test example given an EQL, I am doing lots of unnecessary work changing brackets into braces and splicing sexp's. That being said, my exposure to Clojure and Datomic/Pathom/Fulcro is limited, and I truly want to hear if there are reasons why EQL was designed the way it is rather than my intuitive version. I apologize my above arguments spiralled into a small rant.

favila23:06:26

Not a complete answer but some historical context: eql is based on pull expressions, which existed prior to the namespaces map features you mention

👍 3
favila23:06:58

So to some degree this is historical accident.

favila23:06:31

Also the symmetry of your proposal breaks down once you consider parameters

favila23:06:43

Key renaming, limits, defaults, etc. some of that you can smuggle into the value slot, but you need a plan for nested maps

souenzzo00:06:12

I dont think that the pull notation is import and do not think that one is better then other. We can have many notation/representations and talk about the same AST.

zhuxun205:06:50

@favila I can support params the same way the current EQL does -- on the key slot, no?

#:user{(:id {:with "params"}) _
           :name _
           :email _
           (:projects {:with "params"}) #:project{:id _
                                                  :name _
                                                  :start-date _
                                                  :end-date _
                                                  :comleted? _}
           :contacts #:user{:id _}}

zhuxun205:06:39

Sure I lifted the keys this way, but when I start doing params, the query is already so much deviated from regular static data that it becomes a DSL, so I don't care the structural simplicity as much

souenzzo09:06:46

also, "eql query notation" isn't the same of "datomic selector notation" https://github.com/souenzzo/eql-datomic/

souenzzo09:06:47

in other words: it's OK to have many "query languages". each one has it's own benefits/facilities. all then can talk about the same AST