This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2020-06-16
Channels
- # announcements (8)
- # aws (28)
- # babashka (26)
- # beginners (125)
- # calva (18)
- # chlorine-clover (2)
- # cider (12)
- # cljs-dev (6)
- # cljsrn (4)
- # clojure (134)
- # clojure-europe (31)
- # clojure-italy (2)
- # clojure-nl (14)
- # clojure-uk (83)
- # clojurescript (81)
- # conjure (4)
- # cursive (2)
- # datomic (145)
- # emacs (13)
- # events (3)
- # figwheel-main (14)
- # fulcro (30)
- # graalvm (23)
- # graphql (15)
- # helix (21)
- # jackdaw (20)
- # juxt (1)
- # lambdaisland (4)
- # leiningen (2)
- # malli (12)
- # meander (22)
- # observability (22)
- # off-topic (27)
- # pedestal (3)
- # re-frame (12)
- # reitit (1)
- # releases (2)
- # rewrite-clj (3)
- # shadow-cljs (67)
- # spacemacs (7)
- # sql (1)
- # tools-deps (19)
- # unrepl (2)
- # xtdb (25)
Hello
What am I missing here?
user=> (d/q '[:find ?genre
#_=> :where [_ :movie/genre ?genre]] db)
[["Drama, Action"] ["Drama"] ["Sci Fi"]]
user=> (d/q '[:find ?e ?a
#_=> :where [(fulltext $ :movie/genre "Drama") [[?e ?a _ _]]]] db)
Based on what the manual says, this ought to work.
I’ve even tried a parameterised variety and didn’t get it to work. I am sure I am doing something stupid, just don’t know what it is
This is the query syntax: https://lucene.apache.org/core/2_9_4/queryparsersyntax.html
😄 That was it. Thanks
Anybody?
@srijayanth Try “Drama*” maybe?
We have a lambda ion handler that gets invoked daily with a CloudWatch event. It never gave problems, but since this last night it throws this exception:
No implementation of method: :->bbuf of protocol: #'datomic.ion.lambda.dispatcher/ToBbuf found for class: clojure.lang.PersistentArrayMap: datomic.ion.lambda.handler.exceptions.Incorrect
clojure.lang.ExceptionInfo: No implementation of method: :->bbuf of protocol: #'datomic.ion.lambda.dispatcher/ToBbuf found for class: clojure.lang.PersistentArrayMap {:cognitect.anomalies/category :cognitect.anomalies/incorrect, :cognitect.anomalies/message "No implementation of method: :->bbuf of protocol: #'datomic.ion.lambda.dispatcher/ToBbuf found for class: clojure.lang.PersistentArrayMap"}
at datomic.ion.lambda.handler$throw_anomaly.invokeStatic(handler.clj:24)
at datomic.ion.lambda.handler$throw_anomaly.invoke(handler.clj:20)
at datomic.ion.lambda.handler.Handler.on_anomaly(handler.clj:171)
at datomic.ion.lambda.handler.Handler.handle_request(handler.clj:196)
at datomic.ion.lambda.handler$fn__3841$G__3766__3846.invoke(handler.clj:67)
at datomic.ion.lambda.handler$fn__3841$G__3765__3852.invoke(handler.clj:67)
at clojure.lang.Var.invoke(Var.java:399)
at datomic.ion.lambda.handler.Thunk.handleRequest(Thunk.java:35)
Nvm, I found that it's the return value from the server handler
@raspasov - that didn’t work! 😞
does anyone have any experience running a dockerized peer server or peer application? There seems to be some networking requirement that I'm not fully understanding
outbound or inbound?
the host= or alt-host= in the transactor properties file must name the transactor and be resolveable by peers
gotcha. So it needs to be able to hit 172.17.0.1:4334, in my case?
then peers connect to storage, lookup the transactor name, and connect to the transactor on 4334
ah ok, keep going
so they need whatever the txor writes to storage to be resolveable to the transctor in whatever network they are in
interesting ... digesting
and it's a one way connection?
looks like from your logs that the peer can find storage, but either 172.17.0.1 doesn’t resolve to the txor, or it’s not allowed to connect to it, or the destination port isn’t open
there's no inbound from AMQ?
gotcha. hmm okay thank you. This gives me what I need to work on the puzzle
sorry back
I developed it locally and now I'm attempting to dockerize it
I was considering converting the app code to client process but I'd be back in the same boat with the peer server needing to be dockerized
interesting -- found this and I'm not seeing any extra exposed ports: https://github.com/frericksm/docker-datomic-peer-server
I'm trying to dockerize my peer application code, not the transactor
so using this peer server as inspiration -- sorry for the confusion
transactor is on my host machine right now
peer application code is on the docker
OH IT DOES EXPOSE 9001 !!! great catch
Also I notice it is going with "0.0.0.0" instead of the docker bridge network, interesting
gives me some stuff to play around with, great eye.
well worst case scenario I can rewrite the code with client API and try to talk to the peer server instead
have you tried this peer-server docker image and gotten it to connect to your transactor?
good thinking
I'll give that a shot
interesting, new error anyway
sorry about formatting 😕
now it's clearly psql that's pissed
yeah this is interesting, postgres wants a DIFFERENT host identifier than the transactor does. Transactor is connecting on 0.0.0.0, but postgres wants the docker bridge:
I think you are misunderstanding something. this error shows you got even less far along
you are setting up three things: 1) postgres, exposes 5432, needs routable hostname 2) transactor, exposes 4334, needs routable hostname, needs to connect to postgres. 3) peer; needs to connect to postgres, needs to connect to transactor
transactor.properties host= is what the transactor binds to for port 4334 (for peers to connect to it). Both host= and alt-host= are written to postgres for peers to discover
alt-host= is an alternative host/ip in case there’s some networking topology where what the transactor binds to isn’t the same thing other peers should connect to
If you're running on a single machine, all you need to do is 1) name the running containers, 2) use this name as the host name when connecting to a running container, and 3) run all containers on the same bridge network. Ports don't need to be exposed explicitly if they're only accessed by other containers on the same bridge network. All services should bind to 0.0.0.0
, not localhost. For example, we start the peer server like $DATOMIC_RELEASE/bin/run -m datomic.peer-server -h 0.0.0.0 -p 8998 -a "$DATOMIC_ACCESS_KEY_ID","$DATOMIC_SECRET_ACCESS_KEY" -d $DATOMIC_DATABASE_NAME,datomic:mem://$DATOMIC_DATABASE_NAME
Awesome, thank you! I'll be sure to use this when we transition to client API
What is the significance of port 9001 with regards to the peer server? Would this also apply to peer application code?
interesting. So this should not apply to peer application code? :thinking_face:
no. BTW that port is configurable. 9001 is just the one that docker image you were looking at happened to use
hrrrm
okay creeping closer.
.properties
file modifications:
alt-host=172.17.0.1
host=0.0.0.0
protocol=sql
#host=localhost
port=4334
peer connection string from docker:
datomic:
[main] ERROR org.apache.activemq.artemis.core.client - AMQ214016: Failed to create netty connection
java.net.UnknownHostException: 172.17.0.1
at java.base/java.net.InetAddress$CachedAddresses.get(InetAddress.java:797)
hmm good question. well this is what the transactor says:
$ ./run-transactor.sh
Launching with Java options -server -Xms1g -Xmx1g -XX:+UseG1GC -XX:MaxGCPauseMillis=50
Starting datomic:sql://<DB-NAME>?jdbc:, you may need to change the user and password parameters to work with your jdbc driver ...
System started datomic:sql://<DB-NAME>?jdbc:, you may need to change the user and password parameters to work with your jdbc driver
datomic:sql://<DB-NAME>?jdbc:
(postgres and the transactor are on the same host right now)
(but in the future they could be on different hosts)
hmm the transactor is outside, the postgres is also running in a docker probably on the default network
if the peer could connect to postgres on 172.17.0.1 to get the transactor IP, how is that an unknown host?
are you absolutely sure this is what the peer used? (d/connect "datomic:<sql://search?jdbc:postgresql://172.17.0.1:5432/datomic?user=datomic&password=datomic>")` ?
good question
I'll hardcode it just to be sure
(ns search.config
(:require [clojure.tools.logging :as log]))
;; todo -- parameterize
(def dockerized? (System/getenv "DOCKERIZED"))
(log/info (str "Dockerization detected:" dockerized?))
(def db-host (if dockerized?
"172.17.0.1"
"0.0.0.0"))
(log/info (str "Using host: " db-host))
(def db-uri (str "datomic: " db-host ":5432/datomic?user=datomic&password=datomic"))
(log/info (str db-uri))
(def db-version "0.6")
is technically what it's doingah, no wait, this is the same error facepalm
it just used that IP to talk to postgres, and got the host and alt-host info from there
yeah, it's nuts. It's def a networking thing, when I use the --network=host
option on the Docker everything works fine
ah I see so the observation here indicates that the peer process looked up the location of the transactor in storage (the host/alt-host settings) and then attempted to use those to connect to the transactor?
is there a way to "ping" the transactor?
because then I could test what the host is supposed to be from the Docker perspective
ah okay, I have a new hypthesis.
docker_gwbridge: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 172.17.0.1 netmask 255.255.0.0 broadcast 172.17.255.255
ether 02:42:b5:8e:77:df txqueuelen 0 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 2302 bytes 377328 (377.3 KB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
that's the docker bridge (as seen from host)postgres is dockerized, the peer service is dockerized ... the transactor is not
in my mind it makes sense then what you said about the baffling behavior of the transactor failing when it just talked to postgres to get the information
so perhaps if I dockerize the transactor it will alleviate this issue
I'm writing up the current state of affaris
re “ping” if you just want to test reachability from various machines use nc -z hostname port
Either there is a missing piece or I just need to learn more about networking/docker. Current setup: • transactor on docker0 • postgres on docker1 • peer application on docker2 observations: tranasctor can connect to psql
root@docker0:/opt/datomic-pro-1.0.6165# psql -h 172.17.0.1 -p 5432 -U datomic -W
Password:
psql (11.7 (Debian 11.7-0+deb10u1), server 12.3 (Debian 12.3-1.pgdg100+1))
WARNING: psql major version 11, server major version 12.
Some psql features might not work.
Type "help" for help.
datomic=>
peer application can connect to psql
:
root@c4c329971ae7:/app# psql -h 172.17.0.1 -p 5432 -U datomic -W
Password:
psql (11.7 (Debian 11.7-0+deb10u1), server 12.3 (Debian 12.3-1.pgdg100+1))
WARNING: psql major version 11, server major version 12.
Some psql features might not work.
Type "help" for help.
datomic=>
current error message from peer application:
[main] INFO search.config - Dockerization detected:true
[main] INFO search.config - Using host: 172.17.0.1
[main] INFO search.config - datomic:
[main] INFO datomic.domain - {:event :cache/create, :cache-bytes 2086666240, :pid 6575, :tid 1}
[main] INFO datomic.process-monitor - {:event :metrics/initializing, :metricsCallback clojure.core/identity, :phase :begin, :pid 6575, :tid 1}
[main] INFO datomic.process-monitor - {:event :metrics/initializing, :metricsCallback clojure.core/identity, :msec 0.47, :phase :end, :pid 6575, :tid 1}
[main] INFO datomic.process-monitor - {:metrics/started clojure.core/identity, :pid 6575, :tid 1}
[clojure-agent-send-off-pool-0] INFO datomic.process-monitor - {:AvailableMB 3880.0, :ObjectCacheCount 0, :event :metrics, :pid 6575, :tid 13}
[clojure-agent-send-off-pool-0] INFO datomic.kv-cluster - {:event :kv-cluster/get-pod, :pod-key "pod-catalog", :phase :begin, :pid 6575, :tid 13}
[clojure-agent-send-off-pool-0] INFO datomic.kv-cluster - {:event :kv-cluster/get-pod, :pod-key "pod-catalog", :msec 8.89, :phase :end, :pid 6575, :tid 13}
[main] INFO datomic.peer - {:event :peer/connect-transactor, :host "172.17.0.1", :alt-host nil, :port 4334, :version "1.0.6165", :pid 6575, :tid 1}
Execution error (ActiveMQNotConnectedException) at org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl/createSessionFactory (ServerLocatorImpl.java:787).
AMQ119007: Cannot connect to server(s). Tried with all available servers.
transactor config:
host=172.17.0.1
#host=0.0.0.0
protocol=sql
#host=localhost
port=4334
sql-url=jdbc:
sql-user=datomic
sql-password=datomic
As I write that up I think the host might be wrong. host is supposed to be host of transactor, not sql, yeah?
that was it 🙂 @favila you are THE MAN!! thank you!!!
I should probably do a blog thing about this for the benefit of others cause this was a little tricky
I should probably do anything productive with my time at all 😓
so that's a new error, the UnknownHostException
@favila if there was an equivalent of upvotes or reddit gold for this slack channel I'd be throwing them at you, thank you
Is qseq
not available in datomic.api as suggested here?
https://docs.datomic.com/on-prem/clojure/index.html#datomic.api/qseq
This fn is brand new and only available on the latest version (yours is not) https://docs.datomic.com/on-prem/changes.html
Earlier I asked a question about EQL in #fulcro but I figured it might concerns datomic users as well so I felt compelled to drop it here as well. Apologize for the long post. 🙏 --- original post --- I have a question about the design of EQL which I'm not sure this is the right place to discuss. However, I feel EQL is something fulcro users deal with heavily so I think it might be valuable for me to hear your opinions: There are couple of quirks that made me wonder why EQL was designed like it is. Let's take a look at a typical EQL -- the kind of which probably show up in your app hundreds of times
[:user/id
:user/name
:user/email
{:user/projects [:project/id
:project/name
:project/start-date
:project/end-date
:project/completed?]}
{:user/contacts [:user/id]}]
EQL claims to have "the same" shape as the returned data. That's awesome! However, why don't we go a step further? Consider the return value of the above query:
{:user/id #uuid "..."
:user/name "Fred Mertz"
:user/email ""
:user/projects [{:project/id #uuid "..."
:project/name "Mertz and Kurtz"
:project/start-date #inst "..."
:project/end-date #inst "..."
:project/completed? true}]
:user/contacts [{:user/id #uuid "..."}
{:user/id #uuid "..."}]}
Why wasn't EQL designed to completely mimic that structure:
{:user/id _
:user/name _
:user/email _
:user/projects [{:project/id _
:project/name _
:project/start-date _
:project/end-date _
:project/completed? _}]
:user/contacts [{:user/id _}]}
Or, if the explicitness of pluarity is not desired here:
{:user/id _
:user/name _
:user/email _
:user/projects {:project/id _
:project/name _
:project/start-date _
:project/end-date _
:project/completed? _}
:user/contacts {:user/id _}}
The immediate benefit is that now I can use the map namespace syntax to make it much more succinct and DRY (and easy on the eye):
#:user{:id _
:name _
:email _
:projects #:project{:id _
:name _
:start-date _
:end-date _
:comleted? _}
:contacts #:user{:id _}}
IMHO many important semantics are much better aligned this way. For example, in a return value, the order of the keys in a level of map should not matter, and there should not be duplicated keys. However, EQL uses an ordered collection (vector) to denote the keys, the semantics of which has a sense of order while ensures no uniqueness. Also, it feels like in EQL maps are used in place of pairs. I understand that Clojure doesn't have a built-in literal for pairs so it makes sense to use maps, but maps seem to be a poor fit for this role -- here they are only allowed to have one key-value pair, and pushes the key into the next level when it should really belong to the outer level. I feel that the ad-hoc-ish design not only misses mathematical simplicity but also make everything unnecessarily complex. If I were to write a function to retrive all root level keys given a EQL (which should have been trivial), the implementation would be a few lines unnecessarily longer since I need to consider those ref-type attributes. If I am using Emacs to manually write a test example given an EQL, I am doing lots of unnecessary work changing brackets into braces and splicing sexp's. That being said, my exposure to Clojure and Datomic/Pathom/Fulcro is limited, and I truly want to hear if there are reasons why EQL was designed the way it is rather than my intuitive version. I apologize my above arguments spiralled into a small rant.
Not a complete answer but some historical context: eql is based on pull expressions, which existed prior to the namespaces map features you mention
Key renaming, limits, defaults, etc. some of that you can smuggle into the value slot, but you need a plan for nested maps
I dont think that the pull notation is import and do not think that one is better then other. We can have many notation/representations and talk about the same AST.
@favila I can support params the same way the current EQL does -- on the key slot, no?
#:user{(:id {:with "params"}) _
:name _
:email _
(:projects {:with "params"}) #:project{:id _
:name _
:start-date _
:end-date _
:comleted? _}
:contacts #:user{:id _}}
Sure I lifted the keys this way, but when I start doing params, the query is already so much deviated from regular static data that it becomes a DSL, so I don't care the structural simplicity as much
also, "eql query notation" isn't the same of "datomic selector notation" https://github.com/souenzzo/eql-datomic/