Fork me on GitHub
#datomic
<
2017-03-19
>
yonatanel01:03:03

@riadvargas What about using clojure.string/lower-case instead of regex?

sova-soars-the-sora03:03:31

@val_waeselynck thanks val, I think I have a timestamp attrib on my data, that will come in handy.

isaac11:03:14

why datomic client query does not support bindings? eg: [find .... :where [(quot ?time 60) ?minute]]

isaac11:03:31

got this error:

{:cognitect.anomalies/category :cognitect.anomalies/incorrect, :cognitect.anomalies/message "The following forms do not name predicates or fns: (quot?)", :dbs [{:database-id "58ce6a9d-287a-434a-b282-0107\
237fc4f9", :t 1003, :next-t 1008, :history false}]}

favila13:03:27

@isaac Error message is about quot not binding. Try fully qualifying it: clojure.core/quot

isaac13:03:05

@favila I tried, but still got the same error

favila13:03:41

Exactly the same?

favila13:03:37

Do you maybe have a typo? It says it can't find quot? (note the question mark)

isaac13:03:43

yeah,

{:cognitect.anomalies/category :cognitect.anomalies/incorrect, :cognitect.anomalies/message "The following forms do not name predicates or fns: (clojure.core/quot?)", :dbs [{:database-id "58ce6a9d-287a-434a-b282-0107\
237fc4f9", :t 1003, :next-t 1008, :history false}]}

isaac13:03:05

yeah, I also tried mod, it’s neither not working

isaac13:03:08

it is strange that the same query working in peer

favila14:03:37

Could you have copy+pasted a control character or something? The question mark is still suspicious.

isaac14:03:40

(<!!
 (client/q conn {:query '[:find ?e ?tx
                          :where
                          [?e :db/txInstant]
                          [(quot ?e 2) ?tx]]
                          :args [(client/db conn)]}))

nickik18:03:18

Has anybody explored datomic transactor in kubernetes?

nickik18:03:19

Could kubernetes handle the hot failover?

devth18:03:36

i've played with it but haven't fully productionalized yet. HA would simply be a Deployment with 2 replicas, right?

nickik18:03:20

Thats what I am hoping.

devth18:03:38

i'm not totally sure what the readiness and liveness probes would look like

nickik18:03:25

Im a real beginner with all this kubernetes stuff so I have no clue. I think for starters I will put the transactor and AWS directly and only deploy the peers in the kubernetes. That should be easy.

devth18:03:52

yes, peers on k8s should be straightforward

nickik18:03:02

Maybe put memcachd in the cluster to reduce the pressure on dynamodb

nickik18:03:42

Putting everything into the cluster seems hard, all the stateful stuff in kubernetes is confusing as hell.

devth18:03:59

stateful stuff only recently became supported with StatefulSets

devth18:03:11

there's a (very active) k8s slack btw

nickik18:03:34

The question is with what storage backend it would be best used.

devth18:03:34

the transactor isn't actually very stateful (besides caches i assume) - mainly just the storage

devth18:03:38

could work with a variety of storages

devth18:03:17

if on aws dynamo makes sense. if gcp, cloud sql (mysql or postgres) maybe

nickik18:03:31

Work yes. Easy to set up and run is the question.

nickik18:03:33

I will use whats native to the cloud right now, but I think it would be much cooler to have setup that is all in kubernetes and can be moved to different places.

devth18:03:11

you mean including storage?

devth18:03:50

i'm running a postgress + transactor + peers all in a k8s cluster but like i said, not productionalized. been a few months since i played with it.

nickik18:03:56

Yeah, that would be fun.

devth18:03:18

postgres is a statefulset with attached ssd persistent disks

nickik18:03:20

Is that opensource? I have been thinking about that setup as well.

devth18:03:48

i haven't open sourced it .. just stuff i was playing with

nickik18:03:27

Could you show me the container img for the transactor?

nickik18:03:51

I am also just playing around, so it does not need to be production quality.

devth18:03:34

i used https://github.com/pointslope/docker-datomic to build a datomic docker image

devth18:03:59

my Dockerfile:

FROM pointslope/datomic-pro-starter:0.9.5390

ADD restore-db.sh /usr/local/bin/restore-db

ENV MYSQL_CONNECTOR_URL 

# from 
# download mysql connector ()
# and put it in datomic's lib dir

RUN \
  wget $MYSQL_CONNECTOR_URL -O mysql-connector-java-5.1.24.zip \
  && unzip mysql-connector-java-5.1.24.zip \
  && mv mysql-connector-java-5.1.24/mysql-connector-java-5.1.24-bin.jar $DATOMIC_HOME/lib \
  && rm -fr mysql-connector-java-5.1.24 mysql-connector-java-5.1.24.zip

ENTRYPOINT ["bin/transactor", "-Xmx4g", "-Xms4g"]

devth18:03:37

oh right, i remember now i switched to Cloud SQL (on GCP)

devth18:03:53

so scratch that postgres part.. still - it'd be relatively straightforward

devth18:03:25

if you want to get into K8S you should check out Helm Charts for canonical ways of running standard open src projects https://helm.sh/

nickik18:03:50

Cool. I have tried that docker img, but I had some problems with it. Maybe I did something wrong. I guess I will try again.

nickik18:03:05

I have heard of helm, but I was not sure what it was doing for me.

devth18:03:26

basically you can helm install postgres or helm install zookeeper (those aren't the exact commands)

nickik18:03:35

What I would like is a oneclick deployment thing that gives me storge with datomic and all I have to put in is my clojure program.

devth18:03:05

it packages up the deployment(s), service(s), ingress, persistent disks, etc into a single logical unit

nickik18:03:06

So that I get a hole cluster with storge, monitoring around my Clojure server.

devth18:03:19

it could do exactly that

devth18:03:29

but no Datomic chart exists

devth18:03:35

anyone could make one though

nickik18:03:49

Nice. I need to get into that. There are just so many different new developments in this space.

devth18:03:12

yes. very active, but k8s seems to be beating everyone πŸ™‚

nickik18:03:58

Thats why I am jumping on the bandwagon as well. Im trying to set up a cluster in AWS and doing my own projects inside of it.

devth18:03:17

here's the site for all the stable Helm charts https://kubeapps.com/

devth18:03:30

includes both mysql and postgres

nickik18:03:28

Those do persistence as well?

devth18:03:46

haven't checked but i doubt they'd be recommended if they didn't

devth18:03:22

that gives you a persistent volume

devth18:03:38

cloud agnostic (this is why k8s is awesome)

nickik18:03:47

but its probably overkill.

devth18:03:01

looks cool. not familiar

nickik18:03:48

I thought if everything is so awesome and automatic, why can't I get a full setup for my 5 line clojure ring web server πŸ™‚

nickik18:03:11

Thanks for your help.

devth18:03:24

sounds like a good blog post πŸ™‚

nickik18:03:48

Well, I first need to setup the cluster to deploy a blog πŸ™‚

devth18:03:55

just use a managed cluster

nickik18:03:58

I don't actually have one

nickik18:03:18

like on google cloud?

devth18:03:18

aws: EC2 Container Service gcp: Google Container Engine

devth18:03:24

even Azure has one

nickik18:03:49

Yeah, I have started with that as well. But I also wanted to explore doing more low level stuff myself.

devth18:03:05

i haven't tried setting up my own cluster but there's a lot automation i've seen around it

devth18:03:09

check CoreOS' Tectonic

nickik18:03:23

Thats what I am setting up.

devth18:03:31

good luck πŸ™‚

nickik18:03:31

A friend of mini is hosting DNS for me, but I cant continue until he makes some changes. So in the meantime I am looking into deploying datomic once I have it running.

nickik19:03:39

@devth Another question. In the transctor file, there is the holt and alt-host. How did you configure those while running in kubernetes.

devth19:03:15

host: 0.0.0.0
alt_host: datomic
where datomic was the name of the service i.e. dns

nickik19:03:52

@devth So that is the kubernetes service name? Is that configured in the kubernetes yaml file?

nickik19:03:40

ok. that makes sense. The service is the one that routes to the transactor pod.

devth19:03:44

e.g.

apiVersion: v1
kind: Service
metadata:
  name: datomic
  namespace: {{.Values.global.namespace}}
  labels:
    app: datomic
    tier: db
    release: {{ .Release.Name | quote }}
spec:
  ports:
    - port: {{.Values.datomic_port_1}}
      name: datomic1
    - port: {{.Values.datomic_port_2}}
      name: datomic2
    - port: {{.Values.datomic_port_3}}
      name: datomic3
  selector:
    app: datomic
    tier: db
(the template syntax is because i'm doing it as a Helm Chart template - replace with real values)

nickik19:03:32

So you actually have a Helm package?

devth19:03:44

yeah just a local one. they're very easy to create

nickik19:03:57

Thanks for the file. I think Im slowly getting the hang of how everything connects.

devth19:03:05

basically:

Chart.yaml with some metadata
values.yaml with any vars you want
templates/k8s-yaml-files-here.yaml

nickik19:03:06

Cool. Make sense.

devth20:03:04

maybe i'll write it up in a blog post if there's interest.

nickik20:03:13

Whats the namespace usually?

devth20:03:35

depends on your conventions. i was using env as my namespaces, e.g. dev, prod, staging ...

nickik20:03:36

I am defiantly interested in all things "clojure stack" on kubernetes.

devth20:03:39

if you don't care just use default

devth20:03:01

others use teams as the namespace

nickik20:03:02

Why are there 3 ports?

devth20:03:12

standard datomic ports

devth20:03:23

don't remember their purposes offhand

nickik20:03:31

ah, I did not remember there were 3.

devth20:03:53

in my values.yaml i'm defining them as:

datomic_port_1: 4334
datomic_port_2: 4335
datomic_port_3: 4336

nickik20:03:21

Whats your opinion on using peer or client api.

nickik20:03:33

I am still a little unsure on what to use when.

devth20:03:59

always prefer peer when possible (i.e. when your app is on the jvm)

devth20:03:10

client api won't be as performant iiuc (i haven't used it)

devth20:03:19

# The dev: and free: protocols typically use three ports
# starting with the selected :port, but you can specify the
# other ports explicitly, e.g. for virtualization environs
# that do not issue contiguous ports.

# h2-port=4335
# h2-web-port=4336

devth20:03:31

there's some docs on ports in the transactor props sample

nickik20:03:58

Ok. I can figure that stuff out. I just tought there might be another reason.

devth20:03:51

key to understanding clj on k8s is really just docker. you need to understand two mostly-orthogonal pieces: 1. how to containerize a clojure app 2. how to run containers on k8s there's nothing really clojure-specific about #2

nickik20:03:23

Sure. Datomic was more of a question for me. Clojure itself should be straight forward (I might try with OSv but that is besides the point)

nickik20:03:12

I had my eye on that.

nickik20:03:20

But pedestal already provides a docker file

nickik20:03:24

and one for osv

devth20:03:05

that repo is what produces the official image i linked (https://hub.docker.com/_/clojure/)

devth20:03:38

but yeah, many ways to do it. should be straightforward any way you choose

nickik20:03:13

I really want to have a kubernetes and datomic setup, so that I can have quick devlopment process where I can automatically deploy the apps into that cluster.

devth20:03:39

that's what i wanted too πŸ™‚ i got it working with GitLab repo + GitLab CI

devth20:03:04

you can also develop locally against a remote transactor in a k8s cluster, if you choose

nickik20:03:08

I am sort of betting on the CoreOS stuff and I am using Quay, but I have only just registered.

devth20:03:21

quay's been good IME

nickik20:03:30

It strange, I tried to hock a repo up to a github repo and it did not work.

nickik20:03:34

But I will figure it out πŸ™‚ Nothing as easy as you hope it is.

devth20:03:04

it's only easy after you go through the pain of figuring it out the first few times

nickik20:03:32

With the Quay thing its just strange because there is little you can do wrong "create new repo" -> "add github hook" done.

nickik20:03:49

But it fails to add the github hook. Authentication has worked.

devth20:03:59

oh, weird. maybe a bug

nickik20:03:37

Yeah, I was thinking about contacting support, but I did not want to seem like a idiot. So I will just wait until tomorrow and try again.

devth20:03:27

can also use your cloud's docker registry

devth20:03:39

but you'd have to push the image there manually or setup your own CI

nickik20:03:34

I want to try this security scanning stuff, that sound cool.

nickik20:03:22

Also I sort of like how CoreOS approches a lot of stuff, so if I really like quay I would not mind paying for it.

devth20:03:24

i think gcp's registry does that

devth20:03:38

yeah, i like their stuff

nickik20:03:41

I just love that they work on trusted boot stuff.

nickik20:03:13

I think even docker has added something as well.

devth20:03:19

only on paid plans πŸ˜ͺ

fenton21:03:05

Here is a datomic tutorial I worked on... Aimed at the beginner... Feedback appreciated: https://www.reddit.com/r/Clojure/comments/5zu1oc/my_datomic_tutorial_feedback_sought/