Fork me on GitHub
#onyx
<
2017-04-06
>
devth00:04:36

alternatively, if i only have a single Onyx node would it be storing any data (PII specifically) in ZK?

devth00:04:50

if it's just checkpoint/state data then no ssl might be ok.

lucasbradstreet00:04:15

@devth we don’t currently expose a way to do this, but nippy does support encryption, so we would need to expose it somehow https://github.com/onyx-platform/onyx/blob/d1d0db97332b52659947bf88499f7228b9f5d2f0/src/onyx/compression/nippy.clj#L19

lucasbradstreet00:04:36

Even with a single Onyx node, you will still need to place the job data in ZK

lucasbradstreet00:04:53

but that’s purely the job definition that you submit, not the data that is pushed through Onyx

devth00:04:03

right, so I might be ok

devth00:04:12

i'm streaming Datomic -> ElasticSearch

devth00:04:28

that means basis-t would be in ZK right? but never entities/attributes/values

devth00:04:21

still interested in encryption but this might allow me to backlog it

devth00:04:14

also: any plans to make the coordination pluggable?

devth00:04:22

e.g. consul, etcd, ...

lucasbradstreet00:04:55

No plans currently. I’d accept a PR, but it’s too much work for not enough gain for us, especially considering that I’d want to jepsen test it

devth00:04:16

ok. ZK is great from my experience but lack of SSL and the weirdness around 3.5.x release makes me a little nervous :thinking_face:

devth00:04:37

not in any released version. just 3.5.x (alpha, beta)

devth00:04:48

which means there's no docker image. (i had to create my own)

lucasbradstreet00:04:04

Ah wow. That’s been in the works for a while.

devth00:04:05

also that's only client-to-zk ssl. not server-to-server ssl

devth00:04:16

yep. years 😴

Travis00:04:22

@lucasbradstreet what check pointing storage systems are supported/planned for 0.10 . I know s3

Travis00:04:20

Think I used the right terms there, lol

lucasbradstreet00:04:22

Only s3 currently. HDFS and google's S3 equivalent would be the nice to have, but would need to be supplied via PR or as contracted work.

devth00:04:19

if you need an S3-compatible service but aren't using AWS you can run a Minio (we do in GCP) https://github.com/minio/minio

Travis00:04:17

Cool, yeah our plan is in aws on mesos/dcos. I have seen minio

Travis00:04:22

Just curious because we have an odd ball use case for an on prem solution

devth20:04:27

is there an endpoint i can monitor or a proc i can run to ensure my onyx infra is healthy?

michaeldrogalis20:04:32

onyx-peer-http-query is a must.

devth20:04:50

@michaeldrogalis thanks. not sure why google didn't find that :thinking_face:

devth21:04:45

trying to figure out why after adding [org.onyxplatform/onyx-peer-http-query "0.9.15.1"] to my dependencies, lein test now errors with:

java.lang.ClassNotFoundException: io.aeron.CommonContext
i included onyx.http-query in my test namespace. if i don't include it i get the error:
clojure.lang.ExceptionInfo: No http query server implementation found. Did you include org.onyxplatform/onyx-node-query in your dependencies and require onyx.http-query in your peer bootup namespace?

lucasbradstreet21:04:47

what version of onyx are you using? 0.9.15?

lucasbradstreet21:04:25

weird. Looks like a problem with it not bringing in the right aeron dependency

devth21:04:37

ah core was on 0.9.10. all the other plugins were on 0.9.15

devth21:04:48

yep that caused it

devth21:04:06

thx. guess i need to make sure those are all kept in sync