Fork me on GitHub
Slack Archive

This page is not created by, affiliated with, or supported by Slack Technologies, Inc.

2020-09-01

Channels

#xtdb
<
2020-09-01
>
Nejc11:09:50

Hi, I'm looking into Kafka - Connect, the docs says "Download the connector from Confluent hub" but I can only find old alpha version there. no juxt-kafka-connect-crux-20.08-1.10.1-beta.zip as the documentation suggests. Is there any other source where I can get that connector?

refset11:09:05

Hi, thanks for checking out the connector - sorry we've not uploaded a more recent version! You should be able to build the latest version without much trouble if you have the Crux repo cloned See: https://github.com/juxt/crux/tree/master/crux-kafka-connect#packaging-the-connector Please let us know if you have any trouble 🙂

Nejc11:09:54

Yes I have it.. let me try

Nejc09:09:40

@U899JBRPF I ended up writing my own sink connector in Java.. but I encountered the following problem when starting the connector: 

[2020-09-02 09:00:08,906] ERROR WorkerSinkTask{id=CruxSink-0} Task threw an uncaught and unrecoverable exception (org.apache.kafka.connect.runtime.WorkerTask)
Syntax error compiling new at (taoensso/nippy/compression.clj:43:15).
...
...
Caused by: java.lang.SecurityException: sealing violation: package org.tukaani.xz is sealed
	at java.net.URLClassLoader.getAndVerifyPackage(URLClassLoader.java:400)Caused by: java.lang.SecurityException: sealing violation: package org.tukaani.xz is sealed
	at java.net.URLClassLoader.getAndVerifyPackage(URLClassLoader.java:400)
....
mvn dependency tree is stating that org.tukaani.xz is included only once via crux-cor--> nippy --> xz Any idea how to get around this issue?

refset09:09:12

Hmm! Is this for 1.10.1 ?

Nejc09:09:41

<crux.version>20.08-1.10.1-beta</crux.version>        <clojure.version>1.10.1</clojure.version>

refset09:09:17

thanks, I'll have a look. We had a similar looking issue earlier in the year, so I need to read my notes

đź‘Ť 3
Nejc09:09:36

thank you

Nejc09:09:25

I excluded tukaani.xz package from crux-core dependency and I got around this. Not sure if this will cause any other problems but at the moment it seems that Sink is working

refset09:09:05

cool, thanks for the update. I will keep you posted also 🙂

Ivan Fedorov20:09:46

Also in Kafka-Connect / Confluent area – I can’t understand yet how to estimate how much I will need pay for their awesome storage and how much storage I will need.

refset21:09:20

Hey! GCP is usually cheaper, from what I've seen. Unless you're doing something super serious (requiring multi-AZ HA) the Basic service is probably good enough:

refset21:09:06

Why they haven't created a calculator themselves...I have no idea 🙂

refset21:09:12

When you're using Confluent for Crux then you can assume that all data store will need to be retained indefinitely

Ivan Fedorov05:09:06

Thanks for pointing out pricing section!

🙂 3
Ivan Fedorov20:09:45

I’ve skimmed the surface on Confluent website and didn’t find a “calculator” for that. Maybe if you could include some estimates on opencrux website that would improve the funnel.

đź‘Ť 3