This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2021-06-21
Channels
- # announcements (5)
- # babashka (81)
- # beginners (26)
- # calva (6)
- # cider (7)
- # clojure (26)
- # clojure-czech (1)
- # clojure-europe (19)
- # clojure-nl (4)
- # clojure-spec (5)
- # clojure-uk (21)
- # clojuredesign-podcast (2)
- # clojurescript (19)
- # conjure (6)
- # cursive (13)
- # datomic (2)
- # depstar (1)
- # editors (2)
- # graalvm (25)
- # honeysql (5)
- # jackdaw (4)
- # jobs (5)
- # lsp (8)
- # malli (13)
- # music (1)
- # polylith (3)
- # practicalli (1)
- # releases (1)
- # remote-jobs (2)
- # sci (10)
- # shadow-cljs (5)
- # sql (14)
- # tools-deps (25)
- # xtdb (65)
Hi. I have a requirement to periodically push data into kafka topic in batches. If its a one time run, i could have directly run it in with bb script only construct the data. But since i need to run on a periodic basis, i need to push to kafka through bb script. But i keep running into dependency problems with every clojure jar i use. The java import
(import '[org.apache.kafka.clients.producer KafkaProducer ProducerRecord])
also didnt help. Can anyone help me out with this. Is it possible to use babashka for kafkaHey @U016GSFNDNY so the lib you're using is written in Java and bb is meant for primarily interpreting clojure code and not java byte codes. Some of the java classes have been baked into bb builds for convenience and to help out libs but not the Kafka one.
The way you're expecting to use wont be possible in bb unless we compile bb with it being baked in. I had a similar need once and gotten around by shelling out to https://github.com/deviceinsight/kafkactl and sending things. Not so smooth but gets the job done
I tried adding various clojure dependencies too @U7ERLH6JX. It gets errored out for missing dependencies
Most of the kafka libs are wrappers around the canonical java lib and we need to bake that in to make it work like you want
Thanks @U7ERLH6JX. That helps.
as far as i know, none of the popular libs would work, unless they somehow implement in pure clojure, still there could be issues
Yes. thats exactly what happened. Thanks for your help. I will try to approach with a different solution
kafkactl
with babashka.process
sort of solved my usecase, which was quite simple though 😄
you could also implement it as a Pod: https://book.babashka.org/#pods Would be a really useful one! 😄
There is a pod here: https://github.com/tzzh/pod-tzzh-kafka But it doesn't have any releases yet. @UJADF897F?
ah nice!
nice. thanks @U04V15CAJ. will try it out
yeah @U04V15CAJ there is no manifest available for this
How did you try to compile it @U016GSFNDNY?
just cloned and ran go build
seems to compile for me
what errors do you see?
also what's your go version
?
#
In file included from ..\..\go\pkg\mod\\confluentinc\[email protected]\kafka\00version.go:24:
./librdkafka/rdkafka.h:83:10: fatal error: sys/socket.h: No such file or directory
83 | #include <sys/socket.h> /* for sockaddr, .. */
| ^~~~~~~~~~~~~~
compilation terminated.
so it seems librdkafka, the c lib powering the confluent connector isnt supported on windows
could you try the steps mentioned here: https://github.com/confluentinc/confluent-kafka-go/issues/128
basically you need to have MinGW gcc
installed
if thats there, the compile may work
alternatively you could try using Windows Subsystem Linux
Hey guys yeah sorry there is no release as it's using CGO (ie C extensions for go) and so it's harder to cross compile it (I didn't know about the issues about confluent-kafka-go on windows but I guess that makes it even harder)
@UJADF897F We have a pod which is using CGO here: https://github.com/babashka/pod-babashka-go-sqlite3 If you want, you can copy the configs :)
is this intended behavior?
{:tasks {repro {:doc (str "This tasks repro is: " "showing a doc string")
:task (shell "ls")}}}
❯ bb tasks
The following tasks are available:
repro (str "This tasks repro is: " "showing a doc string")
I'm trying to build a doc string from some default values, but it interprets the value of :doc
literally
Currently the docstring isn't evaluated, except those on functions, if you refer to them with :task foo.bar/baz
yeah, it would come in handy, but looking at most commandline tools that support 'actions', the overview doesn't give a lot of detail, and only does so when you would bb repro --help
thanks for babashka tasks by the way, it's super awesome for writing dev tooling
I’m looking at this libary to do consistent hashing https://github.com/replikativ/hasch. The lib has a platform ns: (:import java.io.ByteArrayOutputStream java.nio.ByteBuffer java.security.MessageDigest) . Are these dependencies under consideration?
Or, any more bash-like idea for consistent hashing? I’m simply trying to cache which rows/objects/maps have been already processed in a data pipeline.
digging into the code, the lib uses protocols as well: that’s the larger hurdle, right?
If I hunt in here: https://github.com/babashka/babashka/blob/master/src/babashka/impl/classes.clj and don’t e.g. ByteArrayOutputStream then it’s not in bb, yet. Correct?
ah, but it is there. https://github.com/babashka/babashka/blob/d90f4e7bacb0af902764f66be4ba0255528d795c/src/babashka/impl/classes.clj#L93 Well, thanks for your help anyway!
borkdude@MBP2019 /tmp $ export BABASHKA_CLASSPATH=$(clojure -Spath -Sdeps '{:deps {io.replikativ/hasch {:mvn/version "0.3.7"}}}')
borkdude@MBP2019 /tmp $ bb -e "(use 'hasch.core)"
----- Error --------------------------------------------------------------------
Type: clojure.lang.ExceptionInfo
Message: Unable to resolve classname: java.io.FileInputStream
Location: hasch/platform.clj:162:24
Phase: analysis
----- Context ------------------------------------------------------------------
158: java.io.File
159: (-coerce [f md-create-fn write-handlers]
160: (let [^MessageDigest md (md-create-fn)
161: len (.length f)]
162: (with-open [fis (java.io.FileInputStream. f)]
^--- Unable to resolve classname: java.io.FileInputStream
163: (encode (:binary magics)
164: ;; support default split-size behaviour transparently
165: (if (< len split-size)
166: (let [ba (with-open [out (java.io.ByteArrayOutputStream.)]
167: ( fis out)
yeah, you can clone babashka and invoke it using clojure -M:babashka/dev ...
when you make an alias for it, that's how I do it
$ clojure -M:babashka/dev -e "(use 'hasch.core) (uuid5 (edn-hash \"hello world\"))"
#uuid "1227fe0a-471b-5329-88db-875fb82737a8"
I will commit to master. New binary will appear in #babashka-circleci-builds in a few minutes
@U068BQFJ9 ok, it's there now and it seems to work.
$ bb -e "(use 'hasch.core) (edn-hash [\"hello world\" {:a 3.14} #{42} '(if true nil \f)])"
(245 139 93 212 32 95 155 217 230 42 204 224 210 124 22 156 241 230 65 199 21 108 160 143 225 185 228 16 141 66 80 96 35 189 202 198 252 99 12 214 23 106 43 113 138 129 16 131 7 110 102 1 55 15 4 148 118 187 201 28 144 90 192 21)
@U068BQFJ9 To be clear, you can just download the binary from appveyor and test it out, right now
You don't have to wait for a release and you can do some testing pre-release if you want
I have a windows server, so I’ll check that later. Safe to assume windows binary always works out?