Hello! 😜

Hello there, im having trouble with Aeron not releasing <udp://127.0.0.1:40200> when restarting my app from the repl, im using the embedded media driver in a dev setup, is there anything beyond calling the shutdown-env needed ?

The aeron media driver is started up as part of the peer group. Try shutting down the peer group as well as the env?

im calling shutdown-onyx before shutdown-env

i mean, shutdown-peer, shutdown-peer-group shutdown-env in that order.

Ah ok. Hmm. It should release it then

ah, wait a sec, it might have todo with clojure.tools.namespace.repl/refresh

Yeah, if you're repl refreshing beforehand that would be a problem

Hello, are there any plans on HDFS plugin 🙂 ?

or some pointers on how to kickstart one ?

It's next on our list but we might not get to it for a bit. Are you interested in input or output or both? Part of the problem is that we will be changing our input plugin interface soon.

Output is easier and won't be changing much

I’m looking for output to HDFS

want to try out an ingestion job from Kafka to HDFS

Ah, k, cool. I don't have anything to kickstart you at the moment but I'd be happy to advise. Alternatively, I am about to do some HDFS work in the next week and might be able to bang out an output plugin while I'm at it

@drankard: I can point you to something a bit later

@lucasbradstreet: cool, I’ll see if I can wrap my head around plugin-dev

not in a hurry though

what would be a good example to get started ? I thought s3 could be interesting one ...

S3 could be a good place to start

ok, thanks! will try reading up some code

AFAICS, s3 is output only .. right ?

@vijaykiran: I've developed output plugin and it's not that hard actually

cool, that’s encouraging 🙂

@v.solovyov: or are you saying you already have hdfs-out plugin ?

I wrote onyx-http output plugin, no hdfs from me 🙂

@vijaykiran: S3 is output only for now, yes

I touched HDFS only once, and it was more than 1.5 years ago

🙂 ok, I’ll come back with questions soon then

Hoping to expand the number of plugins greatly later in the year, and to especially cover more batch-oriented mediums like HDFS and S3. That's the area that we're lacking most right now. Currently working through the next release to enable more rapid plugin development.

Speaking of S3, been using this lately, and I'm pretty over the moon about how well it works. https://github.com/jubos/fake-s3

It's especially for when I'm working offline.

thats weird, i was just thinking “how can i test s3 offline"

@drewverlee: 🙂