Fork me on GitHub
#onyx
<
2016-07-27
>
Travis01:07:58

Hello! 😜

drankard12:07:08

Hello there, im having trouble with Aeron not releasing <udp://127.0.0.1:40200> when restarting my app from the repl, im using the embedded media driver in a dev setup, is there anything beyond calling the shutdown-env needed ?

lucasbradstreet12:07:05

The aeron media driver is started up as part of the peer group. Try shutting down the peer group as well as the env?

drankard12:07:23

im calling shutdown-onyx before shutdown-env

drankard12:07:27

i mean, shutdown-peer, shutdown-peer-group shutdown-env in that order.

lucasbradstreet12:07:02

Ah ok. Hmm. It should release it then

drankard12:07:35

ah, wait a sec, it might have todo with clojure.tools.namespace.repl/refresh

lucasbradstreet12:07:45

Yeah, if you're repl refreshing beforehand that would be a problem

vijaykiran12:07:09

Hello, are there any plans on HDFS plugin 🙂 ?

vijaykiran12:07:15

or some pointers on how to kickstart one ?

lucasbradstreet13:07:23

It's next on our list but we might not get to it for a bit. Are you interested in input or output or both? Part of the problem is that we will be changing our input plugin interface soon.

lucasbradstreet13:07:45

Output is easier and won't be changing much

vijaykiran13:07:53

I’m looking for output to HDFS

vijaykiran13:07:14

want to try out an ingestion job from Kafka to HDFS

lucasbradstreet13:07:31

Ah, k, cool. I don't have anything to kickstart you at the moment but I'd be happy to advise. Alternatively, I am about to do some HDFS work in the next week and might be able to bang out an output plugin while I'm at it

lucasbradstreet13:07:53

@drankard: I can point you to something a bit later

vijaykiran13:07:12

@lucasbradstreet: cool, I’ll see if I can wrap my head around plugin-dev

vijaykiran13:07:19

not in a hurry though

vijaykiran13:07:22

what would be a good example to get started ? I thought s3 could be interesting one ...

lucasbradstreet13:07:36

S3 could be a good place to start

vijaykiran13:07:21

ok, thanks! will try reading up some code

vijaykiran13:07:02

AFAICS, s3 is output only .. right ?

v.solovyov13:07:27

@vijaykiran: I've developed output plugin and it's not that hard actually

vijaykiran13:07:03

cool, that’s encouraging 🙂

vijaykiran13:07:18

@v.solovyov: or are you saying you already have hdfs-out plugin ?

v.solovyov13:07:43

I wrote onyx-http output plugin, no hdfs from me 🙂

lucasbradstreet13:07:49

@vijaykiran: S3 is output only for now, yes

v.solovyov13:07:33

I touched HDFS only once, and it was more than 1.5 years ago

vijaykiran13:07:51

🙂 ok, I’ll come back with questions soon then

michaeldrogalis15:07:23

Hoping to expand the number of plugins greatly later in the year, and to especially cover more batch-oriented mediums like HDFS and S3. That's the area that we're lacking most right now. Currently working through the next release to enable more rapid plugin development.

michaeldrogalis17:07:20

Speaking of S3, been using this lately, and I'm pretty over the moon about how well it works. https://github.com/jubos/fake-s3

michaeldrogalis17:07:46

It's especially for when I'm working offline.

Drew Verlee18:07:22

thats weird, i was just thinking “how can i test s3 offline"