Fork me on GitHub
#onyx
<
2017-03-21
>
georgek02:03:09

Hi, I’m working at NCEI (The National Centers For Environmental Information)… It’s a part of NOAA that warehouses the worlds climate data. Like the data itself, some of which goes back over 100yrs of observations, it’s code is oooold. Anyway, I’m re-architecting a system that ingests data from a network of instruments (ASOS instruments) that has modems and is mostly FORTAN code. We are creating a workflow system that can be easily generalized (we have a ton of projects like this, well mostly all like this..) and are using Onyx. Long story short, we are going to use Onyx-Java and have gotten the Java API refactored. Myself and my jr developer are going to add tests, polish and do a pull request. We didn’t expand on the full API but intend to nibble on that once we are using it ourselves and have shaken the bugs out. Most of our system will have java classes that run native code and we are planning on cleaning that up and creating and Onyx-Native project with proper documentation. Once we have things working and a little more polished with utilities we have to build (like being able to bind methods on instances in the catalog, etc.. native objs are hella-stateful) then we do the pull request!

georgek02:03:39

Thanks, I’m super excited. It’s a really amazing fit for scientific computing. I’m just feeling lucky I snuck it through - in no small part because I could sell it as “just another java dependency” 😉

lucasbradstreet02:03:57

Let us know if there’s anything we can do to review. I’m super keen to see it happen

lucasbradstreet02:03:11

or any questions you need answered.

michaeldrogalis02:03:19

Huge news @georgek, thanks for running this home.

georgek02:03:38

Thanks! I appreciate all the support - y’all rock!

georgek02:03:55

“just another jar” hahaha, IT security review mantra

michaeldrogalis02:03:33

Worked for Rich 🙂

georgek02:03:38

@lucasbradstreet As for reviewing we forked the code so once that’s a little better organized I’ll let y’all know!

georgek02:03:45

It’s so true!

lucasbradstreet02:03:58

Sounds good 🙂

michaeldrogalis02:03:16

This calls for a beer.

georgek02:03:56

Beverages! clink I have some woodford reserve! 🙂

Drew Verlee02:03:03

@georgek is this open source code by any chance?

georgek03:03:50

Will be, we forked Onyx-Java and are going to do a pull request when things are tidier, reviewed, and used a bit

georgek03:03:29

When things are tidier I’ll post the fork we are consolidating on for reviews

yonatanel13:03:57

@georgek How do you bind an instance method? Does it mean you can break the dataflow model with your stateful object?

georgek13:03:42

@yonatanel We have things in our catalog that are backed by native code. So we need to find a way refer to a method on an instance (that’s can be initialized at runtime for a submitted job). We aren’t trying to subvert the dataflow-model, per-se, and ideally we are going to try to represent all the context needed to create the instance in the catalog. Of course, ensuring that the additional assets (like shared libs) are available in the cluster is a different problem and in that sense, yes, kinda does break the dataflow-model. 😉

theblackbox13:03:05

hmmmm nvm.... seems the issue was the logging =/

theblackbox13:03:39

@michaeldrogalis sorry for not answering you about the kafka serialiser question I asked on the git-chat thing last week. My question wasn't really about how, but why, because I'd figured I could pass a handler-fn but I was in the habit of passing a nippy serialiser such as with the franzy client. Thanks for the help!

yonatanel14:03:50

@georgek Will be interesting to see once done.

Drew Verlee19:03:54

random shot out. Even though i’m not able to use Onyx at work, learning the design has been a positive influence. Keep up the good work.

Drew Verlee19:03:54

I’m shamelessly re-using some of the larger design choices a much much simpler application :0