Fork me on GitHub
Drew Verlee00:07:18

@gardnervickers: yes its because i wasn’t visiting the docker host ip. I vaguely remember that being addressed in the docker tutorial i did.

Drew Verlee00:07:51

i get the dashboard & and it seems to be recieving data...

➜  onyx-dashboard git:(update-docker-run-command-in-readme) ✗ docker run -p 3000:3000 onyx/onyx-dashboard:latest ""
Starting Sente
Starting HTTP Server
Http-kit server is running at 
Connected: 4e4ca526-ae1f-4aa4-b058-1f90bf45161d
Connected: 4e4ca526-ae1f-4aa4-b058-1f90bf45161d
Connected: 4e4ca526-ae1f-4aa4-b058-1f90bf45161d
Connected: 4e4ca526-ae1f-4aa4-b058-1f90bf45161d
Connected: 4e4ca526-ae1f-4aa4-b058-1f90bf45161d
But where i would expect to be able to select a job id in the dashboard there is nothing.


@drewverlee: Looks like it is not connected to Zookeeper. Can you ping the zookeeper address from inside the docker container?


@michaeldrogalis: I'm thinking about tools for an ETL problem I may be tackling with a client and I was looking at Onyx but I noticed onyx-etl is not actively maintained. I'm interested in the reason?


@michaeldrogalis: no worries about the doc, let me know if I can help


Im not sure i undestand the windows and triggers correctly, are the windows passed on to the next tasks in the workflow ? the docs doesnt seem to address it. And its a bit hard to explain, but i try.. Lets say a task is splitting a stream of un-ordred but timestamped input sentence segments into words, the task have fixed windowing with a watermark trigger, now the word segments is passed on to the output, it transact the words to a table in a database holding the words and window range, how can the output know which window the words is comming from ?


No the window state is passed to the trigger fn when the trigger conditions are met or the job is terminated


We don't support putting window stat back into your pipeline yet, that's coming in our next major release.


damn ok.. i guess i then can pass the timestamp on thougout the workflow and move the windowing to the last task and have the trigger fn do the transact…


Sure, you can also trigger to durable storage and then use an input plugin to read it again.


Is there any best practices for upgrading a live onyx job? In my case I will be using marathon ( still a relative newbie ) just wondering how this might work.


@agile_geek: Too many projects on our plate, not enough immediate interest. It was really just a thin wrapper around Onyx anyway. If you know what you're doing with Onyx, you were better off just using Onyx directly anyway.


@vijayakkineni: Actually, that'd be great. I've been trying to get the user guide into a more pleasing state for a long time. If you have any ideas about how we could set it up better, would love to get some help there. 🙂


Of course, I would like to help in anyway I can - first order of business would be to order the docs in more palatable way ?


@vijaykiran: Yeah, ordering by topic would be best. The reason we don't do that now is because we're using Jekyll, and we're maintaining multiple versions of the docs for every release. Jekyll is doing an automatic sweep of the docs directory everytime we build the time, and since not all versions have every page, its hard to order by any other way.


I think the real thing is that we need to get off of Jekyll. We outgrew that like a month after we picked it up. 😕


hmm - any plans on moving to something like ?


We were on ReadTheDocs for a while. It was alright, we wanted something that would integrate directly into our larger website - like Postgres or Datomic.


Ah - okay sounds like a nice challenge 🙂


We might have to go back to that though. There's a lot of work to make things "nice" otherwise.


@michaeldrogalis: Check out asciidoctor. You can do some really cool stuff with it.


either way, I’d like to help out, because I think this hurts pretty bad for people who want to get started with onyx 😞


@dominicm: That's what I've been experimenting with lately. I had to put it down though. The problem I ran into is that it doesn't let you render the content into an arbitrary HTML template. They only seem to have preformatted templates to render into.


@vijaykiran: Cool, it's much appreciated. There's a lot to maintain these days, we can use all the help we can get.


@michaeldrogalis: I think it's possible, this is built with AD:


is there any list of concrete tasks that I can pickup or more research needs to be done ?


@dominicm: Oh, wow. Yeah, that'd be perfect.


@vijaykiran: Hm. Let's see. I'm pretty set on moving all the User Guide files from Markdown to AsciiDoc. I actually have a branch that starts doing that. I moved all the user guide pages into a directory called "staging" while I was experimenting -


My idea was to take all the adoc files, concat them together, and do a single-page user guide with AsciiDoctor.


@michaeldrogalis: thanks. I thought it was something like that but just wanted to confirm you hadn't suddenly decided ETL was a bad use case 😉


@michaeldrogalis: So, for conversion check out pandoc


it can do MD->AD now


I actually got it to render a nice looking page. If we could come up with a script to generate something like that and substitute that in the website, I'd be very happy.


@dominicm: Dang. I spent a whole evening doing the conversion, ha. Ill know for next time.


Anywho. @vijaykiran I think a good plan of action would be to work on that branch and move all those user guide files from staging back into their proper home (one directory above), and make a script to compile a one-page user guide, as seen above (I just sorted of hacking it into place to see what it would look like).


I could take that script and plop it into our website build, and we should be in a better place.


okay - will take a look this weekend, I’ll poke you now and then 🙂


@vijaykiran: Thank you very much ^^