Fork me on GitHub
#onyx
<
2018-11-21
>
Lutz08:11:14

I remembered that the "not in a catalog" way was explained in Onyx Template Webcast (starting at 4:43: https://youtu.be/5hFV_W9-10s?t=283). The mechanism is called "task bundles" and are introduced as "nice units of composition". They "group together schema checks with the tasks themselves". You can also specify tasks by parametrizing them. At 7:45 (https://youtu.be/5hFV_W9-10s?t=465) they show an example. And at 8:45 (https://youtu.be/5hFV_W9-10s?t=525) they introduce "bundle modifiers", that can be used e.g. to add logging to a task easily.

lmergen08:11:39

actually, under the hood, i believe add-task does add things to the catalog

lmergen08:11:53

it's just a convenience function when you want to compose things rather than define them as data

dbernal13:11:18

I'm trying to run a job with multiple peer nodes (each with 5 virtual peers) but I'm having a hard time getting it to run correctly. The nodes seem to join the cluster and create their peers but once I start the job it hangs and kills the virtual peers and then starts them again. This repeats indefinitely until one I kill one of the peers. And if the job has enough peers to run from just the one peer node it runs correctly until completion. Any ideas on what I could be doing wrong in my set up? Currently I have 2 machines running the start-peer command as a java process that runs forever and I submit the job through my dev laptop.

dbernal02:11:38

Figured out my issue. I had to add port 40200 in UDP between all node's ingress rules in the cluster.

guillaume15:11:13

isn’t it a crucial piece of the onyx platform?

😶 4
ClashTheBunny15:11:55

My guess would be that it was a first pass at the platform that became pyrostore or something? Why would it be crucial? It has nothing to do with actual data processing, right?

guillaume16:11:10

ok maybe crucial was a strong word 🙂 but it’s part of the getting started instructions https://github.com/onyx-platform/onyx-starter#dashboard

guillaume16:11:21

just read about pyrostore. looks like it was part of the distributed masonry/confluent acquisition.project page is down right now http://pyrostore.io/

ClashTheBunny18:11:27

Dashboards are sadly what people pay money for...