Fork me on GitHub
#onyx
<
2016-12-10
>
stephenmhopper00:12:51

@lucasbradstreet Thanks, I'll look into that for #1. Do you have any recommendations for question #2?

lucasbradstreet00:12:25

@stephenmhopper not sure whether the job tracking bit should be done within Onyx itself. As for the segments fitting in memory issue, yes you will want to setup an input source using an onyx plugin. onyx-seq is a good way to do this to get started as it’s fairly flexible, though you may outgrow it later

stephenmhopper00:12:15

I see. So I can use onyx-seq with something that returns a lazy-sequence of values and it should avoid memory issues? At what point does one "outgrow" this plugin?

lucasbradstreet00:12:15

the primary problem is that the way it recovers from failures isn’t really ideal for all use cases

lucasbradstreet00:12:10

since it expects a seq, the way it recovers is to just drop the first n elements. For example, if you’re doing something like reading a big file / parsing it / etc, maybe a plugin that knows more about the medium and that could directly seek to the recovery point would be more efficient in the face of failures

stephenmhopper00:12:29

that makes sense

stephenmhopper00:12:37

Okay, I think I have enough to keep moving for now

aaelony22:12:46

a little while ago a nice library to create flowcharts, graphs and other diagrams was mentioned. Does anyone recall the name of that library? (It wasn't graphviz...)

aaelony22:12:10

awesome, thanks

aaelony22:12:26

how could I forget a name like that 🙂