Fork me on GitHub

@sirsean: that's perfectly valid, I just prefer to replace Kafka/MySQL with core-async and regular in memory datastructures


Like, for dev mode?


I wouldn't do it like that because I find Kafka too heavy and difficult to manage for dev mode where you want to iterate quickly


But it's supported


It comes down to in my tests I would not want to have to manage external infrastructure


Oh certainly, running Kafka locally would be too big of a pain. That’s kind of why we were thinking of “only running via Compose”.


So our tests would cover our functions that do work, rather than exercising the Onyx infrastructure.


That's fine too, it might get tricky if you want to test your assumptions about the Onyx specific features like flow conditions, state/aggregation etc...


Hm okay. Thank you! That’s exactly the sort of “doing it that way is going to cause me a lot of pain later on” advice I was looking for.


Can I ask what drove you to wanting to do that? we are actively trying to simplify the template, were any parts confusing at first glance?


Well, a big part of it is that when we backported the new Kafka stuff in the latest version of the template into our project (thank you for that), our code coverage dropped by 25% and it just seems like most of this code is Onyx Dev Mode Stuff that doesn’t necessarily do anything for us? So we’re trying to move toward having dev and prod being as similar as possible, including running the same way inside Docker, and focusing on tests for each step of the workflow to make sure they’re doing what we expect.


(I hesitate to answer the “were any parts confusing” part of that question, because I’m not familiar enough with Clojure to answer that confidently. I’m at the point where I understand smaller functions and then there’s just some insanity that, while obviously powerful, I don’t understand. So sure there were confusing parts 😜 but that’s probably on me.)


Gotcha @sirsean thanks for the input.


If you’re fine with setup/teardown in your tests, or write your tests in such a way that you can avoid that, its perfectly valid to just use :prod. If you checkout the way we use regular functions to transform the job map, modifications to that will still apply to your job-map no matter how you construct it initially. Hopefully that addresses your concerns around backporting


@sirsean: I understand what you're saying. I prefer the dev mode tests to allow for quicker test turn around time - but it does have some overhead


Almost feels like we could make a --in-memory-io switch for the template. Sometimes you'd want core.async for dev mode, sometimes not. Just a thought.


It’d have to strip out or stub a lot more things, like the tests


That might be OK though


do you guys ever sleep?


Oh yeah, solid 8-9 hours a night simple_smile


great work, folks!


You sleep when your dead.. simple_smile


or when you reach 1.0? simple_smile


Whichever comes first, haha.