Fork me on GitHub
#onyx
<
2018-01-25
>
kenji_signifier00:01:45

Hi, I’m in early design phase for the next project, and looking at Datomic Cloud as the storage. I think onyx datomic plugin uses peer api datomic.api, which is not supported by Datomic Cloud. Do you have a plan to add support for client api to the plugin?

souenzzo12:01:22

After a bit of research, there is no easy/one-line solution to support client and peer with code reuse. This weekend I will PR a possible solution.

kenji_signifier05:01:48

glad to hear you already started to look into the issue. I’m on hold and help you out to test your PR.

kenji_signifier07:01:44

@souenzzo I noticed com.datomic/client-cloud depends on clojure 1.9. It bombs with java.lang.RuntimeException: Unable to resolve symbol: halt-when in this context, compiling:(datomic/client/api.clj:60:11) to load in clojure 1.8. @michaeldrogalis @lucasbradstreet is it feasible to use a plugin that requires 1.9 at this moment?

niamu07:01:29

onyx-sql was just bumped to version 1.9.0 recently, so yes.

michaeldrogalis16:01:33

Bring it up to 1.9.0 - yup

kenji_signifier08:01:37

I spent some time to get familiar myself to datomic cloud and onyx. Here are the gaps I face;

kenji_signifier08:01:44

the following fns don’t exist in client-api. entity, ident, log, next-t, and tempid. as @souenzzo suggested some fns may be re-written in client-api, but not sure for all of them. Majority of them are used in tests so re-writing tests might be an option but I’d like to minimize the deviations.

kenji_signifier08:01:41

I managed to handle selective lib loading and abstraction by pushing down the impls under protocol.

kenji_signifier12:01:15

Tomorrow I’m gonna try the approach to keep using peer api for tests, and incorporate on-prem client lib. The differences between on-prem and cloud client api are primarily in connection area so I hope it has good enough fidelity.

kenji_signifier00:01:00

I made some progress and now can handle connections with peer, client via peer-server, and client via cloud. I’m going to spend some more time this weekend to convert tests to use client API so that I can run tests against Datomic cloud, too.

kenji_signifier11:02:37

I managed to pass tx_output_test.clj in both peer and cloud. 8 more tests to go…

kenji_signifier07:02:50

Finally I managed to pass all tests on both peer and datomic cloud. Please review PR. https://github.com/onyx-platform/onyx-datomic/pull/29

lucasbradstreet21:02:29

Nice work! I’m pretty busy at the moment but I will try to get a quick review in ASAP.

niamu00:01:43

They were going to investigate further to see what changes were necessary to support it, but the initial reaction was that it would be a small change.

kenji_signifier00:01:06

@niamu, thx, i searched in slack but didn’t find it. Looks promising!

michaeldrogalis01:01:24

Happy to take a patch if someone can make both version compatible, or add a parallel set of tasks

kenji_signifier04:01:47

@michaeldrogalis I’m gonna give a shot this weekend.

lucasbradstreet04:01:15

@kenji_signifier Great! Any questions, hit us up

souenzzo12:01:22

After a bit of research, there is no easy/one-line solution to support client and peer with code reuse. This weekend I will PR a possible solution.

dbernal19:01:00

Do all Onyx plugins have to be utilized as an input or output task? I'm guessing that's the only usable interface for a plugin right?

lucasbradstreet19:01:52

output plugins can also be used on reduce tasks, but that’s almost the same thing as outputs

lucasbradstreet19:01:32

We have discussed being able to make intermediate tasks outputs, and then flowing the messages further down, but we haven’t been able to figure out how to make it make sense.

dbernal19:01:26

ah gotcha, that's kinda where I just got stuck. I was trying to make an output task flow into another output task and it didn't seem to work

lucasbradstreet19:01:52

What’s the use case?

dbernal19:01:48

just for dev experimentation. I wanted to save to Elasticsearch and make the result available in a channel as an async out task

lucasbradstreet19:01:28

K. Yeah, I think the general approach we’d recommend for now is to make an intermediate task that does the transformation, then have it flow to an elasticsearch task as well as a core async task

dbernal19:01:24

Right. Gotcha, yea that makes sense. Thanks!

kenji_signifier05:01:48

glad to hear you already started to look into the issue. I’m on hold and help you out to test your PR.

kenji_signifier07:01:44

@souenzzo I noticed com.datomic/client-cloud depends on clojure 1.9. It bombs with java.lang.RuntimeException: Unable to resolve symbol: halt-when in this context, compiling:(datomic/client/api.clj:60:11) to load in clojure 1.8. @michaeldrogalis @lucasbradstreet is it feasible to use a plugin that requires 1.9 at this moment?