Fork me on GitHub
Slack Archive

This page is not created by, affiliated with, or supported by Slack Technologies, Inc.

2018-08-22

Channels

#onyx
<
2018-08-22
>
rustam.gilaztdinov10:08:37

really, really strange things happens in onyx-sql if I added to project.clj logs libs 

[org.apache.logging.log4j/log4j-core "2.8.1"]
[org.apache.logging.log4j/log4j-api "2.8.1"]
[org.slf4j/slf4j-log4j12 "1.7.25" :exclusions [org.slf4j/slf4j-api]]]
Then, I have batch size = 50 and in catalog -- :sql/rows-per-segment 1000 I submit job and onyx processed 1000 record from db If I exclude logs libs, onyx process 50 records What is that? And of course I have only one batch processed, after him - NPE, in both cases

rustam.gilaztdinov10:08:34

this is so frustrating, anybody can help me and fix that?

rustam.gilaztdinov10:08:57

what is a difference between batch-size and rows-per-segment?

rustam.gilaztdinov11:08:21

anyone test this plugin?

rustam.gilaztdinov11:08:47

this is not working at all

polymeris16:08:51

It's probably possible via JNI, but not sure if Easy

polymeris16:08:05

That is, clj->java->JNI->C->fortran

ben16:08:46

That sounds pretty grim… So probably out of reach for a relative beginner in clojure?

polymeris16:08:55

Most of the complexity would probably be in the JNI->C part, so not even very accessible for someone experienced in clojure.

polymeris16:08:28

The clj->java part is relatively straightforward, I think

mpenet17:08:12

I did exactly that in the past, clj - java - jni - c - fortran, luckily for me the c part was already written

mpenet17:08:31

JNI is quite easy to use, depending on how large the api you need to cover is. Also depending on your requirements you can prolly get away with jna or better, jnr

mpenet17:08:23

Another idea I was toying with lately is to use luajit for the ffi and fennel-lang for these kind of things, but if you need stuff from java land this is a no go

gigasquid17:08:10

I think you might be able to execute Fortran on Graal

gigasquid17:08:44

so maybe you could do something like that using polyglot Clojure https://github.com/gigasquid/graal-test/blob/master/src/graal_test/core.clj

gigasquid17:08:41

Would be interesting to try 🙂

gaverhae18:08:08

Maybe looking at core.matrix -> clatrix -> jblas -> LAPACK can provide useful insights?

jsa-aerial18:08:03

If that (matrix math on the metal) is what you are looking for, I'd strongly recommend having a look at Neanderthal

ben09:08:48

Wow, thanks everyone - it looks like I have a lot to look through 🙂

lmergen18:08:28

@rustam.gilaztdinov like I said, it’s most likely a bug in the Onyx sql plugin that is triggered upon sequence realization. the logging library most likely causes the sequence to be realized in a different place, causing different behavior.

lmergen18:08:44

if you can help debugging the issue that would be great

rustam.gilaztdinov19:08:21

@lmergen thank you! I'll post here my steps, hope we can resolved that harold