Fork me on GitHub
#data-science
<
2017-08-23
>
hswick04:08:39

Hi fellow data scientists! Recently been working on an easy to use wrapper for ND4J and released my latest version. It is a linear algebra library like core.matrix or neanderthal but runs on top of ND4J arrays, the benefit of using ND4J is that it works on both CPU and GPU by swapping out appropriate dependencies. https://github.com/hswick/jutsu.matrix

blueberry08:08:56

@hswick Fantastic! Now it'd be easier for me to do the comparisons with ND4J. Please ping me when you feel it is ready for the prime time, and we can do some comparisons? Thank you for the good work.

jsa-aerial16:08:31

@hswick I haven't looked in detail, but how does this differ fundamentally from https://github.com/ds923y/nd4clj which is noted as 'mature' (which conflicts with your note that providing a nd4j impl in c.m is not doable)??

hswick16:08:36

nd4jclj hasnt been worked on in 10 months and the documentation isnt great. Not to brag, but jutsu.matrix already supports significantly more features than nd4clj. The hope is that my library will differentiate itself by being maintained and user friendly.

jsa-aerial16:08:48

That's a different lib than the posted one. Yes, I read that section and didn't get the bit about not being able to provide nd4j impl for c.m. In the c.m main github page, nd4j lib noted there is amont the 'mature' impls

hswick16:08:34

Sorry, but not sure what you mean. That issue is from the same github page you posted. And idk what ‘c.m.’ means. Basically the conclusion I got from that github issue was that a full core.matrix implementation for ND4J wouldn’t be possible without a lot of work around. Which seemed like unnecessary complexity to me

hswick16:08:33

And nd4clj isn’t very mature. It barely supports a minimum amount of features for ND4J

hswick16:08:18

nvmd I guess you meant c.m. was core.matrix

blueberry17:08:44

nd4clj looks more to me as a proof of concept or an exploratory experiment than a library that anybody really uses or tries to use and develop further (including the author). I am puzzled why it is marked as mature on the core.matrix site. OTOH, clatrix is also marked as mature, an it hasn't seen any development for years, and it is not because it is complete or without bugs....

jsa-aerial17:08:31

Well, what is confusing is that issue refers to a prior incarnation of the lib - which only makes things more confusing since there are 3 variants!

jsa-aerial17:08:16

So, even worse though, I figured I would simply try it - and it just fails. So, probably should make an issue on c.m that this thing is not only not mature, but actually doesn't even seem to load! 🙁

blueberry17:08:11

@U06C63VL4 looking at the issue trackers of all of those libraries, it seems only vectorz-clj more or less works, while all other implementations (probably) worked at some point in time, but either do not work at all, or carry many bugs of various discoverability and severity.

jsa-aerial17:08:07

Hmmmm, it is actually weird. For the specific issue in question the final comment (last sept) notes that "There are 0 tests failing now." indicating the thing should not only load, but pass all tests.

jsa-aerial17:08:19

@U086AG324 I'm guessing you are right that many of these (nd4clj in particular) have not tracked required changes in c.m and stopped working at some point.

jsa-aerial17:08:50

Clatrix does still seem to work - at least for many things...

blueberry17:08:42

Until you realize that it gives wrong results for unspecified number of operations.

jsa-aerial17:08:57

At least nd4clj has the good sense to not even work!

blueberry17:08:00

Clatrix hasn't seen any commits in more than 2 years

blueberry17:08:44

And even before that, most commits are small changes related to core.matrix. The original author left for greener pastures years ago.

jsa-aerial17:08:36

I think this is probably the primary problem with Clojure in data-science land. Too many projects are cases of 'wow, proof of concept kinda sorta works. cool Now what should I work on?' Others either don't notice something already done and start their own (which ends in the same way) or like their variant more (but it also ends in the same way). So, things end up being 'bushy' with not much depth...😞

blueberry18:08:32

OTOH, why is that a problem if there are enough libraries that work as advertised? A little more work to sift through the weed, but you can't criticize people for publishing their free stuff even if it is not the greatest thing in the world.

jsa-aerial23:08:14

not a criticism per se more an observation of current state. And the primary point was that things haven't congealed to an agreed set of things that are being continually improved and refined. And as such will attract multiple resources to work on them and thereby attract even more and ...

blueberry09:08:49

that was the theory at least. but how to agree to a set of things? everyone has their own set of needs, often quite different from others'. after all, those who show up to work decide what is going to be done.

blueberry09:08:49

that was the theory at least. but how to agree to a set of things? everyone has their own set of needs, often quite different from others'. after all, those who show up to work decide what is going to be done.