Fork me on GitHub
#data-science
<
2021-09-29
>
Carsten Behring08:09:20

I am developing a in my view interesting approach to a pure functional way of evaluating / selecting ML models. The code is here: https://github.com/scicloj/metamorph.ml/blob/main/src/scicloj/metamorph/ml.clj#L121 and a tutorial here: https://scicloj.github.io/scicloj.ml-tutorials/userguide-advanced.html in chapter "Pipeline evaluation and selection". At this point in time the code is "working for me and my use cases", but I would like to get a fresh view on it. Especially regarding 2 question about existing function : 1. is it generic enough to cover the most important forms of model evaluation and model selection ? 2. Are the usability / performance trade offs and the control on memory usage acceptable ?

Carsten Behring08:09:24

There is quite some tutorial material available here: https://github.com/scicloj/scicloj.ml

Carsten Behring08:09:14

If somebody here wants to take a look at it, I would be very grateful.

xceno17:10:44

I'll tackle some ML tasks in the coming months, but I'm not there yet. I'll gladly give it a spin though. Thanks for your work and the user guide, very intriguing stuff! FWIW you'll have a bigger audience in the zulip channel

Carsten Behring07:10:54

Thanks, I am usually in zulip, just tried here as well. It seems that ML in Clojure is a tiny group of people.

Carsten Behring07:10:16

I count a lot on interop as well. For this you might be intrested in: https://github.com/behrica/clj-py-r-template This allows to "start in clojure" and continue in python / R if certain ML things are missing.

Carsten Behring07:10:33

You need Docker, though.

Carsten Behring08:09:43

Regarding point 2: The function need to "balance/support 2 very different usecases": 1. Exploratory evaluation of a few models and a few pipelines (= sets of hyper parameters) -> this requires to return "all possible information on the model evaluation process", including the data passed into evry model, the model instanced and so on (apart from only the evaluation metrics)

Carsten Behring08:09:58

2. Systematic searches for the best model across a large number of hyper parameter combinations and cross validations -> This will not work if the methods returns "all evaluation information" as needed in case 1 -> We could have easily thousands / hundred thousands of model evaluations, so we would run out of heap space quickly return data for each of it -> Ideally in this process, the "used heap space" would be only a single float number per model evaluation (= the performance metrics) --> even this is not realy need, as we only needed to keep track of the "best" --> this becomes strange to do in a functional style --> lazy evaluation is neither an option, as model training is slow and can even take up to hours in the case of deep learning models

Carsten Behring08:09:55

My proposed solution consists in being able to configure to a large extend via function parameters, which precise information "per model evaluation", the function will return. This allows the user to balance between "few evaluations and lot of (debug) information on teh evaluations" vs "lots of models and very few information per evaluation"

Carsten Behring08:09:07

Is this reasonable to do ?

Carsten Behring08:09:52

A single model evaluation is a rather complex map in a fixed structure, and the user can basically configure which parts of that map should be removed for each model evaluation. Additionally there are two boolean flags, which decide if only the best models (per cross validation / per hyper parameter) gets returned

Carsten Behring08:09:59

An other very interesting "use case", I could add to the function, would be to allow a kind of "write all to disk" mode. A "model evaluation result" is currently just a map, which I could just serialize to disk after each evaluation. In this way: • the function would use zero-heap space (better said: only the heap space needed to evaluate a single model), after writing to disk, we could discard it completely

Carsten Behring08:09:55

It would be more suitable to "run for days", as a crash of the JVM would not result in "loosing" all evaluation result.

Carsten Behring08:09:57

This would requires some more functions, which could "analyse" what was written to disk and make model selection in that way