This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
Is it fair to say that all bugs are exhaustiveness faults? i.e. our techniques (mental model, tests, types) failed at being exhaustive enough to foresee a problematic code path
> 32. Programmers are not to be measured by their ingenuity and their logic but by the completeness of their case analysis. >
This autotuning feature out of Mojo (the Python superset) has people raving: https://docs.modular.com/mojo/programming-manual.html#autotuning-adaptive-compilation
@didibus, when discussing my injest library for auto transducing and parallelification of thread macros, https://clojureverse.org/t/parallel-transducing-context-fight-night-pipeline-vs-fold/8208/17?u=john_newman:
> "Now what you need is an auto-tune feature, maybe a ?>> macro that runs the thread with various amount of test data and then outputs a recommendation to use: x>>, |>> or =>>."
I was on the fence about implementing full autotune because it would obviously produce very unpredictable code, sometimes acting with one kind of complexity and sometimes in another. Seemed better to just let the developer experiment and decide. But maybe it makes sense where you want an algorithm to adaptively optimize itself at runtime.
One way to make this more controllable and predictable would be to return a data structure describing the level of parallelism deemed optimal by the tuning function, which could be used to reproduce the tuning configuration again in other contexts.
yeah, that might be for the goal of "find the best average settings for this dataset" Whereas another goal might be, "adapt to incoming datasets at runtime" In the later case you probably don't care about the current averages, unless you're building up a long term average to find dataset presets
In the former case, you may not be able to predict all cases ahead of time, but it could still be deterministic for repeatable tests
If people really want auto tuned things like that, we could be doing it a lot more. I think most of us figure that consumers of code will want more deterministic behavior, but maybe not. Any strong opinions on it out there?
Perhaps a simple-enough primitive would be a self-tuning thread pool, one could hint it (e.g. start with n threads) and let it increase/decrease threads based on a metric that is easy to compute. Within some boundaries perhaps. Then one could
(binding [p/*pool* (self-tuning-executor n ,,,)]
(p/->> x f g h))Pretty sure fork/join is already doing some autotuning, keeping work on a single thread if the thread comms cost is higher than the compute benefit
I'll probably add a *> macro to injest that lets you autotune a bunch of things like that, either automatically or within parameters.
Chunk size, thread count, lazy vs transducers vs parallel transducers, etc.
Can you think of any other opportunities for autotuning?
I already have a debugging feature in there that automatically rotates the impl between +>, x> and => and produces a report telling you the performance of each. I could just turn that into a dynamic switcher at runtime
Another parameter that could be auto-tuned in clojure/script is chunk size when chunking sequences of work, for parallelization or concurrency management, we often don't know what the correct chunk size could be as it depends on how heavy weight the elements in each chunk are, in memory space and compute time. And those values could change mid sequence too, so a good chunk size for the first half of the sequence might not be a good one for the last half.
Do you know of any ERP systems made in Clojure? Or any Hospital Information Systems?
Not sure how closely related, but maybe will provide some leads? Maybe you can ask the author for some pointers. https://clojurians.slack.com/archives/C06MAR553/p1679600430771489
Oh yes. I have Clojure in production for a European medical research project and deployed in NHS in UK as part of an EPR. Hermes, referenced above, used in both.
I also wondered whether Cerner used or had used Clojure in their stack given their release of https://github.com/cerner/clara-rules