It sounds like you might like a visual, like a graph or boxes and lines to document the flow of data?

Priority is definitely functionality that would make it easier for a developer to take over and manage (like auto-generated docs, visuals, etc). My end goal would be that non-technical people could also use the system to be able to use the data in the content they write, but that's v2.

In Clojure , I like the techascent stack with tablecloth. This takes care of extract, and is succinct. Plus you can do any work needed in Clojure. I usually skip loading to a typed model or database and query source files stored in an object store directly using tablecloth. This is efficient for millions of rows. For documentation I would look at notebook style (clay library ) https://github.com/scicloj You could leverage malli as well, for providing documented apis. I also like meta base but haven’t used it for a while https://github.com/metabase/metabase

Thanks @U068BQFJ9! I'll look into using these. I like the idea of using notebooks for documenting things.

@U03QQS7341W At Crossbeam, we have been moving our home-rolled Data Pipeline to https://temporal.io/ (which has a nice Clojure SDK btw). I'm working towards having ELT capabilities within our product stack. We are already using dbt for our analytics stack. For extract, we have a variety of things in use - https://www.stitchdata.com and https://www.singer.io/ Taps, plus various home-rolled things. We are looking at https://airbyte.com/ as a candidate for a more generalized approach to managing data sources. Since you are interested in AirFlow, that's probably already on your radar, but wanted to flag it in case you weren't aware.

@U05EF0987B7 so are you setting up the ETL in Temporal as well? What does that look like? Does it play nice with dbt? That's another library I'm thinking of using for some transformations. Am I correct in saying that Temporal is an alternative to Airflow? Yeah I'm aware of AirByte, even set up a local instance to mess around with it. Looks pretty cool.

@U03QQS7341W Haven't gotten to that stage yet but that's the hope. I imaging triggering dbt as a concluding step in a data processing workflow.

We looked at Airflow, Airbyte, Overseer and Temporal. Not quite 1:1 equivalents but it was a learning process. We settled on Temporal as a nice generic 'distributed cron' with DAG workflows. The Clojure-friendliness was a bonus.

Slightly circular but this is interesting: Airbyte uses Temporal and dbt: https://airbyte.com/blog/scale-workflow-orchestration-with-temporal

@U05EF0987B7 thanks for all the info! I'll be spending today comparing Temporal and Airflow. It's nice that Temporal has a clojure SDK.

Vegas / Vegas+ - https://arxiv.org/pdf/2009.05112.pdf

h-Cubature - https://github.com/stevengj/cubature

Gauss-Kronrod - https://en.wikipedia.org/wiki/Gauss%E2%80%93Kronrod_quadrature_formula

Nice! Emmy has a bunch of numerical quadrature as well for anyone interested in this stuff - https://github.com/mentat-collective/emmy/blob/7b8cddc17473d342323cf20d9bdff7fd36b1bea6/src/emmy/numerical/quadrature.cljc#L55

The numerical differentiation follows a Sussman paper and uses Richardson extrapolation to get the numerical derivatives to converge very quickly. It’s a nice read if you’re interested in this sort of algorithm https://github.com/mentat-collective/emmy/blob/v0.31.0/src/emmy/numerical/derivative.cljc

Great to know! I'll take a look at this soon. Probably some duplications are between emmy and fastmath.

I really like how you write the code! I definitely need to change my habits...

btw. regarding Richardson, I've implemented several orders of accuracy which is explicit unrolled version of Richardson method (though, you can't say what error you will get for arbitral accuracy and x)