Fork me on GitHub
#data-science
<
2023-12-13
>
Daniel Slutsky16:12:46

Yesterday's recording: https://clojurians.slack.com/archives/C03RZRRMP/p1702468932436079 Yesterday's recording: https://www.youtube.com/watch?v=skMMvxWjmNM @kiraemclean: - a detailed live intro to Tablecloth - brief intro to Hanami - combining Tablecloth & Hanami through Noj - visualizing data with Clay - using that to analyse git repos and their commit lots @daslu: - a bit more about Clay - keeping track of data collection, visually, using Portal - continuing Kira's analysis a bit @timothypratley: - Kindly - a convention for requesting data visualizations - the Clay & Claykind tools for datavis and literate programming - broader thoughts about literate programming, the visual tooling scene, and the state of community

phronmophobic01:12:13

Great Talk! Not sure how much interest there is on analyzing clojure projects on github, but I would be happy to add new analyses as part of https://github.com/phronmophobic/dewey. I have a weekly AWS batch job that: • finds all the clojure repos with more than 3-4 stars • aggregates all the info needed to include these libs as git urls • runs clj-kondo static analysis on each repo • https://github.com/phronmophobic/dewey/releases the raw and processed data on github • updates a https://phronmophobic.github.io/dewey/search.html that can search for clojure libs by name, topic, description If there's other stats, charts, analyses that might be interesting, I could add them to the pipeline and host the results.

phronmophobic01:12:31

The clj-kondo analysis is around ~900MB gzipped. Is that too large to be processed by tablecloth or is that something it could handle?

Daniel Slutsky07:12:09

> add new analyses as part of dewey Dewey is so nice. Great, let us discuss it. What do you think might be useful?

Daniel Slutsky07:12:09

> The clj-kondo analysis is around ~900MB gzipped. Is that too large to be processed by tablecloth or is that something it could handle? I think it is worth trying, depending on the types of Columns, etc. We may also try it with out-of-memory approaches such as https://techascent.com/blog/just-ducking-around.html.

👍 1
phronmophobic09:12:30

> Great, let us discuss it. What do you think might be useful? I think there's lots of cool opportunities: • searching for individual functions via doc strings • tallies of individual functions • tallies of namespaces • comparisons of clojure vs clojurescript and other dialects • analyzing git histories for common bugs and troublesome functions • a common pitfall for beginners is java interop. use statistics to figure out what common which java libraries might be good targets for clojure focused documentation or wrapper libraries • database of specs • analyze n-grams for common idioms I wrote about some previous work at https://blog.phronemophobic.com/dewey-analysis.html

👍 1
phronmophobic09:12:54

I would really like to improve https://phronmophobic.github.io/dewey/search.html • It's kind of ugly • It would be nice to incorporate data from clojars, https://github.com/clojars/clojars-web/wiki/Data • Search code be improved to include docs and doc strings

👍 1
Daniel Gerson10:12:46

Good talks. Would be good to see how to set up Clay from scratch in vanilla Emacs, VSCode, IntelliJ and a demo project set up from scratch in the video. I haven't converted my Clerk & Tablecloth project, as I'm lazily waiting for step by step guide 😅 . If I'm any indication then this may also prevent others from venturing in. I think the talks want to demonstrate what you get "once you're in". However, not demonstrating "getting in" is an indication that "getting in" is finicky.

Daniel Slutsky12:12:13

@U7RJTCH6J fantastic, please let me know if I can help.

Daniel Slutsky12:12:24

@U03B2SRNYTY great advice, thanks. Yes, these docs & demos are of high priority.

Daniel Gerson22:12:14

@daslu Looks good, but you probably need feedback from a VsCode user. Emacs (Spacemacs) user here.

Daniel Gerson22:12:16

Oh, just saw https://www.youtube.com/watch?v=fd4kjlws6Ts . Will try to get to it in the week.

🙏 1
Frank Henard19:01:29

This video was a huge help, thanks!

🙏 2