Fork me on GitHub
#off-topic
<
2022-02-22
>
Kelvin14:02:13

Has anyone noticed that certain Slack workspaces are down?

Kelvin14:02:34

My work Slack is down but (obviously) the Clojurians Slack is still up. Weird.

flowthing14:02:43

Same here, actually.

spinningtopsofdoom14:02:17

I'm unable to see the threads window in any slack but able to access everything else

functional-brian17:02:31

was done for a while for me; all the workspaces. things are returning to normal

adi21:02:40

In the vein of "stuff I learn in Clojureland I apply everywhere", I'm trying to write a series about (my) FP-style shell-fu, heavily influenced by Unix tools philosophy + Simple made easy + Design, Composition, and Performance + my original initiation into the FP way via Scheme and Clojure. 🙇 gratitude https://www.evalapply.org/posts/shell-aint-a-bad-place-to-fp-part-1-doug-mcilroys-pipeline/

🎉 2
adi07:03:18

s'more Clojurish functional Bash... a https://github.com/adityaathalye/shite I'm writing in anger.

Dimitar Uzunov11:03:14

This was a great read! I really like this part of shell programming. Unfortunately most functions in the wild have many side effects and environment variables which make them hard to reason about. I really appreciate clojure namespaces when working in bash

adi13:03:10

Thank you! :) If you liked that, you will love "Concise GNU Bash" by James Pannacciulli: https://www.youtube.com/watch?v=BJ0uHhBkzOQ > I really like this part of shell programming. Same... as long as one keeps the code purely functional, it is so useful and scales very well. Wanna process nginx logs weighing half a TB gzipped? No problem. zcat, zgrep, and sed away. > I really appreciate clojure namespaces when working in bash Oh boy, if there was one thing I could improve about Bash, it would be introducing "real" namespaces. Mercifully, functions occupy their own namespace (`type -a thingname`) so they don't mutate aliases, builtins, and reserved keywords. So we can do this...

$ sed() { echo lol; }

$ sed="phew!"

$ type -a sed
sed is a function
sed () 
{ 
    echo lol
}
sed is /bin/sed

$ man bash | sed
lol

$ man bash | command sed 1q # sort of Common Lispy, I suppose.
BASH(1)                               General Commands Manual                               BASH(1)

$ echo $sed
phew!
But that's not quite enough and manual syntax hacks like func.name() { ... ; } don't cut it because then we are way out of POSIX territory. In a way that is good, because no namespaces forces me to drop Bash when the code starts getting too involved (like > 1,000 LoC of clean FP Bash). Beyond that size, the sharp corners start poking holes and forcing bugs (quoting, expansion, trap/error handling etc.). That said (or um, sed), 1 Kloc FP-style Bash can do a crazy amount of stuff sufficiently fast. And if you use awk, I hear that it is known to out-perform hand-optimized C for a variety of data processing problems (genomics people awk a lot).

Dimitar Uzunov13:03:25

yeah bash is still strong for ETL-type work in a single file. I’d rather use babashka for anything non-throwaway nowadays, especially if it has to deal with data serialised in yaml or json

1
adi13:03:45

> I’d rather use babashka for anything non-throwaway nowadays, especially if it has to deal with data serialised in yaml or json Absolutely! Still, in situations where I really can't introduce Clojure in a reasonable way (needs institutional support), I'm grateful I can use structured data thanks to jq etc. I will die fighting on the hill called "Structured Logging". https://github.com/adityaathalye/bash-toolkit/blob/master/log-analysis.sh#L193

adi13:03:13

> ETL-type work in a single file Hm, if input files on disk are sensibly namespaced, then it can be one of those "embarrassingly parallel" problems. The nginx logs example I gave was like that. {6 nodes}/yyyy/mm/dd/hh/logfile{1..n}.gzip. Even a naive one-liner with xargs -P can help there. I know because I wrote said naive one-liner, and managed to narrow down where the malicious requests were coming from. 🤓

adi13:03:30

But overall, I do agree. Shell has a sweet spot, and most other spots aren't. Proper modern proglangs are very much preferable. And we are super lucky to have #babashka!

Dimitar Uzunov13:03:47

some gnu tools use multiple cores quite efficiently when available, I remember rewriting a tool written in JS to bash and the bash version being significantly faster without using xargs and such and fully saturating the available cores. I don’t remember which tools it used that were able to do it

1
Dimitar Uzunov13:03:43

no I mean it didn’t require anything like that

adi13:03:12

Aah, I see. I haven't done big enough data processing to go far enough into shell parallelism/multicore. I've primarily written repo-specific tools and/or utilities, and ad-hoc "little data" processing (a few dozen gigs at most). That's shaped my little library of functions-as-cmd-line-tools... the set of "stuff I could use quickly and cheaply, if only I had written it down somewhere" :)

adi13:03:24

Actually, now I don't even care I never did anything "big" with shell. I'm happy with little tools, lots of fun, and of having received a sed-fu/Unix-fu masterclass from Douglas McIlroy (https://www.evalapply.org/posts/shell-aint-a-bad-place-to-fp-part-1-doug-mcilroys-pipeline/#appendix-an-unexpected-masterclass)!!! Gonna ride that high for a long time :))))

Dimitar Uzunov14:03:58

yeah that is pretty awesome

Dimitar Uzunov14:03:07

similarly it was I think something to the tune of 3-4GB of data, which was JSON logs of an application

adi17:03:24

Ah, I see. My last adventure was to write a server logs -> billing calculator (api call based monthly billing against 40-50 GB of gzipped logs dump). Just to see what was feasible. It turned out that my Bash version: • weighed in at 10x less lines of code than the extant JVM version (~1.5Kloc Kotlin), • ran about 20% faster than the JVM job (~ 1GB/min throughput with no optimization whatsoever) • consumed way less memory and CPU (background job, with rest of the PC free for Intellij and Slack) • was idempotent and re-startable against intermediate data • and was a collection of composable functions fully reusable in any combination for ad-hoc log processing at the terminal, which people had to do anyway for debugging/post-mortems. It may have been slightly worse from a parsing/type-checking PoV --- correct tally of calculation, but probably within an acceptable margin of error.

adi17:03:22

I came away pleasantly surprised... it had little to do with any personal cleverness, and much to do with ease of doing pipelined, streaming, modular, data-oriented architecture.

adi17:03:21

Yeah, I like that post by Adam Drake. You may also like https://datascienceatthecommandline.com/ and https://www.datafix.com.au/BASHing/

respatialized22:02:44

tell me you think software is collapsing under the weight of its own complexity without telling me you think that software is collapsing under the weight of its own complexity

😂 4
🙏 1
p-himik23:02:44

create-react-app is like a step-brother of RoR or Django. Only worse.

Cora (she/her)23:02:51

it feels like one of those old python web frameworks where they were just a big ball of mashed up random libraries that mostly worked

Cora (she/her)23:02:13

RoR and Django feel a lot more coherent to me, much more coherent than CRA

p-himik23:02:38

Probably. But I've been to the deep guts of Django, and I couldn't leave unscarred. :(

p-himik23:02:02

(CRA is still worse though)

Cora (she/her)23:02:51

I totally understand the scarring

Cora (she/her)23:02:56

I survived rails v1 to v5

Cora (she/her)23:02:01

I need that on a t-shirt

😄 2
Cora (she/her)23:02:07

hmmm maybe it was rails 2? it was so so so long ago

Cora (she/her)23:02:23

I did some django and turbogears back then, too

p-himik23:02:33

I'm fortunate enough that I've never even heard of Turbogears.

respatialized23:02:25

create-react-app now has powers that have grown far beyond its abilities to incorporate and control them

respatialized23:02:43

the counter-cliche to "worse is better" is "victim of its own success"

lilactown00:02:40

i'm not even mad at CRA. i think it successfully helped bootstrap the React ecosystem and lead to the evolution of things like next.js & remix

lilactown00:02:00

it's hard to start a project. it's also hard to start a rich & cohesive ecosystem. creating a sort of "industry standard boilerplate" has the effect of making a baseline that everyone can get mad about and then go and create something better for their use case

lilactown00:02:28

IMO spec basically did this for validation and coercion 😂

p-himik02:02:38

Spec didn't become a monster though. :)

p-himik02:02:24

Also, Plumatic schema was quite popular before Spec existed, I think.

lilactown02:02:19

my point is that spec spurred a lot of innovation, e.g. malli

👍 3
adi03:02:39

> tell me you think software is collapsing under the weight of its own complexity ... Does this bio qualify? https://confengine.com/user/aditya-athalye-1

Cora (she/her)03:02:06

I used CRA before I convinced coworkers to move to nextjs

Cora (she/her)03:02:19

and now I'm eyeballing remix

Lennart Buit08:02:14

I've argued with CRA for much too long. It's zero configuration model just gets in the way sometimes. Ejecting is terrible and react-app-rewired/ customize-cra are just hacks

kennytilton14:03:38

I have Redux Toolkit to fix Redux to fix Flux to fix MVC. #justayin

kennytilton14:03:48

But software is not collapsing. Our mistakes are failing. They do that. But Tilton's Un-Compliment: "The brightest engineers will make anything work. " https://allpoetry.com/In-Broken-Images This is Clojure's opportunity.

kennytilton14:03:41

Thx, @UK0810AQ2. I knew it. 👨‍🦯 The rugrats are to blame. https://youtu.be/ZSRHeXYDLko?t=866

Ben Sless14:03:51

@U0PUGPSFR as a rugrat who worked on silicone chips, I apologize 🙂

kennytilton15:03:10

:rolling_on_the_floor_laughing: Well, the M1 turned out OK, I guess you did not do too bad. I recently had a lead engineer half my age give a candid explanation for why the DB was not 3NF, meaning I could not look at it and understand it. "We find it easier this way." Great callback to Barbie and math.

Ben Sless15:03:23

I may have worked on M1, but I'm not sure. Willing to take criticism regarding Intel's Cannon Lake

kennytilton12:03:01

"...making Cannon Lake not only one of the shortest-lived microarchitectures of Intel, but also the shortest-lived 10 nm x86 CPU microarchitecture (with only one CPU model to be released and manufactured for 1.5 years)" History passes faster in hardware!

Ben Sless13:03:18

The fabrication process allegedly had terrible issues with it. Blame the material engineers and physicists 🙂