Fork me on GitHub
#off-topic
<
2020-01-08
>
andy.fingerhut01:01:03

Looking for suggestions for some public data set that would make for an interesting example to play with in a small demo Clojure project, preferably one that you can download the whole thing in at most 50 Mbytes or so, and my personal preference would run along astronomy/geography/geology/mapping/chemistry/physics subjects, and ideally would have an optional open source way to make a charts/graphs/simple 3D visualization.

mloughlin10:01:41

https://freegisdata.rtwilson.com/ is a huge list of geographical datasets

mloughlin10:01:58

most of them can be visualised in a program like QGIS

mloughlin10:01:20

for my sins, I have a MSc in GIS and I'm happy to give advice

andy.fingerhut01:01:41

Hoping for something significantly smaller/simpler than SciPy kinds of things right now, and doesn't involve a database, e.g. all data is in a JSON/EDN/CSV format file or similar, and ideally the optional chart/graph/3d part would be a small-ish download, too.

dpsutton01:01:05

Tim baldridge used a set like that in his Denver build a logic engine talk

wcalderipe02:01:29

Does someone around knows good resources about banking domain and systems?

adamfeldman16:01:15

Monzo is a challenger bank with a nice tech blog: https://monzo.com/blog/technology

wcalderipe22:01:45

Thanks for sharing @UCHV4JZ7A

👍 4
wcalderipe23:01:26

yesterday i stumble on open banking specifications materials. sharing here for curiosity purpose https://openbanking.atlassian.net/wiki/spaces/DZ/overview

👍 4
dpsutton13:01:28

i'm giving an ignite talk tonight on using shadow cljs to local devs. Made a presentation here and would love feedback from anyone willing. Ignite talks are 5 minutes long, 20 frames advancing every 15 seconds. I'm taking people from an empty directory to a deployed netlify reagent app in 5 minutes. Would love to hear what is missing, what is extra, etc. if anyone has time. I'll be talking during it so details are flushed out by speaker but if any topics are obviously missing would love to hear. Link to presentation in thread

p-himik14:01:49

Slide 17 seems to have a duplicate.

p-himik14:01:01

Slide 18 - can you give an example of something that cannot be used with shadow-cljs? Also the last point about not as many people making cool stuff - we still have JS interop, so that counts IMO. :)

p-himik14:01:58

Slide 19 mentions re-frame but it seems completely orthogonal to the topic of the presentation.

dpsutton14:01:16

i can't figure out what's up with 17. bizarre. very true on interop. 19: that's a learning course mentioned because i think its done well

dpsutton14:01:21

(thanks for looking over it)

p-himik14:01:24

Also maybe it's worth mentioning Zulip and http://ask.clojure.org at the last slide.

dpsutton14:01:36

for sure on ask.clojure!

p-himik14:01:22

Slide 17 has a vertical navigation as well - maybe because navigating to the next slide just adds a link to the current slide? Maybe that's what causing that duplicate. Regarding re-frame - it is of course up to you, but I'm still hesitant that it's worth it to put it in there. Re-frame brings some concepts that have nothing to do with CLJS and Reagent. And knowing those concepts is required to use re-frame properly. Feels like extra cognitive load right off the bat.

hindol16:01:57

Clojure has a nice, "data-driven" API for AWS but a similar library does not exist for Azure. It's bugging me. Microsoft publishes the Swagger specs for all the APIs. I want to generate a Clojure client library from the Swagger specs but I am afraid my Clojure skills fall short. Anyone want to join me in this quest (hopefully with better Clojure skills than mine)? P. S. I looked at Swagger's own generators and the generated API seems overly complicated.

gklijs17:01:02

We do generate Java classes from Swagger, but they are not nice in Clojure. Maybe this, https://github.com/zalando-stups/Swagger1st works for you. Most Clojure libraries with Swagger only seem to support generating the Swagger json.

hindol17:01:08

Thanks for the link. I am willing to write custom code if that makes the resulting Clojure API pleasant to use.

hindol18:01:22

Wow, that's super helpful. I want to do something similar but want to tune it more for Azure, like an Azure SDK for Clojure. Martian seems to be a great starting point.

adamfeldman17:01:01

In SME/SMB/startup businesses, should software be built with the assumption it will be re-written from scratch every ~4 years? What is the typical/practical lifetime of a software project in non-enterprise businesses? What is your experience with the lifetime of Clojure(Script) projects vs. other ecosystems? By software I mean any of the following (non-exhaustive): • Internal, web-based tools used by employees of the business • In the case of a SaaS company, the web-based product itself (which is used by customers) • “Non-product” customer-facing systems (customer portal, etc)

mloughlin17:01:19

IME working at several SMB the revenue generating products stick around for decades. Occasionally someone in management will decide it's a good idea to do A Big Rewrite, but the project is usually abandoned in favour of incremental improvements when it inevitably takes too long. There usually aren't enough developers to tackle full re-writes without employee churn ruining the project. (Dev team sizes < 10)

seancorfield17:01:53

Full rewrites are, indeed, very hard to pull off successfully. I've been involved with several over my career. They always take far longer than anyone can imagine, so they cost more, eat more resources, and ultimately often deliver a product that doesn't have feature parity with the system they replaced.

seancorfield17:01:23

That said, we've been through an incremental rewrite of our systems, from a legacy JVM stack to Clojure, over the last several years and we're happy we made the switch. That's one of the huge benefits of being hosted (on the JVM), you can mix'n'match languages and shift your stack gradually over time. I'm not sure how (or even if) that can apply to JS->cljs on the front end tho' -- I get the impression that cljs basically assumes it is a full app?

Alex Miller (Clojure team)18:01:01

I think I've seen more full rewrites fail than succeed. It's possible and can even be transformational to the business but there are a lot of ways for it to go wrong.

seancorfield18:01:07

(and there's also the downside that while some of your developers are working on the rewrite, there's less bandwidth for enhancements to the existing product so either your current product goes stale or you add more resources and then the rewrite starts chasing a moving target, having to implement all those new features in addition)

seancorfield18:01:40

So, yeah, even tho' I've done it "successfully" a handful of times, it's not something I recommend 🙂

adamfeldman18:01:41

Thanks for the thoughtful responses y’all, I really appreciate the feedback. Sometimes I’ve seen advice around the web that says to treat software as disposable (due to pace of business and technical change). Sounds like if disposability applies, it’s not ever in SME/SMB — perhaps true in rapidly-scaling well-funded mega-startups, if anywhere, but that’s a very special case relative to the world as a whole.

mloughlin19:01:04

There's the notion that writing decoupled code leads to sub-systems being easily replaceable/disposable -- keeping logic out of views, for example, which I find chimes with Clojure's decomplect mantra. It enables the incremental rewrites (e.g. scrapping an ORM when you find out it's slow as molasses). But realistically re-writing an entire project is an organisational challenge not a technical one.

mloughlin19:01:04

Are you referring to Sandi Metz's output on code half life? https://www.sandimetz.com/blog/2017/6/1/the-half-life-of-code > Code that isn't easy to replace doesn't get replaced, instead it gets expanded. Its conditionals get bigger. The number of class names it knows about grows larger. These sorts of expansions tightly couple the code you're changing to other parts of your application. This coupling makes it difficult to swap in alternative implementations, which it turn leads to a long half-life for the code.

orestis20:01:37

We are currently mid-rewrite (node/jquery -> clj/cljs) using the strangler pattern. So we are adding new features and overtake parts of the app as needed. It’s working fine so far, and the business sees it as just a faster way of getting new features out.

seancorfield21:01:50

@U7PBP4UVA I'd love to hear more about how you're piecemeal converting a JS app to clj/cljs!

👍 8
orestis06:01:22

I’d love to write it down someday or perhaps a talk - it’s nothing really groundbreaking but perhaps worth sharing?

seancorfield06:01:18

Definitely worth sharing 🙂 But I understand the constraints that mean it might not get written down...

danielstockton15:01:28

I think the https://martinfowler.com/bliki/StranglerFigApplication.html is a good strategy, rather than rewriting everything in one go. Similar to what @U04V70XH6 said above, with mixing Java and Clojure over time. The hard part is finishing it off, after you've already replaced most of the important parts, so the old application doesn't limp on for years.

Daniel Hines18:01:20

I'd like to understand what data structures/operations are commutative, and which are not. I feel like this is probably well understand by computer scientists (which I am not). Does anyone have suggestions on where to start looking into this?

dpsutton18:01:05

do you know a definition of commutative?

Daniel Hines18:01:48

> involving the condition that a group of quantities connected by operators gives the same result whatever the order of the quantities involved, e.g., a × b = b × a.

dpsutton18:01:08

and i mean that in the sense that the defintion is (f a b) = (f b a) for all a, b in the set that f works on

dpsutton18:01:31

so then start reasoning from there. string concatenation is certainly not

dpsutton18:01:34

merge isn't either

dpsutton18:01:43

set/union is

Daniel Hines18:01:15

I could imagine some other kind of merge with a different way of determining priority that is commutative.

dpsutton18:01:19

but for a general list of all operations that are that sounds like a tough order

dpsutton18:01:35

in general merge isn't. but if all the values of a map are sets then it can be

dpsutton18:01:57

so (merge-with f a b) would be commutative if f were commutative

Daniel Hines18:01:59

I was envisioning maybe someone has created a small language for distributed programming that consists entirely of commutative operators.

dpsutton18:01:18

interesting

Daniel Hines18:01:19

I figured Bloom had to be. I'm hoping there's something smaller than bloom I can start with.

Daniel Hines18:01:29

Some tiny "commutative calculus"

dpsutton18:01:00

you could look up abelian groups

hiredman18:01:11

I think most of the interest in this kind of thing has moved on to crdt based approaches

dpsutton18:01:14

might be what you are thinking of if you want the abstract properties of an abelian gorup

Daniel Hines18:01:30

Nice @U11BV7MTK, that's what I was looking for.

Daniel Hines18:01:59

I'll also take a look at CRDT's @U0NCTKEV8. Thank you both.

hiredman18:01:28

Commutativity for distributed programs usually starts with sets, but some operations on sets don't commute (additions and removals of the same item) and which point you start needing to build something more which is where you start getting crdts

dpsutton18:01:27

you could also look for commutative semigroup. relaxes the assumptions of elements in the set (inverses and identity)

Daniel Hines18:01:50

Ok... so it's looking like some basics in abstract algebra would do me good. Is that right?

hiredman18:01:00

the idea of folds, commutativity and associativity are very useful

Daniel Hines18:01:49

Fold? When I hear "fold" all I think is "Oh, they mean reduce", but I suspect there's more than I'm getting.

hiredman18:01:03

reduce is one possible way to do a fold

hiredman18:01:01

https://en.m.wikipedia.org/wiki/Fold_%28higher-order_function%29 has some diagrams of two fold variants, foldl and foldr, reduce is foldl

hiredman18:01:25

but those are both ordered folds(the order comes from the structure being folded), where commutativity of the function being folded doesn't matter as much, in a distributed setting where data is partitioned you might want to fold over each partition and then merge the results, in which case commutativity is going to be very important