Fork me on GitHub
#off-topic
<
2020-08-06
>
Mno10:08:20

I, personally, wouldn’t try to get around it, and just learn how they do it at the company, learn the style (even if only as a curiosity) and experience the tradeoffs yourself so you know why FP does FP things. Less conflicts, more perspective.

👍 3
Daniel Tan10:08:54

how does one even do FP in golang

Aron11:08:22

Or when the 'Senior Business Analyst' who is also a "Software Developer" messes up the R functions on the backend before the data gets loaded into the database, the job becomes "please invert the axis on the frontend so that they tests pass".

6
noisesmith15:08:17

no joke, yesterday I got tagged for a "display issue" in the frontend of an internal tool, after reaching out to multiple other teams it turned out that the root cause was a nightly script on a microservice had silently decided not to run the night before

😆 3
😭 3
flowthing18:08:55

So there’s this quote in the “History of Clojure” paper: > Type errors, pattern matching errors and refactoring tools are venerated for facilitating change instead of being recognized as underscoring (and perhaps fostering) the coupling and brittleness in a system. I’d be interested in hearing more on this topic. I’ve seen the “Maybe Not” talk and I appreciate how nullability being tied to the type of a thing rather than the context where it’s used can be problematic, but I don’t quite see how type errors or refactoring tools, for example, would be conducive to fragility, as such. An example would be particularly enlightening. Any thoughts? (I’m not interested in pitting static and dynamic typing against one another. I’m just interested in the reasoning behind the quote.)

ghadi18:08:22

my take on it is that people falsely venerate the idea of changing something and having the compiler tell you the 57 places that need to be fixed

ghadi18:08:36

that's a signal that there are 50+ things that shouldn't care, but do

ghadi18:08:11

"but the compiler helps me with the coupling" is tool-assisted coupling

💯 6
jaihindhreddy08:08:00

I think I'm gonna steal "Tool-assisted coupling"

folcon16:08:44

I think that this sort of argument is in my experience met with a level of suspicion that I find surprising... A lot of which in my mind boils down to, "so how do you handle it then?" There'd be a lot of benefit in my mind to clearly spelling out a better approach...

👍 3
ghadi18:08:34

the problem is coupling

rich 21
👏 3
flowthing18:08:23

I see, that does makes sense. Thanks!

seancorfield18:08:58

That's a lot more eloquent than the long reply I was still halfway through writing but... yeah, what he said!

seancorfield18:08:48

I view it as a bit similar to the way folks seem happy to accept vast amounts of boilerplate in Java because "my IDE writes that code for me".

borkdude18:08:42

The people from "strongly typed" (whatever this may mean) languages will tell you that their language is a whole different experience from Java, but you still have the same coupling problems I think.

☝️ 3
seancorfield18:08:23

This is all contributing to why I like really simple tooling in my editor and around my language: I don't want to end up depending on "magic" happening and lose my connection with what's really happening under the hood.

seancorfield18:08:31

(`lein ring ...` is a good example of that, in my opinion: beginners who start out that way don't know how to start the HTTP server themselves and so they don't adopt a "healthy" REPL workflow around working with server processes)

borkdude18:08:33

Type systems aren't so much magic though. Garbage collection, JIT, etc, is more magic than types I would say.

seancorfield18:08:17

Code generation of getters/setters, code modification through refactoring tools...

noisesmith18:08:01

there are a lot of things that can shore up bad designs, static type checks can do that, editors with "jump to source" shortcuts facilitate bad code organization / factoring

Mark Gerard14:08:59

Is this similar to open source tools in the repl? Or are you talking about something entirely different?

noisesmith17:08:46

I'm talking about the ability to instantly jump to definitions and usages for editing - it can encourage disorganized code bases with coupling between unrelated components

Mark Gerard18:08:42

@noisesmith I could be missing something but I am unsure how rummaging through text files to find a symbol is a good idea that encourages good design. Sure, you could use regex | grep | equivalent, but my understanding is that the option was introduced in IDEs like eclipse to help with finding methods, which are the primary method of dispatch (`indirection?)` in a language like Java. The verbosity of the language can lead to a proliferation of methods, and jumping from lexical scope to find one can take long. I think CIDER supports a similar functionality with M-., but I do not see how it encourages bad structuring of code. Any tool can be abused, guess is what I am saying

noisesmith18:08:25

having to jump to usages and definitions should have friction to it my experience is that codebases made by people that use their jump to source editor feature frequently are more likely to be poorly factored, and leak implementation details if I have to edit 12 files to make one feature, and 40 files if I change an implementation detail, these are signals that my design is broken, and I want to see those signs and fix them early rather than later, I want to directly experience that friction so I can fix it early

noisesmith18:08:28

that said, I don't mean "never use jump to source" I mean: if your project is impossible for me to collaborate with you on, because I choose not to leverage jump to source in my workflow, that is a flaw in the project

Mark Gerard18:08:30

I am curious - Do you mean there is no project that can be sufficiently complex to warrant the existence of such a feature? And would you eschew such a project still? 🙂

noisesmith18:08:34

it's the clojure version of those java spring framework projects that are impossible to understand or work on if you don't use a powerful IDE - it's using an external tool to patch over flaws in your core design

noisesmith18:08:08

I can navigate clojure.core just fine, it's probably bigger than your project, it's bigger than the apps I am describing

Mark Gerard18:08:13

I agree with you on the last point. I think java frameworks are designed around the tooling

noisesmith18:08:19

it's not the project size that causes the problem, it's bad design

Mark Gerard18:08:22

Minus the tooling, you are lost, which is a shame

noisesmith18:08:05

right - but the jvm and its built in offerings? I can get by with the javadoc function in the repl - it's a tooling and community failure, not a problem with the language

noisesmith18:08:16

and I want to help our community avoid similar mistakes

Mark Gerard18:08:37

@noisesmith You are right, but none of the issues you are raising are because of the jump to - that is a feature which is abused. Bad design can leak into any project since it arises from a human flaw (lack of understanding or architecture and design), and not necessarily tooling. Yes, tooling can encourage it, but I doubt it is the source.

Mark Gerard18:08:57

I respect your nobility in trying to help the community look in the right direction, that's admirable mate

noisesmith18:08:55

> there are a lot of things that can shore up bad designs, static type checks can do that, editors with "jump to source" shortcuts facilitate bad code organization / factoring I was at least attempting to be a bit more nuanced - I'm also not saying "never statically check types"

noisesmith18:08:27

but I want to encourage being careful about ways tools can facilitate bad designs

Mark Gerard18:08:12

I saw that, but I couldn't wrap my mind around the jump to source feature. I certainly agree with your points, but want us to be clear that certain feature x does not necessarily lead to code smell or bad design y. That is a function of dev knowledge + experience -

Mark Gerard18:08:54

Someone can quip that macros are bad (I have heard such language out there). Some language have even gone further to introduce hygienic macros - whatever that means. But you agree that it would be wrong to imply that having macroexpand feature in clojure can be a source of bad design

noisesmith18:08:24

right, but if I can't understand any of your code as I read it, without a pop up tooltip that automatically macroexpands the source, that would be the same flavor of problem I am talking about

noisesmith18:08:05

the problem isn't reading the source, or expanding macros, or having tooltips - it's lazy development enabled by tooling which turns design flaws into negotiable tech debt

Mark Gerard18:08:02

Gotcha yer, thanks for painstakingly explaining your thinking through to me. I appreciate it -

borkdude18:08:58

The same could be said of garbage collectors and functional programming: they facilitate writing slower code :)

noisesmith18:08:36

there are multiple metrics of good vs. bad code, legibility is the only one that is fungible, it can literally be traded in to gain other improvements

noisesmith18:08:01

if gc and fp help me improve legibility, they can free my time to improve performance or resource usage where it matters

noisesmith18:08:09

if I don't have legibility, I have nothing

☝️ 3
noisesmith18:08:42

what I don't like are language features / tooling features that help people skate further on reduced legibility

borkdude18:08:47

Some people would argue that types help them read code better, since you don't have to guess what f and xs are.

noisesmith18:08:04

but they use type inference so they aren't even able to read the type in their code

noisesmith18:08:18

it's something that informs the tool, they then leverage that tool

noisesmith18:08:39

I'm not saying "no types", I'm saying they are over emphasized

lilactown18:08:57

us advocates of dynamic typing generally assume that without types, people will structure their code in a way that is less coupled and better organized

lilactown18:08:38

advocates of static typing generally assume that people will be able to encode useful properties of their system into their types, and that doing this is helping solve some business problem

lilactown18:08:44

IME both of these are wrong lol

💯 9
noisesmith18:08:38

the only times that typing problems have been a serious issue in my clojure projects have been when I'm collaborating with people who are trying to code the way they would in ml, without any of the tooling that makes this productive

borkdude18:08:02

@noisesmith can you give an example of that?

noisesmith18:08:38

@borkdude refactoring the unabstracted implementation of a concrete data type touched by multiple microservices and marshalled via a shared io library

noisesmith18:08:56

in a language with strong types, this is trivial, though tedious

noisesmith18:08:03

in clojure, it's a huge waste of time

noisesmith18:08:20

root problem: you didn't abstract the data properly in the first place

noisesmith18:08:37

which you can get away with in ml where the compiler takes care of it for you

noisesmith18:08:00

weeks of time wasted on adding a few bytes to a field

borkdude18:08:23

@noisesmith I wonder what they did to marshall that thing to e.g. JSON? In Clojure we do (cheshire/generate-string ...), in typed languages you often describe a per-field mapping to JSON, which is tedious (but can also yield much more performant code if you don't want to serialize everything). How did they do that in Clojure?

noisesmith18:08:14

this was an overly coupled combination of the apache avro lib (used with kafka) tied to clojure.spec and some overly clever glue code

noisesmith18:08:21

diagnostically, it's the failure to create an explicit boundary between "data from the outside" and "data inside my application"

noisesmith18:08:44

this was also enabled / festered by using a monorepo, so each service could exploit internals of others

borkdude18:08:52

I'm not sure if monorepo has anything to do with it. I once worked in a Scala team where they had many repos and one of the repos was called types :P

flowthing18:08:48

That is probably the most Scala thing I’ve ever heard. 😛

noisesmith18:08:55

anyway - half assed imitation of strong typing ended up with the worst of clojure and ml in one sweeping endless-layers-of-slowly-exposed-bugs refactor

borkdude18:08:40

@noisesmith Do you think just throwing EDN on Kafka and use spec on that EDN would have been better?

noisesmith18:08:52

sure - monorepo didn't cause the problem, attempted typing didn't cause the problem, clojure's laxity with data didn't cause the problem, but they all contributed to the storm that occurred

noisesmith18:08:24

@borkdude I think it would have been more tractable, but this is more about the way that contracts between services are managed, and how they are designed in order to evolve together IMHO

flowthing18:08:24

Once again proving that the really difficult bits of software development have nothing to do with things like typing. 😛

💯 3
noisesmith18:08:22

also there was an attempt to make spec generate avro consumers/producers and this was exciting but in the end a source of a lot of problems, it would have been less work and less brittleness to just have the two definitions, or a DSL that was smart enough to create a spec and avro schema that agreed with one another

borkdude18:08:31

I heard Mia (https://twitter.com/buttpraxis) talk on defnpodcast about her work with Kafka and spec. She said that they would almost never write specs directly but write code generating the specs for them. Do you perhaps work at the same company? ;)

noisesmith18:08:44

we were on the same team

noisesmith18:08:15

I think my memory / long term experience with the avro generator is less happy than hers

noisesmith19:08:15

then again, thinking out loud at this point, if we were using a strongly typed language, we probably wouldn't have tried to pretend that our message exchange format was the same abstraction as our data types / language level type verification

noisesmith19:08:30

I mean, that would be absurd

noisesmith19:08:40

but if engineers on our team didn't think that data types were the solution to contracts between services, that also would have prevented the problem

noisesmith19:08:52

uncanny valley, worst of both worlds IMHO

noisesmith19:08:55

but I wouldn't characterize the company, our team, or those collaborators by this situation, it's just the worst thing technically that came up, IMHO

borkdude19:08:18

does the team still exist?

noisesmith19:08:50

oh no this was years ago, and many people left shortly after the service went live

borkdude19:08:32

I guess this is one of those things where you can write tons of books "Patterns of Enterprise Blah architecture" about

noisesmith19:08:31

Zach Tellman (who consulted with us about architecture), was writing Elements of Clojure (a very good book about how to do architecture correctly) at the same time he consulted with our team

noisesmith19:08:02

in fact if people had followed his advice more carefully we wouldn't have had this issue

borkdude19:08:15

I confess I have that book but still haven't read it

3
flowthing19:08:28

One counterargument I believe I’ve heard is that in some projects, the realities of software development can drive teams into situations where you just need to make a change that requires a fix in 57 places, and if you have static typing, you at least have some help in making that happen. Yes, the real problem lies elsewhere, but whatcha gonna do? Something along those lines.

noisesmith19:08:08

right - it's a strategic enhancing of brittleness to make things fail fast so you know they are fixed

noisesmith19:08:55

but it mixes very poorly with a strategic flexibility that lets you not care about the implementation details of other parts of the code

noisesmith19:08:11

unless the relationship between the two is very well defined

flowthing19:08:54

Yeah, I can see how compound interest can start accumulating there.

noisesmith19:08:04

it's as if a tent maker and a skyscraper maker were trying to make modular parts of one structure

borkdude19:08:30

Maybe a campsite on a skyscraper would work

noisesmith19:08:44

sure, that's a very well defined relationship

noisesmith19:08:47

on a human level, it turns so easily into a type-advocate bitter about the sloppiness of the language making these problems hard, and a dynamism advocate bitter that coupling is making the problems hard, and hey they are both right - the mismatch and ill defined boundaries are the problem, not the things on each side

borkdude19:08:40

so no matter what language you throw at it, the system architecture is the more important issue. which is also what Rich Hickey has emphasized in his talks I think

noisesmith19:08:57

that's my take, but hey I'm a clojurist :D

noisesmith19:08:29

imagine how terrible ring would be, if the core data abstraction inside your middleware and handler code was the TCP packet

noisesmith19:08:54

we recognize how wrong that is, but do the equivalent too often in microservices

borkdude19:08:22

what would be a microservice equivalent of what TCP is to ring?

noisesmith19:08:42

whatever data abstraction describes your IPC - so the avro message, or json, or transit, or edn - I'm advocating for having an explicit boundary between app data and data sent to other services

richiardiandrea03:08:01

I think the concept nowadays is called Hexagonal Architecture...I have been reading a bit about it mainly triggered by curiosity. The Nubank microservice example contains a reference to it so I discovered it from there the first time. https://github.com/nubank/basic-microservice-example

Drew Verlee18:08:49

Is the issue about ipc or is it that different readers (clients, services, etc...) Can have different interpretations of the same data? For example, you might think of a name as meaning full name asc where another reader might expect just the first name. Internal, we always have to store the data as granular as possible to service all caller's. So what we need a translator for is from the readers request to an internal representation. I have felt it would be interesting to capture that translation model in the database itself using a rule engine. E.g if a name gets added we create a version for the internal system and one for each representation. Internal.Fname : drew Internal.Lname: verlee A.fullname: drew B.fullname: drew verlee The alternative is to have code that does the transition ever time someone asks. The downside is that logic it self isn't queryable.

noisesmith18:08:48

It's that the "interpretation" is implicit in sloppy clojure code, and doesn't have a separate transition layer to relevant values

noisesmith18:08:02

"it's just data" but misapplied across the wire

noisesmith18:08:26

putting the translation model in the db itself is the worst possible choice

noisesmith18:08:52

it doesn't fix the problem here, in fact it amplifies it

Drew Verlee18:08:55

I'm not sure I understand the problem then.

Drew Verlee18:08:24

I was interpreting it as Different caller's mean different things with the same request. So who is asking matters. That has to be handled somewhere.

Drew Verlee19:08:36

I wasn't saying you build an API which depends on the caller. I was thinking that ingested data has to be categorized. In that light, yes it doesn't make sense to store multiple representations

borkdude19:08:54

unless you control all those services, it's the same problem with internal APIs and public or library boundary APIs?

noisesmith19:08:12

people learned the lesson earlier with XML / CORBA but less generally than they should have I think

borkdude19:08:48

e.g. don't return field :foobar/id unless it's important. you can never remove it, since that would be breaking

noisesmith19:08:03

@borkdude yeah actually, that's a fair point as well, though within some inner domains it makes sense to keep one data format for sanity's sake instead of having onion layers of transforms

noisesmith19:08:18

the process itself is a nice place to have the explicit boundary

seancorfield19:08:03

This is why I advocate for three separate representations, if only in naming: API (external) data, Domain (internal) data, and Persistence (external) data -- what you expose to clients, what you traffic in within your business logic, and what you write out/read in from constrained external sources.

💯 15
seancorfield19:08:45

That said, in many cases, you can control enough of the Persistence format to let it and your Domain format overlap substantially.

noisesmith19:08:35

another classic example of doing this very wrong: ORM

noisesmith19:08:16

same basic problem at root - trying to impose coupling across a boundary that's better left standing

seancorfield19:08:18

ORMs are "wrong" on so many different levels...

Alex Miller (Clojure team)19:08:42

the O level, and the M level mostly :)

3
💯 3
seancorfield19:08:59

(one of the inherent problems is conflating identity and state there, between objects and relations)

noisesmith19:08:14

I find it interesting that I've felt smug about ORMs for a while, being a functional programmer, but fell into the same mistakes on microservice design

noisesmith19:08:53

the problems can persist even with immutable data as values, if you don't have the right overall structure

seancorfield19:08:51

We're lazy programmers. We look at the two domains and we squint a bit and we convince ourselves they're "close enough" and we don't need to bother designing and building an actual "mapping" to separate them.

emccue23:08:17

I've just given up and assume anything I do where I feel clever about it was a bad idea

12
emccue23:08:17

I might still do it

emccue23:08:26

But I'm waiting for the other shoe to drop