Fork me on GitHub
#beginners
<
2017-09-09
>
tagore04:09:10

While it's written in CL notation, the best book on macros I know of is Paul Graham's "On Lisp."

tagore04:09:52

Clojure's semantics differ from CL's in many respects, but when it comes to macros themselves they are very similar, so "On Lisp" is essentially as useful for Clojure as for CL.

tagore04:09:26

I think something similar could be said for SICP.

tagore04:09:16

(Not about macros of course, which SICP doesn't even touch on- I mean that most of what SICP teaches is transferrable to Clojure)

Drew Verlee12:09:50

What is the purpose of a docstring? What are we trying to convey beyond what the code does already? In Dynamic languages i feel compelled to mention the function paramaters types and its return type, which seems like an admission that I should be using a static language. Beyond that, i generally try describe what a function does at a Domain Level. However i notice a lot of reputable Clojure Devs dont bother with them at all. Is there a way to objectively look at this, or is it all context specific?

Alex Miller (Clojure team)13:09:34

Re the types, I think spec is a better way to say those things now and if you have them, doc automatically includes them

hmaurer13:09:31

@drewverlee code doesn’t always convey its intent at first glance. For example, if I look at the code of clojure.core/map, I am glad it has a docstring 😄

nimblerabit13:09:18

I'm trying to do something somewhat related to chess, where I'm generating all possible moves a piece can make. For example, if I have a king at position [2 2], that king can move to positions [1 2], [1, 1], [2, 1], [3 2], [2 3] [3 3], [1 3], [3 1]. I can describe this easily logically (the set of all moves where y2 - y1 <= 1 and x2 - x1 <= 1), but I'm having trouble executing in clojure. I'm trying to come up with some way to do this generation for various pieces (king being the simplest), does anyone have any ideas how I might go about it? Right now I'm messing with iterate but I just can't seem to get this to work.

noisesmith13:09:37

@nimblerabit why not express the moves by distances and calculate the positions based on the start position?

hmaurer13:09:10

(defn king-moves [x y]
  (for [dx [-1 0 1] dy [-1 0 1]] [(+ x dx) (+ y dy)]))
?

hmaurer13:09:42

(you would need to remove the “no-op” move from that list)

noisesmith13:09:35

@hmaurer I think it would abstract better if you left the math to the "mover" function, that way the moves themselves only differ y the precise thing that makes them different (the distances) and the mover can do the actual calculation instead of having the same logic in every one

hmaurer13:09:39

this could get complicated with other pieces where you’ll have to take “blocking” into account though

hmaurer13:09:21

@noisesmith yep; ideally you would have some sort of composable mover functions, right?

noisesmith13:09:53

if the pieces moves are just relative numbers, both the "threat / block" function and the "move" function can reuse the same data

noisesmith13:09:16

if the pieces moves were a function, you can't reuse the inside, so you end up copy/pasting it into the threat / block code

noisesmith13:09:19

(or doing redundant calculations I guess)

hmaurer13:09:05

@noisesmith how would you abstract the movement of a bishop, which can move diagonally until it reaches an enemy piece?

nimblerabit13:09:20

So that solution makes sense to me, but the reason I didn't go in that direction is I was unsure how to do something like the queen that can move different number of spaces

noisesmith13:09:25

@hmaurer as I said, a "distances" data to get all the possible distances (if they don't leave the board or pass through a piece) and then a "block / threat" function which tells you where there's an intersection (if any)

noisesmith13:09:13

@nimblerabit well, it's possible to iterate them completely - since the board is only N tiles

hmaurer13:09:22

@noisesmith how does that constrain motion along a particular path, e.g. diagonals only?

noisesmith13:09:41

@hmaurer by only including the diagonal steps in the comprehension

noisesmith13:09:11

a for, like yours, that generates moves where the absolute value of the x distance and y distance are always equal

noisesmith13:09:33

so, the lazy way (logic wise) is (for [x (range -11 12) y (range -11 12) :when (= (Math/abs x) (Math/abs y))] [x y])

noisesmith13:09:57

of course there's a more efficient way to calculate that, but that's concise

noisesmith13:09:22

and since that's all the moves for the bishop, it only needs to be calculated once

noisesmith13:09:35

and the queen moves can be calculated by concatenating bishop moves and rook moves

hmaurer13:09:27

(apply union ((juxt rook-moves bishop-moves) pt)) ?

nimblerabit13:09:31

Alright I think I follow what you're saying here, I'm going to give it a shot. Thanks for the help.

noisesmith13:09:11

@hmaurer (concat rook-moves bishop-moves) or (distinct ...) if you really can't repeat the [0 0]

noisesmith13:09:20

union on lazy-seqs isn't good

hmaurer13:09:32

@noisesmith ah, why so? doesn’t work?

noisesmith13:09:50

clojure.set functions silently generate garbage for inputs that are not sets

noisesmith13:09:24

oh, I guess union actually works here

hmaurer13:09:29

(I was assuming the moves functions returned sets)

noisesmith13:09:33

but I consider that an accident, not something I can count on

noisesmith13:09:57

I guess sets make sense

hmaurer13:09:16

Calculating chess moves is a fun problem though; I tried to do that a while back in javascript and it’s non-trivial to do cleanly

noisesmith13:09:22

(but none of the code so far, including yours, returned sets)

hmaurer13:09:39

Actually, I wonder what the cleanest way would be to write a “line of sight” function for a 2d grid like a chess board

hmaurer13:09:56

actually it would depend how you define line of sight; nevermind

noisesmith13:09:54

right, and there's the one piece that doesn't have a line of sight constraint

nimblerabit14:09:46

My actual task is creating a program that determines reachability (every spot a piece can reach in one move) on a 3d board, for any arbitrary piece (can have user defined movement capabilities). It's a fun problem so far.

nimblerabit14:09:30

Ahh that's what my professor calls it, it's a 3d grid of x, y, z positions

hmaurer14:09:49

oh, so not a standard chess board

nimblerabit14:09:03

Nah it actually has nothing to do with chess

nimblerabit14:09:03

just a useful jumping off point for modeling this

hmaurer14:09:42

Oh nice, that does sound like a fun problem

tagore14:09:46

@drewverlee How much code should be commented is a subject on which opinions vary widely.

tagore14:09:30

I think there's some truth to the idea that too much commenting is a bit of a code smell- ideally the code would be self-explanatory.

tagore14:09:45

But that's actually very hard to achieve in practice, in my experience.

tagore14:09:49

I see a lot of code written by people who internalized that idea, but then didn't really work very hard at making the code clear...

tagore14:09:23

The problem with comments/docstrings/non-executable documentation in general. is that they can go out of sync with code pretty easily if people aren't diligent about maintaining them. And an incorrect documentation is worse than no documentation at all.

noisesmith14:09:07

I try to always document protocols, other things mostly just as needed

tagore14:09:00

It's that "as needed" that is the sticking point though 😉

tagore14:09:46

Also, different programmers have different tolerance for reading comments.

tagore14:09:21

I'm pretty verbal, but I work with a couple of people who aren't very...

tagore14:09:31

We butt heads about this occasionally- one of them was reviewing some code I wrote and said "I'm never going to read all that... I find comments like that painful" about a comment I'd made.

tagore14:09:18

I looked at it and was like... it's a four line comment, each about 60 characters of actual text.

tagore14:09:47

Even though I think that's kind of weird, if he's not going to read my comments they aren't going to be very useful to him if he has to work on that bit of code.

rcustodio15:09:51

Using protocol + record for state managing is really nice, right?

rcustodio15:09:41

I was reading component, I'm not using it but I'm using the idea of it

noisesmith15:09:16

there's nothing inherently stateful about protocols, but I like the way component uses a protocol

noisesmith15:09:29

protocols let us reuse code with better separation of concerns than you would have with inheritance (usually) - by thinking about the actions you define alone, as an abstraction, and encouraging you to write code in terms of only those actions

noisesmith15:09:38

and not implementation details of the things you are using to fill them

noisesmith15:09:10

protocols are why we can map or reduce over just about anything

rcustodio15:09:57

So, before I was using (def state (atom {})), and then I read their code and Riemann, so I changed to protocol and record, and pass through arg (the record) to execute functions

rcustodio15:09:36

And I like the way it keeps organized... and functional

rcustodio15:09:42

And isolated

noisesmith15:09:00

that's definitely true - you can of course do the same thing by passing the stateful things as args

noisesmith15:09:25

but having a protocol and record helps clarify what the abstraction boundaries are (and how to properly extend, etc.)

rcustodio15:09:23

Yes, using in TCP and UDP which uses the same protocol but diff records

Lucas Barbosa16:09:46

Guys, I’d like to have some advice here. I’m experimenting some other things alongside with learning Clojure. Among them is Continuous Integration (using Jenkins) and automated acceptance testing. I would like to build a pipeline like this to automate : unit test (lein test) -> build (lein ring uberwar) and checkstyle (any style checking tool for Clojure you’d recommend?) -> deploy to staging environment (with database and everything setup) -> acceptance tests (using gradle and Java with RESTAssured framework) -> deploy to production environment What do you think about this? Am I missing something? Is this approach correct for automating the acceptance tests? Thanks!

noisesmith16:09:59

@lvbarbosa I have a task defined in project.clj that runs clean, cljfmt fix, check, test, eastwood (in that order, failing if any of them fail)

noisesmith16:09:14

eastwood is the style checker I recommend

Lucas Barbosa16:09:56

so you think it’s better to collapse together all the lein tasks?

Lucas Barbosa16:09:11

I mean, in one build task

noisesmith16:09:41

I do that because I like to run it regularly (usually each morning) locally while developing

:aliases {"morning" ["do"                                                     
                       ["cljfmt" "fix"]                                         
                       ["clean"]                                                
                       ["check"]                                                
                       ["test"]                                                 
                       ["eastwood"]                                             
                       ["repl"]]}

noisesmith16:09:54

so I can just run lein morning, get my repl, and start working

noisesmith16:09:18

or fix things before I go further if I discover my work from the night before (or whatever merge I might have done) put something in a bad state

noisesmith16:09:05

you could absolutely just run lein do check, test, eastwood, uberjar from the build task

noisesmith16:09:10

it will fail if any of the steps fail

Lucas Barbosa16:09:35

What do you think about the rest of the pipeline? Does it sound nice? I’m thinking about REST Assured because I know how to use it

Lucas Barbosa16:09:58

maybe using a similar clojure framework would be a better learning experience

Lucas Barbosa16:09:08

or maybe it’s too much information

noisesmith16:09:35

I don't know that deployment framework / acceptance test framework and I don't know much about the alternatives either

noisesmith16:09:57

I do know that CircleCI is implemented in clojure, so I would be surprised if their support for clojure deployment wasn't top notch

noisesmith16:09:39

well no, I did use jenkins, but it could have been a shell script as far as I was concerned (it tested, built a jar, scp'd the jar to a server, sent a rest command to make the container reload) - I assume it has actual features we weren't leveraging properly

Lucas Barbosa16:09:00

That’s the situation here on my thoughts. Jenkins is just doing what I could do in a shell script

Lucas Barbosa16:09:13

I’m doing it just for fun/learning

rcustodio16:09:25

I must learn how to write tests, some good guide?

rcustodio16:09:46

And... another thing that I still don't understand... when to write macros

sundarj16:09:27

only when you must

rcustodio16:09:56

And that is when??? Lol

sundarj16:09:26

when you try doing it with a function and it doesn't work, or you need some sugar around a particularly verbose api

noisesmith16:09:10

IMHO macros are for two kinds of programmers: people who are just starting (so you get some intuition of their uses and limits, helping you understand what goes on when you use other people's macros) and those who are very experienced (needing to make a new syntax to simplify a codebase)

noisesmith16:09:32

if you don't need a new syntax, or compile time evaluation, you don't need to write a macro

noisesmith16:09:06

and most people who think they need those things still don't need them

rcustodio16:09:19

Well, at least for now I'm using functions only.. if I get to its limits I expand to start learn about macro

noisesmith16:09:22

I've had the painful experience that using a powerful feature just because tends to lead to less than satisfactory results

sundarj16:09:30

that is true of mutation also 🙂

tagore17:09:59

@noisesmith I agree with the general thrust of what you're saying, but...

tagore17:09:13

I'm not sure I'd restrict them quite that heavily.

tagore17:09:49

I do think that you probably shouldn;t be writing macros in production code unless you already know when to write them though 😉.

tagore17:09:29

OTOH, you'll never really learn when they are a good idea if you don'y write some that you shouldn't have, and learn how and why they can bite you.

noisesmith17:09:46

I've found that I rarely find macros provided by libraries useful, usually the get in the way

tagore17:09:04

That's often the case.

tagore17:09:41

OTOH, I imagine you lean heavily on macros that come with Clojure...

tagore17:09:33

The main thing about macros, I think, is that you shouldn't reach for them unless there's no really good alternative.

tagore17:09:14

To know if that's the case you have to know enough to know what the alternatives are.

seancorfield17:09:56

As a data point, our 65K line code base has just 23 macros (compared with about 3,000 functions).

seancorfield17:09:12

And those macros are nearly all with-* syntactic sugar for an underlying function.

rcustodio17:09:59

A friend said that macros are just a simple "replace"... I think it will take time that I understand where to use it... for now functions, records and protocols are being enough

tagore17:09:40

Well, there's no softwre you can write with macros that you can't without them.

tagore17:09:53

And higher order functions can often do what you want a macro to do while still being first-class citizens at run-time (which is valuable.)

tagore17:09:56

Cleverness should be eschewed, except when it shouldn't.

tagore17:09:20

Macros are an invitation to cleverness.

rcustodio17:09:17

So macro would be a improved high-order function?

rcustodio17:09:39

The business logic?

tagore18:09:11

Not improved...

tagore18:09:16

different...

tagore18:09:38

In most cases where HOFs will perform the duties of a macro they should be preferred.

tagore18:09:05

Macros, in the end, are a notational convenience, and nothing more than that.

tagore18:09:39

But the importance of notational convenience should not be dismissed out of hand.

sundarj18:09:31

(defmacro alias-macro [new-name old-name] (defmacro new-name ['& 'body] '`(old-name @body)))` what am i doing wrong here? (just an academic exercise)

rcustodio18:09:19

I see, I think I understand a little more... but just writing it I will understand better... I will start to write more macros to reach its limits

rcustodio18:09:33

And then understand when to write it

tagore18:09:19

@rcustodio Probably worth abusing macros in order to discover their downsides 😉. I think everyone who has written a lot of Lisp has done that at some point.

tagore18:09:51

But the warnings about macro abuse are not idle...

tagore18:09:27

One of the very nice things about Lisps, and Lisp-like languages is that they make many things first-class at runtime that are not in most languages.

tagore18:09:50

To pass around functions in C you have to go through gyrations that make it hardly worthwhile in siple cases.

rcustodio18:09:36

I see, thanks for the tips @tagore

tagore18:09:50

In a Lisp (and I'll broadly include languages like Python and Javascript as basically Lisps, if not very good Lisps) functions are first-class. You can pass them around just like anything else. You can have functions that return functions.

tagore18:09:24

This is a powerful thing (though, as with macros, it is an invitation to cleverness.)

tagore18:09:50

Macros are not first-class.

tagore18:09:58

At least at runtime.

tagore18:09:17

(And there are caveats there I won't get into...)

tagore18:09:11

So you need to understand that when you write a macro you break one of the fundamental "nice things" about Lisp.

tagore18:09:00

It's worth it in some cases- especially when you need to control when forms are evaluated, the most common place I write macros for.

tagore18:09:20

Or when you want to abstract certain types of cleanup away, even in the face of exceptions (what @seancorfield was referring to, I think.)

tagore18:09:39

You could do the same thing with higher order functions, but what a pain... with-* is so much nicer.

tagore18:09:06

There's another use for macros- embedding little languages in Lisp.

tagore18:09:29

Common Lisp, for instance, is far less functional than Clojure.

tagore18:09:44

Part of the CL spec is a little language called "Loop."

tagore18:09:32

And I'm inclined to think it is the most comprehensive loop construct I've ever seen....

hmaurer18:09:39

@tagore is it similar to clojure’s loop?

tagore18:09:48

Not at all..

tagore18:09:47

Clojure's loop is essentially a recursion marker.

tagore19:09:27

CL's Loop is an entire imperative language embedded in CL.

tagore19:09:34

It allows you to say things like:

tagore19:09:36

(loop for i from 1 to 100 if (evenp i) minimize i into min-even and maximize i into max-even and unless (zerop (mod i 4)) sum i into even-not-fours-total end and sum i into even-total else minimize i into min-odd and maximize i into max-odd and when (zerop (mod i 5)) sum i into fives-total end and sum i into odd-total do (update-analysis min-even max-even min-odd max-odd even-total odd-total fives-total even-not-fours-total))

tagore19:09:12

Whether or not this is a good idea is an open question 😉

tagore19:09:23

The important thing to note here is that this is, past the first paren, not Lisp syntax.

hmaurer19:09:25

@tagore I assume it’s parsable by the lisp reader? or does it use a reader macro?

tagore19:09:39

Good lord no!

tagore19:09:02

One advantages of Lisp is that the reader is very simple.

tagore19:09:31

It would be very ugly to modify the reader to understand loop- which is why it's a macro.

tagore19:09:39

There is a separate step in the CL "build" process called "macro-evaluation-time"

akiroz19:09:41

The only time I've ever used macros is when generating interop glue code and slurping files for CLJS builds~

tagore19:09:39

At that time the loop syntax is translated into vanilla Lisp by the loop macro.

tagore19:09:10

Which is written in Lisp, and takes in Lisp data structures as arguments.

hmaurer19:09:14

@tagore I’ve seen a few Clojure DSLs being based around keywords (instead of an approach like loop which really embeds a language). Is that a philosophical difference in clojure?

hmaurer19:09:23

e.g. sticking to clojure constructs when building a DSL

hmaurer19:09:32

or have I just not seen enough clojure code?

tagore19:09:43

I think so, but I'm the wrong person to ask about that.

tagore19:09:25

I can't tell you what the Clojure philosophy is (I can tell you what I think it looks like.)

tagore19:09:51

@seancorfield might be a better guy for a definitive answer to that question.

hmaurer19:09:53

also, since you seem to be familiar with common lisp: is there any interest for a new lisper/clojurist to learn CL?

hmaurer19:09:07

are there valuable lessons there that I won’t get to learn in clojure?

tagore19:09:47

Yes, absolutely, just as a new Clojurian should learn basic Scheme (which takes about five minutes.)

akiroz19:09:59

seen some inline JSON embeded in CL code using reader macros... fun stuff 😄

hmaurer19:09:29

@akiroz I guess you could even embed ES6 in Clojurescript 😱

tagore19:09:55

The thing about Lisp is that it's a tradition, and it's an important one.

hmaurer19:09:04

Or embed java in clojure 😱

tagore19:09:16

You need to be able to read what the people who established that tradition wrote, because many of them were brilliant, and Clojure is a step in that tradition.

akiroz19:09:42

Hmm.... ES6 probably can't get pass the clojure reader, since there are no reader macros in clj I doubt you can do that without wraping it in a string

tagore19:09:25

For instance, if I talk about the differences between the CL Loop macro, and the later Series macro (not part of the spec...)

hmaurer19:09:42

@tagore it’s quite funny; I have been “aware” of lisp for many years now, but never got into it because I felt the absence of infix operators and the abudance of parentheses would bother me. Clojure, and to a lesser extent parinfer convinced me to make the jump, and I’ve been surprised how natural it feels

tagore19:09:48

There's a lot to be learned there about Lisp, about macro systems, etc.

akiroz19:09:18

You can just write a macro for infix maths operators 😛

tagore19:09:21

But you'll have to be able to at least follow CL to understand that.

tagore19:09:36

The good news is that...

tagore19:09:41

Lisp is kind of Lisp.

tagore19:09:10

And the better news is that Clojure is more idiosyncratic than most Lisps.

tagore19:09:24

If you understand Clojure you already understand every construct used in SICP, for instance, and did from about the first hour you spent learning Clojure.

akiroz19:09:22

I actually like Clojure's data DSLs since you can build/manipulate them using the rich built-in functions instead of string concat and regexes.

akiroz19:09:26

gives me nightmares just thinking about the builder and factory classes in traditional OOP code...

akiroz19:09:52

the scariest thing I've seen are C macros that wraps C struct builders 😨

tagore19:09:44

Well there are two Perlisisms that apply here (when in doubt reach for a Perlisism.)

tagore19:09:17

34. The string is a stark data structure and everywhere it is passed there is much duplication of process. It is a perfect vehicle for hiding information.

tagore19:09:53

106. It's difficult to extract sense from strings, but they're the only communication coin we can count on.

tagore19:09:09

Lisp, and Lisp's macros are about this: are programs strings?

tagore19:09:34

Well, in C they really are, until they hit the compiler.

tagore19:09:30

But that is painful- we want our programs to write programs, that write programs....

tagore19:09:35

The first step here is to write programs that are functions of programs, and that return programs.

tagore19:09:28

If your program is just a very natural data structure in the language you program in this becomes trivial. Thus Lisp.

tagore19:09:37

To be a Lisp programmer is to be a programmer who writes programs that write programs, until they hit the base case, whatever that might be.

tagore19:09:35

That sounds very difficult until you understand that functions are just data.

hmaurer19:09:28

@tagore thanks! I didn’t know Perlisisms

tagore19:09:40

@hmaurer Ah, you should. The older I get, the more Perlisisms make sense to me.

tagore19:09:42

There are a couple that are just wrong:

tagore19:09:28

51. Bringing computers into the home won't change either one, but may revitalize the corner saloon.

tagore19:09:39

OK then... I've got a computer in my pocket and that is going to kill the corner saloon.

tagore19:09:57

But most Perlisisms hold up remarkably well

tagore19:09:37

Speaking of macros:

tagore19:09:40

56. Software is under a constant tension. Being symbolic it is arbitrarily perfectible; but also it is arbitrarily changeable.

tagore20:09:41

And of course hs remarks on Lisp:

tagore20:09:58

55. A LISP programmer knows the value of everything, but the cost of nothing.i

tagore20:09:20

Worth understanding that one.

tagore20:09:42

49. Giving up on assembly language was the apple in our Garden of Eden: Languages whose use squanders machine cycles are sinful. The LISP machine now permits LISP programmers to abandon bra and fig-leaf.

tagore20:09:58

The poor Lisp machine... turns out that the machine is now so fast that the Lisp machine is irrelevant.

Lucas Barbosa20:09:57

Are there any good resources about TDD in Clojure? In Java, I can inject mocked dependencies in an object. In Clojure, I don’t feel like doing this in a function. If I always pass the dependencies as function arguments, I’ll end up with a lot of extra arguments. I’m looking for guidance, examples of beautifully designed code that has a nice unit testing code coverage

Lucas Barbosa20:09:02

Because I don’t “declare dependencies” in the function arguments, my code is not testable and I don’t write unit tests

Lucas Barbosa20:09:21

Indeed I should be writing the tests first, but I was just learning and would like to add the tests to my project

tagore20:09:15

There are testing frameworks for clojure, certainly, but...

tagore20:09:51

My opinion is that REPL driven development is superior to TDD.

tagore20:09:08

The problem I have with TDD is that I wind up with hundreds of tests that have to be deleted or maintained, but are not actually important past a certain point.

tagore20:09:59

Have you tried just opening a REPL and using it as you would use tests in a TDD scenario?

tagore20:09:40

You might be pleasantly surprised.

akiroz20:09:19

There are state management management micro frameworks that handles dependency injection in Clojure, take a look at stuartsierra/component and weavejester/integrant

Lucas Barbosa20:09:02

I do develop with a REPL open on Atom (forgot the plugin name), but I don’t feel like that is enough, you know… I feel like there’s something missing

hmaurer20:09:10

@tagore TDD gives you reproduciblity though, which might be useful when refactoring. Do you not not miss that part?

Lucas Barbosa20:09:25

especially because TDD helps me a lot with the class design

tagore20:09:04

@hamaurer Emac buffers can be saved.

tagore20:09:09

Tests that matter can also be written.

tagore20:09:07

My issue with TDD (well, one of my issues with TDD) is that it looks to me like a harness for running code... not something anyone with a REPL would have bothered with. But it leave cruft behind.

tagore20:09:05

If you lack a decent REPL, sure, it looks like magic.

tagore20:09:57

I wrote a large and comprehensive computational geometry library a few years ago, in C.

tagore20:09:29

I did it TDD style because I had no good alternative...

tagore20:09:03

I'd write a test, then write some code, then see if the test passed.

tagore20:09:05

If I'd written it in Lisp though...

Lucas Barbosa20:09:54

It is definitely an overkill to test what I am doing right now

tagore20:09:55

I would have just used the REPL to verify that, say, vector addition worked.

Lucas Barbosa20:09:09

I am building a very simple RESTful API

Lucas Barbosa20:09:27

there’s no complex business logic, it’s basically a CRUD with authentication and authorization

Lucas Barbosa20:09:08

But I’m trying to do it with unit tests for learning purposes

Lucas Barbosa20:09:23

The thing is.. how do I test a function that invokes a database module?

tagore20:09:01

Well- unit tests are worth writing (though not as many as TDD would have you write, IMHO)

Lucas Barbosa20:09:02

I’d normally mock that invocation, but the function is choosing which function to call

Lucas Barbosa20:09:18

I thought about putting the dependency in an argument

Lucas Barbosa20:09:22

and that looks very ugly

Lucas Barbosa20:09:33

@akiroz pointed me to some management libraries

Lucas Barbosa20:09:55

I would love to see examples of what people consider good practice

tagore20:09:07

But the main question is, if your endpoint is exposed to the world, how does it handle input you didn;t expect?

hmaurer20:09:13

librairies @akiroz pointed out essentially boils down to putting the dependencies in an argument

tagore20:09:30

Are your unit tests going to handle that? Only if you have thought very hard about them.

Lucas Barbosa20:09:32

:thinking_face:

tagore20:09:07

Your unit tests can all pass and your code can still be completely broken.

tagore20:09:30

Because your tests are likely to recapitulate your assumptions about what you've written.

hmaurer20:09:37

@tagore I feel one area where unit tests are useful is when working on someone else’s code. It gives some degree of confidence that you didn’t break the first guy’s expectations about how the code should behave

mobileink22:09:59

not to mention the case where the other guy is me, 6 months ago.

tagore20:09:35

I like tests that are essentially tripwires...

tagore20:09:49

Like "Wow, this blew up!"

tagore20:09:22

But they are only useful if you have a way of feeding them lots of random data...

tagore20:09:00

Of course my code works when you feed it what I expect....

hmaurer20:09:00

generative testing ❤️

akiroz20:09:21

I don't think there's anyway around putting deps in arguments, if your function has too many deps maybe it's possible to split it up...

Lucas Barbosa20:09:24

I would like to show you guys the code I wrote before, but I’ve been rewriting this application from scratch and I am on the start of a new cycle. This might sound dumb, but I like to learn this way hehe. I always learn something new or at least ways of how not to do things. On this turn, I would like to introduce TDD from the beginning, to influence the way I design my functions

hmaurer20:09:41

I get what you mean though, I’ve seen tests so many tests that are just tautologies

hmaurer20:09:59

and even integration tests that basically repeat what unit tests checked, but on a higher level

hmaurer20:09:07

when you read those you can’t help but feel they’re useless

Lucas Barbosa20:09:26

But what do you guys think about “system tests” then? I am thinking about doing this too

Lucas Barbosa20:09:31

like testing the RESTful API

Lucas Barbosa20:09:51

deploying on a staging environment and running a test suite that issues requests and checks the expected responses

tagore20:09:39

functional testing is not the same as TDD

tagore20:09:07

You absolutely want tests that verify that nothing blows up before you deploy.

Lucas Barbosa20:09:16

There’s a book… let me try to find its title

tagore20:09:25

You can't really do CI without those.

Lucas Barbosa20:09:37

Growing Object-Oriented Software: Guided by Tests Book by Nat Pryce

Lucas Barbosa20:09:58

The message on that book is on my head all the time

tagore20:09:11

Where I work we are allowed about 15 minutes of downtime a year.

tagore20:09:39

We could exceed that of course.... we might if we really fuck up.

Lucas Barbosa20:09:43

It’s basically: “Start writing the use case system test. Then, implement the functionality writing unit tests."

akiroz20:09:29

darn, that sounds hardcore... even github had longer downtimes

Lucas Barbosa20:09:55

Does that apply to FP? Looks like.. but I am not feeling comfortable with unit testing yet here. @tagore is driving me away from them 😁

tagore20:09:07

But the fact is that we rely more on manual testing than on automated tests before we push to production.

hmaurer20:09:22

@tagore what’s your business area, if I may ask?

tagore20:09:27

We have automated tests, but we don't trust them.

Lucas Barbosa20:09:36

@tagore doesn’t that take you a lot of time?

Lucas Barbosa20:09:44

isn’t there a way to automate it?

tagore20:09:07

We kind of sort of do...

tagore20:09:21

And much of our process is automated.

tagore20:09:11

But we always push to staging and have a junior developer bang on it before we push to production.

tagore20:09:36

We just can't afford to have downtime.

tagore20:09:20

15 minutes a year is a pretty strict measure.

akiroz20:09:18

yeah, that's like 6 nines of uptime....

tagore20:09:44

Yeah, it's kind of nuts, IMHO...

tagore20:09:19

But we promise that, and it's a selling point for us, so...

hmaurer20:09:05

@tagore do you operate highly critical medical / financial software?

tagore20:09:54

@hmaurer No, manufacturing monitoring

tagore20:09:20

Like we plug an interface into a million dollar machine you have in a factory and...

tagore20:09:30

e tell ou what it's doing 😉

tagore20:09:24

The world won't end if we go down, but...

tagore20:09:35

We can't go down. Ever.

gonewest81820:09:58

We have far less stringent uptime SLAs. Even so we do all unit tests on most libraries (but we're definitely not TDD). We also have a test harness that automatically calls all REST functionality on every endpoint and that runs at least after every push to staging. devs can trigger that harness to run whenever they want. We have jmeter testing on some of those end points after the functional validation. And we have human and automated application testing on the end user UI. Unit tests take seconds to run. API testing is minutes. Automated (Selenium) UI testing is minutes. And the manual UI testing takes days.

tagore21:09:32

Yeah, I'd like to have much more comprehensive testing across out entire coe base.

tagore21:09:40

*code-base

tagore21:09:31

But the reality of working in a venture-backed startup is that you inherit the CEO's code...

tagore21:09:56

And the CEO was likely a complete idiot when it comes to coding.

hmaurer21:09:54

I hope he doesn’t hang out on Clojurians 😄

tagore21:09:30

I'd love to re-write it all, but it's like >100k lines of terrible Javascript

tagore21:09:08

And we're looking for a series A, so...

tagore21:09:33

Sometimes you have to live with shitty code.

gonewest81821:09:36

I hear you. We rewrote the testing frameworks and boosted the coverage over the course of roughly a year with offshore assistance, but that was after the investment was in the bank.

tagore21:09:58

Yeah- on some level just getting the features out in order to get customers is what matters.

tagore21:09:23

I won't claim to love it, but...

tagore21:09:05

In the end it's the business that matters

tagore21:09:17

But boy a lot of our code is shit, and boy are we going to eventually pay for it.

gonewest81821:09:56

My strategy for that would be to start setting aside a little energy in each sprint, and apply it to pay down some of that tech debt. If your CEO/CTO will allow it. Depends how tight the money is. But arguably the debt is hurting you now by making feature development more trouble than it ought to be.

tagore21:09:12

@gonewest818 Yep, that might be a reasonable approach, but it's not a good one here.

tagore21:09:42

The truth is that our CTO let this happen.

tagore21:09:09

So I need to wrest control of code quality from hjm.

tagore21:09:24

Instead, we're moving the entire site from Angular to Redux, and I get to do that...

tagore21:09:56

I'm not sure this would be a generally good idea, but it's a chance to reset and enforce some standrads.

tagore21:09:46

Also, as e bring more people on I get to train them.

tagore21:09:56

Honestly, the most efficient path would have been to continue with AngularJS, but the CTO got a hair up his ass about React so I took the opportunity.

tagore21:09:35

Far easier than trying to convince him of what he already knows, which is that the codebase is shit.

tagore21:09:52

Unfortunately that's how the world tends to work...

tagore21:09:32

Well, I'm just putting money in the bank in order to do my own (probably CLojure-based) thing.

tagore21:09:45

Nothing like having a long runway.

hmaurer21:09:22

@tagore I take it your current codebase isn’t Clojure?

tagore21:09:55

Nah- JS/Java/C#

tagore21:09:47

And my other company is Python/C++

seancorfield21:09:27

(with my admin hat on: we're pretty relaxed about discussions at weekends but a lot of this really belongs in #off-topic rather than #beginners -- thank you, for consideration with future discussions!)

tagore21:09:10

It doesn't mater much though- one you've programmed enough you start to understand that languages are just notation + platform.

tagore21:09:33

Now, notation should not be dismissed out of hand...

tagore21:09:26

Good notation is powerful.

tagore21:09:26

And, in the end, hard to separate from platform.

gonewest81823:09:58

Ok, fair. Then getting back to what @lvbarbosa originally asked: "how do I unit test a function that calls a database?" then my suggestion is, wrap the test with a with-redefs to temporarily substitute whatever calls the database with a lambda. The lambda returns test data without any external dependencies. Do as many cases as you feel would be necessary.

noisesmith23:09:16

with-redefs comes with a lot of problems

seancorfield23:09:56

If possible, separate out the database interaction to a separate component, perhaps defined with a protocol, and then it's easier to mock, or point to a testing database.

seancorfield23:09:57

Then your function-under-test would be passed that component and call a function on it (to read/write on the "database") and you can test it with a mocked component of some sort.

tagore23:09:29

I'd argue that you can't unit test a function that hits a database.

tagore23:09:19

Unit tests shouldn't even hit the filesystem.

tagore23:09:28

Now, if you really want functions that cause side-effects to have unit tests there are ways to do that.

tagore23:09:19

You can, for instance, make them emit data that describes the side-effects they want to cause.

tagore23:09:46

And have other code actually effect the effects.

noisesmith23:09:49

right, the interpreter pattern

noisesmith23:09:00

as long as the interpreter part is simple, it works great

tagore23:09:49

Sure- also something that comes up a lot in CQRS systems.

tagore23:09:27

There are some real advantages to doing that, but it does make things more complicated.

hmaurer23:09:46

@tagore completely tangential question, but how would you deal with functions that “do some writes, do some reads, do some writes” with that approach?

hmaurer23:09:52

(out of curiosity)

tagore23:09:31

Well, that depends on the nature of transactions in the system, but...

tagore23:09:14

You'd emit an event that asked for a read, and wait on it.

tagore23:09:18

And then, based on the data you got back from the read, emit data that requested a write.

tagore23:09:46

The point is that you'd never read or write yourself...

tagore23:09:58

You'd just emit data.

hmaurer23:09:00

@tagore that sounds like an interesting approach (both modelling side-effects explicitly and what you just said); I’ll try it out

tagore23:09:45

One of the advantages to this approach is that it's very testable...

rcustodio23:09:24

Any examples on that?

tagore23:09:26

Instead of having to mock a database, etc, you just test that your code emits the requests you expect it to.

tagore23:09:36

Take a look at redux-saga.

tagore23:09:27

Which, btw, clearly took some inspiration from Clojure's core.async in it's channel implementation.