Fork me on GitHub
#beginners
<
2017-09-12
>
theeternalpulse01:09:52

a couple code-structure/practice questions 1. Should the code be structured from the top down based on function use order, or bottom up. ie. if func-a depends on func-b, shoud it be

(defn func-b (something))
(defn func-a [] (func-b))
or the other way around 2. Should my tests cover only the functions I expect to provide an api, or use defn- any helper functions and only test my public api functions

noisesmith01:09:35

@theeternalpulse clojure won't even compile the opposite order (unless you go out of your way to use declare, which is rarely done)

noisesmith01:09:22

with tests I start with public API but I also make regression tests as needed for implementation details (and "needed" for regression tests is subjective, but I tend to find that my problems often come up repeatedly in one place in the code, and regression tests help expose that and can sometimes help you find your way out of it too)

theeternalpulse01:09:47

I see, yeah the first one I've forgotten from doing javascript for a career -_-, that and adding commas to my clojure list structures

theeternalpulse01:09:00

what would be an example of the regression tests

noisesmith01:09:43

a regression test is a test written to 1) prove that you can replicate a bug 2) prove that the bug is fixed 3) make sure you know if some future change makes the same bug come back

noisesmith01:09:34

arguably you never need them if your design fits your problem properly, but for stuff that gets quickly iterated they can be a lifesaver

theeternalpulse01:09:36

ah yes, I was more wondering how a regression helped in the development process, I tend to just write negative test cases if I can see a point of failure through the input to the function

noisesmith01:09:14

I'm not talking about TDD here at all

noisesmith01:09:39

if you are, my comment was accidentally off the topic

Zor01:09:22

regarding "TDD", my 2 cents: 1) write tests for what you aim to achieve - don't test for errors 2) don't write narrow tests for impure stuff. large 'integration" type tests > small "unit" type tests for side-effecting stuff 3) feel free to test your pure functions as narrowly as you want 4) if you can cover 95% of your codebase with a short (10-20) lines main, that's a useful test

didibus01:09:16

I'd just say don't do TDD with Clojure

Zor01:09:28

assuming your functions and namespaces are named in a way that Makes Sense (tm) [ideally for more than just yourself], getting backtraces from your test main will help you fix stuff just as quickly as a test suite

didibus01:09:30

But do code with a REPL open

theeternalpulse01:09:16

yeah, I'm always bouncing between functional vs integration of functions. The thing I'm confused about is 1) then my unit tests become fiddling with repl to get them right 2) expanding functionality requires more repl fiddling if it's a pervasive change.

theeternalpulse01:09:12

like I made a macro recently, it uses about 4 functions. The defmacro body is only really 1 line

Zor01:09:27

TDD is ok for pure stuff imo, especially if you have a nice IDE like cider or cursive (that will run the tests for you when you edit the test file). usecase would be, some data massaging namespace, or some maths/number crunching

Zor01:09:56

do not test file i/o, databases, and the like, in a "unit" way. it's a path to hell, paved with mocks and nonsense

noisesmith01:09:22

@didibus having worked on a single app written by a group of devs over many years, I find the relative lack of tests incredibly frustrating if I have to change anything. I don't know what you did in the repl to establish your code "works", and I don't know what to try in the repl to see if my enhancement or course change or simplification worked without doing a lot of tedious work and printlns and... ugh

noisesmith02:09:02

if nothing else the tests would establish what the fuck someone was trying to do (as opposed to what the code does...)

Zor02:09:32

I approve your sentiment on this noisesmith

didibus02:09:46

Workflow is personal. But my recommendation is to think about why you are testing or writing tests. Failure to truly understand why (its a best practice or duh is not a good answer), would be a problem, and you'd need to think more about why you are doing it.

theeternalpulse02:09:47

what helped me was just having #_ exprs with each test in the repl, but I can't imagine not covering those in tests. The repl helped me get there, adding the tests show other's who would use it that it works

theeternalpulse02:09:12

but I don't particularly Beleive in TDD if you have a REPL

theeternalpulse02:09:30

sorry, not each test, each function call with example inputs

Zor02:09:31

well you may run the tests from the REPL, it's not one or the other

theeternalpulse02:09:19

so I'd have #_(some-macro (example-form etc etc))

noisesmith02:09:22

@didibus it's personal if nobody else has commit access to the code

didibus02:09:50

Often, there are the: How do I know this function works? And the answer is because I tried it with a bunch of test cases, and asserted it did what it should. This in Clojure is best done with the REPL. You can choose to put them into unit test also, but I find that's mostly a waste of time.

Zor02:09:26

farewell world, the 4am gong just rang

noisesmith02:09:26

@didibus that's fine for write-only code

didibus02:09:52

There's the: What is the expected behaviour, those are worth putting test cases, as a way to make sure the desired behavior is not broken in the future. Or documentation.

didibus02:09:09

And you have regression finally. Also worth having unit tests

didibus02:09:55

@noisesmith This has been my practice, with my team, its worked great. I can more easily understand what a function does by reading its doc + looking at the code + exploring its behavior at the REPL. I find unit tests often harder to get a good grasp, and they often assert things badly.

noisesmith02:09:21

OK - my experience is every time I try to change something I didn't personally author I have a hard time distinguishing what it does intentionally from what it does accidentally or erroneously, and I need to get invasive with the running system to even have an idea what its input and output are supposed to look like

noisesmith02:09:03

one factor here is that I have worked with people who are learning clojure on the job, so there's plenty of unidiomatic code in the code base

didibus02:09:44

Hum, maybe. But do you trust the tests to be any more accurate? Like what if they tested the unintentional behavior?

noisesmith02:09:39

I can simply comment out a bad test - commenting out the code leaves a broken app

didibus02:09:57

I mean, I'm not saying not to have unit tests, you should have some. Just I feel you need a lot less then for languages that don't have a REPL, since in those, a lot of your unit tests are there to help you figure out the correct implementation and fix bugs.

didibus02:09:19

find bugs*

noisesmith02:09:59

OK - I think we are in agreement actually, I'm not talking about high test coverage percentage here, I'm talking about like, at least having one per important top level pure function with a basic test of typical inputs / outputs

noisesmith02:09:34

if that calls 20 other functions that are not tested, but are easy to read, I'm fine with that

didibus02:09:45

Ya, I think we do agree, just different wording.

noisesmith02:09:29

perhaps you've dealt with the "cargo cult testing" extreme case, while I live with the "nobody but me ever writes tests, period, or even knows how" side of things

didibus02:09:02

Write tests, just don't go writing a test for every single small thing you would try out in your REPL, and don't write the tests first, write them in parallel or after the fact, the cases that make the most sense for the functions that make the most sense.

didibus02:09:13

Haha, yes I believe so

didibus02:09:24

We're trained to protect ourself from a different extreme

theeternalpulse02:09:22

also the occasional "stay safe' test as @noisesmith said

theeternalpulse02:09:59

I'm learning that when you work for a software company, no matter what, testing will eventually pop up with incentives, or simply as decree every once in a while

noisesmith02:09:07

I think I keep getting jobs with people who suffered through "everything must be tested I don't care if it makes sense" bosses, and now they refuse to do it at all

didibus02:09:22

I've seen code bases with 100% code coverage, yet when deployed it doesn't do what it should. When you don't test the right things, who cares if you have a test for every line.

noisesmith02:09:28

I've never experienced actual encouragement to do testing at work, hah

didibus02:09:05

That's funny, I guess it can be very different from places to places.

theeternalpulse02:09:12

we just got patted on the back for getting to 60, and then we recently got ordered to make it 70%

didibus02:09:34

A good advice I got from a principal engineer which I've personally tried to apply since, was that never write a unit test, when an integ test is required. So if you are doing I/O, don't bother mocking anything, instead, properly setup an integrated integ test with a real DB, etc. and have it test the real IO.

noisesmith02:09:44

definitely I prefer to keep things that do IO so simple that they are tested by the library in question, and then use pure functions to generate their input

noisesmith02:09:59

that's an ideal that doesn't always work out 100%, but you can get pretty damn close

didibus02:09:24

Ya, I tend to do that too

didibus02:09:41

BTW, a good trick for getting more coverage in Clojure (which you shouldn't really do, but I'm not here to stop you), is to put more things on the same lines. Since most coverage tool will measure lines and most managers look at that, since nobody understand what a form is at that level. So if you put more forms on a single line, you get more coverage 😛

didibus02:09:50

(if (something?) (bla bla bla) (blip blop bloup))
Now it doesn't matter which branch the if goes into, the line was touched.

didibus02:09:01

And will count as 100%

noisesmith02:09:54

see, I think that leads to terrible code formatting - anything that worries about line count does. I want one useful unit of information per line (eg. I usually have hash-map literals with one key/value pair per line, and I am not going to write a unite test for each key value pair)

didibus02:09:05

Oh ya, I discourage this. Unless you really want to game the numbers

didibus02:09:39

Actually, the forms covered metric is really nice in clojure. Most languages can't get that precise.

didibus02:09:00

I'd like something that could track my REPL session coverage

didibus02:09:54

Not really, but it could be fun

theeternalpulse02:09:40

that sounds like a more fun macro than I made lol

seancorfield02:09:21

We seem to have a lot of chatter about testing here -- we have a #testing channel -- perhaps folks didn't know that exists?

funkrider03:09:41

if I wanted to map within a nested structure like this

(map #(map (partial + 2) %) '((1) (2) (3)))
is map, map the way to go or is there a more functional way to do this?

seancorfield03:09:35

map-over-`map` seems reasonable.

didibus04:09:24

Nothing wrong with map over map. But for some reason, I tend to do for over map, like the difference in construct seems to help my brain clearly differentiate which level I'm at.

didibus04:09:32

(for [e '((1) (2) (3))]
  (map (partial + 2) e))

didibus04:09:33

The for macro can also be used by itself, depending what you want:

(for [lists '((1) (2) (3))
      item lists]
  (list (+ 2 item)))

rcustodio03:09:30

What is recommended for writing tests on clojure? Deftest? I really dunno how to write tests, I never did

seancorfield03:09:46

Starting with clojure.test and deftest is fine when you're getting started. But there's a lot more available once you get into it. Feel free to join #testing and chat in more depth.

noman07:09:28

hey all, can someone school me on a nicer way to do this?

(def user {:age 100})
(def user2 {:name "Noman"})

(defn punctuate [user]
  (def name (:name user))
  (and 
    name
    (str ", " name)))

(defn greet
  "This greets a user."
  [user]
  (str
    "Hello"
    (punctuate user)
    ". How are you?"))

(println (greet user)) ;Hello. How are you?
(println (greet user2)) ;Hello, Noman. How are you?

noman07:09:52

i wanted to avoid defining a variable for name but i couldn't figure out a way to not write (:name user) twice

tap07:09:27

You should use let instead os def for binding local value

(defn punctuate [user]
  (let [name (:name user)]
    (and
      name
      (str ", " name))))
You can also use map destructuring like this
(defn punctuate [user]
  (let [{:keys [name]} user]
    (and
      name
      (str ", " name))))
or
(defn punctuate [{:keys [name]}]
  (and
    name
    (str ", " name)))
https://clojure.org/guides/destructuring#_associative_destructuring

noman14:09:50

ahh, i was trying to figure out how to destructure, that's great, thank you

noman01:09:26

where can i learn more about this {:keys} thing? i'm having trouble finding it in the documentation

noman01:09:20

looks like i can do this as well, which seems a little nicer

(defn punctuate [{name :name}]
  (and
    name
    (str ", " name)))

noman02:09:46

is the :keys thing a way to turn all the keys of a mapping into an array?

noman02:09:09

and then you're destructuring the first element and calling it name?

tap03:09:16

:keys is a shorthand for destructuring and using the same name. You can do like this to assign value from :name to n

(defn punctuate [{n :name a :age}] ...)
But if you use :keys, you will get variables with the same name
(defn punctuate [{:keys [name age]}] ...)
Detailed explanation is in the lower section of that doc page.

tap03:09:58

There is also :as and :or

noman03:09:58

clojure rules

noman03:09:04

yeah i saw those two

noman03:09:06

pretty cool

noisesmith16:09:02

@noman another option

(some->> user
      (:name)
      (str ", "))
this stops if it finds a nil and returns nil, otherwise you get the string

noisesmith16:09:39

user=> (defn punctuate [user]
  #_=>    (some->> user
  #_=>             (:name)
  #_=>             (str ", ")))
#'user/punctuate
user=> (punctuate {})
nil
user=> (punctuate {:name "Joe"})
", Joe"

donyorm16:09:45

So how can I cause boot to pass arguments passed to a task onto the program like is done if I do something like lein run -p 3000. How do I define the arguments to (deftask run) in boot? Basically the equivalent of [& args] for tasks

chris16:09:24

there's a #boot channel for more, but iirc you just use (with-pass-thru ...)

donyorm16:09:26

whoops, meant to post that in #boot. Sorry

chris17:09:42

actually, with-pass-thru might not be right, I was just basing that off the boot for lein users lein run implementation, but (doc with-pass-thru) looks like it does something different, I'm interested to see what they say over there

donyorm17:09:28

yeah accoring to doc, with-pass-thru just passes on the fileset, which isn't as useful in this case

yanis17:09:50

Hello, does clojure have smth like reduce but which can have several seqs as arguments?

noisesmith17:09:25

@yanis what do you want to do with the several seqs? process in parallel like map does?

yanis17:09:52

@noisesmith yeah, exactly, thank you, didn't know map can take several seqs

noisesmith17:09:18

yeah, I use map as a helper for this

user=> (reduce (fn [acc [a b c]] (conj acc [c b a])) #{} (map vector [1 2 3] [4 5 6] [7 8 9]))
#{[9 6 3] [7 4 1] [8 5 2]}

noisesmith17:09:47

I guess using a set as the accumulator makes it less clear, but the three sequences are consumed in parallel with the help of map

noisesmith17:09:00

and then destructuring lets us pretend they are separate args in the reduce

yanis17:09:51

@noisesmith thank you for the example, very clear, very beautiful

noisesmith17:09:26

as a more advanced usage, you can use (map vector) standalone as a transducer arg to transduce with a small adjustment to the reducing function to support the one argument arity

noisesmith17:09:08

this has the same behavior but the added flexibility of using a transducer (these compose nicely) and a moderate boost in performance

noisesmith17:09:02

oh, wait, that wouldn’t work because we want to consume multiple collections so it can’t be the transducing map

noisesmith17:09:56

so, you could replace my_f with something like #(into acc (filter some?) %) - then you don’t need that ->> or into

noisesmith17:09:13

(this time a transducer does actually help)

yanis17:09:22

it's from 4clojure 🙂 I see other solutions now & mine sucks https://www.dropbox.com/s/oddzzwwx58qvz1r/img-2017-09-12-20-44-13.png?dl=0

noisesmith17:09:50

@yanis that’s what 4clojure is for of course, is finding out that other people know some tricks you don’t know yet, and learning their tricks 😄

dadair19:09:33

I have a Clojure service that I want to extract into a Java library so that a separate Scala app can call the service directly rather than incur network overhead. I have a single entrance method: (defn process [m1 m2] ..) where m1 and m2 are both maps, and the return value is a map from keywords => vectors. I'd like to be able to do Engine e = new Engine(); m3 = e.process(m1, m2) from Java. Can anyone point me in the right direction to enable this? I have a :gen-class to build Engine and provide init/state/methods, but I'm not sure which data types to use in the methods definition for process. clojure.lang.IPersistentMap? java.util.Map?

noisesmith19:09:53

Map is probably the safe/convenient one

chris19:09:09

java.util.Map

chris19:09:20

it can still be a PersistentMap under the covers

dadair19:09:32

will I need to modify my method definition for process if I use java.util.Map?

noisesmith19:09:37

if you want to be defensive about it you can call (into {} m) in the function body to get the PersistentHashMap version

noisesmith19:09:49

but even a Map should work with most clojure code

dadair19:09:58

ok I'll give that a try, thanks!

dadair19:09:22

(also if this idea seems crazy and there's a simpler way to get to clojure from Scala, let me know!)

noisesmith19:09:12

also, an alternative to gen-class is to make a stub java class that uses clojure.java.api.Clojure to require and then launch your code https://clojure.org/reference/java_interop#_calling_clojure_from_java

noisesmith19:09:10

then your type declarations come from java code (which might be friendlier anyway, eg. that way you can end up with javadoc and annotations as consumers might want)

dadair19:09:49

so I'd just have a wrapper Java class, with a method process which under the hood redirects to IFn process = Clojure.var("some.ns.engine", "process"); m3 = process.invoke(m1, m2)?

noisesmith19:09:12

right, be sure to call require in there too (maybe in a static initializer)

noisesmith19:09:50

but require is idempotent and not super expensive so you could put it in the method, depending on what you are doing and what the use case looks like

rinaldi19:09:23

Any ideas why println calls within futures are not showing up on the logs?

ghadi19:09:34

Are you in a lein repl rinaldi?

rinaldi19:09:32

But I'm seeing the same issue on AWS CloudWatch too.

ghadi19:09:41

it's possible that standard *out* is not being captured properly. I forget the details exactly, but in some lein repl contexts it might not work as you expect

ghadi19:09:24

I can't speak for your cloudwatch setup

rinaldi19:09:23

Gotcha, thanks for the input @ghadi

ghadi19:09:59

@rinaldi not sure how much your app relies upon lein, but if you do java -cp $(lein classpath) clojure.main to start a bare repl (non nREPL-based), see if it fixes your issue

ghadi19:09:26

if it does, then the culprit is nREPL+lein

irasantiago19:09:14

hey all! i also have a question related to future. i have something like this:

(map #(and @(future (process-a %) true) @(future (process-b %) true)) list-of-strings)
i don't want to complete the function unless every value in this collection is true but if i do
(let [proccessed (map #(and @(future (process-a %) true) @(future (process-b %) true)) list-of-strings)]
   (when (every? true? processed)
     (something)))
i don't get any logs for process-a or process-b. why could that be? is there a way to get these logs to show? (just to clarify these are AWS logs) i'm actually not too concerned about this also is that when block the proper way to evaluate that processed is complete?

didibus21:09:20

@irasantiago It could be that's because the whole thing is lazy, so your map wasn't actually executed yet

didibus21:09:55

@irasantiago But also, I just don't think what you're doing works at all.

didibus21:09:36

Your futures are really adding no value here, since you're dereferencing them at their creation site

noisesmith21:09:40

if you want them to execute in parallel, bind the futures in a let block, then deref in the let body

noisesmith21:09:38

something like

(let [processed (doall (mapcat  #(vector (future (process-a %))
                                        (future (process-b %))))
                        list-of-strings)]
  (when (every? (comp true? deref)
                processed)
    (something)))

didibus21:09:06

So you're code is doing: 1. Loop on every string in a list-of-strings 2. For each string, call process-a and wait for it to be done, then call process-b and wait for it to be done. Once both are done, (and) the results. 3. Once all string have been mapped, then check that all results were true, and if so, do (something).

didibus21:09:28

Nothing is happening in parallel

ghadi21:09:31

you're misreading the code in #2. The wait (deref) doesn't happen during the mapping

john21:09:33

when does not wait for any results, if that is what you were looking for. It is essentially just an if without an else clause.

noisesmith21:09:00

like most other forms it will wait for the first arg to return a value though

noisesmith21:09:13

@ghadi I wonder if I’m misreading what you say, because while the map form is lazy, the values inside the function passed to map are consumed one at a time in a blocking way and the new futures aren’t going to start until previous ones complete, right?

ghadi21:09:46

the code above is launching two things at a time per element in the list of strings, then it's dereffing everything to force realization

ghadi21:09:52

so it's not fully parallel

noisesmith21:09:19

oh, right, thanks

ghadi21:09:33

you could (doall (mapcat.... whatever)) in the bindings to realize all the launches

ghadi21:09:37

then deref

noisesmith21:09:17

fixed (I think - I think there’s still a problem compared to the original logic (I’m not sure what the and was supposed to accomplish, I’m sure it didn’t do the right thing in the original, but I also don’t think my code does what that and was intended to do))

john21:09:48

so you can remove when (every? true? processed) completely

didibus22:09:43

@ghadi What do you mean? You talking about my #2 which was describing @irasantiago 's code? Or you talking about @noisesmith 's code?