I suspect i’m stepping into ancient territory, but is the lack of a contains-value function for sequential collections (vectors, lists) a hint on the designers that your supposed to be using a set? If so, then that seems to contradict my mental model of clojure of putting compostability above performance. Or is there something else to consider here? Another way to think about the trade off?

i think some is pretty close to that

esp using a set as a test pred

( @drewverlee, in case you're afk)

@seancorfield can I improve it my code more then or is it good enough for a beginner like me ?

Another spec question. I want to test a function that has as input something like this : [{ :name "Roelof", :place "internet"} {:name "Jane Doe" :place "unknown" }] and as output {"Roelof", "Jane Doe"} . If I want to test this , can I use the same idea as this :

(def email-regex #"^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,63}$")
(s/def ::email-type (s/and string? #(re-matches email-regex %)))

(s/def ::acctid int?)
(s/def ::first-name string?)
(s/def ::last-name string?)
(s/def ::email ::email-type)

(s/def ::person (s/keys :req [::first-name ::last-name ::email]
                        :opt [::phone]))  

this passes

(apply dissoc  single-elemement [:id :user_id])

(map (apply dissoc [ :id :user_id ]) all-elements)


with-error ->  ClassCastException clojure.lang.Keyword cannot be cast to clojure.lang.IPersistentMap  clojure.lang.RT.dissoc (RT.java:848)

(map (apply dissoc [ :id ]) all-elements)

@kevinbheda (map #(dissoc % :id) all-elements)

@st but i want to do this for two keys :id and :user_id

@kevinbheda (map #(dissoc % :id :user-id) all-elements)

@kevinbheda if you can, avoid user_id in favor of user-id

thanks @st yeah i wanted to use the convention user-id but the map is generated as a result of honeysql query with returns the column name as underscore. Is there any way i can make honeysql return - instead of underscore ?

also #{ } makes it a set

and here we have # () , % any reference to read up on these operators ?

@kevinbheda don’t worry about the _ (I’ve used honeysql a little bit and I think the safer is to keep _) #{1 2 3} is a set, and #(inc %) is an anonymous function You’ll find complete explanation in any Clojure book/tutorial.

Might be useful about anonymous function shorthand:

;; Single argument:
> ((fn [arg] (str arg)) "Hello")
"Hello"

> (#(str %) "Hello"))
"Hello"

;; Multi argument:
> ((fn [arg1 arg2] (str arg1 arg2)) "Hello" " World!")
"Hello World!"

> (#(str %1 %2) "Hello" " World!")
"Hello World!"

;; Multi-arity (unknown number of arguments):
> (#(str %&) "Hello" " " "World!") ; returns a list
"(\"Hello\" \" \" \"World!\")"

@roelof I think using clojure.spec to transform [{ :name "Roelof", :place "internet"} {:name "Jane Doe" :place "unknown" }] to {"Roelof", "Jane Doe”} is going too far — that’s too much coercion. You’d be better conforming (validating) the (s/coll-of (s/keys :req-un [::name ::place])) and then using regular Clojure code to transform the (validated) collection into whatever shape you want.

(and I think you meant [”Roelof”, “Jane Doe”] — a vector or sequence, rather than a hash map?)

yes, you are right

oke, I thought I could test one of my functions with spec instead of using a testing library

so test if the right output is being made

@kevinbheda FWIW, we used to auto-convert _ in SQL entity names into - in identifiers in Clojure when we first started working with databases in Clojure but it actually proved to be more of a pain than a benefit in the end so we switched to just using them as keywords “as-is”. You have to remember to do the translation in both directions in every place where you touch the DB, and also your SQL strings don’t match your SQL DSL so it can get very messy.

@roelof Well, yes, you can test your functions with spec but that has nothing to do with coercing data like that.

(and you generally need some sort of test library to run tests)

So I try to find out I can of have to made a fdef for that

or maybe the same idea as this part : http://clojure.org/guides/spec#_entity_maps

Remember that spec is about the shape of your data, more than anything else. If you have a function that does that transformation and you want to spec it, you will say that it accepts a collection of maps with certain required keys, and you may say that it returns a collection of strings.

You could also specify :fn for an invariant between :args and :ret — but you may end up specifying the exact same logic as the function!

that is not handy

After all, that transformation is merely (mapv :name data) so it’s not worth testing — you’d be testing mapv which is pointless.

clojure.spec is best for validating data at the edges of your application: user input (or API parameters), and using generative testing for complex parts of your system that need a “correctness boundary”.

oke, so spec is not the right choice to test these functions : https://github.com/rwobben/paintings/blob/master/src/clj/paintings2/api_get.clj

then I could better use just clojure/test

Those are exactly the functions clojure.spec is good for: they are boundary functions that bring data from a 3rd party into your system.

You want to validate that the data you get back from those functions matches your expectations.

But that does not necessarily mean fdef — you might actually want to validate that data inline, using s/conform and testing s/invalid? and doing error handling.

For your functions that transform that 3rd party data to a data format you need, you could spec those with fdef and use clojure.spec.test/check on the function to do generative testing.

oke, so I can use spec for validating that the external api returns the right input so the function will work

But testing behavior and validation of data are two very separate considerations. You seem to be mixing those concepts up?

that could be

I try to find out when the use spec and when not

So if I understand you I can use spec for validating that the spec returns the right response

not for testing if the transformation will go right , then I have to use clojure/test

Since you really want read-data-front-page to be able to detect bad input and handle errors, I would write a spec for the format of response that it expects, and I would use s/conform and s/invalid? directly inside that function.

the same we did with the home-function

and I can do the same for read-numbers , and the other read-functions

I would say read-data-detail-page and read-numbers should also use s/conform and s/invalid? — since they need to validate their input (which comes from a 3rd party) and handle errors.

None of that is “testing”. It’s “input validation”.

yep, that part is clear

and if I want to look if the functions work right ? so they produce the right output ?

In addition to that — and separate from that — you could also write fdef specs for those functions and test them generatively to ensure that, given valid input, they generate the expected shape of result data. And that would need some sort of testing framework — even if it is only a call to clojure.spec.test/check.

oke

I think I want both

OK. Just keep them separate in your mind.

so I have to find out what is :ret , :args and :func of all the functions

Write the data specs first, then add the input validation and error handling. Then write the fdef specs and use clojure.spec.test/check to generatively exercise them.

Don’t worry about the fdef stuff until you have the data specs and the input validation working.

oke, the first two you helped me when we did this with the home-page function

I can do the same with these functions and adapt them

I think I need to use the idea of this part of the guide : http://clojure.org/guides/spec#entity-maps

Right ?

@seancorfield last question. The response is always bigger then I need. Is it good I only test for the parts I need

is there any reason why for or map wouldn’t iterate over a seq? I’m trying something like this and no output

(for [cook cooks] (js/console.log cook))

suppose that i have json map that is something like this: {:quote [{:values [{"tLabel" "Hour"} {"tLabel" "Price"}]} {:values [{"tLabel" "Hour"} {"tLabel" "Price"}]}]}]}} i want to filter over the tLabel value, and was thinking using a map for that. however when i try to do this: `(map (extract-entry #(get % :values) ["Hour"]) (:quote json-data))` It claims that a LazySeq cannot be cast to a clojure.lang.IFn. I'm at a loss why. The extract-entry function works as expected when given a single entry

@roelof Yes, specs should specify the minimum shape you need in order to do your work. One of the concepts behind clojure.spec is to allow “additive correctness” — so if data gets more elements, it remains a valid input for your function and the spec doesn’t need to change.

oke, then I could try it in the next few days to make it work

I cannot work as long as I want. My daugther has X-mas holidays from school till begin January . So she needs a lot of quality time too

@wilcov what is extract-entry?

@seancorfield it's a function i made. `(defn extract-entry [coll wanted-keys] (filter #(.contains wanted-keys (get % :tLabel)) coll))`

For that expression to work, extract-entry must return a function.

Since you want (map some-function (:quote json-data))

So extract-entry must return (fn [entry] …)

(sorry for the format btw, i still have to figure out why sometimes formats as code and why sometimes it doesn't )

Three backticks formats multi-line code, single backtick formats inline code.

This uses three
backticks
on multiple lines

ah i see, thanks

(defn extract-entry [coll wanted-keys]
   (filter #(.contains wanted-keys (get % :tLabel)) coll))

i'm reading clojure for the brave and true. I think i have to take another look at collections as sequences

In your map call your first argument to extract-entry is a function, not a collection.

and (extract-entry some-coll some-keys) would return a (lazy) sequence — a subset of some-coll

Mm, I have to wrap my head around the concepts and syntax some more. Thanks so far!

Given your data structure, what exactly are you trying to get back?

The data structure is a list of maps that have multiple keys for each hour. I'm trying to filter out the relevant keys. i.e. ` {:quote [ {:values [{"tLabel" "Hour", "Value" 01} {"tLabel" "Price", "Value" 22} {"tLabel" "Unwanted", "Value" "something" ]}} {:values [{"tLabel" "Hour", "Value" 02} {"tLabel" "Price", "Value" 41} {"tLabel" "Unwanted", "Value" "something" ]}]} I would like to return a list of maps with the Hour and Price key/value pairs

If the structure isn’t so big that converting it all to hash maps first isn’t a big overhead, I might consider that first… Do you have lots of :values that don’t match what you’re looking for?

(I mean, you can certainly filter first and then convert what’s left to hash maps…)

So I would use a set of label values as part of the filtering process: #{”Hour” “Price”} since you can use that as a predicate on the ”tLabel” value.

(defn extract-entry [wanted-keys]
  (fn [coll]
    (filter #(wanted-keys (get % “tLabel”)) coll)))

then (map #(extract-entry #{”Hour” “Price} (:values %)) (:quotes json-data)) should get you the raw data you want (as a sequence of sequences)… and then you’ll still need to map over that to convert the results to hash maps

(this is why I’d probably convert the :values sequences to hash maps first and then filter on the result)

There aren't a lot of values that won't match. about 3/5th will match, 2/5th won't

And how big is the :quotes structure? Tens of values? Hundreds? Thousands?

a couple of hundreds

I wouldn’t be surprised if the overhead of filtering on the sets / strings and rebuilding sequences actually outweighed the cost of a single pass to create a hashmap for each value followed by a simple filter and/or select-keys call…

I would not mind if the function would need a second or two to run, it will run only a couple of times each day. But still, performance is always good to have 🙂

Hmm… how about this:

(defn filtered-value-map [wanted-keys]
  (fn [quotes]
    (not-empty (reduce (fn [m {:strs [tLabel Value]}]
                                        (if (wanted-keys tLabel) (assoc m tLabel Value) m))
                                      {}
                                      (:values quotes)))))

(keep (filtered-value-map #{”Hour” “Price”}) (:quotes json-data))

I’d forgotten you can destructure on strings...

Edited filtered-value-map to use not-empty so you get nil or a hash map, then keep will keep any non-`nil` results of applying its function argument… I think that might work… untested 🙂

I have to catch a train but i'll be sure to test is when i get the chance. Thanks for helping me so far 🙂 I'll post a update when it works 🙂

@wilcov just tested that and it produced ({"Price" 22, "Hour" 1} {"Price" 41, "Hour" 2}) 

@seancorfield I forget to thank you with your explanation and patience with me

Enjoy your Xmas holidays and your break from learning Clojure @roelof !

Thanks, I not leaving Clojure but instead of 4 or more hours , I can maybe 1 hour programming