Fork me on GitHub
#beginners
<
2020-03-20
>
mario.cordova.86200:03:15

I am saving a map into a JSONB postgres column but was having issues with cheshire so decided to use clojure.data.json since the custom encoder/decoder seemed a bit more straight forward. What is wrong with this solution?

mario.cordova.86200:03:42

(def opt_data {:sheet-data-structure
               {"RD" {[1 4] {:backgroundColor "#0000"}
                      :skip #{nil [5 5] [6 6]}}
                "Model" {:mods {[3 4] {:borderRight "none"}}
                         :test [1 2 3]}
                "Tax" {[99 9] ":should-not-be-keyword"}}})

(defn value-serializer
  [_ v]
  (cond
    (= (type v)
       (type #{})) (str "#SET~" v)
    :else          v))

(defn value-deserializer
  [_ v]
  (cond
    (str/starts-with? v "#SET~") (-> (subs v 5) read-string set)
    :else v))

(defn keys-serializer
  [k]
  (cond
    (keyword? k) (str "#KEYWORD~" k)
    (vector? k)  (str "#VECTOR~" k)
    :else        k))

(defn keys-deserializer
  [k]
  (cond
    (str/starts-with? k "#KEYWORD~") (-> k (subs 10) keyword)
    (str/starts-with? k "#VECTOR~") (-> k (subs 8read-string)
    :else k))

(-> (json/write-str opt_data :value-fn value-serializer :key-fn keys-serializer)
    (json/read-str :value-fn value-deserializer :key-fn keys-deserializer))

mario.cordova.86200:03:32

The reason I am doing this is that I need to keep sets as sets and vector keys as vectors and not strings. This data I am reading from the DB is being merged in with other another data of the same structure.

noisesmith00:03:15

(-> k (subs 8read-string)
is obviously wrong - looks like a ) ( got eaten

noisesmith00:03:23

and replaced with 8

mario.cordova.86200:03:22

Yea I kept looking back n forth and I think it was a slack thing

noisesmith00:03:46

I'd use pr-str instead of str inside the serializer I'd use pr-str on the k in the serializer, because it can preserve things that str might not

noisesmith00:03:46

(cmd)user=> (str (map inc [1 2 3]))
"[email protected]"
(cmd)user=> (pr-str (map inc [1 2 3]))
"(2 3 4)"

noisesmith00:03:49

it's a common gotcha - a good rule of thumb is that if what you want is the string you'd see in the repl for a given object, you want pr-str and not str

mario.cordova.86200:03:06

Is this a bad approach though? Is there a better way of doing this that I am not seeing?

mario.cordova.86200:03:20

There is discussion tomorrow about whether to do this manually or using Transit. Personally I would rather use Transit since it works out of the box and can probably do the job faster

noisesmith00:03:00

what transit creates when you ask for "json" is not as likely to be useful in a jsonb field (eg. its shorthand for backreference of repeated keys in one document)

noisesmith00:03:18

but it definitely works and its tagged readers/writers are mature and tested unlike yours

mario.cordova.86200:03:42

yea thats main pushback

mario.cordova.86200:03:46

65ms vs 890ms lol

mario.cordova.86200:03:44

woops more like 65ms vs 250ms. I was counting the read and write

thoneyvazul06:03:53

I got stuck with a problem. Could someone help? I would like to call a -main function from a clojure script file with a namespace, but I do not know to do that. Here is what I tried. I have a Code.clj with Code namespace and a -main function. My current dir is the same directory as the source code (`/tmp/test`)

$ pwd
/tmp/test
$ cat Code.clj 
(ns Code
  (:gen-class))

(defn -main []
  (println "hello world"))
$ clojure -m Code Code.clj 
Execution error (FileNotFoundException) at clojure.main/main (main.java:40).
Could not locate Code__init.class, Code.clj or Code.cljc on classpath.

Full report at:
/tmp/clojure-90572729667538797.edn
$ 
I tried exporting CLASSPATH env variable to ".", or pass it somehow with command line parameters, but none of these helped.

dromar5607:03:45

@thoneyvazul you need a deps.edn file adding the current directory to the classpass (you also need to add & args to your -main function args)

dromar5607:03:25

I'd be interesting on knowing if there's a way to modify classpath without a deps.edn file

thoneyvazul07:03:33

Awesome, it works! Thank you @dromar56

aviv12:03:58

How can I sum up all pair freq (bi-grams) to a map using reduce? (def pairs '(("ab" "bd") ("ab" "bc") ("ab" "bk"))) (def freq '(1 2 3)) (defn f [p f]) => {"ab" 6, "bd" 1, "bc" 2, "bk" 3} {:low 5 :lowest 2} pairs will be (('lo', 'ow'...) ('lo', 'st'))....etc freq (5, 2) applying f will yield: => {"lo" 7, "ow" 7 "we" 2,...etc}

aviv12:03:58

How can I sum up all pair freq (bi-grams) to a map using reduce? (def pairs '(("ab" "bd") ("ab" "bc") ("ab" "bk"))) (def freq '(1 2 3)) (defn f [p f]) => {"ab" 6, "bd" 1, "bc" 2, "bk" 3} {:low 5 :lowest 2} pairs will be (('lo', 'ow'...) ('lo', 'st'))....etc freq (5, 2) applying f will yield: => {"lo" 7, "ow" 7 "we" 2,...etc}

ordoflammae12:03:41

Ah, I thought def freq was a typo.

ordoflammae12:03:17

You probably just need to write your own function.

ordoflammae12:03:22

But there's a much simpler way without doing reduce.

ordoflammae12:03:55

The function frequencies takes a sequence and returns a map.

ordoflammae12:03:03

You just need to flatten your data structure first.

aviv12:03:11

but I want the frequencies to be determined by the freq itself

ordoflammae12:03:34

I'm sorry, now I'm confused again.

aviv12:03:34

'ab' is 3x with frequencies but in my situation it's 6

ordoflammae12:03:50

Why do you want it to produce 6?

ordoflammae12:03:06

It only appears 3 times.

aviv12:03:09

freq is a constant, it determines each pair with it frequency

aviv12:03:35

i'm not trying to get the number of it appereances

ordoflammae12:03:37

Oh, THAT's what that's for.

aviv12:03:17

think of that I have some frequency already calculated, and I want to generate pairs from it with each -sums freq

aviv12:03:24

for example

ordoflammae12:03:08

Well... I'd write a function that multiplied each sequence by the number in the frequency that matched the sequence.

ordoflammae12:03:20

You could use reduce for that.

aviv12:03:29

{:low 5 :lowest 2} => {"lo" 7, "ow" 7 "we" 2,...etc}

aviv12:03:09

I get a frequencies map and I want to create it pairs-frequency (from it keys)

hindol.adhya12:03:55

For generating pairs, you can use (partition 2 1 <string>).

aviv12:03:10

yep that's generated my def pairs

hindol.adhya12:03:28

Focus on each key value pair individually. Later, merge two maps using merge-with and +.

aviv12:03:20

no way to do this with reduce in one time? accumulating the result-map?

hindol.adhya12:03:10

I don't know of a more direct way. This problem is sufficiently complex.

jaihindhreddy12:03:43

@aviv you can use repeat to repeat each pair the corresponding number of times, mapcat that together into one long seq of strings, and then apply frequencies on that. Not the prettiest but looks like this:

(frequencies (mapcat (comp #(apply concat %) repeat) freq pairs))

hindol.adhya12:03:03

You can use seq on a map to get the entries, like this, (seq {:low 5 :lowest 2}) => ([:low 5] [:lowest 2]).

jaihindhreddy12:03:00

One inefficiency in my suggestion above is, if there's a huge number in freq, then we'll end up creating a massive seq instead of simply multiplying the counts. This approach doesn't have that limitation:

(defn mapvals
  "Maps f on values of map m"
  [f m]
  (into (empty m)
        (map (fn [[k v]]
               [k (f v)]))
        m))

(->> (map frequencies pairs)
     (map (fn [n fs]
            (mapvals #(* n %) fs)) freq)
     (apply merge-with +))
We get the frequencies of each pair, then multiply those values with the corresponding frequencies, then merge all those maps together with + to add 'em up.

aviv12:03:49

@ how would that help to count the pairs (bi-grams), i can generate pairs then use map f' pairs freq then merge result as you have noted,

hindol.adhya12:03:51

I tried this,

(defn pair-freqs
  [m]
  (reduce (partial merge-with +)
          (for [[k v] m]
            (zipmap (partition 2 1 (name k)) (repeat v)))))

(pair-freqs {:low    5
             :lowest 2})
;; => {(\l \o) 7, (\o \w) 7, (\w \e) 2, (\e \s) 2, (\s \t) 2}

hindol.adhya13:03:44

Here, for internally calls seq on the map and I am destructuring that into key-value pair.

hindol.adhya13:03:56

We don't usually need to call seq on a map but many functions that expect a sequence can take a map and what they'll get is a sequence of entries. That is very useful.

aviv13:03:29

Interesting, i'll check it out, thanks!

hindol.adhya13:03:24

Small update, (cycle [v]) can be changed to (repeat v). Edited.

ordoflammae12:03:00

vals will return a sequence of all the values in your map, with no keys.

ordoflammae12:03:29

Just use reduce on that.

aviv12:03:38

I do not have a map

aviv12:03:04

each pair is corresponding to it freq, so 'ab':1 then 'ab':2 then 'ab':3, total freq of 6

ordoflammae12:03:08

That's what freq returns.

ordoflammae12:03:13

A map of all the frequencies.

ordoflammae12:03:22

Unless I'm completely confused.

aviv12:03:29

let's thread on this

skoude13:03:03

is there a way in spec to add additional description?? I mean if i create (s/def application-id int?) i could add a written description and explain what is the application-id. Then the same “description” would also be shown in swagger

alexmiller13:03:06

No, this is the highest voted item in the tracker though

alexmiller13:03:18

So we are aware of the desire :)

ikitommi13:03:32

with spec-tools, you can say:

(require '[spec-tools.core :as st])
(require '[spec-tools.swagger :as swagger])

(swagger/transform
  (st/spec
    {:spec integer?
     :name "integer"
     :description "it's an int"
     :swagger/default 42}))
;{:type "integer"
; :title "integer"
; :description "it's an int"
; :default 42}

skoude13:03:01

okay, maybe I need to go and vote it also (if possible).. And thanks, I will check spec-tools out 👍

aviv15:03:43

curious - are there any production-teams using Property Based Testing? if so - with/without TDD? mixing? what are your thoughts? new to it and thinking on adding to our pipeline..

sova16:03:15

@aviv what am Property Based Test?

iagwanderson17:03:59

is that any way to adjust max-body size in jetty served apps?

hiredman17:03:32

We have mostly a traditional test suite with a few generative/property based tests. We don't have a separate generative test suite that takes a long time to run, which I have seen mentioned as a best practice. Our generative tests if they potentially take forever to run default to a small iteration count, but can be increased using a environment variable if you want a more complete test run. I am definitely not doing TDD

aviv17:03:05

Not doing TDD as this have replaced / over-taken your testing pipeline?

hiredman17:03:51

no, I just don't find TDD to be helpful

aviv17:03:37

so your test pipeline includes only property-based-testing?

aviv17:03:49

I currently use TDD and looking for alternatives

hiredman18:03:10

I think we are not using the same definition for TDD

aviv18:03:43

by TDD I mean having test-by-example, and lots of them

hiredman18:03:05

*T*est *D*riven *D*evelopement is the practice of writing tests first and using those tests to guide creation of the implementation

hiredman18:03:58

as I said, I don't practice TDD, but the majority of our tests are a traditional test suite (example based tests) with a few generative/property tests sprinkled around

seancorfield18:03:57

I use TDD for some things -- such as where we have a clearly defined spec with success and failure behavior well-documented up front, such as new REST API endpoints. I'll write failing tests for that around the specific types of failure and a simple success case, then write code to make them pass, then more tests and more code, and refactor while "green".

seancorfield18:03:23

Where things are less well-speced, I'm more likely to do RDD to explore the shape of the problem/solution.

aviv18:03:51

I know hiredman, I skipped the introduction because the relevancy to PBTesting is the test-by-example vs test-by-property

seancorfield18:03:21

If I have something that is based on ranges of data with boundary conditions, I'll use PBT (earlier this week, I wrote PBT for some conversion/formatting functionality).

seancorfield18:03:54

(disclosure: hiredman and I work together -- so our responses, taken together, reflect the wide range of our approach to development/testing)

aviv18:03:40

heh 🙂 awesome

aviv17:03:38

using generatorsfrom test.check I'm receiving exception with error Couldn't satisfy such-that predicate after 100 tries. when I try to generate none-empty vectors with 'tokens' -> strings which are lowercased + not empty

(gen/such-that (complement empty?)
                                (gen/vector (gen/such-that lowercase? gen/string-alphanumeric 100)))
I understand that the generators are unable to create my desired input - but this seems like a very 'decent' request, is there any alternative generating such? or anyway to stop from exception being thrown?

hiredman17:03:56

it is a limitation of such-that

hiredman17:03:32

such-that can only filter (remove stuff that doesn't match) from the given generator (string-alphanumeric in this case)

hiredman17:03:00

and pretty much every random string is going to be filtered out

hiredman17:03:12

it is much more efficient to generate exactly what you want, but that is usually going to require writing a custom generator

hiredman17:03:28

you could also transform the output of the generator instead of filtering, e.g. instead of throwing out all the generated strings that are not all lowercase, you can take whatever generated string and make it lower case

neo255119:03:23

On the same topic, can you spec a key which would be a value? I used fn? as predicate, but who can I fake the return values, or define the return values?

neo255119:03:21

I guess I could use a generator of a set.

ghadi19:03:26

use ifn? not fn?, it's much more general

ghadi19:03:19

what do you mean by "key which would be a value"?

neo255119:03:10

{:f (fn [x] (* 2 x))}

neo255119:03:35

How would you spec :f?

ghadi19:03:06

a function would be ifn? as the predicate

ghadi19:03:51

I still didn't understand "key which would be a value" -- did you mean "key whose value is a function?"

ghadi19:03:03

not sure if there was a word missing

neo255121:03:29

@ghadi Yes, this is what I desired to mean

thurmondmb22:03:11

If I was just doing a API/ server in Clojure , are people still using ring/compojure? Any new templates people would recommend?

hindol.adhya03:03:45

There was a survey by Eric Normand (?) where he points to ring/Pedestal/yada for production services.

jr0cket00:03:33

https://github.com/metosin/compojure-api is a great template for creating an API in Clojure. It also includes swagger (open api) to automatically generate live documentation and test web interface for the API.

seancorfield22:03:41

At work, our (REST-ish) API Server is Ring/Compojure (and Component and a bunch of other stuff). @thurmondmb

thurmondmb22:03:40

Thanks @seancorfield wasn't sure if there was anything new in the past year or so

seancorfield22:03:58

If we were starting over, maybe we'd look at other libraries more focused on APIs. Maybe compojure-api, or reitit, or something. But Ring/Compojure is fine as basis for most stuff. And I almost never use a template when starting a project (even tho' I created boot-new and clj-new for that purpose!).

seancorfield22:03:38

(well, I might use clj-new to create a very basic app template, but that's it)

seancorfield22:03:42

FWIW, I like to start from the basics because most templates that do more than very simple stuff are far too opinionated for my tastes 🙂

thurmondmb23:03:53

If I went the ring/ compojure route I could then add in Clojure spec or schema for validation?

thurmondmb23:03:22

And a swagger docs dependency if I needed it later on?

seancorfield23:03:01

Sure. It's all just libraries.

seancorfield23:03:45

Even when I start from just a basic app template, I usually end up with dozens of dependencies by the time I'm finished. I just find it easier to build things up as I go.

seancorfield23:03:47

Also, clojure.spec is pulled in automatically if you're using Clojure 1.9 or 1.10 (1.10.1 is the current stable release, 1.10.2 is in alpha right now).

seancorfield23:03:18

(I just checked our API project at work and it looks like it has 16 top-level dependencies that bring in around 150 total libraries via transitive dependencies -- most of those top-level deps are actually other subprojects at work, so second-level deps would be a more realistic measure which is around 90 libraries)

seancorfield23:03:57

(our "core" business project, reused by everything else, has almost 60 top-level dependencies)

thurmondmb23:03:13

I'm going to try to build it from the ground up. When I was working in Clojure full time , my boss created our server that way was well. This was also before a lot of templates were available