Fork me on GitHub
#clojure-uk
<
2017-09-22
>
chrisjd04:09:14

@conan Looks really nice. Thanks!

practicalli-john07:09:55

Anyone suggest coding kata sized challenges (1-4 hours long) for those just starting clojure / clojurescript? Links to simple solutions a bonus. Thanks.

Rachel Westmacott08:09:33

I always like to plot a mandlebrot or julia set when I first start in a language - you can do that in under an hour, but you can also spend more than 4 hours on it.

Rachel Westmacott08:09:59

it does require some maths and maybe a headstart with graphics code for beginners

Rachel Westmacott08:09:12

final solution is usually quite simple; one nested for loop over the coordinates (x & y), one iteration loop doing the (in fact quite simple) maths, and one drawing command + language-dependant overhead for actually spitting the fractal out somewhere

yogidevbear09:09:32

Morning ☀️

maleghast09:09:20

Morning All…

maleghast09:09:54

@conan - I had no idea that ClubHouse was a Clojure app; how cool 🙂

conan09:09:18

no i've been using it for a couple of months and only just found out myself!

maleghast09:09:28

So, lovely UK Clojure people, I have a question… If you had this function:

(defn init-noaa-canonical-db-pt
  "Command Function to create PostgreSQL databse for NOAA daily observations data - PT"
  []
  (let [st-data (rest (ingest-csv-data "/rawdata/NOAA_DailySummaries_Portugal_1992_to_2016_normalised.csv"))
        st-data-prepared (get-station-data-prepared st-data (:station-keys ingest-cfg))]
    (doseq [row st-data-prepared]
      (sql/populate-noaa-stations-pt! {:? (ordered-values-vector-from-map
                                        (:station-keys ingest-cfg)
                                        row)}))
    (doseq [row st-data]
      (sql/populate-noaa-daily-measurements-pt! {:? (ordered-values-vector-from-map
                                                  (:noaa-measurement-keys ingest-cfg)
                                                  (first (noaa-prepare-measurements-data
                                                          row)))}))))
and you wanted functions that would do the same thing, but for other database tables, would we be in macro club territory?

maleghast09:09:20

(I am using Yesql, so there are different functions for each table Insert, i.e. sql/populate-noaa-stations-pt! as above for Portugal, and as an example sql/populate-noaa-stations-ci! for Côte d’Ivoire)

maleghast09:09:59

i.e. Can one use a macro to re-write the symbol of a function call or several function calls?

conan09:09:23

Does Yesql let you enumerate the functions it generates in some way?

maleghast09:09:27

(Clearly the string param could become a config value)

maleghast09:09:54

@conan - Not that I am aware of, and I have spent some time spelunking in the docs, but I may__ be wrong…

maleghast09:09:59

(i.e. probably am)

conan09:09:31

you could probably just read the files from your sql directory actully, the naming is based on those, right? then you could generate the symbols

conan09:09:49

(it's been a while since i used yesql)

maleghast09:09:55

In the “here and now” I am prepared to create 5 other, practically identical functions, but overall that much repetition feels wrong.

maleghast09:09:30

@conan - I am using the “one file” approach, with annotations / comments that name__ the queries / functions

conan09:09:52

could you put all the country codes in a db table and iterate over them with the same sql function?

conan09:09:58

or are they all different

conan09:09:10

i'm trying to think of anything that is not a macro!

maleghast09:09:54

Right, so because Yesql ONLY allows for parameterised replacement of values, i.e. you can’t parameterise a tablename, that would not help - using Yesql I HAVE TO have a query per table.

maleghast09:09:19

So looping does not really help, unless I store the query / function names.

danielneal09:09:38

yeah, yesql gives you back the names of the functions it has defined in defqueries

danielneal09:09:48

might it be possible to look through those and find the right one?

danielneal09:09:52

if you have a convention?

maleghast09:09:48

It might be, I am not familiar with how to get that info back out, but I will re-check the docs… I am also not sure how to invoke a function using a value returned from a function… (I realise that this might seem supremely n00b-ish, but I’ve never had to do it before and I have no idea how one would go about doing it in Clojure)

danielneal09:09:14

(defn query-with-name [s query-vars]
  (some #(when (= (:name (meta %)) n) %)
        query-vars))

maleghast09:09:39

I am going to have to unpack that, but thanks 🙂 (full disclosure, I think I understand about 70% of that expression)

danielneal09:09:48

I typed it wrong which didn't help 😄, fixed now

danielneal09:09:56

so when you do defqueries you get the vars back

danielneal09:09:21

(defqueries "somefile.sql") => [#'insert-country #'insert-something]

danielneal09:09:36

if you store those vars in a variable

danielneal09:09:49

(def query-vars (defqueries "somefile.sql"))

danielneal09:09:54

you could look in it for a var with a matching name

danielneal09:09:00

(defn query-with-name [s query-vars]
  (some #(when (= (:name (meta %)) n) %)
        query-vars))

danielneal09:09:16

(query-with-name "insert-country") => #'insert-country

danielneal09:09:37

(let [x "england"
q (query-with-name (str "insert-country-"  x)] (q ...)

maleghast09:09:19

I see, I think… So I would run the query like this:

((query-with-name "populate-noaa-daily-measurements-pt!") valuescoll)

maleghast09:09:52

or as you show above, let-bind the query to a scoped var.

danielneal09:09:02

yeah exact;y

maleghast09:09:04

🙂 Very awesome 🙂 Thanks

maleghast09:09:36

Just one thing… What is “n” doing / representing on the second line of that function “query-with-name”?

maleghast10:09:05

I tried to implement the function and I am getting an “Unable to resolve symbol…” error on “n” and so I looked at it again and I thought, huh?

maleghast10:09:32

Should it be “s” instead of “n” ?

maleghast10:09:00

It should be - also you need to wrap

(:name (meta %))
with str, like this:
(str (:name (meta %)))
else it never matches, ‘cos the result of pulling :name out of the meta result is NOT a string, it’s a symbol… 🙂

maleghast10:09:24

But other than that, AWESOME! Totally working a treat, thanks so much 🙂

danielneal11:09:07

ha sorry yeah was just typing without testing

maleghast12:09:08

No worries - I figured it out 🙂

maleghast12:09:31

I am having a data integrity problem now, with my CSV files… 😞 You approach works a treat - thanks again 🙂

danielneal12:09:40

programming_lyfe.txt

maleghast13:09:17

Basically the HUGE CSV for Indonesia has a broken field in it somewhere, and as such I am buggered. Still the other 5 countries work! 🙂

maleghast09:09:01

I am starting to be of the opinion that I need to migrate to HugSQL as that allows for parameter replacement of SQL Identifiers, like table names.

maleghast09:09:58

I don’t really want to write a macro - I mean my question boils down to “can one write a macro that will, based on input params, rewrite the invoked functions?”

maleghast09:09:04

(I don’t even know if that’s possible)

conan09:09:05

i was thinking of doing the parameterisation in the SQL query inside a transaction, by loading the data from a table; I might be off here though. That said, you don't have to use Yesql for everything, in this case you could swap it out

maleghast09:09:48

@conan - Indeed, and I am starting to think that HugSQL is a better fit for my use cases

conan09:09:49

it looks like yesql has a raw-sql function

maleghast09:09:08

It does, but that rather defeats the object of using the library imho.

maleghast09:09:28

(the object being to segregate Clojure and the SQL)

conan09:09:01

it does feel like the thin end of the wedge

maleghast09:09:45

HugSQL would allow me to have a generalised query that can do these Inserts on any of the tables that conform to the structure of the data. All my munging functions are already generalised, and are being re-used, so as long as the CSV data is in the right format / structure I already have re-usable functions that can deal with all the CSVs / data sources.

maleghast09:09:14

It’s the problem of having a discrete function to INSERT into each of 6 tables that leads to the repetition.

maleghast09:09:18

If I swap Yesql out and put HugSQL in I have a (small) hill to climb in terms of adoption and so even temporarily having a macro seems better as I have a deadline to hit…

mattford09:09:08

Workflow help: I have some code which I'm writing hand-in-hand with some extensions to a library. How best to update the library and have it reflected in the REPL (my code) that uses it?

mattford09:09:26

i.e, can I avoid the lein install song'n'dance.

maleghast09:09:15

@mattford - I am sad to say that my experience doesn’t go that deep, but that I’ve got a feeling that the lein install dance may be required…

glenjamin09:09:24

lein checkouts are probably the best way to work across the two in sync

glenjamin09:09:35

other two options are: decide what you need from the lib and build in isolation from the actual code throw in an extra (ns) form or similar on your side and write the lib code inline until it’s stablised

maleghast09:09:40

@conan - In case you are interested, @danieleneal has offered a really interesting take on my question above ^^ which I am certain will work and is very elegant. Ping me if you’d like to see it and I’ll send you the “solution” 🙂

conan09:09:03

oh yes, i am interested

conan09:09:40

@mattford use leiningen checkouts

conan09:09:14

you can just reload the files in the repl too though

maleghast10:09:00

@conan - Will get it working and then send you the overview in a chat… 🙂

conan10:09:14

great, thanks!

mattford10:09:43

hah! that worked a treat. Awesome thanks.

maleghast17:09:00

@danieleneal - Just wanted to say “Thanks!” one more time - my code is working very nicely on live and is currently loading LOADS of data into my production databases 🙂

danielneal19:09:20

Aw thanks! Glad to hear :)