Fork me on GitHub
#beginners
<
2020-03-12
>
robert.nadler00:03:39

@michael401 You may want to post this question on the #emacs channel.

michael40100:03:29

Thanks. Didn't see that one.

decim02:03:19

So how would I go about pulling JSON from an API that requires a key? Using curl I would do curl “https://api.propublica.org/congress/v1/116/senate/members.json” -H “X-API-Key: my-key-here”

decim02:03:19

So how would I go about pulling JSON from an API that requires a key? Using curl I would do curl “https://api.propublica.org/congress/v1/116/senate/members.json” -H “X-API-Key: my-key-here”

bfabry02:03:23

putting it together

~/C/clj-scratch $ clj -A:clj-http                                                                             19:51:57
Clojure 1.10.1
user=> (require '[clj-http.client :as http])
nil
user=> (:body (http/get "" {:headers {"X-Api-Key" "foo"} :as :json}))
{:status "OK"}

decim02:03:56

Ahh cool! thanks =P

decim03:03:15

So I’m almost there but it says cannot open (and then shows me all of the data in the json that i’d want) (def senate-data (:body (client/get “`https://api.propublica.org/congress/v1/116/senate/members.json" {:headers {"X-Api-Key" "fGP6pB50eqy6tliOnDOCx44l3wsKCFAxMzDYTEKs"} :as :json})))` senate-data (def cleaned-data (cheshire/parsed-seq (`http://clojure.java.io/reader senate-data) true))`

bfabry03:03:46

did you add cheshire as a dependency to your project?

bfabry03:03:13

oh, you don't need to read etc

bfabry03:03:30

if cheshire is a dependency and you do :as :json the content of :body will be clojure data

decim03:03:41

so this is what it gives me…

bfabry03:03:56

yes, because clj-http already parsed the stream to clojure data using cheshire. you're trying to do it a second time. the contents of senate-data is already what you want

decim03:03:02

oh ok so the two libraries work together and I didn’t have to take that second “cleaned-data” step. Hmm…

bfabry03:03:05

that's what the :as :json directive does

decim03:03:18

gotcha..ok lets see how to break this thing up then.

bfabry03:03:43

if you remove :as :json then the value of :body will return to being a stream

decim03:03:18

ahh i see. everything is embedded in :results so i have to go a few levels in. Ok I’ll play with this but I’m sure i’ll be back

decim04:03:45

Sorry, shouldn’t this be correct? (with-open [wrtr ( "~/Code/neoamerica/resources/senate.json")] (doseq [data (seq senate-data)] (.write wrtr data) (.newLine data)))

decim04:03:31

i even created a senate.json file in that exact location and it tells me there’s no such file or directory. <--newbie

didibus06:03:59

You can't use the ~

didibus06:03:03

inside the path

didibus06:03:17

~ is actually something that your shell magically expands

didibus06:03:56

You need to put the absolute path to your home

didibus06:03:53

Or if you want to "find the user home" the same way your shell does it, you need to call: (System/getProperty "user.home")

johnwesonga05:03:17

I have the following data in a CSV file:

National/ County,Cows,Goats,Sheep,Total
county_A,23548056,24014716,1524,47564296
county_B,610257,598046,30,1208333
county_C,425121,441681,18,866820
county_D,704089,749673,25,1453787
county_E,158550,157391,2,315943
county_F,76103,67813,4,143920
county_G,173337,167327,7,340671
county_H,458975,382344,34,841353
and I’m using Oz to build some viz, I want to build a stacked bar graph, I’ve tried:
(def populations-county-viz
  {:title "Distribution of population by County"
   :data {:values (population/population-by-county)}
   :encoding {:x {:field "county" :type "ordinal"},
              :y {:aggregate "count" :type "quantitative"
                  :fields ["sheep", "goats", "cows"]
                  :axis {:title "population"},
                  }
              :color {:field "animal"
                      :type "nominal"
                      :scale {
                              :domain ["sheep", "goats", "cows"],
                              }}
              }
   :mark {:type "bar" :color "#9085DA"}})
but that doesn’t work, I can do a simple bar graph with individual columns like “sheep” etc but not stacked, any clue?

decim05:03:44

How would I go about getting the data out of results… :chamber specifically for starters. {:status “OK”, :copyright ” Copyright (c) 2020 Pro Publica Inc. All Rights Reserved.“, :results [{:congress “116”, :chamber “Senate” …}]}?

hindol.adhya05:03:04

Result is an array. So, you want :chamber out of all maps in the array?

hindol.adhya05:03:11

Or just the first one?

decim05:03:40

just for the first one. I want to pull a few things but so far i’m lost as to how to get into it

decim05:03:56

but yes i’ll want to pull fields from all of the maps

hindol.adhya05:03:15

(->> result :results (map :chamber))

decim05:03:50

hmm nice… what about… all :first_name

hindol.adhya06:03:32

(->> result :results (map #(->> % :members (map :first_name)))) ^ This if you want a one liner. But ideally, you should capture intermediate values with let for complex cases like this.

hindol.adhya06:03:26

You can change those maps to mapcats if you want to flatten the result a bit.

decim06:03:36

hmm yeah I’m going to break it down more, so as far as using let i could make a function with variables for all of the fields that I want and use a modified version of the form you just wrote to sift through each value and plug them in?

decim06:03:15

mapcat on the outer works well… is that returning a seq?

hindol.adhya06:03:29

I meant something like this,

(let [members (->> result :results (mapcat :members))]
  (->> members (mapcat :first_name)))

hindol.adhya06:03:29

map returns a seq and if it is a seq of seqs, you can flatten it with mapcat.

decim06:03:21

hmm that last one may have gone a bit too far?

decim06:03:17

(->> senate-data :results (mapcat #(->> % :members (map :first_name))))

decim06:03:28

^ i think thats the way to go…for now

hindol.adhya06:03:34

Oops, 🙂. Change to (map :first_name). I did not actually try out what I wrote since you only pasted a screenshot of the data. Too lazy.

decim06:03:00

Ok well that works…thank you! I was stumped for the last hour.

hindol.adhya06:03:00

No problem. There are many ways to do the exact same thing though. For example, you can also use for. Happy discovery!

hindol.adhya06:03:54

ClojureDocs is where I go to, to find examples. http://clojuredocs.org/clojure.core/for

decim06:03:17

LoL =( I’m there… and the essential reference book

skoude07:03:10

anybody used honeysql with postgresql and jsonb column type? I’m having a problems getting it to work..

kelveden11:03:42

You mean for building up queries? You can extend honeysql to handle it of course but personally I've found just using hsql/raw sufficient so far. E.g.

{:select   [:*]
 :from     [:mytable]
 :where    [:= (hsql/raw "data->>'myField'") my-value]
 :order-by [(hsql/raw "data->>'someOtherField'")]}

kelveden11:03:08

(where data is the JSONB field.)

mafcocinco13:03:51

Not particularly helpful for your specific question but it might be generally useful: We use hugsql and it has some very nice features for handling different types coming out of Postgres (or whatever DB you are using).

seancorfield20:03:44

I'm a bit surprised https://github.com/nilenso/honeysql-postgres doesn't have any support for that sort of stuff. Maybe you could petition that library maintainer? 🙂

dharrigan10:03:33

Hi @skoude Come to #sql and #honeysql

stebokas15:03:35

How to map nested data structure? I have maps in lists in maps

bfabry15:03:23

there's a lot of different ways. you can map over a map (it will yield [key val] tuples). so you can do (map (fn [x] (map (fn [y] (map (fn [z] ...) y)) x)) data)

bfabry15:03:04

the clojure.walk namespace is also very useful for this kind of thing https://clojure.github.io/clojure/#clojure.walk

bfabry15:03:37

all depends on what specifically you want to do

stebokas15:03:01

this walk thing looks like what I need, thanks!

bfabry15:03:57

it can take a little time to grok, but once you get it it's super useful. feel free to ask more questions!

stebokas15:03:33

Actually I need to find all objects of certain type and apply some function to them

noisesmith15:03:56

that's a common operation

stebokas15:03:15

but data is very nested 🙂 maps and lists

noisesmith15:03:33

I mean, that's a common operation with postwalk

noisesmith15:03:35

user=> (clojure.walk/postwalk (fn [x] (if (string? x) (clojure.string/reverse x) x)) {:a "abba" :b ["cedrik" :d "ef"]})
{:a "abba", :b ["kirdec" :d "fe"]}

bfabry15:03:43

^ the important bit that trips people up about the fn above is the else condition of the if btw, if it's not the type you care about changing (in this case not a string), you need to return it unchanged. the fn gets invoked not just on the leaves but the nodes as well,

noisesmith15:03:00

alternatively

(cmd)user=> (defmulti replacer type)
nil
(cmd)user=> (defmethod replacer :default [x] x)
#object[clojure.lang.MultiFn 0x58a120b0 "[email protected]"]
(cmd)user=> (defmethod replacer String [s] (clojure.string/reverse s))
#object[clojure.lang.MultiFn 0x58a120b0 "[email protected]"]
(cmd)user=> (clojure.walk/postwalk replacer {:a "abba" :b ["cedrik" :d "ef"]})
{:a "abba", :b ["kirdec" :d "fe"]}

noisesmith16:03:08

mainly useful if you have multiple transforms to make

(ins)user=> (defmethod replacer (class :k) [k] (keyword "foo" (name k)))
#object[clojure.lang.MultiFn 0x58a120b0 "[email protected]"]
(cmd)user=> (clojure.walk/postwalk replacer {:a "abba" :b ["cedrik" :d "ef"]})
#:foo{:a "abba", :b ["kirdec" :foo/d "fe"]}

stebokas17:03:09

why this one (clojure.walk/postwalk #(if (instance? js/Object %) (js-obj-read %) %) data) only returns me modified data but not from false branch (single %)

noisesmith17:03:58

I assume everything is an instance of Object

noisesmith17:03:05

you probably want to check for something else

stebokas17:03:27

I have mixed data , not only objects

noisesmith17:03:57

mixed with what? all clojure data structures and literals are isntances of Object as I understand it

noisesmith17:03:09

$ lumo
Lumo 1.8.0
ClojureScript 1.9.946
Node.js v9.2.0
 Docs: (doc function-name-here)
       (find-doc "part-of-name-here")
 Source: (source function-name-here)
 Exit: Control+D or :cljs/quit or exit

cljs.user=> (instance? js/Object {})
true

stebokas17:03:52

oh no, how do I distinguish from say #object[cljs.core.BitmapIndexedNode] and Clojure datastructures?

stebokas17:03:42

I mean Clojure stuff thom JS stuff

bfabry17:03:04

{} [] '() [] and #{} will all respond true to seq? if that helps

bfabry17:03:17

but :foo will not

noisesmith17:03:19

this might help also

cljs.user=> (satisfies? ICollection [])
true
cljs.user=> (satisfies? ICollection {})
true
cljs.user=> (satisfies? ICollection "")
false

bfabry17:03:22

which is also clojure data

noisesmith17:03:41

cljs.user=> (seq? {})
false

bfabry18:03:06

oh what am I thinking of...

noisesmith18:03:14

maybe this is what you had in mind

cljs.user=> (coll? {})
true

stebokas18:03:20

so to filter if object but not coll?

bfabry18:03:22

probably coll?

noisesmith18:03:48

yeah @ in your test, check for coll? - that only returns true for clojure data structures

noisesmith18:03:28

then you probably want special cases for number? and keyword? etc. - the list is limited

noisesmith18:03:49

the things that aren't any of the "clojure native" you can pass to your transformer

stebokas16:03:12

what is version of println that returns the arg?

bfabry16:03:21

it sounds like you might want something like spyscope https://github.com/dgrnbrg/spyscope

bfabry16:03:25

if this is for debugging

erp1216:03:51

Or you can write a simple function yourself if you don't want to add another dependency. For example,

(defn tap [f x]
  (f x)
  x)

(tap println "Hello")
I typically write a couple functions like this. Its nice to have versions where f is the first or second arg so that it can quickly be added when using -> or ->>.

alexmiller16:03:48

doto can be used for that too

alexmiller16:03:57

user=> (-> 100 inc (doto println) (* 2))
101
202

alexmiller16:03:13

that will only work well with ->, not ->>

erp1216:03:46

Oh cool. I hadn't considered ussing doto in that way. thanks!

juri18:03:33

Is there some way to "unwrap" for example a vector? Approximately the same way apply works, but for JS interop for example, where the following won't work (apply .invoke connection args) . I would like to be to do something like this: (.invoke connection (unwrap args))

bfabry18:03:16

if you know the number of arguments for the js function you can just wrap it in an anonymous fn, (apply #(.invoke connection %1 %2 %3 %4 %5) args)

bfabry18:03:22

ah, seems like you could use the javascript method apply()

bfabry18:03:33

I guess that'd be something like (-> connection .-invoke (.apply args))

juri18:03:06

Hmm, looks like it could work, will try it. Thanks!

noisesmith18:03:54

well, clojure function invocation already calls the invoke method so (apply connection args) ... ?

noisesmith18:03:52

never mind, that doesn't apply to cljs

bfabry18:03:06

cljs.user=> (.apply (.-indexOf "abab") "abab" #js ["b" 2])
3

bfabry18:03:38

pretty weird, but you could wrap the weirdness