This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2020-03-12
Channels
- # announcements (65)
- # aws (1)
- # babashka (12)
- # beginners (111)
- # bristol-clojurians (1)
- # cider (32)
- # clj-kondo (55)
- # clojars (3)
- # clojure (71)
- # clojure-europe (17)
- # clojure-france (4)
- # clojure-italy (36)
- # clojure-losangeles (8)
- # clojure-nl (6)
- # clojure-uk (115)
- # clojurescript (2)
- # datomic (99)
- # fulcro (32)
- # graalvm (12)
- # graphql (20)
- # hoplon (203)
- # meander (56)
- # mount (3)
- # off-topic (17)
- # pathom (17)
- # reitit (22)
- # shadow-cljs (32)
- # spacemacs (9)
- # tools-deps (19)
- # vim (25)
- # vscode (3)
@michael401 You may want to post this question on the #emacs channel.
Thanks. Didn't see that one.
So how would I go about pulling JSON from an API that requires a key? Using curl I would do curl “https://api.propublica.org/congress/v1/116/senate/members.json” -H “X-API-Key: my-key-here”
I would use clj-http https://github.com/dakrone/clj-http and set the headers as needed https://github.com/dakrone/clj-http#get
and I would include cheshire as a dependency https://github.com/dakrone/clj-http#optional-dependencies and use :as :json https://github.com/dakrone/clj-http#output-coercion
putting it together
~/C/clj-scratch $ clj -A:clj-http 19:51:57
Clojure 1.10.1
user=> (require '[clj-http.client :as http])
nil
user=> (:body (http/get " " {:headers {"X-Api-Key" "foo"} :as :json}))
{:status "OK"}
So I’m almost there but it says cannot open (and then shows me all of the data in the json that i’d want)
(def senate-data
(:body (client/get “`https://api.propublica.org/congress/v1/116/senate/members.json" {:headers {"X-Api-Key" "fGP6pB50eqy6tliOnDOCx44l3wsKCFAxMzDYTEKs"} :as :json})))`
senate-data
(def cleaned-data (cheshire/parsed-seq (`http://clojure.java.io/reader senate-data) true))`
if cheshire is a dependency and you do :as :json the content of :body will be clojure data
yes, because clj-http already parsed the stream to clojure data using cheshire. you're trying to do it a second time. the contents of senate-data
is already what you want
oh ok so the two libraries work together and I didn’t have to take that second “cleaned-data” step. Hmm…
ahh i see. everything is embedded in :results so i have to go a few levels in. Ok I’ll play with this but I’m sure i’ll be back
Sorry, shouldn’t this be correct?
(with-open [wrtr (
(doseq [data (seq senate-data)]
(.write wrtr data)
(.newLine data)))
i even created a senate.json file in that exact location and it tells me there’s no such file or directory. <--newbie
Or if you want to "find the user home" the same way your shell does it, you need to call: (System/getProperty "user.home")
I have the following data in a CSV file:
National/ County,Cows,Goats,Sheep,Total
county_A,23548056,24014716,1524,47564296
county_B,610257,598046,30,1208333
county_C,425121,441681,18,866820
county_D,704089,749673,25,1453787
county_E,158550,157391,2,315943
county_F,76103,67813,4,143920
county_G,173337,167327,7,340671
county_H,458975,382344,34,841353
and I’m using Oz to build some viz, I want to build a stacked bar graph, I’ve tried:
(def populations-county-viz
{:title "Distribution of population by County"
:data {:values (population/population-by-county)}
:encoding {:x {:field "county" :type "ordinal"},
:y {:aggregate "count" :type "quantitative"
:fields ["sheep", "goats", "cows"]
:axis {:title "population"},
}
:color {:field "animal"
:type "nominal"
:scale {
:domain ["sheep", "goats", "cows"],
}}
}
:mark {:type "bar" :color "#9085DA"}})
but that doesn’t work, I can do a simple bar graph with individual columns like “sheep” etc but not stacked, any clue?How would I go about getting the data out of results… :chamber specifically for starters. {:status “OK”, :copyright ” Copyright (c) 2020 Pro Publica Inc. All Rights Reserved.“, :results [{:congress “116”, :chamber “Senate” …}]}?
just for the first one. I want to pull a few things but so far i’m lost as to how to get into it
(->> result :results (map #(->> % :members (map :first_name))))
^ This if you want a one liner. But ideally, you should capture intermediate values with let
for complex cases like this.
hmm yeah I’m going to break it down more, so as far as using let i could make a function with variables for all of the fields that I want and use a modified version of the form you just wrote to sift through each value and plug them in?
I meant something like this,
(let [members (->> result :results (mapcat :members))]
(->> members (mapcat :first_name)))
Oops, 🙂. Change to (map :first_name)
. I did not actually try out what I wrote since you only pasted a screenshot of the data. Too lazy.
No problem. There are many ways to do the exact same thing though. For example, you can also use for
. Happy discovery!
ClojureDocs is where I go to, to find examples. http://clojuredocs.org/clojure.core/for
anybody used honeysql with postgresql and jsonb column type? I’m having a problems getting it to work..
You mean for building up queries? You can extend honeysql to handle it of course but personally I've found just using hsql/raw
sufficient so far. E.g.
{:select [:*]
:from [:mytable]
:where [:= (hsql/raw "data->>'myField'") my-value]
:order-by [(hsql/raw "data->>'someOtherField'")]}
Not particularly helpful for your specific question but it might be generally useful: We use hugsql
and it has some very nice features for handling different types coming out of Postgres (or whatever DB you are using).
I'm a bit surprised https://github.com/nilenso/honeysql-postgres doesn't have any support for that sort of stuff. Maybe you could petition that library maintainer? 🙂
How to map nested data structure? I have maps in lists in maps
there's a lot of different ways. you can map over a map (it will yield [key val] tuples). so you can do (map (fn [x] (map (fn [y] (map (fn [z] ...) y)) x)) data)
the clojure.walk namespace is also very useful for this kind of thing https://clojure.github.io/clojure/#clojure.walk
this walk thing looks like what I need, thanks!
it can take a little time to grok, but once you get it it's super useful. feel free to ask more questions!
Actually I need to find all objects of certain type and apply some function to them
that's a common operation
but data is very nested 🙂 maps and lists
I mean, that's a common operation with postwalk
user=> (clojure.walk/postwalk (fn [x] (if (string? x) (clojure.string/reverse x) x)) {:a "abba" :b ["cedrik" :d "ef"]})
{:a "abba", :b ["kirdec" :d "fe"]}
^ the important bit that trips people up about the fn above is the else condition of the if btw, if it's not the type you care about changing (in this case not a string), you need to return it unchanged. the fn gets invoked not just on the leaves but the nodes as well,
alternatively
(cmd)user=> (defmulti replacer type)
nil
(cmd)user=> (defmethod replacer :default [x] x)
#object[clojure.lang.MultiFn 0x58a120b0 "clojure.lang.MultiFn@58a120b0"]
(cmd)user=> (defmethod replacer String [s] (clojure.string/reverse s))
#object[clojure.lang.MultiFn 0x58a120b0 "clojure.lang.MultiFn@58a120b0"]
(cmd)user=> (clojure.walk/postwalk replacer {:a "abba" :b ["cedrik" :d "ef"]})
{:a "abba", :b ["kirdec" :d "fe"]}
mainly useful if you have multiple transforms to make
(ins)user=> (defmethod replacer (class :k) [k] (keyword "foo" (name k)))
#object[clojure.lang.MultiFn 0x58a120b0 "clojure.lang.MultiFn@58a120b0"]
(cmd)user=> (clojure.walk/postwalk replacer {:a "abba" :b ["cedrik" :d "ef"]})
#:foo{:a "abba", :b ["kirdec" :foo/d "fe"]}
why this one (clojure.walk/postwalk #(if (instance? js/Object %) (js-obj-read %) %) data)
only returns me modified data but not from false branch (single %)
I assume everything is an instance of Object
you probably want to check for something else
I have mixed data , not only objects
mixed with what? all clojure data structures and literals are isntances of Object as I understand it
$ lumo
Lumo 1.8.0
ClojureScript 1.9.946
Node.js v9.2.0
Docs: (doc function-name-here)
(find-doc "part-of-name-here")
Source: (source function-name-here)
Exit: Control+D or :cljs/quit or exit
cljs.user=> (instance? js/Object {})
true
oh no, how do I distinguish from say #object[cljs.core.BitmapIndexedNode]
and Clojure datastructures?
I mean Clojure stuff thom JS stuff
this might help also
cljs.user=> (satisfies? ICollection [])
true
cljs.user=> (satisfies? ICollection {})
true
cljs.user=> (satisfies? ICollection "")
false
cljs.user=> (seq? {})
false
so to filter if object but not coll?
yeah @UU0EZB7M3 in your test, check for coll?
- that only returns true for clojure data structures
ok ta
then you probably want special cases for number?
and keyword?
etc. - the list is limited
the things that aren't any of the "clojure native" you can pass to your transformer
what is version of println
that returns the arg?
it sounds like you might want something like spyscope https://github.com/dgrnbrg/spyscope
also available as a function in tools.logging http://clojure.github.io/tools.logging/#clojure.tools.logging/spy
println-str
Or you can write a simple function yourself if you don't want to add another dependency. For example,
(defn tap [f x]
(f x)
x)
(tap println "Hello")
I typically write a couple functions like this. Its nice to have versions where f
is the first or second arg so that it can quickly be added when using ->
or ->>
.doto
can be used for that too
user=> (-> 100 inc (doto println) (* 2))
101
202
that will only work well with ->
, not ->>
Is there some way to "unwrap" for example a vector? Approximately the same way apply
works, but for JS interop for example, where the following won't work (apply .invoke connection args)
. I would like to be to do something like this: (.invoke connection (unwrap args))
if you know the number of arguments for the js function you can just wrap it in an anonymous fn, (apply #(.invoke connection %1 %2 %3 %4 %5) args)
well, clojure function invocation already calls the invoke
method so (apply connection args)
... ?
never mind, that doesn't apply to cljs