Fork me on GitHub
#beginners
<
2017-07-08
>
Lone Ranger00:07:18

so I hear the word "reflection" used a lot when discussing <this> and <that> in Clojure... what exactly does "reflection" mean? 😅

seancorfield00:07:05

It means that code that interoperates with Java has to use Java's "reflection" machinery to deduce types for resolving overloaded functions etc.

seancorfield00:07:30

(that's a bit of a circular definition but it'll depend on how much you know about Java @goomba ?)

Lone Ranger00:07:42

😅 zippo, nada

Lone Ranger00:07:57

but I'm guessing it has to do with the overhead of type based dispatch?

Lone Ranger00:07:40

(seriously wish I came from a Java background --- when are they going to make Python-hosted-Clojure!!)

Lone Ranger00:07:39

(just kidding... but seriously 😅 )

seancorfield00:07:48

Basically, when you have a call like (.method some-object) where some-object is a Java object and .method is how you call someObject.method() in Java -- but the type of some-object is not known (likely Object)... in order to determine which specific .method to call, it has to lookup the type information (of some-object and then of all the (overloaded) methods called method) to determine exactly which one to call.

Lone Ranger00:07:10

hey it's your birthday??

seancorfield00:07:20

Java has a whole package of "reflection" methods to deduce types, look up declared methods and their arg types etc.

seancorfield00:07:27

I'm 55 today 🙂

Lone Ranger00:07:29

Happy Birthday!! 😄

Lone Ranger00:07:50

Thank you for answering all my garbage questions for the past six months

seancorfield00:07:03

Happy to answer Qs any time!

seancorfield00:07:15

Remember: there are no dumb questions.

seancorfield00:07:23

(there are, sometimes, dumb answers 🙂 )

Lone Ranger00:07:44

I don't understand how you manage to code, write a blog, and answer questions on here. Probably other stuff too

seancorfield00:07:49

I don't sleep 😐

seancorfield00:07:50

Here you can see I turned on *warn-on-reflection* then defined a function q that returns a string.

seancorfield00:07:38

Then I tried to call (.length (q)) -- and the function call's return type is assumed to be Object so reflection is needed to figure out the value is a java.lang.String and that's the .length that should be called.

seancorfield00:07:08

Then I defined it again but gave a "type hint" that it returns a String -- and reflection wasn't needed in the second call.

seancorfield00:07:17

I don't write much on my blog these days. I guess I don't have much long-form stuff to say since I say all the short-form stuff here and on Twitter. Twitter's really killed my blogging. I used to write a blog post every day back when I worked at Macromedia...

seancorfield00:07:36

Anyways @goomba I hope that's a useful example of reflection in action?

Lone Ranger00:07:38

ahhhhhh fascinating yes that explains the type hinting too

moogey01:07:19

ayyyy happy birthday @seancorfield

bschrag02:07:21

Any idea what’s wrong here? On Windows, with jdk1.8.0_131 installed.

(ns edit-server.core
  (:require [org.clojure/data.json "0.2.6"])
 (:gen-class))

1. Unhandled java.lang.ClassCastException
   java.lang.Character cannot be cast to clojure.lang.Named

seancorfield02:07:41

@bschrag That's a dependency declaration -- it belongs in project.clj (or build.boot). A :require expects a namespace (not an artifact ID) and an optional alias.

seancorfield02:07:57

(:require [clojure.data.json :as json]) for example

seancorfield02:07:00

Then you can reference functions from the clojure.data.json namespace as json/read-str for example.

bschrag02:07:23

@seancorfield So I need both the dependency in project.clj and the :require form in the ns form (in core.clj)?

bschrag02:07:39

@seancorfield And happy birthday!

dpsutton02:07:10

yeah. the idea is you tell the project infrastructure which dependencies you require for your project and then in your own code you refer to particular namespaces in that dependency you happen to need for a particular namespace

seancorfield04:07:15

@bschrag Yup. Just got back from dinner (& drinks). The dependency -- the group/artifact and version -- tells Leiningen (or Boot) what JAR file to fetch from Maven Central or Clojars and the :require is how you tell Clojure which specific namespace(s) from that JAR you need to reference in your code.

seancorfield04:07:30

A JAR -- a library -- might contain one or ten or twenty or more namespaces. The group/artifact doesn't necessarily relate to the namespace. For org.clojure/data.json, the (main) namespace is clojure.data.json but the clj-http dependency has several namespaces you might need. When you see a group/artifact that has just one name, it indicates the group and artifact have the same name and it is shorthand: clj-http means clj-http/clj-http, for example.

bschrag09:07:44

@seancorfield Thanks for your help. I've been programming in Common Lisp for what has apparently been way too long, so I guess that makes me kind of a refugee. :-) Looking forward to learning a lot!

mathpunk19:07:45

I don't know if I'm holding yogthos's config library wrong, or if I'm misunderstanding something about http://clojure.java.io/file:

mathpunk19:07:15

( "~/options/index.org") => #object[java.io.File 0x3b3bc0eb "~/options/index.org"]

mathpunk19:07:41

(.exists ( "~/options/index.org") => false

mathpunk19:07:55

well, wait, that's not even a file I'm getting at through config

mathpunk19:07:17

so, why doesn't this file .exist even though it seems to exist?

mathpunk19:07:09

Ah --- it's that dang tilde

ajmagnifico19:07:33

I was reading at https://github.com/clojure/test.check . This paragraph stood out to me. “Clojure supports namespaced keywords and symbols. Note here we are just talking about namespace-qualified names, not Clojure namespace objects. These are tragically underutilized and convey important benefits because they can always co-reside in dictionaries/dbs/maps/sets without conflict. spec will allow (only) namespace-qualified keywords and symbols to name specs. People using namespaced keys for their informational maps (a practice we’d like to see grow) can register the specs for those attributes directly under those names. This categorically changes the self-description of maps, particularly in dynamic contexts, and encourages composition and consistency.”

xiongtx20:07:12

ajmagnifico: @U06UY677Y is a fan of namespaces in general: https://twitter.com/richhickey/status/249545807718670336 And why not? Namespaces are one honking great idea: https://www.python.org/dev/peps/pep-0020/

ajmagnifico19:07:02

Can someone tell me where I can read about namespace-qualified keywords and symbols and why they are “tragically underutilized?”

ajmagnifico19:07:29

I get what they are, but I’m not understanding where they fit in, where I should definitely be using them.

ajs19:07:25

@ajmagnifico clojure.spec is one of the reasons why namespaced keywords have become more popular

ajmagnifico19:07:34

Okay. I’

ajmagnifico19:07:38

I’m new to spec

ajmagnifico19:07:49

so if I keep reading more, then it should become obvious?

ajmagnifico19:07:58

I guess since I’m reading about these in the motivation for clojure.spec, I assumed that the author is lamenting their tragic under-usage in Clojure in general.

ajmagnifico19:07:18

And I just wanted to know where and why people are using those in their general (non-clojure.spec) clojure usage.

jeremys20:07:02

datomic is another use case

jeremys20:07:43

you can also use namespace qualified keywords when processing web request through middleware or interceptors

jeremys20:07:50

you can add data to the context map to be used by interceptors down the chain. Choosing namespaces for the keys you add to the map judiciously,the data you add to the context won’t override other keys.

noisesmith21:07:54

I use namespaced keywords to give some indication of parts of a hash map that belong to a specific namespace - either indicating a context specific to that ns, or specifically meant to parameterize something in that ns' code...

noisesmith21:07:42

when I have big complex data objects that go through a lot of different code, it can be nice to have namespaced keywords to help know where it's been and what's happened to that data - consider that in OO code I would have objects that come from a specific package and that gives some idea of where the data's context comes from, namespaced keywords can help add some of that kind of information to standard reusable hashmaps

Lone Ranger22:07:57

@cgrand I ended up getting a 6 fold speed increase since someone showed me how to use a transducer/pmap... curious if you think xforms could tune this even better yet?

(defn subs-windows-educt
  [n s]
  (eduction (map #(subs s % (+ % n)))
            (range (- (count s) (dec n)))))

(defn fast-ngrams [n contents]
  (->> contents
       (subs-windows-educt n)
       (frequencies)))

(defn fast-book []
  (let [content (slurp (nth files 1020))
        book (extract-book content)
        ws (words book)]
    (time (into {}
            (pmap (fn [[k v]] [k (v ws)])
                  [[:bigrams (partial ngrams 2)]
                   [:trigrams (partial ngrams 3)]
                   [:quadgrams (partial ngrams 4)]])
            ))))

cgrand22:07:35

@goomba you used tricks orthogonal to mine. I'll try to combine both approaches. Maybe on Monday.

Lone Ranger22:07:26

Hey no sweat. Also didn't realize they were orthogonal. On Friday the Python guys at the office were all smug... Monday is going to be Return of the Jedi for Clojure muahahaha

cgrand08:07:22

goomba: This new version just assumes that the inût is as string (while previously I assumed the input was either a random collection or a collection of chars).

(defn ngrams [n s]
  (x/into {}
    (comp
      (x/for [i %] (subs s i (+ i n)))
      (x/by-key identity x/count))
    (range (- (count s) n))))
This version (for n=2) is more than 6x faster than the naive-sequency approach. Now you can still parallelize on top:
user=>  (time (into [] (map #(count (ngrams-naive % data-str)) [2 3 4])))
"Elapsed time: 5201.065 msecs"
[676 17576 405638]
user=>  (time (into [] (pmap #(count (ngrams % data-str)) [2 3 4])))
"Elapsed time: 789.787 msecs"
[676 17576 405638]

cgrand09:07:52

still on the 6x ballpark, let’s sait for your actual numbers with real data

cgrand09:07:13

What’s the python time?

Lone Ranger17:07:33

for this file: Elapsed time: 1507.614613 msecs

Lone Ranger17:07:43

wow, that ngrams function shaved another 30% off my best time

Lone Ranger17:07:55

some serious magic you got going on there!!!

Lone Ranger17:07:23

Previous best: "Elapsed time: 1139.493618 msecs" (no xforms) New best: "Elapsed time: 704.064089 msecs" !!! (w/xforms)

Lone Ranger17:07:19

That's a factor of 10 increase from my original code and more than 50% faster than my Python code!

Lone Ranger17:07:11

I'm stoked. I can't wait to look at your code, damn.

cgrand20:07:54

50% faster than python but multithreaded vs singlethreaded, no?

cgrand09:07:57

Still it’s slow if you compare to the baseline mutable javaesque impl:

(defn ngrams-java [^long n ^String s]
  (let [end (- (count s) n)
        m (java.util.HashMap.)]
    (loop [i 0]
      (when (< i end)
        (let [k (.substring s i (+ i n))]
          (.put m k (inc (or (.get m k) 0))))
        (recur (inc i))))
    m))

user=>  (time (into [] (map #(count (ngrams % data-str)) [2 3 4])))
“Elapsed time: 1199.753 msecs”
[676 17576 405638]

user=>  (time (into [] (map #(count (ngrams-java % data-str)) [2 3 4])))
“Elapsed time: 366.606 msecs”
[676 17576 405638]