Fork me on GitHub
#beginners
<
2018-07-01
>
justinlee01:07:24

is there a partial sort already built in to clojure? e.g., give me the 5 smallest numbers in this list

andy.fingerhut03:07:39

You can do (take 5 (sort my-unsorted-coll)), but that will sort the entire collection before doing the take 5

andy.fingerhut03:07:51

The fastest algorithm I can imagine for doing what you ask would involve a linear time inserting of all N elements into something like a priority queue data structure, then getting and deleting the min element M (e.g. M=5 in your example) times, which would take O(N + M log N) time. Sorting the entire list is O(N log N)

mg03:07:18

In theory you could do quicksort and not realize unnecessary right-hand branches

mg03:07:56

That's still O(N logN) I think, but with a lower constant factor

dpsutton03:07:06

this seems like a good stab at it:

(defn collect-n [n coll x]
  (cond (or (empty? coll) (< (count coll) n))
        (sort-by identity > (cons x coll))
        (< x (first coll))
        (take n (sort-by identity > (cons x (rest coll))))
        :else
        coll))
(reduce (partial collect-n 5) () (shuffle (range 10000)))
(4 3 2 1 0)

dpsutton03:07:29

linear scan collecting up to 5 things and maintaining the largest at the front

justinlee04:07:46

Yea there are a bunch of algos that are n log k where k is the number of elements you take. I just wanted to make sure I wasn’t missing something. I think @dpsutton’s approach is probably fine for small k

matan08:07:07

How would you explain this piece from http://clojure.org to newbs? > Clojure is a compiled language, yet remains completely dynamic Would it be fair to say that it compiles to bytecode, but only as it interprets your code (unless autogen and stuff is used).

andy.fingerhut10:07:54

As your code is loaded from a file, or entered into a REPL, each top level expression is compiled to JVM byte code (for Clojure on the JVM)

andy.fingerhut10:07:17

It compiles as it reads your code. There is no step where it "interprets" your code, if by that you mean an interpreter is involved, the way it is for some languages implemented via an interpreter.

val_waeselynck12:07:45

To be fair, the distinction between compilation and interpretation is not very relevant nowadays, as most runtimes do some of both. For instance, the JVM is an interpreter for JVM bytecode

👍 4
matan11:07:19

So just for doubt avoidance, it compiles all the way down within each top level expression, as soon as the source file is loaded? so how does this interact with AOT compilation again?

matan11:07:40

Why is AOT ultimately 'evil'?

matan11:07:17

And what do we actually mean when we say clojure's also dynamic?

sundarj13:07:52

well, many compiled languages (e.g. Java) are static, and can't be compiled on the fly like Clojure can (eval and such)

kennytilton14:07:04

“Dynamic” to me means, eg, if I have a caller function C that calls a function F, and I recompile F but not C, the next time C runs the new version of F is what runs. I guess it also means I can compile individual functions separately. 🙂

kennytilton14:07:34

This, btw, is oft called “late binding”.

sundarj14:07:11

good point, and it's a deliberate feature in Clojure. (var references within function bodies deref the var at runtime, rather than compile time)

matan14:07:02

@U0PUGPSFR Okay, I can put this in the dev cycle convenience department then... somehow I feel some people bundle additional things with being dynamic v.s. static, probably my imagination

sundarj14:07:04

yeah, it's a feature to support redefing things in the REPL

kennytilton14:07:12

Or the Rescue the Satellite 400 Million Miles Away Department. If you have one. 🙂

4
kennytilton14:07:08

Do you know the story? I am googling for it now.

matan14:07:18

So then, dynamic aside, does every top-level function really compile down to the last s-form nested in it, as soon as the source file gets loaded?

matan14:07:49

@U0PUGPSFR and no, I never knew of a clojure satellite 🙂

kennytilton14:07:56

Lisp, Clojure — what’s the difference? 🙂

sundarj14:07:48

>>>In 1994 JPL started working on the Remote Agent (RA), an autonomous spacecraft control system. RA was written entirely in Common Lisp despite unrelenting political pressure to move to C++. At one point an attempt was made to port one part of the system (the planner) to C++. This attempt had to be abandoned after a year. Based on this experience I think it's safe to say that if not for Lisp the Remote Agent would have failed. We used four different Common Lisps in the course of the Remote Agent project: MCL, Allegro, Harlequin, and CLisp. These ran in various combinations on three different operating systems: MacOS, SunOS, and vxWorks. Harlequin was the Lisp that eventually flew on the spacecraft. Most of the ground development was done in MCL and Allegro. (CLisp was also ported to vxWorks, and probably would have been the flight Lisp but for the fact that it lacked threads.) We moved code effortlessly back and forth among these systems. The Remote Agent software, running on a custom port of Harlequin Common Lisp, flew aboard Deep Space 1 (DS1), the first mission of NASA's New Millennium program. Remote Agent controlled DS1 for two days in May of 1999. During that time we were able to debug and fix a race condition that had not shown up during ground testing. (Debugging a program running on a $100M piece of hardware that is 100 million miles away is an interesting experience. Having a read-eval-print loop running on the spacecraft proved invaluable in finding and fixing the problem. The story of the Remote Agent bug is an interesting one in and of itself.) The Remote Agent was subsequently named "NASA Software of the Year".

kennytilton14:07:49

Yeah, sorry, I was off by 300 million miles…within an order of magnitude, though!

sundarj15:07:45

close enough! 😁

shidima13:07:54

Im tring to get some data out of an atom, but i think im doing it wrong:

(reset! app-state {:articles {:article-0 {:title "Hello world!", :text "This is the first article", :author "Shidima"}}})
(for [{:keys [title text author] :as article} (vals @app-state/articles)]
                 (println title))

shidima13:07:43

I get the error app_state is not defined

shidima13:07:17

but @app-state gives me the right results

mg13:07:10

You want (:articles @app-state)

mg13:07:13

specifically, (vals (:articles @app-state)). app-state/articles is something completely different - the slash is not an accessor into the map, it's the namespace of a symbol

shidima13:07:04

Yes, that is working

mg13:07:37

so if you had a namespace computer, and it had a (def answers {:meaning-of-life 42}), and another namespace, people that had a (require 'computer) in it, you'd refer to answers as computer/answers

mg13:07:03

but you couldn't do something like, computer/answers/meaning-of-life

shidima13:07:23

Aaah, I see where I am going wrong

mg13:07:43

you'd have to do (get computer/answers :meaning-of-life)

mg13:07:56

or (:meaning-of-life computer/answers)

shidima13:07:40

the tutorial I'm referencing is using different atoms in one file that gets required as state (:require [giggin.state :as state]) Bu my code has one atom for all its state

shidima13:07:42

so then you would use the @state/gigs to access one atom from the state

shidima13:07:19

atoms still get me confused

mg13:07:04

think of them like they're normal Clojure values, just behind a layer of accessing them via deref or @ and modifying them via swap! or reset! (or a few others)

mg13:07:57

or in other words, being in at atom doesn't change anything about the value

Drew Verlee14:07:13

hmm can someone explain why those dont have the same outputs?

(transduce identity - 0 [1 2]) ;;=> 3
(reduce - 0 [1 2]) ;;=> -3
it’s due to the completing function, in some way i dont understand. I mean, i dont feel like positive 3 should even exist anywhere in the calculation. I feel like this is the first thing that has really baffled me 🙂.

mg14:07:51

@drewverlee yeah, I think you're right about the completing function. So your complete reducing function there is the result of (identity -) which is just -. Remember that at the end of the process, transduce is going to call the completion arity of the reducing function, which is to call it with the result as the only arg. So right before the completion, your result is -3. Then you call - on that, and (- -3) is 3

mg14:07:52

well, you don't call that at the end, transduce does

mg14:07:21

you can get around this problem with, (transduce identity (completing -) 0 [1 2])

Drew Verlee14:07:54

i’m not sure its a problem. I just think i’m not connecting some dots. identity is the xform function so once thats done i get [1 2]. Then we call` reduce with - 0 [1 2]` which should be -3.

mg14:07:48

@drewverlee identity is not getting called on the input, it's getting called on - to produce the function that will be used with reduce

mg14:07:55

for reference, here's the contents of the relevant arity of transduce:

([xform f init coll]
     (let [f (xform f)
           ret (if (instance? clojure.lang.IReduceInit coll)
                 (.reduce ^clojure.lang.IReduceInit coll f init)
                 (clojure.core.protocols/coll-reduce coll f init))]
       (f ret)))

mg14:07:18

So there are two major differences between this and just reduce. First is the construction of the reducing function - (xform f). Second is the completion call at the end, (f ret)

Drew Verlee14:07:07

I think i was able to gloss of this detail because … comp curries functions?

Drew Verlee14:07:53

so the examples let me think of it as a xform was applied first then the reducing function at the end.

Drew Verlee14:07:17

(def xf (comp (filter odd?) (map inc)))
(transduce xf + (range 5))
;; => 6
(transduce xf + 100 (range 5))
;; => 106

Drew Verlee14:07:52

I suppose the examples confused me a bit as ((comp (filter odd?) (map inc)) +) wouldn’t do anything useful outside the context of transduce, or i’m i wrong?

mg14:07:49

you could still reduce with that.

(reduce ((comp (filter odd?) (map inc)) +) (range 5))
;; => 6

mg14:07:22

you wouldn't get the correct behavior with stateful transducers though

mg14:07:59

(transduce (comp (map inc) (partition-all 2)) conj (range 5))
;; => [[1 2] [3 4] [5]]
(reduce ((comp (map inc) (partition-all 2)) conj) [] (range 5))
;; => [[1 2] [3 4]]

mg14:07:17

also some other niceties are missing, like you can get the right init out of conj in the transduce example, but not in the reduce example

Drew Verlee14:07:41

hmm. ok. A couple things to think about. I guess i’m confused why we need a completing function. As it seems to introduce, at least to me, some oddities.

Drew Verlee14:07:06

or, why, if we do, it isn’t a different function that can supply optional.

mg14:07:06

you need the completing function so that stateful transducers will work

mg14:07:34

sorry, kind of misspoke - it's not a completing function, rather the completion arity of the reducing function

mg14:07:03

the partition-all example above shows that

mg14:07:38

at the end of the reducing process, partition-all has the 5 in its state, waiting for the second item

mg14:07:59

the completion call tells partition-all that there are no more values coming

mg14:07:56

it responds by calling down the transduction stack with (list 5)

mg14:07:32

oh I'm mistaken. It's [5]

Drew Verlee14:07:35

err ok. and why does the function (completing -) make it work as expected? ok i understand, because it has a default which is identity. lol

mg14:07:25

yeah, completing is a way to add a do-nothing completion arity to functions that don't have one

mg15:07:42

or in the case of -, to replace the 1-arity implementation that's causing problems with a do-nothing one

Drew Verlee15:07:14

right. so this is because -’s logic doesn’t have the init and first arity logic.

Drew Verlee15:07:17

I’m guessing it would break stuff to add those now, but its not clear how. I mean, if those arities arent doing anything now, then it shouldn’t be a breaking change to add them?

mg15:07:24

- has the correct mathematical implementation of the 0 and 1-arity, it would be incorrect to change it

mg15:07:15

oh, I'm mistaken, it has no implementation of 0-arity

mg15:07:21

arguably it could return 0, I think that's the subtraction identity? I forget. anyway, the 1-arity is definitely correct and necessary.

sundarj15:07:44

i think - and / don't have the 0-arity because they're not associative, whereas + and * are

Drew Verlee15:07:40

interesting. This makes sense. I was just trying to think of a way to internalize the difference and thats probably it.

Drew Verlee15:07:16

The transducer pattern/function is very useful, i mean, i can imagine its made quite a few very hard libs simply hard to write 🙂 It really feels like one of those, you have to understand it to think to use it type patterns. Not that, the idea as a whole, isn’t clear, it’s just not clear how you would do it without re-writing a lot of idioms. Which i feel is what rich did. Just decoupled existing code. <tangent> Strange, the architects i see in practice tend to build things up and hand them off, rich tends to break things down and hand them off. with maybe datomic being a noticeable difference <tangent>

sundarj15:07:28

absolutely, that's Rich's whole design philosophy, which he elucidates in his talk https://www.infoq.com/presentations/Design-Composition-Performance

sundarj15:07:14

datomic is also a separation of sorts - separating the value of the data in memory from its query engine, separating the time the data is written into its own thing, etc

Drew Verlee15:07:06

interesting perspective, i suppose i was just thinking about how many moving parts there are, but the parts aren’t really the abstraction.

sundarj15:07:47

well, it is a database and an implementation of datalog 😛

sundarj15:07:47

it's a Cognitect product as well, i suppose if it were a library it might be more modular

Drew Verlee15:07:57

I think it was a astute observation. Time, values, query, writes, data all separated with the goal of giving you the best performance (they can engineer) while actually raising the declaratives of your handles. As someone working at a company thats building a graph db inside a relational one, whose leads spend a lot of time on that db and its query language, i can see why they feel its a valuable set of tools worth promoting and selling.

Drew Verlee15:07:39

its shocking how fast, “we should do” turns into “we can’t do” then right into “we shouldn’t let our users do”

Drew Verlee15:07:16

i’m moving way off topic. 🙂 thanks again for the help

sundarj15:07:58

haha, well this is an interesting train of thought nonetheless. working on databases sounds like interesting work!

thaumant17:07:36

Hi all! Could someone tell me (or point to the right docs) why this works:

(->> (range 10)
     (filter even?)
     (into []))
=> [0 2 4 6 8]
...and this does'nt:
(def f (filter even?))
(->> (range 10)
     f
     (into []))
=>
IllegalArgumentException Don't know how to create ISeq from: clojure.core$filter$fn__5610$fn__5611  clojure.lang.RT.seqFrom (RT.java:550)

kennytilton17:07:31

Can you macroexpand the form? Trying now over here..

mg17:07:13

(filter even?) actually calls filter with 1 arg, returning a transducer

mg17:07:51

You can't call the resulting transducer the same way that you would call filter on a collection

👍 4
mg17:07:34

If you did (def f (partial filter even?)) then it would work

thaumant17:07:09

Oh yeah I see it, thanks!

mg17:07:01

In the threaded form, you never actually call (filter even?) although it looks like it, because the thread macro is putting another arg at the end

👌 8
Jay19:07:58

Hello, I was wondering how you could use macros to do a computation at compile time

Jay19:07:59

like if i have (my-macro (+ 1 2)), and i want this to become 3

bronsa20:07:07

user=> (defmacro my-macro [expr] (eval expr))
#'user/my-macro
user=> (macroexpand `(my-macro (+ 1 2)))
3

bronsa20:07:42

but beware that e.g. (let [a 3] (my-macro (+ a 2))) would fail

Jay20:07:20

is it common to use eval like this?

Jay20:07:53

from what i understand eval is an expensive function to use but at compile time i guess it doesn’t matter how long it takes, is that correct?

bronsa20:07:33

eval is not that expensive, but it's generally not a good idea to use it if you can avoid (I would never suggest anybody use that my-macro)

kennytilton21:07:46

If you unquote the expr it will evaluate at macro-expansion time. If it is like Lisp. 🙂

kennytilton21:07:45

But then, yeah, all the operands have to be available at the same time.

bronsa21:07:29

there's no amount of unquoting that will make (my-macro (+ 1 2)) evaluate (+ 1 2) at macroexpansion time

bronsa21:07:02

you need eval if you want to do that

bronsa21:07:39

(or to roll your own little interpreter over the subset of clojure you want to evaluate)

Drew Verlee21:07:45

is there anything wrong with using namespaced keywords this way? (defn foo [{:keys [bar/zoo]}] zoo)

bronsa21:07:33

no, that's perfectly valid syntax

bronsa21:07:48

since clojure 1.8 at least

zlrth22:07:30

not sure where to best ask this: I'd like to run a function via lein run -m ns/function and redirect both standard out and standard error to a file. but i can't get standard error redirected. to reproduce:

lein new app testing
add this function to core.clj:
(defn kinds-of-output
  []
  (println "testing standard out")
  (binding [*out* *err*]
    (println "printing to standard error")))
when i run it, "standard error" is printed, and "standard out" goes into the file:
5:58PM |mfm| ~/hacking/testing|  lein run -m testing.core/kinds-of-output 2>&1 >> log.txt
printing to standard error
 5:59PM |mfm| ~/hacking/testing|  cat log.txt
testing standard out
 6:00PM |mfm| ~/hacking/|  

bronsa22:07:34

you want &>> log.txt or >> log.txt 2>&1 instead of 2>&1 >> log.txt

zlrth22:07:58

indeed! thank you, i see you're right.

bronsa22:07:26

bash is weird like that 🙃

zlrth22:07:13

oh i'm getting

bash-3.2$ lein run -m testing/output-to-some-things &>> log.txt
bash: syntax error near unexpected token `>'
bash-3.2$

zlrth22:07:33

i see that &> doesn't append. working on it.

bronsa22:07:13

@mfm weird? that should work

bronsa22:07:46

dunno about bash 3.2

zlrth22:07:50

>> log.txt 2>&1 works, including appending. looks magical.

zlrth22:07:56

ah wait &>> works in zsh. oh man this is finicky. i'll keep at it. thank you again for pointing me in the right direction, @bronsa

zlrth22:07:24

thanks! (as an aside, i see &> is there as of bash 4.0. i'm probably using the bash that came with macOS. i'll upgrade. )