Fork me on GitHub
#clojure
<
2015-08-26
>
meow01:08:44

I'm having trouble with running out of memory. Anyone got any tips for optimizing this kind of code:

(def foo-axiom [:A])

(def foo-rules {:A [:B :- :A :- :B]
                :B [:A :+ :B :+ :A]})

(defn bar [rules axiom]
  "Returns a lazy sequence of colls, starting with axiom, where each
   subsequent coll is the result of the replacement rules applied to the
   preceding coll."
  (iterate #(flatten (walk/postwalk-replace rules %)) axiom)
  #_(iterate #(flatten (replace rules %)) axiom)
  #_(iterate #(flatten (transduce (replace rules) conj %)) axiom))

(def foo-bar (bar foo-rules foo-axiom))

meow01:08:05

I'm having this weird feeling that some kind of smarter transducer ought to be able to eliminate the need for flatten, for one thing.

rauh01:08:11

@meow: Sorry, no time 😕

rauh01:08:46

Post it on Stackoverflow, it's a good question

meow02:08:17

Flatten is really overkill because this thing will never be more than one level deep. Would love to be able to flatten it on-the-fly but I'm not seeing how to do that.

meow02:08:23

Add this to the list of failed attempts: (iterate #(flatten (into [] (replace rules) %)) axiom)

meow02:08:18

For this particular set of rules on my machine I can't get past (count (nth foo-bar 15))

whacked02:08:51

I can't even get 10

whacked02:08:02

fascinating bit of code simple_smile

meow02:08:46

@whacked: wait til you see what I build using this stuff - going to be very, very cool, I believe...

whacked02:08:55

reminds of the miniKanren stuff from William Byrd

meow02:08:11

I went as high as 12:

(count (nth foo-bar 12))
=> 1062881

meow02:08:30

Then just jumped to 15 and have been trying to optimize since then.

meow02:08:07

I don't know anything about miniKanren other than having seen the term before.

whacked02:08:07

it's similar primarily in the "very very cool" part and doing some simple queries generates massive rule expansions

meow02:08:21

It's pretty crazy what you can do with one line of code in Clojure.

meow02:08:09

I'm doing a variation of Lindenmayer Systems

meow02:08:26

but taking it into an unexpected direction

whacked02:08:26

yeah sorry i can't see a way out of flatten myself here

meow02:08:47

There are alternate implementation of flatten here: https://clojuredocs.org/clojure.core/flatten

meow02:08:13

But for this case I'm not happy with them. I want something more like destructuring.

whacked02:08:25

the my-flatten is quite good actually

whacked02:08:16

(time
   (count
    (nth
     ((fn [rules axiom]
        (iterate #(flatten (walk/postwalk-replace rules %)) axiom))
      foo-rules foo-axiom)
     10)))
;; "Elapsed time: 546.204442 msecs" -- flatten
;; "Elapsed time: 108.045854 msecs" -- my-flatten

whacked02:08:01

I did 15 using my-flatten, it took maybe 2 minutes. Not going to try flatten simple_smile

meow02:08:00

@whacked: sweet! I'll give that a try.

meow02:08:39

might be worth testing it with (iterate #(flatten (replace rules %)) axiom)

meow02:08:07

and (iterate #(flatten (into [] (replace rules) %)) axiom)

meow02:08:20

with my-flatten instead of flatten

whacked02:08:14

what's the into supposed to contribute here?

whacked02:08:50

indeed replacing with replace, "Elapsed time: 237.22869 msecs" -- flatten "Elapsed time: 80.822718 msecs" -- my-flatten

meow02:08:00

the into is just a variation on the same theme

whacked02:08:08

still not sure i understand,

(prn (replace foo-rules foo-axiom)) ;; [[:B :- :A :- :B]]
  (prn (into [] (replace foo-rules foo-axiom))) ;; [[:B :- :A :- :B]]
though FWIW, timeing the added into adds single digit msec to the runtime

whacked02:08:35

oh! you're doing an L-system!!! cool

meow02:08:02

L-system or Lindenmayer system

meow02:08:12

But its just the starting point for what I'm eventually doing

meow02:08:10

I was doing (into [] (replace foo-rules) foo-axiom), not (into [] (replace foo-rules foo-axiom))

meow02:08:21

but it looks like you got the same result?

meow02:08:35

well, for one time through, but not how I actually used it

whacked02:08:56

methinks first expr will give you an exception

meow02:08:21

into can take a transducer

whacked02:08:42

uh oh, i'm on clojure 1.6.0 simple_smile

meow02:08:46

(replace ...) returns a transducer

meow02:08:54

ah, 1.7 😉

meow02:08:04

gotta love transducers

whacked02:08:10

sigh. time to quit my repl and update my project.clj

meow02:08:41

I'm still running out of memory

whacked02:08:24

well whaddya know, into is faster

whacked02:08:49

i don't understand why it should be, but it is!

meow02:08:54

into has become a goto function for me

meow02:08:35

plus its easy to comprehend once you've used it a few times

meow02:08:54

I guess everything is, but into is just really handy

whacked02:08:27

ok, it's not really faster, at least from eyeballing ~20 measurements

whacked02:08:31

it's more or less the same

luxbock02:08:15

you can't really use time to benchmark things like this because of the JIT

whacked03:08:05

learned. thanks

meow03:08:16

yeah, you really need to use criterium - I just haven't gotten around to that myself

meow03:08:00

not super worried about benchmarking at this point, just bummed that I'm getting these memory errors

meow03:08:05

(count (nth foo-bar 12))
=> 1062881
(count (nth foo-bar 13))
=> 3188645
(count (nth foo-bar 14))
OutOfMemoryError Java heap space  clojure.lang.ArrayChunk.dropFirst )

meow03:08:14

I think I need a custom version of replace

luxbock03:08:13

(iterate (fn [x] (mapcat #(if-let [e (find rules %)] (val e) [%]) x)) axiom)

luxbock03:08:33

that avoids flatten

luxbock03:08:14

you are going to run out of memory eventually though, right?

meow03:08:47

rofl, yes, of course, this thing just grows unbounded

meow03:08:13

but we need to push the limits as much as possible simple_smile

meow03:08:26

otherwise it just isn't as fun

luxbock03:08:31

I do wonder how much memory the structural sharing saves you here

meow03:08:28

I'm thinking not much because of all the substitutions, except for the fact that they are all keywords

luxbock03:08:59

my MBP is working hard at (nth foo-bar 16)

luxbock03:08:37

I just noticed cider now gives you an indicator for long running processes, which is nice

whacked03:08:57

is (find rules %) preferable to just using rules as a fn?

luxbock03:08:19

ran out of memory with the mapcat version

luxbock03:08:48

@whacked I just copied it from the source of replace

whacked03:08:07

i didn't try it... because I'm waiting for 16 to finish

whacked03:08:52

~8 minutes so far

meow03:08:19

@luxbock: perfect code - mapcat was just what I needed

luxbock03:08:08

@whacked I think (or (rules %) [%]) is definitely a bit nicer

meow03:08:05

I think there's a problem with the code

whacked03:08:56

ok OutOfPatience error after 16? minutes

meow03:08:07

@whacked: it needs to return the original/unmatched value if a replacement doesn't exist

meow03:08:36

(or (rules %) [%]) looks interesting

meow03:08:52

I don't think I'll be happy until my fractals have billions of line segments... 😉

meow03:08:51

If anyone needs data to simulate a meaningful stream of input for FRP stuff, here's one way, right?

meow03:08:18

I need to see how these variations work in cljs as well, but I think that will have to wait until tomorrow

meow03:08:26

Getting late here.

whacked03:08:13

I suppose your ruleset will also include other attribute transformations

whacked03:08:22

or you have a parallel ruleset for that?

meow03:08:19

@luxbock: I do like the look of this, thanks: (iterate (fn [x] (mapcat #(or (rules %) [%]) x)) axiom)

meow03:08:39

@whacked: I'm still working out how to handle additional data to augment the basic instructions that get rewritten

luxbock03:08:26

I'm getting close to throwing an OutofPatience exception on the 16th as well 😛

meow03:08:46

Yeah, these rewrite systems are definitely limited in terms of how many iterations are reasonable, but they're still fun and powerful

luxbock03:08:58

@meow this was posted on the mailing list yesterday: https://github.com/mschuene/stratege

luxbock03:08:04

might be of interest

meow03:08:14

So the way to scale this up is to call the rewriter more than once, rather than asking it for another iteration.

meow03:08:22

@luxbock: cool, I hadn't seen that. Might be more sophisticated than what I need.

meow03:08:17

I'm done for the day. Thank you so much, @whacked and @luxbock. Hope you had as much fun as I did! Really appreciate the help. simple_smile

luxbock04:08:28

I'm just waking up so it was a good way to warm up simple_smile

whacked04:08:53

@meow @luxbock had fun too and thanks for the criterion tip, added to my profile.clj

escherize05:08:34

Oh look, there's a Clojure article at #1 on HN right now simple_smile

jtackett09:08:32

Not sure if this is the right forum/channel for this or not, but I am currently searching for a Clojure job in Pittsburgh or DC and I wanted to reach out to everyone and see if they are hiring, or know about an open position. I also have experience with Datomic/Datalog, and Clojurescript.

danielcompton09:08:18

@jtackett: try #C05006WDW as well

viesti10:08:06

hum, having a discussion with colleagues about using Datomic in an open source project

viesti10:08:46

Datomic has so many neat things in it, but the question of keeping your data in a closed system is hard to tackle

tcrayford10:08:36

@viesti: might get some better discussion in #C03RZMDSH

viesti10:08:50

ah, true 😄

afhammad10:08:18

is there any way of printing out the compojure routes? similar to Rail's rake routes? Have some nesting going on thats broken but can't spot the problem

tcrayford10:08:12

@afhammad: not easily, because compojure routes are just nested function calls - there's no way to introspect them

afhammad10:08:57

aight, thanks.

malcolmsparks11:08:48

I know this is probably unhelpful, but bidi has a route-seq function to return a sequence of all the routes in the structure

afhammad11:08:57

Good to know, thanks @malcolmsparks

tcrayford11:08:33

@malcolmsparks: periodic ping about bidi supporting flat routes as input

tcrayford11:08:59

(I might actually have some open source time I could push into working on that at some point this/next month, if you're interested)

afhammad11:08:39

Any idea why that inner-most context doesn't work: https://gist.github.com/afhammad/3f0e97a8058f949f4663

malcolmsparks11:08:58

@tcrayford: I think what would be useful is a table to tree transformation - there's a reason why you may want to further transform the tree 1. schema validation 2. tree walking to augment (e.g. security) policies on leaves 3. make use of other tools that transform the tree structure

malcolmsparks11:08:31

on the issue @afhammad is having with compojure, bidi has a prismatic schema definition you can use for validating route structures - right that's enough blatant bidi promotion for one day

malcolmsparks11:08:27

@afhammad: GET /api/users/1/contacts/1

malcolmsparks11:08:31

don't you mean...

malcolmsparks11:08:33

GET /api/users/1/contacts/1/

malcolmsparks11:08:00

you must use a trailing slash otherwise your inner context will not match

afhammad11:08:37

that doesn't work either

afhammad11:08:27

I have a feeling it's because one of the parent context's has a parameter

afhammad11:08:18

also user-id can't be referenced from the inner-most context routes, gives: Unable to resolve symbol: user-id in this context

malcolmsparks11:08:59

(defn foo []
  "I don't do a whole lot."
  (context "/api" []
   (context "/users" []
    (context "/:user-id" [user-id]
      (context "/contacts" []
        ;;; routes here work
        (context "/:contact-id" [contact-id]
          ;;; anything here gives 404
          ;;; GET /api/users/1/contacts/1
                 (GET "/" [] (fn [req] {:status 200 :body (str "user-id:" user-id)}))))))))


((foo) (request :get "/api/users/123/contacts/1/"))

malcolmsparks11:08:26

it works for me - make sure your request uri is what you expect. Are you using this handler in any subcontexts?

malcolmsparks11:08:38

perhaps break it up into testable units

afhammad11:08:42

hmm, yes yours works for me, will need to debug more, thanks

afhammad11:08:55

fixed, need my rainbow parens back...

ricardo11:08:59

Welp, nevermind. Thought I was on #C03S1L9DN.

kushalp12:08:35

@malcolmsparks: Is it common to structure routes like that? It feels like it would be easier to read (for me) as either:

(defn foo []
  "I don't do a whole lot."
  (context "/api/users/:user-id" [user-id]
    (context "/contacts/:contact-id" [contact-id]
      ;; anything here gives 404
      ;; GET /api/users/1/contacts/1
      (GET "/" [] (fn [req] {:status 200 :body (str "user-id:" user-id)})))))
or:
(defn foo []
  "I don't do a whole lot."
  (context "/api" []
    (context "/users/:user-id" [user-id]
      (context "/contacts/:contact-id" [contact-id]
        ;; anything here gives 404
        ;; GET /api/users/1/contacts/1
        (GET "/" [] (fn [req] {:status 200 :body (str "user-id:" user-id)}))))))

afhammad13:08:59

@kushalp: there are additional sub routes under each context not shown in the sample code

kushalp13:08:37

Thanks for clarifying

sdegutis14:08:00

I'm trying to design my application to be friendly to code-reloading, as well as being inspectable such that one namespace can search for vars throughout the whole project matching a specific metadata, in this case ^:schema or ^:route

sdegutis14:08:25

But I'm running into problems using (clojure.tools.namespace.find/find-namespaces (clojure.java.classpath/classpath)) to find and then require all my own application's namespaces.

sdegutis14:08:57

For one thing, some how the current namespace is getting loaded twice.

meow14:08:39

@whacked: @luxbock I'm still working on it, but my take on a D0L system (deterministic context-free Lindenmayer system) is here: https://github.com/decomplect/ion/blob/master/src/ion/ergo/l_system.cljc

sdegutis14:08:40

Anyone here have experience using @stuartsierra's Component to make code-reloading more manageable?

sveri14:08:01

@sdegutis: works very good for me

meow14:08:29

@sdegutis: lots of people use it and like it. Even if you don't use it, be sure to understand the concepts. I also recommend reading this: https://github.com/bhauman/lein-figwheel#writing-reloadable-code

sdegutis14:08:25

Thanks both of you.

tel14:08:04

@sdegutis: what’re your question about it?

sdegutis14:08:07

I was just looking for general opinions about Component by people who have used it.

sdegutis14:08:28

I always value feedback based on actually using something and gaining retrospective perspective.

tel15:08:12

I’m using it rather extensively at the moment in both clojure and clojurescript

tel15:08:31

(and by it, I mean my own version sadly, in order to get the clojurescript support)

tel15:08:53

(but the concept is the same and the code doesn’t take more than an hour or so to spin up)

tel15:08:05

I find it pretty great for managing runtime state

tel15:08:16

the restarting bit is, well, ok, it’s nice

tel15:08:28

it makes repl-based interactive development nice

tel15:08:36

but the real key is just being explicit about state

tel15:08:38

as it always is

tel15:08:52

since clojure lacks monads there isn’t a good natural way to do that

tel15:08:58

but components aren’t a bad way

tel15:08:45

What’s the data type that clojure.zip represents?

tel15:08:53

it is just a big branching nested hashmap?

tel15:08:57

in Haskell all the generic zippers need to be built atop datatype generics, unfolding semantics, or lenses

tel15:08:30

Ahhh, there it is

tel15:08:42

zipper is parameterized on access functions

tel15:08:45

making it generic

tel15:08:25

but it looks like that makes it inherently positional

tel15:08:36

unless you want to send back children in an assoc-list form

tel15:08:12

thus why there are built in vector-zip and seq-zips but no map-zip

malcolmsparks16:08:48

@sdegutis: I use Component in a lot of projects now (about a dozen), and have used it for about 18 months. The code reloading is actually from tools.namespace, also from Stuart Sierra, which component builds upon. It solves a hard problem and doesn't usually give me any bother. I've learned to workaround the odd time it doesn't work for me, and I have to say I can't imagine using a different approach for Clojure server-side work

joelkuiper16:08:38

hmm, I’m doing something wrong. If I have a type (deftype) that extends a protocol (defprotocol), and I have a function that returns an instance of that type … and I access that type in a different namespace the functions defined by the protocol should be available right?

malcolmsparks16:08:49

well, you have to still require them

joelkuiper16:08:44

the protocol?

malcolmsparks16:08:40

you have to require all the functions of a protocol you want to call

malcolmsparks16:08:49

requiring the protocol itself isn't enough

malcolmsparks16:08:11

the only time that works is when you're extending the protocol with reify or defrecord

malcolmsparks16:08:21

but to use the protocol at other times you need to require in the symbols

malcolmsparks16:08:39

I've never questioned why this is, it just is simple_smile

sdegutis16:08:05

@malcolmsparks: Thanks for the excellent feedback on Component.

joelkuiper16:08:33

hmm, lame 😛

joelkuiper16:08:40

maybe I’ll look into reify

potetm16:08:30

@joelkuiper: to be clear, requiring the namespace that the protocol is in is sufficient.

potetm16:08:50

So the following works:

(ns foo
  (:require [ns.with.protocol :as other-ns]))

(other-ns/-protocol-fn (other-ns/create-special-type))

joelkuiper16:08:08

@potem does it also work with :refer :all ?

potetm16:08:20

If you’re into that sort of thing simple_smile

potetm16:08:28

I believe so.

abtv17:08:22

Is it possible to make post a file with ring.mock.request? I would like to test my REST API, which should save files. Is it possible or there is another way?

abtv17:08:08

The proposed solution is a little verbose. How do you test post file to your server?

kitallis19:08:29

what’s the correct way to get an integer back from (/ 1/16 1/16)?

malcolmsparks19:08:02

wrap with (int ...)

malcolmsparks19:08:19

or (.intValue ...)

jackschaedler20:08:57

does anyone have any good resources or examples of implementing a 'fake' mutable collection in clojure using store-passing-style?

Alex Miller (Clojure team)20:08:51

any Cryogen gurus here?

hlship20:08:41

A tricky leinigen question.

hlship21:08:18

I want to add a plugin to perform some hooks at lein startup; to enable pretty exception reporting via https://github.com/AvisoNovate/pretty.

hlship21:08:50

I've created a namespace, pretty.plugin, and a function, hooks.

hlship21:08:10

The trick is that pretty has an optional dependency on clojure.tools.logging; I want to know in my hook function

hlship21:08:23

if that's available, and do some additional setup when present.

hlship21:08:28

here's the hack I've been working on:

hlship21:08:34

The first part of the hook seems to work fine (i.e., via lein repl, and adding pretty to my profiles.clj.

hlship21:08:05

(io.aviso/logging has transitive dependencies on clojure.tools.logging and logback).

hlship21:08:48

Also, it seems like I'm getting different behaviors depending on whether I'm using lein repl (without a project) and lein repl in a project that has its own dependency on io.aviso/pretty.

hlship21:08:39

BTW, I'm ok if the approach requires I create a separate artifact to do the setup, if that's what it takes to make it work in a project with an existing dependency on io.aviso/pretty. For the moment, I'm hoping I can do it all in a single artifact.

nberger21:08:32

There it shows an example where it injects a dependency into the project via merge-profiles

nberger21:08:37

About detecting if a dependency is available, tools.logging uses a similar approach (try & catch), but using (Class/forName ...). See https://github.com/clojure/tools.logging/blob/master/src/main/clojure/clojure/tools/logging/impl.clj#L52

nberger21:08:51

I'm also interested to hear opinions on the different approaches

hlship21:08:35

Right now I'm using hook, and that is called with no arguments; no project available. So far I haven't found a project var global I can use, and I bet I won't.

hlship21:08:51

I could be sneaky and use some middleware.

nberger21:08:51

Ok, sorry, I missed that part