I

I'm having trouble with running out of memory. Anyone got any tips for optimizing this kind of code:

(def foo-axiom [:A])

(def foo-rules {:A [:B :- :A :- :B]
                :B [:A :+ :B :+ :A]})

(defn bar [rules axiom]
  "Returns a lazy sequence of colls, starting with axiom, where each
   subsequent coll is the result of the replacement rules applied to the
   preceding coll."
  (iterate #(flatten (walk/postwalk-replace rules %)) axiom)
  #_(iterate #(flatten (replace rules %)) axiom)
  #_(iterate #(flatten (transduce (replace rules) conj %)) axiom))

(def foo-bar (bar foo-rules foo-axiom))

I'm having this weird feeling that some kind of smarter transducer ought to be able to eliminate the need for flatten, for one thing.

@rauh: Ping!

@meow: Sorry, no time 😕

@rauh: np

Post it on Stackoverflow, it's a good question

Flatten is really overkill because this thing will never be more than one level deep. Would love to be able to flatten it on-the-fly but I'm not seeing how to do that.

Add this to the list of failed attempts: (iterate #(flatten (into [] (replace rules) %)) axiom)

For this particular set of rules on my machine I can't get past (count (nth foo-bar 15))

I can't even get 10

fascinating bit of code

@whacked: wait til you see what I build using this stuff - going to be very, very cool, I believe...

reminds of the miniKanren stuff from William Byrd

I went as high as 12:

(count (nth foo-bar 12))
=> 1062881

Then just jumped to 15 and have been trying to optimize since then.

I don't know anything about miniKanren other than having seen the term before.

it's similar primarily in the "very very cool" part and doing some simple queries generates massive rule expansions

It's pretty crazy what you can do with one line of code in Clojure.

I'm doing a variation of Lindenmayer Systems

but taking it into an unexpected direction

yeah sorry i can't see a way out of flatten myself here

There are alternate implementation of flatten here: https://clojuredocs.org/clojure.core/flatten

But for this case I'm not happy with them. I want something more like destructuring.

the my-flatten is quite good actually

(time
   (count
    (nth
     ((fn [rules axiom]
        (iterate #(flatten (walk/postwalk-replace rules %)) axiom))
      foo-rules foo-axiom)
     10)))
;; "Elapsed time: 546.204442 msecs" -- flatten
;; "Elapsed time: 108.045854 msecs" -- my-flatten

I did 15 using my-flatten, it took maybe 2 minutes. Not going to try flatten

@whacked: sweet! I'll give that a try.

might be worth testing it with (iterate #(flatten (replace rules %)) axiom)

and (iterate #(flatten (into [] (replace rules) %)) axiom)

with my-flatten instead of flatten

what's the into supposed to contribute here?

indeed replacing with replace, "Elapsed time: 237.22869 msecs" -- flatten "Elapsed time: 80.822718 msecs" -- my-flatten

the into is just a variation on the same theme

still not sure i understand,

(prn (replace foo-rules foo-axiom)) ;; [[:B :- :A :- :B]]
  (prn (into [] (replace foo-rules foo-axiom))) ;; [[:B :- :A :- :B]]

oh! you're doing an L-system!!! cool

yes

L-system or Lindenmayer system

But its just the starting point for what I'm eventually doing

I was doing (into [] (replace foo-rules) foo-axiom), not (into [] (replace foo-rules foo-axiom))

but it looks like you got the same result?

well, for one time through, but not how I actually used it

methinks first expr will give you an exception

into can take a transducer

uh oh, i'm on clojure 1.6.0

(replace ...) returns a transducer

ah, 1.7 😉

gotta love transducers

sigh. time to quit my repl and update my project.clj

I'm still running out of memory

well whaddya know, into is faster

i don't understand why it should be, but it is!

into has become a goto function for me

plus its easy to comprehend once you've used it a few times

I guess everything is, but into is just really handy

ok, it's not really faster, at least from eyeballing ~20 measurements

it's more or less the same

you can't really use time to benchmark things like this because of the JIT

check out https://github.com/hugoduncan/criterium

learned. thanks

yeah, you really need to use criterium - I just haven't gotten around to that myself

not super worried about benchmarking at this point, just bummed that I'm getting these memory errors

(count (nth foo-bar 12))
=> 1062881
(count (nth foo-bar 13))
=> 3188645
(count (nth foo-bar 14))
OutOfMemoryError Java heap space  clojure.lang.ArrayChunk.dropFirst )

I think I need a custom version of replace

@meow what about `

(iterate (fn [x] (mapcat #(if-let [e (find rules %)] (val e) [%]) x)) axiom)

that avoids flatten

you are going to run out of memory eventually though, right?

rofl, yes, of course, this thing just grows unbounded

but we need to push the limits as much as possible

otherwise it just isn't as fun

I do wonder how much memory the structural sharing saves you here

I'm thinking not much because of all the substitutions, except for the fact that they are all keywords

my MBP is working hard at (nth foo-bar 16)

I just noticed cider now gives you an indicator for long running processes, which is nice

is (find rules %) preferable to just using rules as a fn?

ran out of memory with the mapcat version

@whacked I just copied it from the source of replace

ic

i didn't try it... because I'm waiting for 16 to finish

~8 minutes so far

@luxbock: perfect code - mapcat was just what I needed

@whacked I think (or (rules %) [%]) is definitely a bit nicer

I think there's a problem with the code

ok OutOfPatience error after 16? minutes

nvmnd

@whacked: it needs to return the original/unmatched value if a replacement doesn't exist

(or (rules %) [%]) looks interesting

I don't think I'll be happy until my fractals have billions of line segments... 😉

If anyone needs data to simulate a meaningful stream of input for FRP stuff, here's one way, right?

I need to see how these variations work in cljs as well, but I think that will have to wait until tomorrow

Getting late here.

I suppose your ruleset will also include other attribute transformations

or you have a parallel ruleset for that?

@luxbock: I do like the look of this, thanks: (iterate (fn [x] (mapcat #(or (rules %) [%]) x)) axiom)

@whacked: I'm still working out how to handle additional data to augment the basic instructions that get rewritten

I'm getting close to throwing an OutofPatience exception on the 16th as well 😛

Yeah, these rewrite systems are definitely limited in terms of how many iterations are reasonable, but they're still fun and powerful

@meow this was posted on the mailing list yesterday: https://github.com/mschuene/stratege

might be of interest

So the way to scale this up is to call the rewriter more than once, rather than asking it for another iteration.

@luxbock: cool, I hadn't seen that. Might be more sophisticated than what I need.

I'm done for the day. Thank you so much, @whacked and @luxbock. Hope you had as much fun as I did! Really appreciate the help.

I'm just waking up so it was a good way to warm up

@meow @luxbock had fun too and thanks for the criterion tip, added to my profile.clj

Oh look, there's a Clojure article at #1 on HN right now

@escherize: nice one too

Not sure if this is the right forum/channel for this or not, but I am currently searching for a Clojure job in Pittsburgh or DC and I wanted to reach out to everyone and see if they are hiring, or know about an open position. I also have experience with Datomic/Datalog, and Clojurescript.

@jtackett: try #C05006WDW as well

hum, having a discussion with colleagues about using Datomic in an open source project

Datomic has so many neat things in it, but the question of keeping your data in a closed system is hard to tackle

@viesti: might get some better discussion in #C03RZMDSH

ah, true 😄

is there any way of printing out the compojure routes? similar to Rail's rake routes? Have some nesting going on thats broken but can't spot the problem

@afhammad: not easily, because compojure routes are just nested function calls - there's no way to introspect them

aight, thanks.

I know this is probably unhelpful, but bidi has a route-seq function to return a sequence of all the routes in the structure

Good to know, thanks @malcolmsparks

@malcolmsparks: periodic ping about bidi supporting flat routes as input

(I might actually have some open source time I could push into working on that at some point this/next month, if you're interested)

@tcrayford: pong

Any idea why that inner-most context doesn't work: https://gist.github.com/afhammad/3f0e97a8058f949f4663

@tcrayford: I think what would be useful is a table to tree transformation - there's a reason why you may want to further transform the tree 1. schema validation 2. tree walking to augment (e.g. security) policies on leaves 3. make use of other tools that transform the tree structure

on the issue @afhammad is having with compojure, bidi has a prismatic schema definition you can use for validating route structures - right that's enough blatant bidi promotion for one day

@afhammad: GET /api/users/1/contacts/1

don't you mean...

GET /api/users/1/contacts/1/

you must use a trailing slash otherwise your inner context will not match

that doesn't work either

I have a feeling it's because one of the parent context's has a parameter

@afhammad: this works for me: https://gist.github.com/malcolmsparks/8145725b8c08c45fac20

also user-id can't be referenced from the inner-most context routes, gives: Unable to resolve symbol: user-id in this context

(defn foo []
  "I don't do a whole lot."
  (context "/api" []
   (context "/users" []
    (context "/:user-id" [user-id]
      (context "/contacts" []
        ;;; routes here work
        (context "/:contact-id" [contact-id]
          ;;; anything here gives 404
          ;;; GET /api/users/1/contacts/1
                 (GET "/" [] (fn [req] {:status 200 :body (str "user-id:" user-id)}))))))))


((foo) (request :get "/api/users/123/contacts/1/"))

it works for me - make sure your request uri is what you expect. Are you using this handler in any subcontexts?

perhaps break it up into testable units

hmm, yes yours works for me, will need to debug more, thanks

fixed, need my rainbow parens back...

Welp, nevermind. Thought I was on #C03S1L9DN.

@malcolmsparks: Is it common to structure routes like that? It feels like it would be easier to read (for me) as either:

(defn foo []
  "I don't do a whole lot."
  (context "/api/users/:user-id" [user-id]
    (context "/contacts/:contact-id" [contact-id]
      ;; anything here gives 404
      ;; GET /api/users/1/contacts/1
      (GET "/" [] (fn [req] {:status 200 :body (str "user-id:" user-id)})))))

(defn foo []
  "I don't do a whole lot."
  (context "/api" []
    (context "/users/:user-id" [user-id]
      (context "/contacts/:contact-id" [contact-id]
        ;; anything here gives 404
        ;; GET /api/users/1/contacts/1
        (GET "/" [] (fn [req] {:status 200 :body (str "user-id:" user-id)}))))))

@kushalp: there are additional sub routes under each context not shown in the sample code

Thanks for clarifying

Hi there.

I'm trying to design my application to be friendly to code-reloading, as well as being inspectable such that one namespace can search for vars throughout the whole project matching a specific metadata, in this case ^:schema or ^:route

But I'm running into problems using (clojure.tools.namespace.find/find-namespaces (clojure.java.classpath/classpath)) to find and then require all my own application's namespaces.

For one thing, some how the current namespace is getting loaded twice.

@whacked: @luxbock I'm still working on it, but my take on a D0L system (deterministic context-free Lindenmayer system) is here: https://github.com/decomplect/ion/blob/master/src/ion/ergo/l_system.cljc

Anyone here have experience using @stuartsierra's Component to make code-reloading more manageable?

@sdegutis: works very good for me

@sdegutis: lots of people use it and like it. Even if you don't use it, be sure to understand the concepts. I also recommend reading this: https://github.com/bhauman/lein-figwheel#writing-reloadable-code

Thanks both of you.

@sdegutis: what’re your question about it?

I was just looking for general opinions about Component by people who have used it.

I always value feedback based on actually using something and gaining retrospective perspective.

gotcha

I’m using it rather extensively at the moment in both clojure and clojurescript

(and by it, I mean my own version sadly, in order to get the clojurescript support)

(but the concept is the same and the code doesn’t take more than an hour or so to spin up)

I find it pretty great for managing runtime state

the restarting bit is, well, ok, it’s nice

it makes repl-based interactive development nice

but the real key is just being explicit about state

as it always is

since clojure lacks monads there isn’t a good natural way to do that

but components aren’t a bad way

What’s the data type that clojure.zip represents?

it is just a big branching nested hashmap?

in Haskell all the generic zippers need to be built atop datatype generics, unfolding semantics, or lenses

Thanks @tel.

np

Ahhh, there it is

zipper is parameterized on access functions

making it generic

but it looks like that makes it inherently positional

unless you want to send back children in an assoc-list form

thus why there are built in vector-zip and seq-zips but no map-zip

@sdegutis: I use Component in a lot of projects now (about a dozen), and have used it for about 18 months. The code reloading is actually from tools.namespace, also from Stuart Sierra, which component builds upon. It solves a hard problem and doesn't usually give me any bother. I've learned to workaround the odd time it doesn't work for me, and I have to say I can't imagine using a different approach for Clojure server-side work

hmm, I’m doing something wrong. If I have a type (deftype) that extends a protocol (defprotocol), and I have a function that returns an instance of that type … and I access that type in a different namespace the functions defined by the protocol should be available right?

well, you have to still require them

the protocol?

the symbols

you have to require all the functions of a protocol you want to call

requiring the protocol itself isn't enough

the only time that works is when you're extending the protocol with reify or defrecord

but to use the protocol at other times you need to require in the symbols

I've never questioned why this is, it just is

@malcolmsparks: Thanks for the excellent feedback on Component.

hmm, lame 😛

maybe I’ll look into reify

@joelkuiper: to be clear, requiring the namespace that the protocol is in is sufficient.

So the following works:

(ns foo
  (:require [ns.with.protocol :as other-ns]))

(other-ns/-protocol-fn (other-ns/create-special-type))

@potem does it also work with :refer :all ?

If you’re into that sort of thing

I believe so.

Is it possible to make post a file with ring.mock.request? I would like to test my REST API, which should save files. Is it possible or there is another way?

All I could find: https://github.com/weavejester/ring-mock/issues/11

The proposed solution is a little verbose. How do you test post file to your server?

Howdy!

what’s the correct way to get an integer back from (/ 1/16 1/16)?

wrap with (int ...)

or (.intValue ...)

thanks!

does anyone have any good resources or examples of implementing a 'fake' mutable collection in clojure using store-passing-style?

any Cryogen gurus here?

A tricky leinigen question.

I want to add a plugin to perform some hooks at lein startup; to enable pretty exception reporting via https://github.com/AvisoNovate/pretty.

I've created a namespace, pretty.plugin, and a function, hooks.

The trick is that pretty has an optional dependency on clojure.tools.logging; I want to know in my hook function

if that's available, and do some additional setup when present.

here's the hack I've been working on:

The first part of the hook seems to work fine (i.e., via lein repl, and adding pretty to my profiles.clj.

(io.aviso/logging has transitive dependencies on clojure.tools.logging and logback).

Also, it seems like I'm getting different behaviors depending on whether I'm using lein repl (without a project) and lein repl in a project that has its own dependency on io.aviso/pretty.

BTW, I'm ok if the approach requires I create a separate artifact to do the setup, if that's what it takes to make it work in a project with an existing dependency on io.aviso/pretty. For the moment, I'm hoping I can do it all in a single artifact.

@hiship: Did you take a look to https://github.com/technomancy/leiningen/blob/master/doc/PLUGINS.md#evaluating-in-project-context?

There it shows an example where it injects a dependency into the project via merge-profiles

About detecting if a dependency is available, tools.logging uses a similar approach (try & catch), but using (Class/forName ...). See https://github.com/clojure/tools.logging/blob/master/src/main/clojure/clojure/tools/logging/impl.clj#L52

I'm also interested to hear opinions on the different approaches

Right now I'm using hook, and that is called with no arguments; no project available. So far I haven't found a project var global I can use, and I bet I won't.

I could be sneaky and use some middleware.

Ok, sorry, I missed that part