@denik: Were you thinking of Slingshot?

(a gazillion messages before) he said it's not slingshot

hi everybody! i wanted to know if I forgot any important link in my post for clojure starters (https://medium.com/this-is-not-a-monad-tutorial/how-to-earn-your-clojure-white-belt-7e7db68a71e5). let me know if I should add anything please 😄

why the memory keep going up when run this query? The table of test is huge. Anyway to let clojure not consume so much memory? I am testing the network performance of db, so didn't count it inside db. (count (j/query db ["select * from test"]))

@wqhhust: http://stackoverflow.com/questions/19728538/clojure-java-jdbc-query-large-resultset-lazily

@beppu , thanks, good information

@nberger: You'll be happy to know that I just added ring-logger-timbre to my middleware stack, and it was painless and effective. It's successfully telling me exactly what the incoming requests are.

Curiously, this test code:

(let [credentials (select-keys user-values #{:username :password})
          request (-> (request :post "/api/login")
                    (content-type "application/json")
                    (body credentials))
          response (app request)]
      (is (= 200 (:status response))))

(Okay, I just realized that I was shadowing request (from ring.mock.request) with request (the let binding), but that's not actually a bug here -- just ugly and confusing.)

I believe I might have found it! When using ring.mock.request in this way, I need to explicitly encode the JSON body as a string, because ring.mock.request/body is a multimethod which handles a map argument by changing the content type to application/x-www-form-urlencoded. Whew!

I would argue that if the content-type is already application/json, ring.mock.request should assume that the body map is a JSON hash, but that's another story.

Indeed, I'm happy ring-logger-timbre is helping. And I agree, would be great if ring.mock.request was smarter there...

And that let me finally drill down to the last bug in my code, which was that I was invoking the raw user creation function generated by YeSQL, not my model-level user-creation function -- so the user I was creating in the test database had a plaintext password instead of a hashed one, which (for whatever reason) caused buddy-hashers/check to bust a gut.

I'll make a note to offer a PR about the ring.mock.request thing. JSON requests built from plain maps are pretty common.

Thanks for all your help! I'm sure I'll be back with more problems.

It seems https://github.com/xeqi/peridot already handles this a bit better. In the README I'm reading the following: > By default, when POSTing data, params will be encoded as application/x-www-form-urlencoded. If you want to use an alternative encoding, you can pass :content-type as an option, and use :body instead of :params

Ah, I've seen people mention it. Maybe I'll try that out instead.

That said, it can't hurt to file a PR on ring.mock.request anyway if it might help someone!

yep!

@jmglov nope :)

Hi. Does anyone can help with something that looks simple but I’m stuck

I have clojure.lang.LazySeq as a result of PGSQL query which contains ~ 35k rows. I’m trying to build atom based in-memory store since those data rarely change with simple

(defn update-cache [cache mac serial] (swap! cache conj {mac serial}))

(defn make-cache [cache records] "Fill in-memory atom cache with data from giver record" (let [recs (vec records)] (dosync (map #(update-cache cache (:mac %) (:serial %)) recs) )))

(def cache (atom {}))

And when I call (make-cache cache 35k-records) I’m getting out of memory …:(

first, calling vec on records makes it no longer lazy

That is my point

since otherwise map will not finish

and you're holding onto the head

map is used for returning a new seq

you want to do side-effects, then use doseq

I want to build my memor-based-cache for further queries

(doseq [rec records] (update-cache cache (:mac rec) (:serial rec)))

let me check

Yes - you are of course right

cl-repl.core=> (count @cache) 0 cl-repl.core=> (count r) 32116 cl-repl.core=> (make-cache cache r) nil cl-repl.core=> (count @cache) 32116 cl-repl.core=>

Thanks.

Now it works like a charm.

cl-repl.core=> (get @cache "647c344792ee" "UNKN") "uaap41940384"

In the book Clojure Applied, while talking about core.async channels and pipelines, the author presents the following snippet - http://media.pragprog.com/titles/vmclojeco/code/cljapplied/src/ch5/pipeline.clj

An intermediate channel is used to separate the filtering of interesting words and sentiment analysis of those words. The idea being we could have a separate state log-words which operates on the intermediate channel in parallel.

I can't figure out how both sentiment-analysis and logging stages will get the same word? Once one of those stages takes a word off the intermediate channel, isnt it lost for forever? How does another consumer see that word?

;---
; Excerpted from "Clojure Applied",
; published by The Pragmatic Bookshelf.
; Copyrights apply to this code. It may not be used to create training material, 
; courses, books, articles, and the like. Contact us if you are in doubt.
; We make no guarantees that this code is fit for any purpose. 
; Visit  for more book information.
;---
(ns ch5.pipeline
  (:require [clojure.core.async :as async]))

;
;; parse message into set of words
(def parse-words (map #(set (clojure.string/split % #"\s"))))

;; filter messages that contain a word of interest
(def interesting (filter #(contains? % "Clojure")))

;; detect sentiment based on different word lists
(defn match [search-words message-words]
  (count (clojure.set/intersection search-words message-words)))
(def happy (partial match #{"happy" "awesome" "rocks" "amazing"}))
(def sad (partial match #{"sad" "bug" "crash"}))
(def score (map #(hash-map :words %1
                           :happy (happy %1)
                           :sad (sad %1))))
;

;
(defn sentiment-stage
  [in out]
  (let [xf (comp parse-words interesting score)]
    (async/pipeline 4 out xf in)))
;

;
(defn interesting-stage
  [in intermediate]
  (let [xf (comp parse-words interesting)]
    (async/pipeline 4 intermediate xf in)))

(defn score-stage
  [intermediate out]
  (async/pipeline 1 out score intermediate))

(defn assemble-stages
  [in out]
  (let [intermediate (async/chan 100)]
    (interesting-stage in intermediate)
    (score-stage intermediate out)))
;

@amithgeorge: that's not actually done here but you would need to use a core.async mult that fans out to multiple output channels -this is covered more in ch 6 and there is an example there connect-and-tap. That could be applied to the intermediate input channel in the example above.

http://media.pragprog.com/titles/vmclojeco/code/cljapplied/src/ch6/async.clj

Hi @unbalancedparentheses . I thought it some nice highlights. Thanks

Clojure 1.8.0-alpha5 is out https://groups.google.com/d/msg/clojure/xZ2YpiKg2FU/VvnojP7rCgAJ

Guys - how to access atom that is in another namespace ? Lets say I have (require `[my-project.db :as db] and within db is (def cache (atom {}))

db/@cache doesn’t work

@db/cache - the same

@db/cache should work, maybe you haven't recompiled properly

also, it's an antipattern

something wrong with my compliation then

transducers gotcha: (comp .. (distinct)) not (comp ... distinct) /facepalm

@gtrak: Yeah, I think cat is the only transducer in core - everything else is some function that returns a transducer.

Speaking of transducers, has anyone had a need to use a custom reducing function instead of just conj?

I've done some work with interesting transducer stacks, including stateful transducers, but I still haven't seen a need for anything unusual for the reducing function.

@meow: re cat - yes, that's the only function that is a transducer rather than a creator of a transducer

building an output collection is certainly one of the most common reducing functions (which really I'd recommend into for that use case)

but any time you are "summarizing" a collection down to a value you will need something different

for example, + would be a "summing" reducing function

@alexmiller: So the code I've been working on is basically building output collections - so you recommend into - any particular reason?

that's what it does

and it's optimized for transients

with transients

ok, makes sense

(into [] xf input)

will be faster than the naive version of (transduce xf conj input)

plus you can easily change to a different output collection type

@alexmiller: thanks. had read only till the end of chap 5, for today

@alexmiller: looks like I need to revise my code, once again... will it never end... 😉

I've been using tranduce where I should use into

(it will never end btw :)

neverending is a good thing

so I've been writing variations on this theme:

([get-xf get-rf]
   (letfn [(process [w]
                    (lazy-seq
                      (when (seq w)
                        (let [word (transduce (get-xf) (get-rf) w)]
                          (cons word (process word))))))]
     (process jumpstart))))

get-xf has been a very useful way to plug in various transducer stacks, but I haven't had a use for get-rf beyond the default that I supply.

So I've been thinking about taking it out.

If I switch from transduce to into I'm wondering if I need any flexibility here or if (into [] (get-xf) w) is all I will ever need?

if you're always collecting then taking the initial output collection (here []) would give you an intermediate level of flexibility

although maybe you don't even need that

this process is recursive and axiomatic, meaning there is an intial collection and it gets fed back into the process

I'm not sure I can think of a need for a collection other than vector

but that's why I'm asking... 😉

and I just learned about transients - cool http://clojure.org/transients

so I can definitely see that if into is making use of transients then it should definitely be better than transduce in my use-case, which because of the recursive rewriting can easily produce collections with millions of items after just a few iterations.

Which I then want to feed into Alda: http://daveyarwood.github.io/alda/2015/09/05/alda-a-manifesto-and-gentle-introduction/

@alexmiller: and now I see what you mean by looking at the source for into:

([to xform from]
     (if (instance? clojure.lang.IEditableCollection to)
       (with-meta (persistent! (transduce xform conj! (transient to) from)) (meta to))
       (transduce xform conj to from)))))

yep :)

so vector, hash-map, and hash-set

hmm, I'm a bit confused by this:

(type {})
=> clojure.lang.PersistentArrayMap
(type (hash-map))
=> clojure.lang.PersistentArrayMap
(type (array-map))
=> clojure.lang.PersistentArrayMap

perf

actually I can't see where that happens either

@meow: My understanding is that the underlying type and implementation of hash map is to use PersistentArrayMap for sufficiently small maps, and convert to PersistentHashMap after a sufficient size is reached. Something like 8 or so pairs

E.g:

user=> (type {})
#<java.lang.Class@6611df92 class clojure.lang.PersistentArrayMap>
user=> (type {:a 3 :b 3 :c 3 :d 3 :e 4 :f 3 :g 3 :h 3 :i 3 :j 3})
#<java.lang.Class@21507a04 class clojure.lang.PersistentHashMap>

then somewhere we should see 'new PersistentArrayMap', but cursory investigation shows that's not called anywhere from PHM.java

http://clojars.org seems to be struggling today… lots of 500 server error responses or very very slow to load / respond…

anyone successfully using a local repository with a "file:" url for local jars? I had one working at one point and for some reason it just stopped.

(probably at least slightly triggered by clojars throwing 500s....?)

the symptom I'm getting is IllegalArgumentException: number of transferred bytes cannot be negative from aether, which is unhelpful.

I’m not sure why Leiningen would even both to contact http://clojars.org when we have the JAR in the local Maven cache 😞

@seancorfield: that too, yes.

if you have snapshots, it will check for newer ones in remote repos

for non-snapshots, not sure why

lein -o work?

that should be local only I think

"The repository system is offline but the artifact [my-artifact] is not available in the local repository."

which actually seems extra weird, because that shouldn't be the case.

by which I mean, there is the local repository, which is one thing, but I also have a "local" (i.e., with a "file:" url) repo in my :repositories for the project.

well the latter is probably treated as a "remote" repo regardless

it's possible that you got a bad jar downloaded in your local cache - I have seen maven download an http error form and save it as a jar before

you might want to do some minimal investigation whether the jars in the .m2 are valid

spot checks of jars indicate cached .m2 versions are identical to my "local" repo versions

but I'm not entirely sure why it's not looking in the "local" repo and only clojars. Or, for that matter, even if it is looking at the "local" repo.

yeah, not sure I can help more at this point

sometimes I have found it useful to do "lein pom" then use Maven to see its errors

"mvn dependency:tree" should be sufficient

and you can get more error reporting with "mvn -X dependency:tree"

I'll give that a shot

two warnings; neither of my libraries have associated poms

pkobrien and gtrak - from your earlier question, (hashmap) will return literal {}, which (since 1.7 I think) will be a PAM, not a PHM. That is probably more accident than intention.

but mvn seems to be able to find the repo just fine

pkobrien: gtrak: actually that didn't change in 1.7, would have been same prior

hmmmm... this could be a bug in aether. lein is using sonatype-aether, which is officially deprecated.

and there's a known bug in sonatype-aether that would cause exactly these symptoms on an existence check.

here's a discussion about it: https://bugzilla.redhat.com/show_bug.cgi?id=1118341

(short version: total transferred bytes is initialized to -1; existence check branch doesn't/didn't reset that to zero)

anyway... I think that's likely the culprit, but the linkage between lein and aether is far too obscure for me to know where to start to address it.

interesting

I think lein uses pomegranate which uses the aether stuff

so pomegranate is likely the place

https://github.com/cemerick/pomegranate/blob/master/pom.xml#L41

given that sonatype-aether was deprecated 4 years ago, that sems like a problem. 😉

though to be fair, pomegranate appears more or less "done" and not actively developed.

i was considering using automat to describe the state transitions that can happen to one of our domain entities here, but i’m struggling to see how you can ‘drop into the middle’ of the FSM.

eclipse-aether was supposedly massively rewritten, but I don't know if it's api-compatible enough to just swap into pomegranate

i.e. if i have a state machine that goes something like [1 (a/* [2 3]) 4]

i’d like to be able to use it such that (f :current-state-of-the-thing :proposed-state-of-the-thing) -> truthy if it could advance the FSM

as near as i can tell, the way you would go about doing that is by constructing an automat state map and passing that in to advance

but it uses ‘internal’ state indexes rather than the edge names

(i’m probably not even using the right terminology)

any thoughts?

i suppose i could cache off somehow the advances of any given domain entity through the fsm but that seems like much more trouble than its worth given that i can express this logic pretty simply otherwise

aha... already an open PR on pomegranate. https://github.com/cemerick/pomegranate/pull/70

Documented for posterity on pomegranate: https://github.com/cemerick/pomegranate/issues/71

with workaround!

@timvisher: If your finite state machine is implemented using state transition tables, it's very easy to make a function that does: (f :current-state-of-the-thing :proposed-state-of-the-thing) -> truthy if it could advance the FSM

I think I just broke http://clojars.org - I was just trying to update by PGP key, and now the entire site errors out 😞

(def fsm {:A {:a :B :b :C} :B {:a :A :b :C} :C {:a :C :b :C}})
(defn new-state [machine state inputs] (reduce (fn [s i] (-> machine s i)) state inputs))
(defn possible? [fsm a b] (-> fsm a vals set (contains? b)))

@timvisher: The possible?function is what tells you if the fsm allows transitions from state a to state b.

Examples:

(new-state fsm :A [:a])       ; => :B
(new-state fsm :A [:a :a])    ; => :A
(new-state fsm :A [:a :a :b]) ; => :C
(possible? fsm :A :A)         ; => false, it is not possible to move from :A to :A
(possible? fsm :A :B)         ; => true
(possible? fsm :A :C)         ; => true

Does anybody know if it’s possible to specify individual files with cljsbuild to compile, instead of all files in a directory?

You should ask in #C03S1L9DN.

beppu: I already did that, but nobody knew over there. I thought someone here might know.

oh sorry.

@tcrawley: https://clojurians.slack.com/archives/clojure/p1442000895000712

^-- he's the guy that's keeping http://clojars.org up and running these days.

At the last Clojure/West, he was looking for people to help him with existing issues. Everything marked ready is stuff that he'd like help with: https://github.com/ato/clojars-web/issues?utf8=%E2%9C%93&q=is%3Aissue+is%3Aopen+label%3Aready

oh thanks

I put out the bat signal. Hopefully he responds. 😛

I saw on reddit that clojars needs help, but I've only been a consumer of clojure libraries up until now

It's not a bad way for an intermediate-to-advanced programmer who is new to Clojure to get their feet wet with programming in Clojure.

It's a nice medium-sized project. It's not too complex. A little crufty here and there, but not terrible.

awesome, looks like my weekend just filled up

I fixed one issue for implementing pagination on the search page. I told him I'd send him a pull request after Clojure/West and I did a few days after, but then I got busy at work and forgot about following up on it. Some months after Clojure/West, I finally did get around to finishing that feature after he pinged me via email.

I'm sure he'll be happy to get help. It's been a thankless job.

@bsima: I can confirm that @tcrawley would love some help, and that there’s plenty to do. It’s also a great way to help the community out, since it’s such a key piece of infrastructure.

okay, I have clojars-web running locally, so the hard part is over and fun part of fixing bugs begins

@beppu: true. i was hoping that automat would implement that sort of behavior for me

If you were having trouble with clojars, it should be fixed now. the server ran out of disk space

What did you delete to free up space?

a 9.7gb error.log from nginx

space is getting tight on that box, I need to move it to a bigger one

sorry for the delay, I was far from a computer for a few hours :)

Thank you for keeping things running.

@timvisher: I looked at automat just now, and I was hoping I could see the inside of a compiled fsm, but it's a mysteriously opaque value.