Can anyone point me to good examples of Clojure database/message queue libraries that present a lazy seq, and don’t load a full result set into memory all at once?

@danielcompton: monger supports pagination http://clojuremongodb.info/articles/querying.html , not sure if thats what you’re looking for.

@pastafari: thanks, that’s what I was looking for

@danielcompton: :thumbsup:

has anyone by any change made a ring middleware for http://prerender.io?

@borkdude: still looking for prerendering stuff? i made/tried https://github.com/sander/precook which may be a useful minimal example. switched to a pure cljs setup though, since i didn’t feel comfortable with starting, stopping and controlling nashorn repls in java

@sander: thanks, I'll check it out. my hope is now on http://prerender.io, it seems a good solution for SEO

@sander: what do you mean by pure cljs?

@borkdude: using a node server instead of a jvm one

lol, I just had this problem with clojure.java.jdbc and reading edn: (read-string (str {(keyword "count(some-col)") 1}))

Is there such a thing as a “SoftAtom”? As in an atom that will clear values once there is memory pressure?

I’ve been looking into SoftReference … which seem to do what I want

Meh, could also try to use the WeakHashMap I guess

@joelkuiper: generally jvm folk don't recommend them these days. They make the GC a bunch slower afaik

@tcrayford: hmm yeah I’ve been reading that too. The Guava Cache seems to be liked though

yeah, but think many folk who use that just lean on the size-based lru caching as opposed to anything fancier

right, that could work too I guess. The thing is that I have to keep a bunch of results (from an HTTP call) accessible for “a while” but not indefinitely (since people will only access them a couple of times, and they become invalid after a while anyway)

Right now the responses are held in an atom, but I fear that holding on to them forever might use too much memory (i.e. the JVM will run out)

so a cache seems perfect, but there are so many to choose from!

core.cache may frustrate you at first, but know that you can stack them: https://github.com/clojure/core.cache/wiki/Composing

@ghadi: thanks, I’ve been looking at core.cache, I guess I could just use an atom with one of the caches, the hit/miss semantic seems a bit out of place for this use case … but oh well 😛

yup. caches are inherently stateful so putting one behind an atom is the way to go

@micha thanks for the tip on http://github.com/strongh/crache. works GREAT

plug it in, switch it on, forget about it. now we have super fast paginated result-sets for our expensive datalog queries, using the database value as one of the cache keys!

robert-stuttaford: 🍻

wait, what? trying to wrap my head around what the looks like.

s/the/that/

dya mean what i said, curtosis?

yes, about the database value as cache-key

you understand how datomic dbs are immutable values?

yep

so.. instead of using a connection directly, you just use crache?

I may be insufficiently caffeinated — and also new to what production datomic code looks like — to get a solid mental picture.

we memoize the function that produces the full unpaginated result set. this function takes filters and the db against which to query. core.memoize caches its return value in redis so that when we call again (to bite off the next page’s worth of data) it uses the cached value. we can do this because a datomic database value can not change

as soon as you call with a new datomic db value or any alterations to the filter args, the query runs again. but if all you do is sort the set differently or view a different page, you use the cached list

makes perfect sense, thanks

we just need to find the ideal TTL for the cache and make sure the webservers have plenty of ram

that seems like it should be pretty easy to instrument and tune based on observed miss/eviction rates.

yup

i was hoping for a memcached backend for core.memoize as we already have memcached for datomic, but redis is perfectly fine too. adding some elasticache redis nodes isn’t that much more complex

Hey all, how would you implement migration/patch code in a clojure server app? These are once-off jobs that (most of the time) patches up data, does once-off setup, convert sets of files & upload to s3 - that kind of thing

@cmdrdats: we have a standalone "migration" app that we run with a list of ticket numbers (from Unfuddle, our bug tracker) and each one represents a function that performs the specified task from that ticket.

We create a ticket for the actual patch / migration, add the function that corresponds to that, then create a ticket in either our pre-production-build or post-production-build milestones for the actual application of that patch, for tracking purposes.

@cmdrdats: I just write one-off functions. Whenever possible, make them idempotent, and then just run on server startup

If they can’t be idempotent, I run them by hand from the repl

I’ve tried using code to track when migrations have run in production, and it’s often not worth the hassle

i.e. store “migration #32 ran” in the DB

but that code is brittle, so you have to debug it

and sometimes you want to re-run migration #32

@cmdrdats: Just a thought: you could probably write boot tasks for that, assuming you'd want to use boot.

Anyone run into the problem of needing two libs occupying the same namespace? Is there a way to re-ns something in lein deps?

@canweriotnow: Which two libs? I've had a low-grade worry about how many people are using single-segment project names, but never seen it in practice before.

@luke actually, two versions of the same lib… instaparse-cljs retains the unstraps ns for clj and cljs, but the clj portion is broken under 1.7; upstream instaparse 1.4.0 has this fixed but downstream is 1.3.5

Crazy edge case, I know. Probably no good solution but a PR

You could put the CLJS side in a separate lein profile that you activate only when doing cljs compilation

depending on your project setup

super hacky but might get you by until someone fixes the underlying issue. Or fix it yourself, like you said.

Hmm, worth a shot. Thanks!

fwiw, I get why I can't do that (now), just hoping there might be another solution along these lines I haven't thought of.

@voxdolo: why can't the above macro just be a normal fn?

pbostrom: the number of calls needs to be persistent, thus the atom. for it to be a regular function, the atom would need to be external to the function. in the above scenario, the idea was that each call to mock-once would essentially yield a fn with what was effectively it's own atom

I guess this opens the door for clojure on AWS Lambda https://aws.amazon.com/blogs/aws/aws-lambda-update-run-java-code-in-response-to-events/?sc_campaign=launch&sc_category=lambda2&sc_channel=SM&sc_content=java&sc_detail=std&sc_medium=aws&sc_publisher=tw_go&adbsc=social_launches_20150615_47567446&adbid=610521356594089984&adbpl=tw&adbpr=66780587

how do i convert a string seq (generated by enlive) nicely to an input source for http://clojure.java.io/copy (used in clojure.java.shell/sh)?

pbostrom: though it's entirely possible I'm just misunderstanding something and needlessly complicating things

@voxdolo currently it wouldn't work because "(atom 0)" is getting substituted multiple times without referencing a common atom throughout the lifetime of the function.

well. bollox. 😄

sorry, I have to run; look at memoize: https://github.com/clojure/clojure/blob/41af6b24dd5be8bd62dc2b463bc53b55e18cd1e5/src/clj/clojure/core.clj#L6067

it might provide a pattern for what you are trying to do

thanks @pbostrom

clojure
(defn mock-once
  [the-fn first-value]
  (let [calls (atom 0)]
    (fn []
      (if (= 1 (swap! calls inc))
        first-value
        (the-fn)))))

(oops, github flavored markdown fail)

@aengelberg: that works precisely how I would expect it to. though, I'm not entirely sure why I guess I need to look into how the let's lexical scoping works there.

Thank you kindly

https://en.wikipedia.org/wiki/Closure_%28computer_programming%29

rephrased: can i easily transform a sequence of strings into a Reader or InputStream with their concatenated content?

(i guess the sequence is lazy but am not sure, it comes from enlive)

apply concat would get a lazy sequence of characters. How performant does it need to be?

alex_engelberg: seems good, thanks. was looking for the most clojurianistic / clean way, and the lazy apply concat seems like an improvement already to my reduce str

apply str is also an option; it returns a string but is more performant than reduce str.

@borkdude: did a quick test with clojure on lambda, but got FileNotFoundException: Could not locate clojure/core__init.class or clojure/core.clj on classpath

just used a plain lein uberjar

clojure/core__init.class is there alright

anyone have a print-method handy for java.time.*?

nm. toString is good enough for 8601. yay progress