Fork me on GitHub
#clojure
<
2015-06-15
>
danielcompton02:06:33

Can anyone point me to good examples of Clojure database/message queue libraries that present a lazy seq, and don’t load a full result set into memory all at once?

pastafari04:06:58

@danielcompton: monger supports pagination http://clojuremongodb.info/articles/querying.html , not sure if thats what you’re looking for.

danielcompton04:06:24

@pastafari: thanks, that’s what I was looking for

borkdude10:06:49

has anyone by any change made a ring middleware for http://prerender.io?

sander11:06:21

@borkdude: still looking for prerendering stuff? i made/tried https://github.com/sander/precook which may be a useful minimal example. switched to a pure cljs setup though, since i didn’t feel comfortable with starting, stopping and controlling nashorn repls in java

borkdude11:06:17

@sander: thanks, I'll check it out. my hope is now on http://prerender.io, it seems a good solution for SEO

borkdude11:06:33

@sander: what do you mean by pure cljs?

sander11:06:58

@borkdude: using a node server instead of a jvm one

borkdude13:06:16

lol, I just had this problem with clojure.java.jdbc and reading edn: (read-string (str {(keyword "count(some-col)") 1}))

joelkuiper13:06:35

Is there such a thing as a “SoftAtom”? As in an atom that will clear values once there is memory pressure?

joelkuiper13:06:28

I’ve been looking into SoftReference … which seem to do what I want simple_smile

joelkuiper13:06:49

Meh, could also try to use the WeakHashMap I guess

tcrayford13:06:13

@joelkuiper: generally jvm folk don't recommend them these days. They make the GC a bunch slower afaik

joelkuiper13:06:07

@tcrayford: hmm yeah I’ve been reading that too. The Guava Cache seems to be liked though

tcrayford14:06:40

yeah, but think many folk who use that just lean on the size-based lru caching as opposed to anything fancier

joelkuiper14:06:29

right, that could work too I guess. The thing is that I have to keep a bunch of results (from an HTTP call) accessible for “a while” but not indefinitely (since people will only access them a couple of times, and they become invalid after a while anyway)

joelkuiper14:06:18

Right now the responses are held in an atom, but I fear that holding on to them forever might use too much memory (i.e. the JVM will run out)

joelkuiper14:06:21

so a cache seems perfect, but there are so many to choose from!

ghadi14:06:09

core.cache may frustrate you at first, but know that you can stack them: https://github.com/clojure/core.cache/wiki/Composing

joelkuiper14:06:29

@ghadi: thanks, I’ve been looking at core.cache, I guess I could just use an atom with one of the caches, the hit/miss semantic seems a bit out of place for this use case … but oh well 😛

ghadi14:06:37

yup. caches are inherently stateful so putting one behind an atom is the way to go

robert-stuttaford14:06:07

plug it in, switch it on, forget about it. now we have super fast paginated result-sets for our expensive datalog queries, using the database value as one of the cache keys!

micha14:06:58

robert-stuttaford: 🍻

curtosis14:06:37

wait, what? trying to wrap my head around what the looks like.

curtosis14:06:43

s/the/that/

robert-stuttaford14:06:12

dya mean what i said, curtosis?

curtosis14:06:44

yes, about the database value as cache-key

robert-stuttaford14:06:40

you understand how datomic dbs are immutable values?

curtosis14:06:43

so.. instead of using a connection directly, you just use crache?

curtosis14:06:46

I may be insufficiently caffeinated — and also new to what production datomic code looks like — to get a solid mental picture.

robert-stuttaford14:06:21

we memoize the function that produces the full unpaginated result set. this function takes filters and the db against which to query. core.memoize caches its return value in redis so that when we call again (to bite off the next page’s worth of data) it uses the cached value. we can do this because a datomic database value can not change

robert-stuttaford14:06:09

as soon as you call with a new datomic db value or any alterations to the filter args, the query runs again. but if all you do is sort the set differently or view a different page, you use the cached list

curtosis14:06:23

makes perfect sense, thanks

robert-stuttaford14:06:42

we just need to find the ideal TTL for the cache and make sure the webservers have plenty of ram simple_smile

curtosis14:06:43

that seems like it should be pretty easy to instrument and tune based on observed miss/eviction rates.

robert-stuttaford14:06:42

i was hoping for a memcached backend for core.memoize as we already have memcached for datomic, but redis is perfectly fine too. adding some elasticache redis nodes isn’t that much more complex

cmdrdats17:06:33

Hey all, how would you implement migration/patch code in a clojure server app? These are once-off jobs that (most of the time) patches up data, does once-off setup, convert sets of files & upload to s3 - that kind of thing

seancorfield18:06:46

@cmdrdats: we have a standalone "migration" app that we run with a list of ticket numbers (from Unfuddle, our bug tracker) and each one represents a function that performs the specified task from that ticket.

seancorfield18:06:50

We create a ticket for the actual patch / migration, add the function that corresponds to that, then create a ticket in either our pre-production-build or post-production-build milestones for the actual application of that patch, for tracking purposes.

arohner19:06:54

@cmdrdats: I just write one-off functions. Whenever possible, make them idempotent, and then just run on server startup

arohner19:06:05

If they can’t be idempotent, I run them by hand from the repl

arohner19:06:46

I’ve tried using code to track when migrations have run in production, and it’s often not worth the hassle

arohner19:06:54

i.e. store “migration #32 ran” in the DB

arohner19:06:06

but that code is brittle, so you have to debug it

arohner19:06:12

and sometimes you want to re-run migration #32

meow19:06:18

@cmdrdats: Just a thought: you could probably write boot tasks for that, assuming you'd want to use boot.

canweriotnow20:06:02

Anyone run into the problem of needing two libs occupying the same namespace? Is there a way to re-ns something in lein deps?

luke20:06:34

@canweriotnow: Which two libs? I've had a low-grade worry about how many people are using single-segment project names, but never seen it in practice before.

canweriotnow20:06:10

@luke actually, two versions of the same lib… instaparse-cljs retains the unstraps ns for clj and cljs, but the clj portion is broken under 1.7; upstream instaparse 1.4.0 has this fixed but downstream is 1.3.5

canweriotnow20:06:30

Crazy edge case, I know. Probably no good solution but a PR simple_smile

luke20:06:38

You could put the CLJS side in a separate lein profile that you activate only when doing cljs compilation

luke20:06:44

depending on your project setup

luke20:06:59

super hacky but might get you by until someone fixes the underlying issue. Or fix it yourself, like you said.

canweriotnow20:06:33

Hmm, worth a shot. Thanks!

voxdolo20:06:20

fwiw, I get why I can't do that (now), just hoping there might be another solution along these lines I haven't thought of.

pbostrom21:06:07

@voxdolo: why can't the above macro just be a normal fn?

voxdolo21:06:12

pbostrom: the number of calls needs to be persistent, thus the atom. for it to be a regular function, the atom would need to be external to the function. in the above scenario, the idea was that each call to mock-once would essentially yield a fn with what was effectively it's own atom

sander21:06:56

how do i convert a string seq (generated by enlive) nicely to an input source for http://clojure.java.io/copy (used in clojure.java.shell/sh)?

voxdolo21:06:18

pbostrom: though it's entirely possible I'm just misunderstanding something and needlessly complicating things

aengelberg21:06:25

@voxdolo currently it wouldn't work because "(atom 0)" is getting substituted multiple times without referencing a common atom throughout the lifetime of the function.

voxdolo21:06:40

well. bollox. 😄

pbostrom21:06:13

it might provide a pattern for what you are trying to do

aengelberg21:06:29

clojure
(defn mock-once
  [the-fn first-value]
  (let [calls (atom 0)]
    (fn []
      (if (= 1 (swap! calls inc))
        first-value
        (the-fn)))))

aengelberg21:06:42

(oops, github flavored markdown fail)

voxdolo21:06:16

@aengelberg: that works precisely how I would expect it to. though, I'm not entirely sure why simple_smile I guess I need to look into how the let's lexical scoping works there.

voxdolo21:06:50

Thank you kindly simple_smile

sander21:06:39

rephrased: can i easily transform a sequence of strings into a Reader or InputStream with their concatenated content?

sander21:06:28

(i guess the sequence is lazy but am not sure, it comes from enlive)

aengelberg21:06:14

apply concat would get a lazy sequence of characters. How performant does it need to be?

sander21:06:57

alex_engelberg: seems good, thanks. was looking for the most clojurianistic / clean way, and the lazy apply concat seems like an improvement already to my reduce str

aengelberg21:06:28

apply str is also an option; it returns a string but is more performant than reduce str.

ulsa22:06:02

@borkdude: did a quick test with clojure on lambda, but got FileNotFoundException: Could not locate clojure/core__init.class or clojure/core.clj on classpath

ulsa22:06:24

just used a plain lein uberjar

ulsa22:06:05

clojure/core__init.class is there alright

ghadi22:06:29

anyone have a print-method handy for java.time.*?

ghadi22:06:51

nm. toString is good enough for 8601. yay progress