Fork me on GitHub
#clojure
<
2017-10-10
>
thedavidmeister06:10:39

anyone know a good way to get a file system based memoize instead of using memory?

andrea.crotti07:10:14

@thedavidmeister mm it will eventually load the data in memory when you have to use it anyway right?

andrea.crotti07:10:57

what's the use case?

thedavidmeister07:10:28

@andrea.crotti i have 100's of network calls that take 20-60s each to complete and they are always the same (historical event data)

andrea.crotti07:10:18

well you can use any DB for something like that

andrea.crotti07:10:30

Redis even handles the TTL for you

thedavidmeister07:10:40

yes, i can write a caching layer

thedavidmeister07:10:08

i was hoping to simply memoize the fns

thedavidmeister07:10:08

i don't want to add a dependency on a specific tech like a db or redis

andrea.crotti07:10:19

well you can just do it yourself as well

andrea.crotti07:10:33

write to some EDN files

andrea.crotti07:10:39

and read from them

thedavidmeister07:10:56

current plan is to copy and paste the clojure core memoize function and swap out the lookup for a file system based logic

thedavidmeister07:10:05

it just seemed like something that might already exist

andrea.crotti07:10:20

it won't save you any memory though, it will only save the computation time, unless you have a different file per input

thedavidmeister07:10:38

yes, different file per fn/args combination

andrea.crotti07:10:56

some answers there

andrea.crotti07:10:26

but depending on what you're doing even sqlite could be better than doing it yourself

andrea.crotti07:10:34

and that doesn't require any other server running

thedavidmeister07:10:58

sqlite seems like a lot of work

thedavidmeister07:10:06

the event data i'm getting has no schema

thedavidmeister07:10:36

so i either have to hammer a schema into place to represent the specific data coming back from the query

thedavidmeister07:10:53

or put k/v into the db, which doesn't seem better/easier than just dumping to files

thedavidmeister07:10:27

this stack overflow answer is exactly what i was thinking

thedavidmeister07:10:51

so the answer is "there's no existing libraries, so hand roll something simple"

rauh07:10:27

@thedavidmeister I wrote that answer 🙂. You could also use: https://github.com/Factual/clj-leveldb if you don't want to deal with multiple files. I have no experience with leveldb though.

thedavidmeister07:10:10

mmmk, i'll start with the simple approach and move to a db if i find a reason to

rauh07:10:37

LevelDB is no "real" db, there is no server. It's just a single file.

danielcompton07:10:39

@thedavidmeister I think you probably want to use a cache instead of memoising

danielcompton07:10:16

a cache gives you more control over where it spills out to

thedavidmeister07:10:19

but a cache also relies on me inserting cache logic

thedavidmeister07:10:36

which will end up being the same as memoize behaviour

danielcompton07:10:30

sure, they're basically the same thing here?

danielcompton07:10:09

you can write a PluggableMemoization if you prefer, see https://github.com/clojure/core.memoize

thedavidmeister08:10:53

it all just seems like overkill... what's the downside of tweaking memoize to use a folder of files instead of an atom?

andrea.crotti08:10:10

By tweaking you mean rewriting it I guess?

andrea.crotti08:10:54

It just means you are reinventing the wheel imho

dominicm08:10:14

Rather than changing memoize, you could have the function write to the fs and return a path to it

thedavidmeister08:10:17

writing a custom plugin for a library?

thedavidmeister08:10:34

those are spokes, not a wheel 😛

rauh08:10:23

@thedavidmeister There is nothing wrong with just using files, but YOU will have to do the coordination if you write/read from multiple threads. Or you'll get garbage. LevelDB library would allow you to just get/write as you like. It's thread safe (as long as you don't use Iterator).

thedavidmeister08:10:32

i don't think i'll run into that issue with what i'm doing

thedavidmeister08:10:40

but that's a fair point in general

thedavidmeister08:10:07

i'm strictly serially hitting api endpoints, and each time i get about 3 million rows back

thedavidmeister08:10:14

and just doing this in a repl

thedavidmeister08:10:03

i don't think there's any threading in there, but i could be wrong

rauh08:10:44

Well I hope you're right, b/c initially yous said it'd take 20-60sec for you to answer the request.

thedavidmeister08:10:11

i can wait that long once or twice

thedavidmeister08:10:14

just not over and over

rauh08:10:01

@thedavidmeister My point is: What if a second request comes in, while you generated your cache from the first request...

thedavidmeister08:10:25

why is that a problem?

thedavidmeister08:10:32

each request has different args and so a different file on disk

rauh08:10:35

@thedavidmeister So why cache it then? If it's never going to be accessed again since they're all different?

thedavidmeister08:10:59

the next time i run the function to run a report

thedavidmeister08:10:15

it's not always different

thedavidmeister08:10:31

it's just not the same for a single set of calls

thedavidmeister08:10:35

the only way to have the same args at the same time is for me to run two repls side by side

thedavidmeister08:10:56

but the same args will definitely be called sequentially, when i call the fn a second or third time in the repl

thedavidmeister08:10:13

anywho, i've already rewritten the memoize to do what i want for now

thedavidmeister08:10:23

so i'm getting some dinner, thanks for all the suggestions everyone 🙂

qqq08:10:49

(def _+ 20) <-- this appears to work, but is it guarnteed by any standard?

bronsa08:10:11

both _ and + are valid symbol constituents

qqq08:10:03

@rauh: the phrasing of Symbols begin with a non-numeric character and can contain alphanumeric characters and *, +, !, -, _, ', and ? (other characters may be allowed eventually). threw me off

qqq08:10:30

by special casing the "_", one could interpret "non-numeric" to mean "non-numeric alpha-numeric" char (or atleast that was my first interpretation)

Bravi10:10:06

is there an equivalent function to flip (in other programming languages) in clojure?

Bravi10:10:09

to flip arguments

reborg10:10:15

No, you'll have to roll your own. For example (defn flip [f] (fn [& args] (apply f (reverse args)))) and then ((flip reduce) (range 10) +) @bravilogy

Bravi10:10:32

thank you

Bravi10:10:14

basically I have this function

(defn game-progress [score]
  (cond
    (zero? (mod score 4)) add-more-colors
    (zero? (mod score 5)) decrease-time-left
    :else identity))
and thought I would refactor that repetitive bit somehow and perhaps use condp instead

thheller10:10:55

I vaguely remember a library that can read EDN data and write it out again while preserving all comments and formatting. Can someone remember the name?

bronsa10:10:00

i believe rewrite-clj has utilities for that sort of things

thheller11:10:08

rewrite-clj was the thing I was looking for. thx.

cgrand11:10:14

Don’t know the reference but that’s clearly the spirit

dominicm11:10:27

I didn't think words would quite express enough no in this case, I just pictured you screaming it. 😂

Ertugrul Cetin11:10:57

Hey guys, new Clojurecademy course uploaded: Code Katas - https://clojurecademy.com/courses/17592186068882/learn/overview

viesti11:10:29

hum, is there a brew formula available for the clj script?

alex-dixon13:10:19

I can’t figure out how to implement a custom pretty print method without mutating the default behavior. Is there an idiomatic way to use a custom pretty print method for one call to pprint?

alex-dixon13:10:28

(def my-atom (atom nil))
(defn pprint-dispatch []
  (do
    (defmethod clojure.pprint/simple-dispatch clojure.lang.IDeref [o]
      (print o))
    clojure.pprint/simple-dispatch))
(clojure.pprint/write my-atom :dispatch (pprint-dispatch))
(clojure.pprint/pprint my-atom)

alex-dixon13:10:24

clojure.pprint/pprint my-atom)
#<Atom@58a7831e: nil> ;; this is the default I'm trying to overwrite
=> nil
(clojure.pprint/write my-atom :dispatch (pprint-dispatch))
#object[clojure.lang.Atom 0x58a7831e {:status :ready, :val nil}]=> nil ;; great, this is the way I want IDerefs to print
(clojure.pprint/pprint my-atom)
#object[clojure.lang.Atom 0x58a7831e {:status :ready, :val nil}] ;; oops -- all subsequent calls to pprint will look this way?
=> nil

alex-dixon13:10:16

@U3DAE8HMG Thanks. I see these use code-dispatch instead of simple-dispatch, but I think I face the same issue. I want to alter the way IDeref is pretty printed and leave everything else the same 😕

alex-dixon14:10:13

^ might also be a question about extending multimethods since that’s how clojure.pprint/simple-dispatch is implemented

chrisblom14:10:27

does anyone know how to disable clojure assertions in AOT compile namespaces?

chrisblom14:10:56

(set! *assert* false) is throwing an exception in that case

chrisblom14:10:38

and alter-var-root also fails

Jim Rootham14:10:36

I would like to have separate names for test and production uberjars compiled from the same project.clj file. Is that possible? Can you put an uberjar-name key is a profile and make uberjars with 2 different profiles?

tomaas15:10:45

hi, can I parse a date string with this pattern Oct 09, 2017 using clj-time?

dpsutton15:10:44

in clj-time.format there's a function called show-formatters that will show you all of the built in parse strings and what they can do

tomaas15:10:33

i've looked at those, and there's no such format

ccann15:10:12

@tomaas you’re going to want to use a custom formatter e.g. (f/formatter "MMM dd, YYYY") < might work

ccann15:10:44

(f/parse (f/formatter "MMM dd, YYYY") "Oct 09, 2017")

phreed15:10:58

I have data from a lisp (common lisp maybe) and I would like to read it in clojure. Any advice? I found this... https://github.com/clojure/clojure/blob/master/src/jvm/clojure/lang/LispReader.java with examples of its use at the end of the file.

ccann15:10:05

are you trying to read common lisp code into clojure as data?

ccann15:10:29

LispReader is a clojure reader, similar to tools.reader https://github.com/clojure/tools.reader#differences-from-lispreaderjava

ccann15:10:44

it doesn’t read common lisp

madstap16:10:05

What does the data look like?

tomaas15:10:36

Thus, "MM" might output "03" whereas "MMM" might output "Mar" (the short form of March) ....

tomaas15:10:40

didn't read until this point

tomaas15:10:58

thanks, a lot @ccann. 3 MMMs work

bfabry17:10:35

I often find myself looking for "something like pipeline-async, but that doesn't care about ordering, but does still deal with signaling 'all of x are finished'". am I silly? Does that already exist?

bfabry17:10:00

like, I'm writing this entirely too much

(async/pipeline-async
              500 ;; some number bigger than our number of tasks
              tasks-finished-for-account

bja17:10:41

You can do that with climate corp's claypoole library

bja17:10:33

from the readme:

(require '[com.climate.claypoole :as cp])
;; We'll use the with-shutdown! form to guarantee that pools are cleaned up.
(cp/with-shutdown! [net-pool (cp/threadpool 100)
                    cpu-pool (cp/threadpool (cp/ncpus))]
  ;; Unordered pmap doesn't return output in the same order as the input(!),
  ;; but that means we can start using service2 as soon as possible.
  (def service1-resps (cp/upmap net-pool service1-request myinputs))
  (def service2-resps (cp/upmap net-pool service2-request service1-resps))
  (def results (cp/upmap cpu-pool handle-response service2-resps))
  ;; ...eventually...
  ;; Make sure sure the computation is complete before we shutdown the pools.
  (doall results))

bfabry17:10:45

certainly seems about it. trying to avoid introducing new libs if possible though

mpenet18:10:31

You can build this on top of async/merge if your ops all return chans/promise-chans

bfabry19:10:37

yeah I generally build it with async/merge

bfabry19:10:43

(starts reading promise-chan docs)

tbaldridge18:10:36

@bfabry nah, it doesn't exist, mostly because it's fairly simple to write.

bfabry19:10:11

yup, that's what I ended up writing (ish). makes sense, ta

bfabry19:10:55

I think you miss that you need to create a channel for each thread though. otherwise you may close! out-c before one of the go-loops has finished its work

lxsameer19:10:28

is it possible to extend hashmaps and arraymaps without repeating the same code ?

lxsameer19:10:41

hmmm so it's harder than i expected.

bfabry19:10:09

yes. in practice though I've found it's very rarely necessary

bfabry19:10:32

what did you mean by "extend" though btw? because you could for instance extend them with a protocol with very little code

lxsameer19:10:10

I have a protocol and I want to extends them with that protocol. but without repeating my self

lxsameer19:10:52

I guess the simpler solution would be just to repeat myself.

noisesmith19:10:49

don’t extend PersistentHashMap and PersistentArrayMap - those are implementation details, extend java.util.Map or clojure.lang.IPersistentMap, which cover both hash-map and array-map

lewix20:10:06

I miss clojure

lewix20:10:26

too bad there are not many opportunities in my area…

lewix20:10:54

I never completed learning it fully like I wanted to but it taught me a lot

lewix20:10:22

Happy to see that the community is still thriving 🙂

dominicm20:10:27

@lewix there's always remote work 🙂 Keep an eye open

lxsameer20:10:46

@noisesmith good point. thanks

eoliphant20:10:49

Hi, I have a quick collections question. I’m doing some stuff with datomic and of course you’re kind of on the hook for your own paging. I’m trying to implement cursor based paging such that I can give a key value then return that item, plus N items ‘in front’ or ‘behind of it’ So for the following

({:id "A" :val..} {:id "B" :val..} {:id "C" :val..}{:id "D" :val..} {:id "E" :val..})
If I have say “C” and ‘2 behind’, I get A,B,C. Or “C” and ‘one after’ would give me C and D. and of course, it’d be nice if it was reasonably performant

lxsameer20:10:08

@noisesmith hang on . java.util.Map or clojure.lang.IPersistentMap ? not both ?

noisesmith20:10:46

there’s no need to extend both - it depends on whether you also want to cover vanilla java hashmaps at the same time

lxsameer20:10:28

I just want to target clojure hashmaps and arraymaps

noisesmith20:10:42

then IPersistentMap suffices

lxsameer20:10:59

cool, but i'm a little bit confused in here

lxsameer20:10:38

then what's the point of using a lib like potemkin

lxsameer20:10:48

i mean for this purpose ofcourse

noisesmith20:10:50

it is for making a new type that acts like a hash-map

noisesmith20:10:12

if that’s what you are trying to do, then I totally misunderstood you

noisesmith20:10:38

also, unless you want to change the hash-map behaviors, you can just use defrecord instead of potemkin

lxsameer20:10:59

no no I don't want to create a new type

lewix21:10:24

@dominicm if you know of anything let me know 😉

dominicm21:10:24

/r/clojure had a post right now

bostonaholic21:10:09

@lewix check out #remote-jobs

qqq21:10:36

1. I have a function that is much easier to write recursively than as an iteration / loop. (It has a tree like branching structure.) 2. Realistically, how many stack frames does the jvm/clojure guarantee me?

noisesmith21:10:57

@qqq it’s a config on java startup - you should be able to find the default pretty quickly