Fork me on GitHub
#clojure
<
2017-10-10
>
thedavidmeister06:10:39

anyone know a good way to get a file system based memoize instead of using memory?

andrea.crotti07:10:14

@thedavidmeister mm it will eventually load the data in memory when you have to use it anyway right?

andrea.crotti07:10:57

what's the use case?

thedavidmeister07:10:28

@andrea.crotti i have 100's of network calls that take 20-60s each to complete and they are always the same (historical event data)

andrea.crotti07:10:18

well you can use any DB for something like that

andrea.crotti07:10:30

Redis even handles the TTL for you

thedavidmeister07:10:40

yes, i can write a caching layer

thedavidmeister07:10:08

i was hoping to simply memoize the fns

thedavidmeister07:10:08

i don't want to add a dependency on a specific tech like a db or redis

andrea.crotti07:10:19

well you can just do it yourself as well

andrea.crotti07:10:33

write to some EDN files

andrea.crotti07:10:39

and read from them

thedavidmeister07:10:56

current plan is to copy and paste the clojure core memoize function and swap out the lookup for a file system based logic

thedavidmeister07:10:05

it just seemed like something that might already exist

andrea.crotti07:10:20

it won't save you any memory though, it will only save the computation time, unless you have a different file per input

thedavidmeister07:10:38

yes, different file per fn/args combination

andrea.crotti07:10:56

some answers there

andrea.crotti07:10:26

but depending on what you're doing even sqlite could be better than doing it yourself

andrea.crotti07:10:34

and that doesn't require any other server running

thedavidmeister07:10:58

sqlite seems like a lot of work

thedavidmeister07:10:06

the event data i'm getting has no schema

thedavidmeister07:10:36

so i either have to hammer a schema into place to represent the specific data coming back from the query

thedavidmeister07:10:53

or put k/v into the db, which doesn't seem better/easier than just dumping to files

thedavidmeister07:10:27

this stack overflow answer is exactly what i was thinking

thedavidmeister07:10:51

so the answer is "there's no existing libraries, so hand roll something simple"

rauh07:10:27

@thedavidmeister I wrote that answer šŸ™‚. You could also use: https://github.com/Factual/clj-leveldb if you don't want to deal with multiple files. I have no experience with leveldb though.

thedavidmeister07:10:10

mmmk, i'll start with the simple approach and move to a db if i find a reason to

rauh07:10:37

LevelDB is no "real" db, there is no server. It's just a single file.

danielcompton07:10:39

@thedavidmeister I think you probably want to use a cache instead of memoising

danielcompton07:10:16

a cache gives you more control over where it spills out to

thedavidmeister07:10:19

but a cache also relies on me inserting cache logic

thedavidmeister07:10:36

which will end up being the same as memoize behaviour

danielcompton07:10:30

sure, they're basically the same thing here?

danielcompton07:10:09

you can write a PluggableMemoization if you prefer, see https://github.com/clojure/core.memoize

thedavidmeister08:10:53

it all just seems like overkill... what's the downside of tweaking memoize to use a folder of files instead of an atom?

andrea.crotti08:10:10

By tweaking you mean rewriting it I guess?

andrea.crotti08:10:54

It just means you are reinventing the wheel imho

dominicm08:10:14

Rather than changing memoize, you could have the function write to the fs and return a path to it

thedavidmeister08:10:17

writing a custom plugin for a library?

thedavidmeister08:10:34

those are spokes, not a wheel šŸ˜›

rauh08:10:23

@thedavidmeister There is nothing wrong with just using files, but YOU will have to do the coordination if you write/read from multiple threads. Or you'll get garbage. LevelDB library would allow you to just get/write as you like. It's thread safe (as long as you don't use Iterator).

thedavidmeister08:10:32

i don't think i'll run into that issue with what i'm doing

thedavidmeister08:10:40

but that's a fair point in general

thedavidmeister08:10:07

i'm strictly serially hitting api endpoints, and each time i get about 3 million rows back

thedavidmeister08:10:14

and just doing this in a repl

thedavidmeister08:10:03

i don't think there's any threading in there, but i could be wrong

rauh08:10:44

Well I hope you're right, b/c initially yous said it'd take 20-60sec for you to answer the request.

thedavidmeister08:10:11

i can wait that long once or twice

thedavidmeister08:10:14

just not over and over

rauh08:10:01

@thedavidmeister My point is: What if a second request comes in, while you generated your cache from the first request...

thedavidmeister08:10:25

why is that a problem?

thedavidmeister08:10:32

each request has different args and so a different file on disk

rauh08:10:35

@thedavidmeister So why cache it then? If it's never going to be accessed again since they're all different?

thedavidmeister08:10:59

the next time i run the function to run a report

thedavidmeister08:10:15

it's not always different

thedavidmeister08:10:31

it's just not the same for a single set of calls

thedavidmeister08:10:35

the only way to have the same args at the same time is for me to run two repls side by side

thedavidmeister08:10:56

but the same args will definitely be called sequentially, when i call the fn a second or third time in the repl

thedavidmeister08:10:13

anywho, i've already rewritten the memoize to do what i want for now

thedavidmeister08:10:23

so i'm getting some dinner, thanks for all the suggestions everyone šŸ™‚

qqq08:10:49

(def _+ 20) <-- this appears to work, but is it guarnteed by any standard?

bronsa08:10:11

both _ and + are valid symbol constituents

qqq08:10:03

@rauh: the phrasing of Symbols begin with a non-numeric character and can contain alphanumeric characters and *, +, !, -, _, ', and ? (other characters may be allowed eventually). threw me off

qqq08:10:30

by special casing the "_", one could interpret "non-numeric" to mean "non-numeric alpha-numeric" char (or atleast that was my first interpretation)

Bravi10:10:06

is there an equivalent function to flip (in other programming languages) in clojure?

Bravi10:10:09

to flip arguments

reborg10:10:15

No, you'll have to roll your own. For example (defn flip [f] (fn [& args] (apply f (reverse args)))) and then ((flip reduce) (range 10) +) @bravilogy

Bravi10:10:32

thank you

Bravi10:10:14

basically I have this function

(defn game-progress [score]
  (cond
    (zero? (mod score 4)) add-more-colors
    (zero? (mod score 5)) decrease-time-left
    :else identity))
and thought I would refactor that repetitive bit somehow and perhaps use condp instead

thheller10:10:55

I vaguely remember a library that can read EDN data and write it out again while preserving all comments and formatting. Can someone remember the name?

bronsa10:10:00

i believe rewrite-clj has utilities for that sort of things

thheller11:10:08

rewrite-clj was the thing I was looking for. thx.

cgrand11:10:14

Donā€™t know the reference but thatā€™s clearly the spirit

dominicm11:10:27

I didn't think words would quite express enough no in this case, I just pictured you screaming it. šŸ˜‚

Ertugrul Cetin11:10:57

Hey guys, new Clojurecademy course uploaded: Code Katas - https://clojurecademy.com/courses/17592186068882/learn/overview

viesti11:10:29

hum, is there a brew formula available for the clj script?

alex-dixon13:10:19

I canā€™t figure out how to implement a custom pretty print method without mutating the default behavior. Is there an idiomatic way to use a custom pretty print method for one call to pprint?

alex-dixon13:10:28

(def my-atom (atom nil))
(defn pprint-dispatch []
  (do
    (defmethod clojure.pprint/simple-dispatch clojure.lang.IDeref [o]
      (print o))
    clojure.pprint/simple-dispatch))
(clojure.pprint/write my-atom :dispatch (pprint-dispatch))
(clojure.pprint/pprint my-atom)

alex-dixon13:10:24

clojure.pprint/pprint my-atom)
#<Atom@58a7831e: nil> ;; this is the default I'm trying to overwrite
=> nil
(clojure.pprint/write my-atom :dispatch (pprint-dispatch))
#object[clojure.lang.Atom 0x58a7831e {:status :ready, :val nil}]=> nil ;; great, this is the way I want IDerefs to print
(clojure.pprint/pprint my-atom)
#object[clojure.lang.Atom 0x58a7831e {:status :ready, :val nil}] ;; oops -- all subsequent calls to pprint will look this way?
=> nil

alex-dixon13:10:16

@U3DAE8HMG Thanks. I see these use code-dispatch instead of simple-dispatch, but I think I face the same issue. I want to alter the way IDeref is pretty printed and leave everything else the same šŸ˜•

alex-dixon14:10:13

^ might also be a question about extending multimethods since thatā€™s how clojure.pprint/simple-dispatch is implemented

chrisblom14:10:27

does anyone know how to disable clojure assertions in AOT compile namespaces?

chrisblom14:10:56

(set! *assert* false) is throwing an exception in that case

chrisblom14:10:38

and alter-var-root also fails

Jim Rootham14:10:36

I would like to have separate names for test and production uberjars compiled from the same project.clj file. Is that possible? Can you put an uberjar-name key is a profile and make uberjars with 2 different profiles?

tomaas15:10:45

hi, can I parse a date string with this pattern Oct 09, 2017 using clj-time?

dpsutton15:10:44

in clj-time.format there's a function called show-formatters that will show you all of the built in parse strings and what they can do

tomaas15:10:33

i've looked at those, and there's no such format

ccann15:10:12

@tomaas youā€™re going to want to use a custom formatter e.g. (f/formatter "MMM dd, YYYY") < might work

ccann15:10:44

(f/parse (f/formatter "MMM dd, YYYY") "Oct 09, 2017")

phreed15:10:58

I have data from a lisp (common lisp maybe) and I would like to read it in clojure. Any advice? I found this... https://github.com/clojure/clojure/blob/master/src/jvm/clojure/lang/LispReader.java with examples of its use at the end of the file.

ccann15:10:05

are you trying to read common lisp code into clojure as data?

ccann15:10:29

LispReader is a clojure reader, similar to tools.reader https://github.com/clojure/tools.reader#differences-from-lispreaderjava

ccann15:10:44

it doesnā€™t read common lisp

madstap16:10:05

What does the data look like?

tomaas15:10:36

Thus, "MM" might output "03" whereas "MMM" might output "Mar" (the short form of March) ....

tomaas15:10:40

didn't read until this point

tomaas15:10:58

thanks, a lot @ccann. 3 MMMs work

bfabry17:10:35

I often find myself looking for "something like pipeline-async, but that doesn't care about ordering, but does still deal with signaling 'all of x are finished'". am I silly? Does that already exist?

bfabry17:10:00

like, I'm writing this entirely too much

(async/pipeline-async
              500 ;; some number bigger than our number of tasks
              tasks-finished-for-account

bja17:10:41

You can do that with climate corp's claypoole library

bja17:10:33

from the readme:

(require '[com.climate.claypoole :as cp])
;; We'll use the with-shutdown! form to guarantee that pools are cleaned up.
(cp/with-shutdown! [net-pool (cp/threadpool 100)
                    cpu-pool (cp/threadpool (cp/ncpus))]
  ;; Unordered pmap doesn't return output in the same order as the input(!),
  ;; but that means we can start using service2 as soon as possible.
  (def service1-resps (cp/upmap net-pool service1-request myinputs))
  (def service2-resps (cp/upmap net-pool service2-request service1-resps))
  (def results (cp/upmap cpu-pool handle-response service2-resps))
  ;; ...eventually...
  ;; Make sure sure the computation is complete before we shutdown the pools.
  (doall results))

bfabry17:10:45

certainly seems about it. trying to avoid introducing new libs if possible though

mpenet18:10:31

You can build this on top of async/merge if your ops all return chans/promise-chans

bfabry19:10:37

yeah I generally build it with async/merge

bfabry19:10:43

(starts reading promise-chan docs)

tbaldridge18:10:36

@bfabry nah, it doesn't exist, mostly because it's fairly simple to write.

bfabry19:10:11

yup, that's what I ended up writing (ish). makes sense, ta

bfabry19:10:55

I think you miss that you need to create a channel for each thread though. otherwise you may close! out-c before one of the go-loops has finished its work

lxsameer19:10:28

is it possible to extend hashmaps and arraymaps without repeating the same code ?

lxsameer19:10:41

hmmm so it's harder than i expected.

bfabry19:10:09

yes. in practice though I've found it's very rarely necessary

bfabry19:10:32

what did you mean by "extend" though btw? because you could for instance extend them with a protocol with very little code

lxsameer19:10:10

I have a protocol and I want to extends them with that protocol. but without repeating my self

lxsameer19:10:52

I guess the simpler solution would be just to repeat myself.

noisesmith19:10:49

donā€™t extend PersistentHashMap and PersistentArrayMap - those are implementation details, extend java.util.Map or clojure.lang.IPersistentMap, which cover both hash-map and array-map

lewix20:10:06

I miss clojure

lewix20:10:26

too bad there are not many opportunities in my areaā€¦

lewix20:10:54

I never completed learning it fully like I wanted to but it taught me a lot

lewix20:10:22

Happy to see that the community is still thriving šŸ™‚

dominicm20:10:27

@lewix there's always remote work šŸ™‚ Keep an eye open

lxsameer20:10:46

@noisesmith good point. thanks

eoliphant20:10:49

Hi, I have a quick collections question. Iā€™m doing some stuff with datomic and of course youā€™re kind of on the hook for your own paging. Iā€™m trying to implement cursor based paging such that I can give a key value then return that item, plus N items ā€˜in frontā€™ or ā€˜behind of itā€™ So for the following

({:id "A" :val..} {:id "B" :val..} {:id "C" :val..}{:id "D" :val..} {:id "E" :val..})
If I have say ā€œCā€ and ā€˜2 behindā€™, I get A,B,C. Or ā€œCā€ and ā€˜one afterā€™ would give me C and D. and of course, itā€™d be nice if it was reasonably performant

lxsameer20:10:08

@noisesmith hang on . java.util.Map or clojure.lang.IPersistentMap ? not both ?

noisesmith20:10:46

thereā€™s no need to extend both - it depends on whether you also want to cover vanilla java hashmaps at the same time

lxsameer20:10:28

I just want to target clojure hashmaps and arraymaps

noisesmith20:10:42

then IPersistentMap suffices

lxsameer20:10:59

cool, but i'm a little bit confused in here

lxsameer20:10:38

then what's the point of using a lib like potemkin

lxsameer20:10:48

i mean for this purpose ofcourse

noisesmith20:10:50

it is for making a new type that acts like a hash-map

noisesmith20:10:12

if thatā€™s what you are trying to do, then I totally misunderstood you

noisesmith20:10:38

also, unless you want to change the hash-map behaviors, you can just use defrecord instead of potemkin

lxsameer20:10:59

no no I don't want to create a new type

lewix21:10:24

@dominicm if you know of anything let me know šŸ˜‰

dominicm21:10:24

/r/clojure had a post right now

bostonaholic21:10:09

@lewix check out #remote-jobs

qqq21:10:36

1. I have a function that is much easier to write recursively than as an iteration / loop. (It has a tree like branching structure.) 2. Realistically, how many stack frames does the jvm/clojure guarantee me?

noisesmith21:10:57

@qqq itā€™s a config on java startup - you should be able to find the default pretty quickly