Fork me on GitHub
#clojure
<
2016-06-05
>
thug.nasty01:06:31

skrat: i love your user pic

bhauman02:06:22

I'd really like to get a stable checksum/hash of an arbitrary persistent data structure, has anyone done this?

(defn canonical-data-structure [build-config]
  (walk/prewalk (fn [x]
                   (cond
                     (map? x) (into (sorted-map) x)
                     #_(string? x) #_(.hashCode x)
                     (set? x) (into (sorted-set) x)
                     :else x))
                 build-config))

(defn canonical-hash [data]
  (-> data
      canonical-data-structure
      pr-str
      digest/md5))

bhauman02:06:18

It hasn't been working for me.

ckarlsen03:06:09

I need an algorithm or method that can calculate missing values from time series. Basicly I have a dataset with hourly readings from energy meters and I need to calculate the hourly energy usage. The challenge is that somestimes alot of data points are missing. example: https://www.refheap.com/ce0efc49f2a894c60a5246ff4

ckarlsen03:06:39

any tips ? maybe Incanter?

dpsutton03:06:56

make a record with slots for hour, interpreted/recorded, meter-total, usage, then just loop over, storing all records without a recorded meter total until you hit the next one with a metered result, and then just put a linear interpolation across all the ones without readings

dpsutton03:06:23

if this makes it up to the current hour, then you are obviously stuck and can make a decision as to leave it without measurements or to estimate based on the last three or four known points

dpsutton03:06:12

oh wow. didn't know you were missing so much data

dpsutton03:06:22

maybe loop twice

dpsutton03:06:04

first loop create a trend, and then have records for each reading as :deviation-from-trend and :predicted-value

dpsutton03:06:30

in this way you can create your model, allow for changing needs over the day, ie, noon usage isn't the same as early morning, etc.

dpsutton03:06:15

and you fill in each value and then when you need to consume you just grab (or (:metered-reading row) (:predictive-value row))

ckarlsen03:06:18

@dpsutton: thank you very much! Was reading about gaussian kernel smoother on wikipedia and my head was about to explode.

dpsutton03:06:51

perfection is the enemy of good

dpsutton03:06:58

sorry, perfection is the enemy of done

dpsutton03:06:02

don't over complicate it

dpsutton03:06:09

come up with a super simple model

dpsutton03:06:27

get that model from the data and then map its predictions onto the data

dpsutton03:06:31

bam problem solved

dpsutton03:06:45

then you can in the future increase the complexity of the model

dpsutton03:06:00

but that's a separate and orthogonal problem from the one at hand

dpsutton03:06:09

simple programming problem, complicated modeling problem

dpsutton04:06:29

you might be able to solve your woes with this one.

dpsutton04:06:00

i had never heard of that library. seems chock full of stuff

ckarlsen04:06:48

very cool! thanks again

ckarlsen08:06:33

looks reasonable I think

crankyadmin08:06:19

Hi, say I'm do a bunch of checks like:

(or (= something "foo")
            (= something "bar")
            (= something "baz"))
Is there a nicer way of doing it? Doing it like this doesn't feel right in Clojure.

niwinz08:06:52

(contains? #{"foo" "bar" "baz"} something)

crankyadmin08:06:44

... I knew that too... Happy Sunday morning 🙂

jrychter09:06:44

Trying out clojure.spec — and I'm having trouble with some of my maps. I want to validate maps of the form {:insert [{} {} {} {}]}, where keys are (s/def ::op #{:insert :replace :delete :state}). How can define a type ::data that will validate a map with (optional and non-namespaced) keys conforming to ::op?

jrychter09:06:46

I realize I can list those explicitly in the :opt-un vector passed to s/keys, but that isn't quite satisfactory — and besides all operations have the same parameter structure, so I would like to later also validate their content without defining the same spec for each op.

benzn09:06:47

hi folks - i have a clojure project in cursive that has become fantastically slow to boot a REPL. everything appears to have parsed and executed in under 10 seconds but then clojure spends the next 50 seconds or so compiling somethingorother and often timing out

benzn09:06:03

...what is clojure so busy compiling? the code has already executed...

benzn09:06:37

and things are garbage collection like mad in the meantime too

Alex Miller (Clojure team)12:06:17

@jrychter: why is :opt-un not satisfactory? you can reuse the same spec for each attribute too

(s/def ::op-val ...whatever...) ;; define the content of an op here
(s/def ::insert ::op-val)
(s/def ::replace ::op-val)
(s/def ::delete ::op-val)
(s/def ::state ::op-val)
(s/def ::command (s/keys :opt-un [::insert ::replace ::delete ::state]))

Alex Miller (Clojure team)12:06:54

@benzn if you can repro outside of Clojure, you might try taking thread dumps of the process (ctrl-break in win, ctrl-\ in *nix, or kill -3 the process, or use jstack) to see what it's doing in that extra time.

jrychter12:06:41

@alexmiller: I tried that, and somehow thought it was too verbose, I thought I could just list all the ops in a single set. But having looked at it again, it doesn't look so bad…

jrychter12:06:52

@alexmiller: ok, now I remember. The next level in my data structure contains a table name (as a keyword). So you could get an {:replace {:stock {more-data here}}. So there again, instead of my current (def ^:const +watched-tables+ #{:stock ...}) I will need to declare each of them in two places. I guess my data structures don't really fit with clojure.spec, and I should probably change them to have the op and table names as values, not as keys…

Alex Miller (Clojure team)12:06:22

@jrychter: you might want to declare the next level down using something like map-of, since these are not fixed keys

jrychter12:06:50

@alexmiller: thanks. The more I think about it, the more I come to the conclusion that I should have a fixed key structure and keep data in values, which would let me use multi-specs. With the added advantage of them being extensible (doesn't make sense for ops, but does for tables).

plexus13:06:14

clojure.spec question, it seems you can you s/or in place of s/alt just fine. Are there cases where this doesn't work? Why do we need both?

plexus13:06:22

(s/conform (s/cat :s string? :x (s/alt :n number? :k keyword?)) ["foo" :bar])
;;=> {:s "foo", :x [:k :bar]}

(s/conform (s/cat :s string? :x (s/or :n number? :k keyword?)) ["foo" :bar])
;;=> {:s "foo", :x [:k :bar]}

xfcjscn13:06:16

About argument order in clojure. I have posted some comments in https://groups.google.com/forum/#!topic/clojure/iyyNyWs53dc, but unfortunitly no reply. Can some one give some comments here?

plexus14:06:15

To partially answer my question, s/or generates a clojure.spec.Spec object, while s/alt returns a regex operator (represented as a map and used by the spec regex machinery), so I suppose these compose differently

plexus14:06:24

(s/alt :n number?)
;;=> {:clojure.spec/op :clojure.spec/alt, :ps (#function[clojure.core/number?]), :ks (:n), :forms (clojure.core/number?), :id #uuid "72ae2d04-43ef-4e33-a8c2-e3ca70cc5844"}
(s/or :n number?)
;;=> #object[clojure.spec$or_spec_impl$reify__11565 0x1a6b7806 "[email protected]"]

plexus14:06:25

still would love to see an example where the result differs, I tried conform, valid?, exercise, explain, and at least for the basic case above alt and or always behave the same

plexus14:06:39

In case anyone else is interested, this snippet illustrates the differences 🦆

nwjsmith14:06:13

@plexus: I was wondering about this, thanks!

plexus14:06:29

yeah it makes total sense if you keep reminding yourself it's just regexes. I'm not used to applying that to anything that isn't a string 🙂

nwjsmith14:06:16

Hm. I think I might be fuzzy on this still. What is difference between specs and regex ops that causes '(:bar :baz) to be consumed by or but not by alt in the second conform above?

plexus14:06:20

Because alt is a regex operator you stay on the same "level", in the way that #"a(bc|d)" matches "abc", (s/cat :a keyword? :x (s/alt :s (s/cat :b keyword? :c keyword?) :d keyword?)) matches [:a :b :c]

plexus14:06:47

in other words, it doesn't match a nested structure, it just matches a flat seq

plexus14:06:35

if you do want to match a nested structure like [:a [:b :c]] you would add an s/spec to start a new spec that matches the nested sequence

plexus14:06:12

because or is not a regex operator but its own type of spec, it implicitly causes such nesting

plexus14:06:06

don't know if that makes sense... it's a bit tricky to explain well

plexus14:06:23

to put it another way, as long as you're only using regex operators (cat, +, *, ?, alt), you're matching a single flat sequence.

plexus14:06:22

Each item in the sequence can be validated/conformed by a predicate or a spec (registered or inline)

plexus14:06:38

if the item is itself a sequence, you can express that with a spec (s/or, s/spec) or a predicate (vector?) inside the regex

nwjsmith14:06:59

@plexus: great explanation. Thanks, this definitely clears it up.

Alex Miller (Clojure team)15:06:25

We're typically used to regexes matching characters in a string. in this case though we are matching Clojure data in a sequence. It's not possible for a single character in a string to also be a substring, but it is possible for an element in a Clojure sequence to be another nested sequence. That's what introduces this complication here.

plexus15:06:52

right @alexmiller, well put!

bhauman15:06:05

In a multiple process situation, say I want to synchronize around a system resource (a directory) and guarantee a release of that resource on process exit. Is there a good Java-y way to do that? File Locks seem too brittle I'd love to have something that just won't survive the end of a process.

niwinz16:06:36

Maybe it is too much over-engineering but, hazellcast offers a distributed lock

niwinz16:06:54

it is pretty easy to use with java interop...

bhauman16:06:18

yeah I really don't want to bring in any more deps

bhauman16:06:25

this is for figwheel

bhauman16:06:36

thanks though

Alex Miller (Clojure team)16:06:50

@bhauman you can use Java finalizers for that although there are whole talks worth of caveats about them

Alex Miller (Clojure team)16:06:39

there are also ways to create temp files and mark them to be deleted on exit

Alex Miller (Clojure team)16:06:54

can't say I've found that to actually work reliably though

bhauman16:06:56

reliability is the caveat

niwinz16:06:47

it seems much more reliable that manual using of files for locking

bhauman16:06:55

deleteOnExit can some sometimes fail as well, would love something like opening a socket, you know its going to go away after the process exits, but sockets don't have arbitrary names

Alex Miller (Clojure team)16:06:37

I'd trust something like FileLock more than the others I mentioned

Alex Miller (Clojure team)16:06:13

but I haven't used it to know how reliable it is

niwinz16:06:09

later, also maybe considered strange option, but if you have postgresql already a dependency in your project, you can use an advirsory locks to hold a lock. It works much more in a "socket way" if connection is closed, lock is released.

bhauman16:06:27

FileLock does look like the best of the options

lvh16:06:11

Is clojure/algo.generic common/idiomatic or is there a better way to get fmap? I’m torn between just implementing fmap myself and using it

lvh16:06:34

I care in particular for maps; I want {k v} => {k (f v)}

lvh16:06:49

(defn ^:private mapvals
  [f m]
  (reduce-kv (fn [m k v] (assoc m k (f v))) (empty m) m))
is not very ugly I suppose:)

plexus16:06:35

@lvh Medley is one of the more common utility libs that has map-vals https://github.com/weavejester/medley

lvh16:06:52

Ah; thanks — I had seen it before but forgot about it

plexus16:06:57

there's another commonly used utility library that also has it but can't remember what it's called now, and it seems http://crossclj.info is still down 😞

jrychter18:06:36

I've been using my own transform-v and transform-kv since Clojure 1.0, and I've always wondered why they aren't in core. It seems everybody implements their own version…

gfredericks18:06:15

there will always be a Most Useful Function Not In Core

jrychter18:06:29

True, but these two (especially transform-v also known as fmap) are the ones I encounter pretty much everywhere. I recently discovered three different versions within our company, written by different people over the years.

james18:06:12

I made a screencast to introduce people to visualising their data in proto-repl-charts: https://www.youtube.com/watch?v=BJUI1ntfPy8

gfredericks20:06:13

I've always been bothered by the use of namespaced keywords as sentinels

bronsa20:06:45

likewise -- I tend to use (Object.) or (reify)

gfredericks20:06:27

bronsa: (reify) is cute -- I guess it's ostensibly more portable than (Object.)?

creese20:06:06

Is it possible to return keys from an execute! in clojure.java.jdbc?

janxspirit20:06:56

is there a preferred library/framework for writing integration tests for compojure api routes?

cddr21:06:20

I have a Dockerfile modelled after the example in Clojure's docker hub and any time I change the project.clj, it seems to download all maven dependencies again. Is that by design?

gfredericks21:06:27

cddr: hard to avoid I'd think

gfredericks21:06:51

I've mitigated that some by linking in my ~/.m2 but that doesn't work when building images, just when running stuff in the container

cddr21:06:53

Yeah I can definitely see why it would be. Just wanted to check I wasn't doing something silly. It's only really a problem in the early stages of a project when you keep adding new libs. And even then it's not too bad

gfredericks21:06:55

I could imagine some sort of fancy programmatic modification of the Dockerfile used in tandem with leiningen's local-repo feature

cfleming22:06:06

@james: That looks lovely - proto-repl is looking really nice.