Fork me on GitHub
#clojure
<
2016-09-29
>
verma00:09:48

fenak that’s actually something I am doing right now, I am using migratus for now, as far as I know right now, I think migrations will run as a component for me at the beginning of everything else and bring the DB state to the most recent migration

verma00:09:32

so its actually a part of process bootup for me, or could be a separate bootstrap process that makes sure your db env is sane before it launches the nodes on a cluster or whatever

verma00:09:43

not sure I am directly addressing your question here

verma00:09:45

there’s also

fenak00:09:41

yeah, i’m using ragtime..

donaldball00:09:44

I use a component system, and I have higher level components depend on a database component that applies pending migrations on startup

jrheard01:09:40

spec question from someone who hasn’t yet personally used the library: in http://clojure.org/about/spec#_data_spec_registration , there’s this passage: "By convention, specs for data should be registered under keywords and attribute values should be registered under their attribute name keyword.” does anyone know what this means? i’m having trouble parsing it

jrheard01:09:28

in this context, what’s meant by “data” vs “attribute values”?

amacdougall01:09:01

I believe "attribute value" refers to a value in a map. If I'm interpreting it right, the idea is that you'd be able to define :my-namespace/width as int? and then say that a map has required keys :my-namespace/width, and spec is able to figure out that the value of something with that key must conform to the spec.

amacdougall01:09:29

(spec/def ::width int?)
(spec/def ::larger-thing (spec/keys :req [::width]))

amacdougall01:09:02

spec sees that ::larger-thing is a map with {::width <val>}, and knows that <val> must be int?.

amacdougall01:09:20

But #clojure-spec is probably a better place to ask.

jrheard01:09:02

aha, didn’t know about that channel, thanks!

yogthos02:09:24

@fenak I package migrations into the app in luminus, and then the app runs them when it starts if you give it a migrate argument, can also be configured to do that automatically

yogthos02:09:47

so I would suggest a similar approach to what @verma outlines, where the app manages its own migrations

fenak02:09:57

Thanks guys, I’ll give a try!

borkdude08:09:30

What’s the best place to ask questions about clojure.java.jdbc?

borkdude08:09:32

I have a query like this:

(db/query  connection {} {} {:fetch-size 10 :auto-commit false})
which queries an enormous table. I want to test if this query is really lazy, as in, the result set is not in memory all at once. How can I verify this?

mpenet08:09:32

(let [x (your-query)] (take 1 x) (println (realized? x)))

mpenet08:09:37

something like this I guess

mpenet08:09:34

assuming it has more than one row, also depends on chunking dunno how/if that lib does chunking or not or if it defers to some of core's defaults

borkdude08:09:46

can you ask how many elements are realized?

mpenet08:09:51

I don't think so

mpenet08:09:14

try to set a breakpoint in the c.j.jdbc code concerned maybe

borkdude08:09:42

Hmm, (let [result-set (db/query …)] (realized? result-set))is true, count is 6673 (on my dev machine, but in production it will be a lot bigger

mpenet08:09:49

actually I wonder if realized? is true for a single take on a lazy-seq

mpenet08:09:01

count will trigger the realization of the full seq

borkdude08:09:00

yeah, I only did count after realized?

mpenet08:09:36

well seems like realized is broken for some stuff that used to work too... I am not sure it's realiable after all

borkdude08:09:53

I guess I could test it with :row-fn (fn [r] (println "hello”)) or some other side effect

borkdude08:09:58

which seems to indicate it’s not lazy right now

horia08:09:19

realized? "Returns true if a value has been produced for a promise, delay, future or lazy sequence." - doesn't it mean that realized? will return true even if the whole seq was not realized but only at least one element ?

horia08:09:42

(let [x (range 10) y (first x)] (realized? x))

horia08:09:47

returns true

borkdude08:09:03

due to chunking the whole range is realized here

mpenet08:09:04

yeah that might be the case (that's what I mentioned earlier)

borkdude08:09:13

32 elements at a time right?

borkdude09:09:23

I think it might have to do with autoCommit on the connection from HikariCP

borkdude09:09:45

with a connection by uri, same problem

borkdude11:09:22

I had to do something like this:

(do
    (def conn {:connection-uri "jdbc:})
    (def conn (j/get-connection conn))
    (.setAutoCommit conn false)
    (def stm (j/prepare-statement conn "select * from mytable" {:fetch-size 10}))
    (def row (atom 0))
    (clojure.java.jdbc/query 
     {:connection conn}
     [stm]
     {:row-fn (fn [r] (swap! row inc))
      :result-set-fn #(doall (take 100 %))})
    @row)

borkdude11:09:43

overriding the result-set-fn was essential, because it defaults to eager evaluation.

borkdude11:09:47

but also I had to manually set auto-commit to false and prepare a statement, because both auto-commit and fetch-size are not handled by query it seems

bcbradley12:09:52

if i had a map that contains vector values and i want to generate a map that has set values, how would i do it?

mpenet12:09:49

with reduce-kv or zipmap ?

mpenet12:09:14

even (into {} ) with an xform that does the convertion, but that's equivalent to reduce-kv i believe

nathanmarz12:09:59

@bcbradley with specter, (transform MAP-VALS set my-map)

bcbradley12:09:18

I did

(into {} (map (fn [v] [(first v) (into #{} (second v))]) registry))

nathanmarz12:09:32

will also preserve the type of the map and be near optimal performance

bcbradley12:09:51

what is specter?

dpsutton12:09:28

@bcbradley specter is highly regarded, and its author is quite active in helping craft operations in it and helping people understand it and use it, as you may have noticed

bcbradley12:09:59

thats interesting but i'd prefer to keep my dependencies low for this one specific thing. If i were working on an application or library I'd consider using it though, it looks quite powerful

bcbradley12:09:30

@nathanmarz I see you want to reduce on graphs in topological order in the future work for specter

bcbradley12:09:15

this may or may not be helpful to you:

nathanmarz12:09:36

@bcbradley thanks, but I've actually already implemented all that graph stuff (on top of Loom)

dpsutton12:09:51

did Loom find a new maintainer?

bcbradley12:09:53

gosh i feel out of the loop

bcbradley12:09:57

what is loom?

dpsutton12:09:00

i remember hearing they were trying to get someone

bcbradley12:09:00

it would be really really cool if you could search gist for clojure code use specs rather than text

bcbradley12:09:41

for instance, instead of searching for keywords, use a spec to specify the constraints on the function you are looking for

bcbradley12:09:53

that way it can return more precise results

mpenet12:09:23

there once was findfn in that flavor

mpenet12:09:50

but it predates specs

bcbradley12:09:43

thats cool, but i was thinking more integrated

bcbradley12:09:50

for instance, you could have something that can integrate into your github-- it automatically generates gists and keeps them updated to match your github code and behind the scenes the gist is indexed according to its spec and other metrics

bcbradley12:09:15

basically if someone has already solved a problem that you are working on in a library today, and they solved it 4 years ago, you can easily find their solution by doing the search.

bcbradley12:09:57

Moreover, you don't actually have to do the search manually. When you specify the spec of your function, the search could be done on your behalf

bcbradley12:09:21

results can be given to you automatically as a step in your continuous integration

bcbradley12:09:35

whats more, if there is more than one solution they could be compared against other metrics

bcbradley12:09:44

latency, throughput, memory use, cpu use, etc

borkdude13:09:04

@seancorfield Should it be possible to pass down :fetch-size from jdbc/query so it will be passed down to prepare-statement? The docs say something like: "The options specify how to construct the result set (and are also passed to prepare-statement as needed)” but not all options that are relevant to it are accepted.

seancorfield15:09:47

@borkdude: there's an open issue to fix that

asbesto16:09:59

hi guys, does anyone have familiarity with the ring-basic-authentication library? https://github.com/remvee/ring-basic-authentication/blob/master/src/ring/middleware/basic_authentication.clj

asbesto17:09:50

i'm trying to get my own wrapper working around it, but i seem to be by-passing the auth step

asbesto17:09:52

(defn mywrap [handler] (fn [req] (if (= (path-info req) "/ping") (handler req) (do (wrap-basic-authentication req authenticated?) ;; <------ this is where I'm trying to call authentication (handler req)))))

andy.fingerhut17:09:38

@bcbradley : There is a function often called map-vals that gets reimplemented or copy/pasted across projects, where your task would become (map-vals set registry) Example implementation of map-vals is in the (small) medley library: https://github.com/weavejester/medley/blob/master/src/medley/core.cljc

anmonteiro19:09:59

is there any way a can avoid calling if in this snippet?

(if (= (type x) java.net.URI)
  (.toString ^java.net.URI x)
  (.toString ^java.io.File x))

jr19:09:13

(.toString (with-meta x {:tag (type x)})) ?

blueberry19:09:33

pardon my javanese, but shouldn't toString be polupymorphic? that is, doesn't just (str x) or at least (.toString x) do the right thing?

grzm19:09:59

Looks like he might be trying to avoid reflection

jr20:09:04

blueberry that requires reflection

anmonteiro20:09:21

definitely trying to avoid reflection

anmonteiro20:09:41

@jr your solution would probably work if this was a macro

trptcolin20:09:55

just hint x to ^Object, right?

anmonteiro20:09:55

I want to add meta to the symbol, not the URI object

grzm20:09:57

But you're doing reflection yourself by checking type, correct?

trptcolin20:09:06

.toString is on java.lang.Object

anmonteiro20:09:22

@grzm I’m not currently using if 🙂

anmonteiro20:09:32

I’m using .toString and getting a reflection warning 😛

blueberry20:09:41

in that case, what is the difference in speed? toString itself is not a speed demon...

grzm20:09:43

Right 🙂

trptcolin20:09:25

(defn get-it [^Object x] (.toString x)) doesn't reflect

anmonteiro20:09:05

@trptcolin that should work, yeah. let me try

trptcolin20:09:18

you could also just use str

trptcolin20:09:40

(which also hints to ^Object)

anmonteiro20:09:57

yeah I’m gonna do that, thx all

trptcolin20:09:39

oops, i see @blueberry already said that, sorry @blueberry!

blueberry20:09:14

why apologise? i don't have a trademark on str :) cheers :)

ghadi20:09:31

str is the correct thing to do -- but if you're going to check class use class not type

bcbradley21:09:02

what does tap actually return?

bcbradley21:09:08

in clojure async

bcbradley21:09:01

it says it copies the mult source onto the supplied channel

bcbradley21:09:13

so why would it return anything?

novakboskov21:09:32

What is so special with clojure.data.json, why it is excluded from AOT compilation? I'm packing my project into uberjar and AOT is used for it, uberjar won't load clojure.data.json and raises ClassNotFound.

bfabry21:09:16

@novakboskov I'm not sure what your specific issue is, but you do not need to AOT your project in order to uberjar it

ghadi21:09:06

(aside: i'd avoid clojure.data.json if you care about performance)

novakboskov21:09:07

@bfabry My issue is that my AOT compiled uberjar can't load clojure.data.json, simple. 😄

josh_tackett22:09:34

App won’t build on heroku, need help quick:

-----> SSH app detected
-----> Installing ngrok...  !     Push rejected, failed to compile SSH app.
 !     Push failed

codefinger22:09:35

@josh_tackett that’s from the heroku-buildpack-jmx. not sure what’s wrong, but you can remove it by running heroku buildpacks:remove

quoll22:09:29

@ghadi re: clojure.data.json, I am using it for the first time, mostly because it’s in a namespace that starts with “clojure”. So I should go back to Cheshire?

ghadi22:09:42

yeah IMHO

ghadi22:09:58

doing it all in clojure is ok... but cheshire is the de facto leader

quoll22:09:51

Suits me. I’ve been using it for years. I’m only too happy to be shuffled back into my comfort zone 🙂

seancorfield23:09:03

@quoll We had to switch from clojure.data.json to Cheshire a while back as we ran into some basic Java types that the former doesn’t handle (dates, I think, but don’t hold me to that).

quoll23:09:14

I’m presuming that’s for encoding to string, since JSON doesn’t support dates

seancorfield23:09:27

Could be. Cheshire supports dates just fine and lets you control how they’re formatted.