Clojurians
#clojure
<
2016-03-18
>

This page is not created by, affiliated with, or supported by Slack Technologies, Inc.

thiagofm00:03:25

How do I reload a config file inside resources in cider?

martinklepsch08:03:39

@thiagofm: you need to re-run whatever mechanism loaded the config file in the first place

martinklepsch08:03:57

nothing to do with cider I'd say

camdez09:03:30

Can anyone recommend resources (blog posts, sample projects, ...) for larger Clojure test suites? I'm struggling to find answers to simple questions like: if I have shared test code, where should that live? Are there any conventions?

dm309:03:25

@camdez test code is treated the same way non-test code is - have your abstractions in the namespaces under test-source paths and build upon that

camdez09:03:49

@dm3: Thanks. Perhaps I should have been a bit clearer though—I’m not asking how I possibly can, but I’m looking for recommendations. For example, with clojure.test one tends to mirror the src namespace tree, adding -test to the end of the namespace name. If I create a file / namespace in my test/ directory with a -test name, that might defy the convention that -test files contain tests. If I don’t add a suffix, the namespace might match something in my src directory. So I thought there might be community conventions.

dm309:03:09

yeah, most testing tools have their own convention of naming test files, this is probably best looked up in the docs for the test tool (expectations/midje/...). Also the namespaces should not clash if they appear on the same classpath. Don't think I'm aware of any more conventions.

camdez09:03:14

@dm3: I appreciate the feedback! 🍻

rickmoynihan10:03:38

@camdez: In my experience there are two predominant styles of naming test namespaces: 1. app.foo => app.foo-test (the -test suffix approach) 2. app.foo => app.test.foo (the .test subnamespace approach) 2 seems to have been popularised by things like the luminus template - however I really dislike this approach... Mainly because the test file name is the same as the apps namespace file - which can be confusing in some editors. And also for purely selfish reasons that the default emacs/projectile jump to test doesn't understand this convention (but does understand the first). I just wish there was one standard for tooling to target though.

camdez10:03:27

@rickmoynihan: Thanks, Rick. Kinda funny—I totally understand your assessment and tend to use naming style number #1, but I just made an app.test.helpers namespace to keep that stuff from colliding with the names in my src namespace, and #2 would dovetail more nicely with that.

camdez10:03:18

I’ve started taking notes on Clojure testing practices with an eye towards putting together a short guide when I have more answers.

rickmoynihan10:03:09

@camdez: not sure I see how.... But for test helpers with style 1 you just avoid the -test suffix for things that don't contain test suites.. e.g. helpers can become: app.test-helpers

camdez10:03:52

@rickmoynihan: Only in that it bundles everything under a common app.test namespace. Not critical, but it has some nice properties. I considered the app.test-helpers name as well, but all things being equal, I would prefer to not have that code in the same folder as the proper tests. ¯\(ツ)/¯ At this point I’m just experimenting and keeping an eye towards reasons to prefer one method over another.

rickmoynihan10:03:23

I don't see that as a problem if you have a convention to find them - if it really bothers you, you can always add another source-path e.g src/ test/ and /test-helpers - or if you don't want to clutter the root directory - set a source path for /test/suite and test/helpers and use convention 1 simple_smile

karolmajta11:03:17

Hey guys, is it possible to call clojure.core/name in a macro?

karolmajta11:03:54

(map name [a b]) results in [nil nil] when seeing output of macroexpand

karolmajta11:03:10

maybe it's just a repl thing?

moxaj11:03:37

@karolmajta Could you post your whole macro?

karolmajta11:03:16

sure:

(defmacro defoperation [name bindings & body]
  (let [requires (map name (filter keyword? bindings))
        arguments (filter symbol? bindings)]
    `(def ~name
       (map->Operation {:name ~name
                        :requires ~(vec requires)
                        :arguments ~(vec arguments)
                        :fn (fn ~(vec (concat requires arguments))
                               [email protected])}))))

moxaj11:03:28

Your argument name shadows the function

karolmajta11:03:05

ah! sure! thank you

thiagofm12:03:52

What testing framework library do you people recommend?

thiagofm12:03:22

(which isn't core.test)

dialelo12:03:54

why not core.test?

dialelo13:03:18

i dig test.check, but is not the classical example-based testing

thiagofm13:03:06

Maybe because I'm used to something else. In core.test I couldn't find for example somewhere I would specify my test setup & teardown

thiagofm13:03:23

It's quite simple in general

dialelo13:03:22

may come in handy

thiagofm13:03:53

@dialelo: thanks!

dialelo13:03:36

you're welcome, you may want to ask in #C08LK2DH7 if you have more questions

thiagofm13:03:33

Good to know, will join there

jmayaalv13:03:57

@thiagofm: midje is also a very popular https://github.com/marick/Midje/wiki

thiagofm13:03:27

@jmayaalv: I'll take a deeper look at it, I saw the project a couple of years ago but ended up going with core.test at the time

niwinz15:03:43

almost all them are clojure (jvm) only

niwinz15:03:01

(alternatives to clojure.test)

richiardiandrea15:03:12

yes, in ClojureScript I usually go for vanilla cljs.test

niwinz15:03:31

I prefer just use clojure.test/cljs.test for both

niwinz15:03:49

that makes context switching a little bit more pain less

dialelo15:03:11

also in my opinion midje and similar libraries introduce another DSL to learn and gives you little value when writing tests

richiardiandrea15:03:29

that allows you to put tests and code in the same namespace

richiardiandrea15:03:46

(I haven't tried myself to see if it is a good idea though)

niwinz15:03:11

I prefer have the code and test separated

richiardiandrea15:03:41

still not tried to to merge them, I wonder if you have a smoother REPL + test experience (less switching)

niwinz15:03:36

Yeah, I understand but the downside of mix things and having bloat files with mixed things...

niwinz15:03:14

makes more complicated to reason about your logic on the namespace just because it contains a lot of more stuff no related to solve the problem...

richiardiandrea15:03:12

yeah that can be a problem, I agree

niwinz15:03:50

I have used python previously to clojure

niwinz15:03:56

and python has the concept of doctest

niwinz15:03:54

and almost in all big projects when I have envolved, doctests are avoded...

niwinz15:03:41

doctests in python exists for a while... and I think that them are not heavy used...

richiardiandrea15:03:39

great so I will heed your advice and save some time 😄

danielstockton16:03:15

Can anyone give me tips on how to work with things like config files or key pairs (things you don't want in git or the uberjar)? Wanted to use (io/resource "path/to/key.pem") and then supply that path at runtime but I get errors building the uberjar.

danielstockton16:03:24

This is the project: https://github.com/danielstockton/aaas/tree/master/resources These are the keys I want to remove from git/uberjar.

bostonaholic16:03:30

@danielstockton: if it's in resources, you should only need to (io/resource "pubkey.pem")

bostonaholic16:03:30

and I'd recommend changing your private key now that it's public

danielstockton16:03:30

Have that: https://github.com/danielstockton/aaas/blob/master/src/aaas/core.clj#L21 Problem is, I wanted to move my keys under a keys directory and then have {:dev {:resource-paths ["keys"]}}. This means that uberjar can't find it and I get Cannot open <nil> as a Reader at this line

danielstockton16:03:52

@bostonaholic: Of course I will change the private key after I get this working

danielstockton16:03:02

Pushed the broken version to github with keys directory committed https://github.com/danielstockton/aaas/tree/master/keys

danielstockton16:03:23

Is the general idea correct, that I should be able to compile this and supply resources at runtime?

noisesmith16:03:05

io/resource is for finding things you expect to find on the class path

danielstockton17:03:25

And it should compile without the resource being present during compilation?

hiredman17:03:36

it depends, if you can copy and paste each line of your code in to the repl, without the resource, without error, then yes

hiredman17:03:08

(because that is compiling)

hiredman17:03:07

yeah, looking at this code, this is not going to work

hiredman17:03:39

I assume by compile you mean aot compile? clojure is always compiled to java byte code, regardless of aot compilation or not

danielstockton17:03:11

any advice on how to achieve what i want? example project that does similar?

hiredman17:03:11

so the way aot compilation works, is basically the same as what I said, each form getting pasted in to the repl one by one, the difference is aot compilation saves the class files off to disk too

hiredman17:03:52

so if you paste

(def a (println "foo"))
in to the repl, it is going to get compiled, print "foo" and bind the result, nil, to a

hiredman17:03:08

(defn a [] (println "foo")) 
will of course not print "foo" when you paste it in to the repl, it will get compiled and then run, binding a function to a

hiredman17:03:03

so

(def a (read-in-some-file))
will fail to compile if the file is not present when you paste it in to the repl

hiredman17:03:55

it fails because (read-in-some-file) is being executed immediately, so you want to delay that somehow

hiredman17:03:38

similar to how the function doesn't immediately execute the code in its body

hiredman17:03:11

there are many ways to do this, the simplest for you may be to use a delay

danielstockton17:03:12

i see, would that be idiomatic for this kind of thing?

danielstockton17:03:29

delay is better than wrapping it in a function?

hiredman17:03:10

well, a delay actually does wrap it in a function, but it is like a memoized zero argument function

noisesmith17:03:09

a cool thing with delay that I figured out recently is that (force x) forces the delay x and derefs the value, but if the thing forced is not a delay, it just returns the value

hiredman17:03:08

I might instead of using a delay, create a function that reads in the keys, call those functions in side new-web-server and pass them around as arguments

hiredman17:03:48

but I have actually used delay exactly for this purpose (reading in crypto keys)

danielstockton17:03:46

delay seems ok to me, thanks for the tips

danielstockton18:03:05

think you're right, i'll have to pass them around as arguments or i get into trouble

tobowers18:03:42

are there any OSS projects for scanning Leiningen dependencies for CVEs? Kind of like brakeman for rails? http://brakemanscanner.org/

lvh19:03:04

Are there any tricks to using core.match for compiling s-exps to s-exps? I’m currently doing this: https://github.com/RackSec/desdemona/blob/master/src/desdemona/query.clj#L35-L44 and it works fine but I don’t know if maybe I’m missing some nifty utilities or something simple_smile

smw20:03:08

@lvh I don’t have an answer for you, but I’d love to subscribe to your newsletter simple_smile Have a blog post or anything about what you’re doing?

lvh20:03:47

smw: well, I’m giving a talk about it soon but I’m guessing you’re not in Chicago and I’m not sure it’ll be recorded

lvh20:03:21

smw: I’m a principal engineer at Rackspace Managed Security, we have a customer-facing SOC, we’re trying to build what they want that addresses limitations in currently available SOC software

smw20:03:23

Ahh, that’s true, I’m not. I wonder if I have someone I can send simple_smile

lvh20:03:43

one of the bets we’re making is that in order to be effective we want a) more power to correlate, b) faster feedback

lvh20:03:02

part of that is “OK let’s run the queries both on recent data in the browser and on less recent data on the JVM backend"

smw20:03:25

Oh, very interesting.

lvh20:03:25

in that the SOC analyst boxes can reasonably have 64GB RAM so who cares if firefox’s RSS is 60GB

smw20:03:38

I resemble that remark.

lvh20:03:43

also I’m hoping to get a bunch of barreleyes so the onyx cluster is 5x 2TB RAM boxes

smw20:03:47

(I don’t know how to close tabs)

smw20:03:48

(I had to look it up, but I probably should have known)

lvh20:03:58

turns out it’s also a fish

smw20:03:26

Are you looking at Riemann at all?

lvh20:03:26

I mostly care that it has 192 hw threads and 2TB of ram

lvh20:03:28

(also runs clojure)

smw20:03:34

Or the design patterns?

lvh20:03:35

smw: Not really

lvh20:03:39

well, beyond that simple_smile

smw20:03:58

Ahh. Ok. Too much data?

lvh20:03:16

That’s a good point actually; I probably should go through it again and see what’s applicable

lvh20:03:32

smw: Probably. We’re currently looking at about ~20PB/day, and some of that is bursty.

lvh20:03:53

(It gets bursty when two dozen different bro sensors suddenly decide to ship a bunch of PCAP in your direction)

lvh20:03:43

the hard part is long term storage, mostly; I have Aaron Sullivan and the OpenCompute/OpenPOWER teams to give me crazy fast CPU/RAM

lvh20:03:59

Ideally we’d store almost all of that long-term; because the other not-so-big bet is that you care about it for both audit logs and training data

smw20:03:34

I seem to remember you guys have object storage as a service? simple_smile

lvh20:03:01

we do; and that’s plan A for storing it

lvh20:03:13

but if you’re ingesting 20PB a day it does not take a lot of days to get to a pretty big object store

smw20:03:25

Or a pretty big anything

lvh20:03:27

also, we’re a SOC, so we can’t use the existing infrastructure; all of that data is quite sensitive

smw20:03:32

I’d assume you could do at rest encryption or something

smw20:03:51

but I would also assume that the economies of scale would make it cheaper to scale a large object storage platform than trying to scale something internally.

lvh20:03:56

if an environment needs to be HIPAA/FedRAMP/PCI/whatever compliant unfortunately you can’t dump all of its logs just wherever

lvh20:03:17

smw: scale’s one thing, unfortunately not all compliance requirements are ok with saying “encryption” and have that be the end of it

lucj0620:03:33

Hi, I’m very new to Clojure and just started to get my hand dirty when building a client for my RabbitMQ server. I have some trouble getting message from my « data » exchange and « ALERT » queue. Any idea what I’m missing ? http://pastebin.com/LKqXLQDQ ?

therabidbanana21:03:06

Anybody know if any good libraries for analyzing memory usage in a similar way to using criterium to microbenchmark runtime exist? I've updated a function to remove a sort and criterium is showing a likely improvement in performance, but I'm wondering if the refactoring is better or worse memory-wise, which we're sensitive to at the same sizes sort time starts mattering.

lucj0621:03:54

Just fixed a couple of option but cannot get the issue: http://pastebin.com/mA8UbAxq any idea ?

stuartsierra21:03:06

@therabidbanana: a Java profiler should give you memory stats. YourKit is best (commercial), VisualVM (free) is adequate for basic stuff.

jamesleonis21:03:37

@lucj06: What are your errors?

sfz-21:03:09

do any of you use jmxtrans to profile?

lucj0621:03:51

@jamesleonis: I do not have errors, but message are not received

lucj0621:03:08

while they are received by the node.js version of the client

therabidbanana21:03:09

@stuartsierra: was hoping something I could play with in the repl - I know ruby has ways to count object allocations (using ObjectSpace), and if Clojure had something similar it'd be possible maybe? In any case, I'll crack it open with VisualVm - thanks!

jamesleonis22:03:33

@lucj06: I can’t find anything obviously wrong 😕

ptrwldn22:03:31

@lucj06: queue name, and topic parameters swapped in bind?

stuartsierra22:03:27

@therabidbanana: Ruby is its own runtime. Clojure's runtime is the JVM. Clojure itself doesn't have any insight into things like object allocation — that's all at the JVM level. So you need JVM tools.

smw22:03:08

I’d think it would be possible to write something in clojure that gets access to plaform MBeans on the jvm and reports stats.

smw22:03:18

Probably fairly easily?

hiredman22:03:29

there is a (somewhat neglected) contrib library for that

nnbosko22:03:37

Does anyone know what the best way to structure a Swagger request in order to include file data and other form data correctly? I haven't found anything useful in the Compojure or Schema docs for something like this

nnbosko22:03:25

Basically I'm seeking to extend the picture gallery example on the Web Development with Clojure 2nd Ed book

sfz-23:03:27

thanks guys