Fork me on GitHub
#clojure
<
2016-12-29
>
Busy escaping the comfort zone01:12:18

Hey, looking for a way of selecting items from a Vector using Specter (like srange but for any index), couldn't find a way of doing: (select [ALL <?>] [[1 2 3 4 5] [1 2 3 6 7]]) -> [[2 5] [2 7]] [12:22] There is multi-path [12:23] (select [ALL (multi-path (keypath 1) (keypath 4)) ] rs) [12:24] but it flattens out the result [12:24] [2 5 2 7]

Busy escaping the comfort zone01:12:27

(sorry about the copy paste)

Busy escaping the comfort zone01:12:25

A possible solution might be to use: (transform [ALL (s/collect-one (keypath 0) ) (s/collect-one (keypath 1) )] (fn [a b c] [a b]) rs) which feels a bit akward (I would like to have the ability so select neste sub vectors)

qqq02:12:02

in cider, when I get an error, it shows "Show: Clojure Java Repl Tooling Duplicates All" -- with Clojure/Java/Repl selected by default. I want only Clojure to be selected by default. Is there a way to configure this in Emacs?

cmal04:12:50

Hi, my project use run-jetty to do a lot of tasks, one of them is http service. We use (run-jetty svc-app {:port port}) to start the service in starting of this project. And restarting whole project takes a lot of time. When I want to restart http service, I had to restart the whole project. Is there a way to only restart the http task without restart the whole project? Thanks.

seancorfield04:12:08

@cmal Yes, you can start and stop Jetty without restarting the whole application.

seancorfield04:12:06

You probably want to look at Stuart Sierra's Component library or Mount (can't remember the repo owner for that...) as a way of encapsulating the start / stop lifecycle.

cmal04:12:01

@seancorfield Thanks. If I run (run-jetty svc-app {:port port}) it will tell me the port is already used.

seancorfield04:12:27

Yes, because you need to stop Jetty first.

seancorfield04:12:33

I haven't read that, but it'll probably give you an idea of what Component is about.

seancorfield04:12:26

Also take a look at this https://github.com/framework-one/fw1-clj/blob/master/examples/usermanager/main.clj#L91 -- it's a project of mine that I've stopped maintaining but it shows how to use Component to wrap starting and stopping Jetty.

cmal04:12:44

Thanks. I'll read those and learn from that. Is there a way to stop jetty in repl?

tolitius04:12:46

@cmal: or this https://github.com/tolitius/stater/blob/master/smsio/src/app/www.clj#L20-L27 🙂 not to confuse you, this is mount, not Component (a different approach). If Component makes sense to you, go for it

seancorfield04:12:14

@tolitius Sorry I blanked on who wrote and maintains Mount! 😐

cmal04:12:18

Does that mean that I must keep the object returned from run-jetty for later usage when to stop it

seancorfield04:12:34

(and you must start Jetty with with :join? false and wait in your main thread until ... well, you have to decide that)

cmal04:12:46

Ahh... Thank you!

tolitius04:12:44

@seancorfield: not a problem at all, I did not expect you to know 🙂 @cmal: check it out https://gist.github.com/tolitius/51efc53e6d3643f5189756b68ca172bc

cmal04:12:33

Thank you.

pesterhazy13:12:14

is there a version of shuffle that shuffles, not by random, but by some predictable characteristic of the import?

pesterhazy13:12:21

so you always get the same order

pesterhazy13:12:12

this works for me:

(defn shuffle
  "Like clojure.core/shuffle except consistent order"
  [xs]
  (sort-by #(some-> s str .hashCode clojure.lang.Murmur3/hashLong) xs))

dhucerbin13:12:05

What is clojure way to deal with recursive data structures? Something like AST represented as deeply nested maps? In scala I'm able to factor out recursion with Fix and recursion schemes, does clojure have something like that? Can I bend transducers to work with recursive data? For context, what recursion schemes give me, is that I can define operations on one level of data structure and all morphisms handle recursion for me, and I can use different kinds of morphisms to transform, annotate, use historic data in transformation of each node.

nooga14:12:00

@dhucerbin there is clojure.walk and zippers

dhucerbin14:12:54

can I use clojure.walk to fold a nested map? like evaluate AST?

nathanmarz14:12:58

@dhucerbin specter makes any sort of work with nested data structures much easier

nathanmarz14:12:26

e.g. here's an example which sums up the numbers in nested vectors, starting at the leaves and working its way up:

(def NODES (recursive-path [] p (if-path vector? (continue-then-stay ALL p) STAY)))
(transform [NODES vector?] #(reduce + %) [1 [2 3] [[4]] [5 6]])

dhucerbin14:12:41

I used specter for nested structures and it worked nicely, but for recursive structures was a bit cumbersome, maybe I need to learn more?

dhucerbin14:12:00

For example thing that I need to do often is to recurse through nested maps and annotate each node with value, calculated from their internal properties and properties of ancestors

nathanmarz14:12:47

I do stuff like that all the time

nathanmarz15:12:29

collect, collect-one and recursive-path make the code extremely concise

nathanmarz15:12:42

if you have particular examples you're having trouble working through can help you out in #specter

dhucerbin15:12:21

I will come there soon, thanks!

jacoelho15:12:15

is there any tutorial how to work with protobuf? I am using the protobuf library, however it doesn't see to load correctly extensions

qqq15:12:58

in clojure.test, what is the rationale for putting foo.bar tests in namespace foo.bar-test , as opposed to just inline the tests in foo.bar ?

gfredericks15:12:10

you can write tests without affecting production code

gfredericks15:12:34

e.g., startup time, dependencies...

qqq15:12:37

I never even considered that. Does this work with just foo/bar.clj vs foo/bar_test.clj, or do I need to have separate src/... for production code and test/... for test code?

dpsutton15:12:35

convention is a separate test/ director

dpsutton15:12:44

tooling (including cider) can navigate you to tests automatically

dpsutton15:12:56

so you shouldn't need to fret about actual layout too much i think

tolitius15:12:19

@jacoelho:

(defproject you.project "0.1.0"

  :dependencies [;...
                 [org.flatland/protobuf "0.8.2" :exclusions [org.clojure/clojure]]
                 [com.google.protobuf/protobuf-java "2.5.0"]]

  :protoc "/usr/local/bin/protoc-2.5"

  :plugins [[lein-protobuf "0.5.0"]]
  
  ;; ...)
make sure /usr/local/bin/protoc-2.5 is installed

joshjones15:12:41

a lein new myproject will create separate src and test so that is what most people use, ime

mingp17:12:25

Does anyone have any recommendations for dependency injection/lookup in Clojure? From my reading, it seems like (but please correct me if I'm wrong) something like Component or Mount will help me manage the stateful and/or side-effecting pieces of my app but it would still be up to me as the programmer to manually thread each piece through the parts of the app that need it? Is there something along the lines of a dependency injector or dependency locator in Clojure? Or, if those patterns aren't a good fit, what are the recommended alternatives to achieve the same goals? Thanks in advance.

seancorfield17:12:05

DI / IoC seems like a particularly OOP construct to me — can you explain what you mean by it in an FP context?

dpsutton17:12:39

maybe you have a data service namespace, but that dependency is hard-coded by the namespace requires form. So it would be hard to test if you have functions calling this: you don't want real db interaction in your tests. So DI would let you basically resolve the dependency at runtime? And in your tests you can easily give fake implementations

dpsutton17:12:49

at least in C# that's how we work at work

dpsutton17:12:08

right now I'm writing some controllers against data service interfaces

dpsutton17:12:53

and in my tests i can easily dictate the behavior of those interfaces. And when the application is running, I've got a container that resolves all dependencies so I'm not "new"-ing up a bunch of objects anywhere

seancorfield18:12:57

@dpsutton You can use with-redefs to stub any called functions for your tests.

seancorfield18:12:51

To make that namespace dependency dynamic you’d need to switch to runtime require / resolve all over the place which would make for really ugly code, not to mention the likely performance impact.

seancorfield18:12:02

Still, Component lets you use DI / IoC at the “system component” level so you could isolate all your DB access into a component and then have a mock component and have your tests build a system-map with the mock instead.

seancorfield18:12:09

That might be a lot of functions to mock tho’…

zentrope18:12:12

I think the idea is that you pass in your "database" to the functions that need it. That database implements a protocol. When you test, just stub out that protocol rather than using the real one.

fabrao18:12:48

@seancorfield so with with-redefs can I throw an exception?

seancorfield18:12:33

Yup, you’re providing a completely new definition of a function for the context of the expression in the “with"

fabrao18:12:42

(it "Test"
    (with-redefs [ssh/execute (fn [& _ ] (throw (Exception. "OLA")))]
     (ssh/execute "10.255.8.2" 1022 "xxx" "xxx" ["get system status"])))
throws before it called

fabrao18:12:17

is that correct what I´m doing?

seancorfield18:12:33

You’re replacing ssh/execute with that function — the original ssh/execute will no longer be called there.

seancorfield18:12:55

Normally you use with-redefs to mock something that your function-under-test will call.

seancorfield18:12:20

(in this case, your test is doing exactly what you told it to by throwing an exception!)

fabrao18:12:40

It worked, but is there any way to do it with conditional parameteres? Like ssh/execute "1" -> do this ssh/execute "2" -> do that?

fabrao18:12:08

*parameters

fabrao18:12:17

@seancorfield thanks for your help

seancorfield19:12:50

@fabrao Sure — define your mock to inspect the arguments passed it: it’s just a regular function, after all.

joshjones19:12:52

Let’s say I want to extend the notion of a Number and define a record called Rational, as such: (defrecord Rational [x y]) (as sophie was doing yesterday). What if I now want to be able to call pos? on my record, which will examine the x and y components and determine whether indeed a Rational is positive. What are the best options for how to implement this? One way is to “override” the function, defining my own pos?, which will call clojure.core/pos? when it’s a regular Number and not a Rational. But what are my other options? Essentially, what’s the best way to “add” functionality to a core function?

bfabry19:12:35

@joshjones unless that core function happens to have been defined in an interface, multimethod, or protocol I don't think there's any good way. it specifically wasn't set up for extension

dpsutton19:12:54

can't you import stuff but under a different name?

bfabry19:12:01

the clojure number system does look like it's been designed with extensibility in mind, but at the language level, not the user level https://github.com/clojure/clojure/blob/0929d1d13d036973a03db78a5a03f29d19c9e4b2/src/jvm/clojure/lang/Numbers.java#L1004

dpsutton19:12:48

so load pos? as pos?-core and defn your own pos? that returns the normal result or your own implementation if its a rational?

dpsutton19:12:30

or is that super janx

bfabry19:12:31

sure, but that's not adding functionality to clojure.core/pos? that's creating a new function that includes clojure.core/pos?'s functionality

bfabry19:12:09

you'll only get the extended functionality in places where you call your new function, any other code that calls clojure.core/pos? will still just call vanilla clojure.core/pos?

bfabry19:12:19

if you only care about code that you control getting the new functionality then that's fine, though personally I'd give it a name that doesn't clash with core just to save the hassle of changing all the namespace declns

bfabry19:12:15

you could replace clojure.core/pos? with your own using alter-var-root, but that would definitely be super janky

joshjones19:12:33

well @dpsutton that’s what I was proposing as a reasonable approach, but just looking for ideas. this is just an exercise fwiw, not a real project, expanding my understanding

joshjones19:12:28

If the java.lang.Number type implemented a protocol that included a pos? function, for example, then I could extend a new record (type) based on that protocol and implement pos?

joshjones19:12:42

I’m looking for a way to use runtime dispatch, much like multimethods. The goal is just to see what my options are for doing this kind of thing.

qqq19:12:57

is there a shorter version of (vector (apply concat ...)) ?

timgilbert20:12:27

Am I the only person who finds http://clojure.org/reference/multimethods to be kind of incomprehensible and not useful for trying to figure out what the arguments to (defmulti) mean?

bfabry20:12:54

@qqq maybe not shorter but (reduce into ...) will be a lot faster

dpsutton20:12:38

@timgilbert no i agree with you. The examples are what you really need

dpsutton20:12:44

and most have the same kind of dispatch

dpsutton20:12:53

and its hard to see how the dispatch functions actually work

sb20:12:05

somebody have experience with nginx-clojure server (config)?

caja20:12:55

Curious as to how people handle more complicated unit testing scenarios across architectural boundaries (e.g. database, external web service)? (e.g. if web service returns 503(temp unavailable), then retry if http status code is 503 else stop etc..) In Java, I would have used mocks.

//mockito-esque pseudo code for web service class webServiceImpl implements webService interface(/protocol)
when <webServiceImpl.save()> called, return 503  //first time
when <webServiceImpl.save()> called, return 200  //second time
//similar mockito-esque pseudo code for database
when <db.insert()> called, return DeadlockException  //first time
when <db.insert()> called, return DeadlockException //second time
when <db.insert()> called, return 1 //third time
There's a lot of variations & combinations of these so I didn't want to go the route of doing mocking implementation of each protocol for the different scenarios etc.. i.e. one implementation of webService protocol where save function returns 503 first time it's called, then returns 200 next time.. then repeating with another implementation where it returns 503 the first 2 times, then 200 the third time etc. Is there a lighter way anyone knows of? Any example code would be great.

qqq20:12:58

@bfabry: that's realy neat; thanks!

geoffs20:12:06

@caja if you have a protocol to mock, I would do something like extend it to clojure.lang.IPersistentMap and then in the protocol implementations define a way of describing your mock usage with data. That way the mock you inject at the base level is just a map of data describing what it will return. I do something sort of like this here: https://github.com/RadicalZephyr/comic-reader/blob/dev/test/clj/comic_reader/site_scraper/mock.clj

geoffs20:12:04

But if you want to return different results on successive calls you'd have to extend that idea to then track how many times the mock has been called so it can know which result to return

caja20:12:10

@geoffs thanks for speedy reply. will take a look at links and have a think about that approach, it does meet my requirements for the most part. would you know if this is the route most people happen to go?

geoffs20:12:17

heh, I definitely can't speak for most people. there was some discussion earlier in this channel about a possibly more idiomatic approach to mocking using with-redefs. But when you want to return different values on sequential calls that's going to get kind of hairy...

noisesmith20:12:04

@mingp: when I inject dependencies via Component, this involves using first class functions or or other value objects that are passed into the component, instead of using values defined statically in namespaces

noisesmith20:12:11

the advantage is that component does all the wiring and it's very flexible, the disadvantage is that it is contagious - now anything that uses injected dependencies must also be a component

noisesmith20:12:11

there's compromises possible via delivering a promise or resetting a var based on the component you consume (if you know your namespace only uses a single static version of a resource once initialized)

danboykis20:12:09

@mingp: mount allows you to swap components on start, so you can swap a test db instead of a real one by simply (mount/start-with {#'app/db my-test-db}) which would start the app with the real db swapped for this test-db. In OO terms: the test-db would be injected into the app instead of the real db.

danboykis20:12:08

@mingp depends on what your use case happens to be though, can you give a little more detail?

mikejcusack21:12:36

So I'm new to Clojure and trying to figure out how everything all works together. I'm trying to find an appropriate library for openlayers and found this: https://clojars.org/cljsjs/openlayers. That would appear to be for clojurescript, no? Are they interchangeable?

mingp21:12:48

An example use-case: Suppose I'm writing a to-do + reminders app. It needs to send SMS from a periodic job. I'd like to have some idea of an "SMS sender" in my app, where the one in production sends actual SMSes using Twilio or similar, while the one in local development and/or automated testing pretends to send SMSes but actually just stores them in memory. Here, the "send reminders job" needs access to an "SMS sender", abstractly speaking.

geoffs21:12:04

@mikejcusack the cljsjs project (where you found that openlayers lib) is a way to wrap up javascript dependencies so they can be consumed easily by clojurescript applications with the same tooling used for clojure projects.

seancorfield21:12:05

@mingp That sounds very much like a fit for Component (or Mount) since you have a (fairly) simple, self-contained “component” that knows how to handle message sending.

geoffs21:12:41

in this case, it's not going to be interchangeable as a clojure dependency, since the code is actually still javascript

mikejcusack21:12:00

@geoffs Thanks. So what would be the appropriate library for using openlayers in Clojure?

geoffs21:12:20

if you're talking about this openlayers https://openlayers.org/ then I'm not sure there is a reasonable way to use it from Clojure, since that is a javascript library. you may want to check out using clojurescript instead 🙂

mingp21:12:29

Yea, seems so. My confusion stems from, seemingly as far as I can tell, Component/Mount solve some parts of the problem (the part about keeping track of stateful pieces) but not other parts (allowing other parts of the code to access stateful pieces conveniently, either by facilitating threading them through [as in traditional OOP constructor/setter based dependency injection] or by providing a central lookup [as in traditional OOP dependency locator]).

mingp21:12:18

To be fair, I think my perspective is probably skewed towards the OOP ways of doing these things, having done them before in OOP land. So, I'm more curious how similar problems are solved in Clojure land (even if the solutions happen to look very different).

mungojelly21:12:05

code is data, data is code, code is data, data is code, but wait if i want to put some clojure code into edn the only thing is this experimental codn!?? is there anything else i can use for the use case of using code as data :/

mikejcusack21:12:30

@geoffs I see. In that case how would I use openstreetmap in Clojure? I'm trying to make a GPS app.

seancorfield21:12:39

@mingp You pass the Component around in your code as an argument. In your case, you’d also want polymorphism on the component so your send-sms function behaved differently in each Component (so it would be defined as part of the defrecord for your component, and you’d have a protocol for SMSSender that each defrecord extended / implemented, along with start / stop).

tolitius21:12:40

(deftest swapping-with-value
  (testing "sms endpoint should send sms"
    (let [sms-ch (chan)
          send-sms (fn [sms] (go (>! sms-ch sms))
                     (future))]                        ;; twilio API returns a future
      (mount/start-with {#'app.sms/send-sms send-sms})
      (http/post (post-sms-url "mars" "earth" "we found a bottle of scotch!"))
      (is (= "we found a bottle of scotch!"
             (:body (<!! sms-ch)))))
    (mount/stop)))

geoffs21:12:12

@mikejcusack: What kind of app are you trying to make (desktop, web, mobile, etc.)?

mikejcusack21:12:46

@geoffs I'm using play-clj to make an app that is built for both Android and iOS.

mingp21:12:51

@tolitius Thank you for the example. To clarify, are you saying that the original implementation in the original namespace should perform the default behavior (e.g. send real SMSes through Twilio), and if/when I want alternate behavior under test, I should stub it with mount/start-with?

mingp21:12:45

@seancorfield Sorry, I'm having some trouble following. Are the users of the "SMS sender" component themselves also necessarily components in this world? Transitively, does that make anything which needs to touch state/side-effects a component also? Or do you mean something different?

seancorfield21:12:59

Yes, you would pass the “SMS sender” component as an argument into anything that wanted to call send-sms (in the Component world).

tolitius21:12:08

@mingp: right, after you do (mount/start-with {#'app.sms/send-sms send-sms}) all the components / functions in the code where #'app.sms/send-sms is referenced will see a send-sms test function instead.

seancorfield21:12:40

So the component would represent any state / configuration associated with the specific type of sender (such as URLs, credentials etc) and the send-sms function itself would be polymorphic (via the protocol) on the actual type of the component passed in. Mount and Component approach this rather differently.

mingp21:12:00

@tolitius Thank you for that explanation. I will go explore the example you posted.

seancorfield21:12:01

@tolitius Interesting to see direct mocking support like that in Mount!

tolitius21:12:01

@mingp: for example this would see the test function once (mount/start-with..): https://github.com/tolitius/stater/blob/master/smsio/src/app/www.clj#L14-L18

tolitius21:12:15

rather than the real send-sms component

caja21:12:30

@seancorfield have you ever attempted/though about something like this? interested in your take https://clojurians.slack.com/archives/clojure/p1483042075007396

tolitius21:12:37

@seancorfield: thanks. yea, I am using it in all the tests. another way is to substitute with other lifecycle'able components (i.e. that would also participate in a start/stop party):

(mount/start-with-states {#'app.neo/db        {:start #(connect test-config)
                                               :stop #(disconnect db)}
                          #'app.neo/publisher {:start #(create-pub test-config)
                                               :stop #(close-pub publisher)}})

sveri21:12:37

@caja I have something like that setup with component: https://github.com/sveri/closp/blob/master/resources/leiningen/new/closp/integtest/clj/web/setup.clj You can see I define my own configuration and own components, just for the tests. You can put inside whatever you want

mingp21:12:31

@seancorfield Let me try to give an example of where my confusion is with this. Let's say, going back to the hypothetical SMS reminder app, it is the reminders periodic job whose responsibility it is to call the SMS sender and send out the SMSes (whether real or mocked). When the reminders job runs, from where does it get a reference to the SMS sender to pass to the SMS send function? Was the reminders job itself passed a reference to the SMS sender, and if so, by whom and when? Does this go all the way up the dependency initialization tree? And if so, is there something which simplifies this passing so that it's not all manual?

seancorfield21:12:19

Whatever creates / starts the process would create the appropriate Component / system-map and pass it through the application call tree — per the Component docs and examples.

seancorfield21:12:49

So you’d either create the Twilio component or the dev component or the test component ...

sveri21:12:19

@mingp This is a scheduler component: https://github.com/sveri/stockfighter/blob/master/src/clj/de/sveri/stockfighter/components/scheduler.clj With immutant you can then just refer to the immutant namespace and schedule any jobs. Other libraries behave different and the component will has to be passed down to the function, which will be done as an argument to the function.

seancorfield21:12:38

Each of them would likely have different arguments at construction time and they would likely have different start / stop behaviors too — for the component lifecycle.

seancorfield21:12:27

In addition, you’d have your SMSSender protocol that declared send-sms and each different component (record) would define the appropriate behavior, based on the component type / contents.

caja21:12:45

@sveri having test specific components is that part i get. it's the next piece of the puzzle i was wondering about- varying the behavior of the test component without having multiple implementations to cover each scenario. (apologies if misunderstood your answer)

seancorfield21:12:16

(so Component is very much like any OOP system you’re used to — construct the appropriate SMSSender “object” at startup)

sveri21:12:27

@caja In my experience you can reuse components, just like any other stuff. And components with different behavior will have to be implemented differently

mingp21:12:13

@tolitius @seancorfield @sveri Thank you all for the explanations and examples. I'm going to take some time to read through both Component and Mount. It seems that some of my understandings about them were mistaken before, so I'll try my best to figure them out.

sveri21:12:38

@mingp I think they are very similar and choosing one over the other is no mistake. There are a few discussions about the differences between these too at reddit

tolitius21:12:34

@mingp: sure, very welcome. I think

When the reminders job runs, from where does it get a reference to the SMS sender to pass to the SMS send function?
is a very frequent and good question. Component answers it by having your application being an integral part of the system: i.e. if you need to access a stateful component in a function A, with Component, this function A should be called from within the system created by Component. Think about it as a Spring application context (if you are familiar with Spring). You don't call functions Components do: i.e. "don't call us, we'll call you" principle. You can call functions as well, and you often do in REPL, in development, by having a reference to all the components (a.k.a. system) and passing pieces of it to functions directly. mount makes components / states on the namespace level (i.e. to be top level beings), which means you can (:require [app.sms :as sms]) and then (sms/send ...) anywhere in your application. Although these components are top level, they are still managed by mount, hence things like starting, stopping, swapping, etc.. are possible.

metametadata22:12:51

@caja you may be interested in https://github.com/metametadata/clj-fakes. Usage example based on your pseudocode:

(defprotocol WebService
  (save [this data]))

; ...

(f/with-fakes
    (let [service (f/reify-fake WebService
                                (save :fake [[:--data--] (f/cyclically [503 200])]))]
      (is (= 503 (save service :--data--)))
      (is (= 200 (save service :--data--)))
      (is (= 503 (save service :--data--)))))

seancorfield23:12:40

@tolitius (and @mingp ) — one last note about Component: it doesn’t have to be an all-or-nothing affair, i.e., your “application” doesn’t have to be part of a Component, although that’s typically the case if you start from scratch with Component. You can just build an appropriate Component as part of your application startup and then “start” it and make it available to other code however you want (either pass it in through the top-level call chain or — shock, horror — make it a piece of global state), if you need to add it piecemeal into an existing app.

seancorfield23:12:52

It certainly does work better if your -main is just (component/start (make-my-app-component {:some “config”})) tho’… (where your AppComponents start function runs your whole application).