Fork me on GitHub
#clojure
<
2016-09-22
>
dfan00:09:00

I am pretty confident that the author was referring to the style of my example above, especially if he’s referring to functional programming, but like @idiomancy says we can’t tell for sure without more context

fenak02:09:18

I’m having a hard time getting used to component (and duct). So my whole system is started when the application boots up, and Duct is glueing components with endpoints. So I have a “processor” namespace that do some queries on the database… how should i inject the db there?

fenak02:09:26

As it’s not and endpoint nor a component?

grzm02:09:35

@fenak you mean a database connection?

fenak02:09:43

or better explained, do i have to then, in my endpoint, call the process function passing on the db :spec and then keep repassing to a lower function within the other namespace?

grzm02:09:39

I'm not sure what you mean by an endpoint, but a database connection is a component/resource that other parts of the system depend on.

grzm02:09:06

and it needs to be connected to (started) and cleaned up (stopped)

fenak02:09:12

@grzm yeah… it didn’t make sense for me to create a component for my processor, as that would make me put on the system map on boot time, and i might be wrong, but then it looks like i have one instance of a processor record on my process...

grzm02:09:31

What's a processor?

seancorfield02:09:41

How are the functions in your "processor" namespace being invoked? (i.e., where from?)

fenak02:09:44

endpoint is just a duct thing i guess, basically a handler

fenak02:09:01

(processor/process) basically

fenak02:09:15

somewhere inside there it runs a query using jdbc/run-query

seancorfield02:09:30

From inside an endpoint function? Which will have access to the component that contains the database connection, yes?

fenak02:09:41

now that i put my db stuff on component, i don’t know how should i be using it.

fenak02:09:13

very simple, as my endpoint is doing something as simple as: (future (processor/process))

grzm02:09:22

In my database component I have the queries that utilize the connection. Then the processor depends on the database component and calls the queries on the component.

seancorfield02:09:43

So when you call (processor/process) pass in either the component containing your db connection (so processor is parameterized on a Component) or pass in the db connection itself (so processor is parameterized on a db-spec).

fenak02:09:45

but then the processor itself needs to be a component right? or not?

seancorfield02:09:13

(processor/process (:db system)) or (process/process (:db-spec (:db system)))

grzm02:09:20

Yeah, the processor would be a component, too. One that depends on the database component

seancorfield02:09:29

The processor does not need to be a Component, but its functions need to be passed a Component.

fenak02:09:07

if I create a component for the processor, that means one instance of Processor will be there right? are there any problems with race conditions on that?

seancorfield02:09:20

I wouldn’t do that.

seancorfield02:09:01

I would just write the processor functions to expect a (database) Component as one of their arguments — or a db-spec which the caller gets from the (database) Component it already has.

seancorfield02:09:51

If the processor namespace has no "state" of its own that it needs, I wouldn’t make it a Component.

fenak02:09:26

@seancorfield it doesn’t.. but let’s suppose it does.. then i create a component..

fenak02:09:59

what happens when 2 different threads call (process {:processor system}) at the same time??

seancorfield02:09:10

At World Singles, our database Component has all our db connections in it (and our MongoDB connections) so we pass the whole database Component into functions that might need to access a database.

seancorfield02:09:08

@fenak I’m not sure what you’re asking...

fenak02:09:31

ok, so, component will create a new system on boot time.. with my component..

fenak02:09:42

that means that will instantiate a record called Processor

seancorfield02:09:59

(which would have a dependency on a Database Component yes?)

fenak02:09:20

and that Processor record instance would be living there on the system map..

seancorfield02:09:42

(assuming Processor is truly a singleton in your system)

seancorfield03:09:12

Are you concerned about mutable state within the Processor record? An atom in there?

fenak03:09:18

that’s what i wanted to ask then.. if it’s not a singleton, it should not be a component..?

seancorfield03:09:39

Right. Component = singleton in essence.

fenak03:09:45

hmm… i don’t think i’d have any mutable state.. i mean, just a function that runs a query and chuck the result in S3.

seancorfield03:09:58

Otherwise you’d just create a hash map (or record) on the fly and pass it around.

seancorfield03:09:23

So Processor could exist as a Component that has dependencies on Database and S3StorageDatabase would start/stop/manage your database connection pool; S3Storage would get the S3 configuration details at startup perhaps. Does that sound reasonable?

seancorfield03:09:42

Then your endpoints (handlers) would need to pass the Processor component into functions in that namespace: (processor/process (:processor system))

seancorfield03:09:38

And you’d have an overall Application Component (`system`) containing :processor, :database, :s3, etc that you build at startup and let Duct weave into the endpoints…

fenak03:09:29

my processor ns functions right now doesn’t have s3 storage as a component.. it just calls a function on files.clj that calls s3/put-object.. and that handles it..

fenak03:09:54

it’s not a component, but it’s opening a connection and running stuff directly from the files/send-to-s3 function..

seancorfield03:09:03

Then you could test your processor independent of your endpoints by having your tests create Processor and pass it into functions as needed.

fenak03:09:08

it that very poorly executed?

seancorfield03:09:30

I’d probably have an S3 Component to encapsulate the configuration so nothing is hardcoded

fenak03:09:31

i see.. that sounds better..

fenak03:09:58

what do you mean? credentials or bucket names?

seancorfield03:09:35

So you could then reuse that S3 files code with different credentials etc more easily.

fenak03:09:35

im getting the bucket as (or (:s3-bucket env) “bucket-name) on a def

fenak03:09:50

the credentials are not there, as we rely on aws IAM roles..

seancorfield03:09:56

Then maybe your env should be a Component?

seancorfield03:09:22

(we have our "environment" as a Component at World Singles so it can populates its configuration at startup)

fenak03:09:31

should i wrap environ as component..?

fenak03:09:43

i was just calling directly from environ.core/env

seancorfield03:09:21

I assume that’s a library that "wraps" environment variables?

seancorfield03:09:14

Think about testability tho’ — do you want to have your tests around the S3 stuff rely on environment variables? Better to encapsulate that in a Component that your tests could provide perhaps?

fenak03:09:43

i’m still trying to learn how to proper test this stuff on clojure..

fenak03:09:17

being on tdd with oop for so long, i’m still trying to shift my workflow for clojure..

fenak03:09:46

in ruby or java i’d just mock/stub the s3 bits.. but in clojure i don’t know yet what i’d do.

fenak03:09:03

i have a function like that that i find very hard to think on how to test.. that’s maybe too poorly designed

fenak03:09:07

(defn content-from-s3
  ([]
   (files-from-s3 s3-bucket rules-dir))
  ([bucket dir]
   (let [s3-objects (->> (s3/list-objects bucket dir)
                         :object-summaries
                         (filter #(re-matches #".+\.json$" (:key %)))
                         (map #(s3/get-object bucket (:key %)))
                         (map :input-stream))]
     (map load-content s3-objects))))

seancorfield03:09:02

I'd say that's doing too much in one function and I'd refactor it into several smaller functions.

seancorfield03:09:25

You can follow a TDD workflow with Clojure... which should stop you creating untestable functions, just like should in Ruby / Java with OOP...

seancorfield03:09:50

Does that help at all @fenak ?

fenak03:09:19

@seancorfield yeah, definitely..

fenak03:09:29

thanks heaps

fenak03:09:07

is there anyway for me to mock that s3/list-objects call?

seancorfield04:09:09

Look at with-redefs.

mhjort07:09:31

@fenak maybe this is not Clojure related per se. We're doing a lot of S3 related stuff with Clojure. Instead of mocking those s3 calls using

with-redefs
we typically use a fake server https://github.com/jamhall/s3rver

fenak07:09:22

hey @mhjort that looks interesting.. i’ll have a look!

mhjort07:09:29

This is how you can use it:

(doto (AmazonS3Client.)
       (.withEndpoint custom-end-point)
       (.setS3ClientOptions (-> (S3ClientOptions.)
                                (.withPathStyleAccess true))))

mhjort07:09:53

So the custom-end-point is the url where you are running the fake-server

abdullahibra10:09:26

is there clojure equivalent to re.sub python ?

val_waeselynck10:09:30

@abdullahibra not sure the semantics are exactly the same, but have you checked out clojure.string/replace ?

abdullahibra10:09:53

val_waeselynck what i want to do exactly is: apply method which will work on lower-case words then return another lower case words but i want to preserve the original format

val_waeselynck10:09:28

I think I'll need an example 🙂

abdullahibra10:09:41

as example: input "Hello;world!" get-words -> ["Hello", "world"] lower -> ["hello"," world"] apply_X_method -> [hell@, w@rld] final -> "Hell@;w@rld!"

abdullahibra10:09:56

that's one example what i need is general solution

abdullahibra10:09:25

the format is changeable

val_waeselynck10:09:47

@abdullahibra ok, so basically split your string into an interleaving of word/non-word substrings, and apply a modification only to the word substrings

val_waeselynck10:09:51

is that correct ?

val_waeselynck10:09:29

oh and there's the lower-case constraint too

val_waeselynck10:09:36

@abdullahibra don't know how to deal with lower-case, but clojure.string/replace-by seems close to what you're trying to achieve

abdullahibra10:09:04

val_waeselynck there is no replace-by

val_waeselynck10:09:12

@abdullahibra I'm on clojure 1.8, you ?

abdullahibra10:09:47

oh 1.7, will update

abdullahibra10:09:57

is there docs for it in clojure ?

pesterhazy10:09:22

replace-by is not public

reefersleep10:09:24

I was looking for it here, but can't find it

reefersleep10:09:12

But it sounds like what you want to be done can be constructed easily. Maybe. I just don't understand your example. Does it show what happens inside the function you desire?

pesterhazy10:09:37

clojure.string/replace can take a fn as replacement though

reefersleep11:09:39

@abdullahibra I find the part about operating on lower-case strings but returning (possibly) differently cased strings a bit weird. What if X_method affects casing?

reefersleep11:09:51

Here's one implementation (concocted in and to be pasted into a REPL, though it'd work fine in a file if you fix the require) :

(require '[clojure.string :as str])

(defn rb [input f] 
  (let [delimiter ";" , 
        words (str/split input (re-pattern delimiter)) , 
        lower (mapv str/lower-case words) , 
        after-fn (mapv f lower) , 
        final (apply str (vec (interpose delimiter after-fn)))] 
  final))

reefersleep12:09:36

... which does not return the original casing 🙂

reefersleep12:09:10

called as such: (rb "Hello;world!" some-fn)

mac12:09:47

@reefersleep checkout as-> to reduce let bindings 🙂

reefersleep12:09:13

Cheers @mac 🙂 I thought I'd make it verbose and stick to "simple" functions in order to aid @abdullahibra , who may or may not be new to Clojure. But I can always use a reminder about handy core fns, I tend to forget them!

wopeers13:09:13

What are recommended libs/frameworks to set up a REST service in clojure?

mac13:09:47

@wopeers Look at liberator, compojure-api and yada

ordnungswidrig13:09:35

@wopeers there’s #liberator and #yada btw.

wopeers13:09:24

thx. liberator looks promising indeed 🙂

ikitommi13:09:42

@wopeers and c-api stuff at #C06GSN6R2

yonatanel14:09:05

@niwinz I was just starting to compare catacumba, aleph etc. I might have some questions for you later :)

yonatanel14:09:36

@niwinz What do you think about a model where you consume requests from a stream and put responses on a stream, instead of having to return something from within the route function?

niwinz14:09:22

Can you elaborate more?

yonatanel14:09:25

@niwinz I want my handling of the request to go forward and deeper into the callstack instead of having to return a value right there in the handling function. I'm not talking specifically about catacumba.

yonatanel14:09:20

maybe a handler get take a deferred or a stream as a parameter and pass it along, instead of creating its own and returning it.

niwinz14:09:07

I have thought about this approach but I found it not much practical

niwinz14:09:50

At this moment you have the ring approach (composition of functions) and catacumba/pedestal (managed pipeline/chain/stream of handlers)

niwinz14:09:15

if i understand your proposal well,

niwinz14:09:33

you want something like the catacumba/pedestal approach but handle the stream yourself

yonatanel14:09:10

Not sure. I will have to read about it.

niwinz14:09:59

in this case and maybe I'm wrong, handling yourself the stream of the request will add additional boilerplate, that something like catacumba/pedestal approaches in more simple approach

niwinz14:09:53

Unfortunately I have to go right now, if you have a concrete questions about catacumba, feel free to post them to me directly or in #funcool channel 😉

kevinmershon16:09:56

Anyone know of a clojure/clojurescript compatible markup language or some such that could be used for marshalling one structure to another? I'm effectively trying to expose the functionality of map to users for custom mutations of data, ideally with a drag-drop editor, for example turning {:user_email ""} into {:email_address ""} without the user having to know how to code, and without having them write functions that we'd have to eval at runtime.

kevinmershon16:09:56

Possibly something along the lines of https://github.com/joelvh/json2json

kevinmershon16:09:17

Nevermind, answered my own question when I found json2json. https://github.com/shaunlebron/hammock

witek17:09:24

I am trying "interactive development" with Clojure (on Emacs and CIDER). But I think I don't get it. My idea is to write a function, eval it, then call it from the REPL. Correct it, eval it and call it again. But there is no single command for this. There is one for evaluation, one for switching to REPL, then open last call and execute it. Then switch back to editor. This seams not right. Do I have a wrong idea about interactive development or is there a command like "eval buffer + send to REPL + execute last REPL expression"? Or should i write my functions directly in the REPL and copy them to my .clj file when it works? Is there any tutorial on how the interactive development workflow is done right?

bolivier18:09:03

if you’re using cider, there’s an eval from buffer command, which is what I tend to use

bolivier18:09:14

then I M-p in the repl to get back to the last thing I executed

bolivier18:09:24

may not be ideal, but it’s worked for me

dhruv118:09:39

C-c C-c to eval it. ;; editing this. C-c C-p to print your expression in a separate bufer

bolivier18:09:14

@dhruv1 but if you’re writing a function, you’ll need some sample call to get a result

dhruv118:09:21

@bolivier and @witek if i understood it right, you want to define a function, compile it, and run that function in the repl with some arguments and then see if it’s correct

bolivier18:09:18

I think what @witek is saying is that the process for interactively writing a function isn’t smooth enough

bolivier18:09:42

eval function -> switch to repl buffer -> write test case for function in repl -> switch back to clj buffer and edit

bolivier18:09:58

I don’t personally know of a smoother approach

bolivier18:09:08

that wouldn’t litter the clj buffer with junk (which I’m sometimes guilty of anyway)

dhruv118:09:16

right. then what i use is: C-c C-c ;; C-c C-c runs the command cider-eval-defun-at-point, which is an ;; interactive compiled Lisp function in `cider-interaction.el’. write the function with it’s arguments in a clj buffer. C-c C-p ;;C-c C-p runs the command cider-pprint-eval-last-sexp, which is an ;; interactive compiled Lisp function in `cider-interaction.el'.

dhruv118:09:01

if i still haven’t understood the question properly and the approach i’ve defined isn’t what you were looking for, i’ll shut up 😛

witek18:09:33

@bolivier explaind exactly my problem. Usually I edit a .clj file with functions calling helper-functions, etc. Then I have to test the main function. Since I jump in the editor between main function and helper functions I need to eval the whole buffer. Then I need to call the main function with some args. I can not realy imagine an other workflow which would work. So why is there no command for evaluating the whole buffer and run the last executed function call again? Or is there?

bolivier18:09:52

can you test your helper functions individually?

bolivier18:09:08

probably lends itself to smoother development

jcsims18:09:54

I mean, it is emacs, so you can just create a command that does that 😄

munk18:09:41

@witek I usually have a comment block at the bottom of my file that contains the tests I’m working with. So I can eval the file, then play around with the statements in the comment block by evaluating them. That way I always stay in one buffer.

witek18:09:03

@jcsims Of course, but I am completely new to Clojure and Emacs. And therefor I think my idea about the interactive development workflow must be wrong. Everybody should have the same problem, right?

jcsims18:09:32

ah sorry, I shouldn't have been facetious

witek18:09:27

@jcmoore Do you uncomment your tests when testing and comment them when done? Or is there a way to execute the commented tests?

jcsims18:09:54

I guess I normally work from the inside out - smaller, independent functions are iterated on, and then work out from there

jcsims18:09:18

for tests, you can wrap them in a (comment ) block, and use C-c C-c to execute them

jcsims18:09:43

or maybe C-x C-e? I forget exactly

witek18:09:29

But to execute the tests inside (comment ...) do I have to move to the bottom of the file? Or is there a command which executes it from anywhere? What is the command name? I am using Spacemacs, so I have different key bindings...

jcsims18:09:22

cider-eval-last-sexp is what I tend to use

jcsims18:09:01

though my C-x C-e is actually bound to eval-last-sexp, that ☝️ fun is called

munk18:09:10

I think if you want to execute them from anywhere, you probably need to put them in a (deftest …) and eval the tests in that ns

munk18:09:37

but I find that I’ll write a function, then I want to play with different inputs to observe it, so moving to the bottom of the file provides that context switch for me

munk19:09:09

another option, if you have a “main” that you want to evaluate after your buffer is loaded, you could make that call the last statement in your file

witek19:09:18

Thank you guys very much. That are good starting points for me 😄

brett20:09:31

@witek When I first started with Clojure, I followed the blog post, Clojure for the Brave and the True, now a book. It has a good Emac and Cider primer. http://www.braveclojure.com/clojure-for-the-brave-and-true/.

smw20:09:50

I use the midje autotest functionality to have my (fast) tests run every time I change anything, from within the repl. I’m sure there’s some similar functionality for clojure.test.