Fork me on GitHub
#beginners
<
2017-04-19
>
sb07:04:37

Where can I find great RabbitMQ tutorial?

sb07:04:07

@timok thanks, I saw that. I would like to see a real example how to use etc like other reps

swizzard14:04:51

i’m trying to write a web-scraping thing, and i think i’m shooting myself in the foot with my use of transducers, but i’m not 100% sure how to fix it

swizzard14:04:31

the site i’m trying to scrape has urls like “foo.html”, “foo-1.html”, “foo-2.html”, etc

swizzard14:04:31

there’s some number x, such that “foo-x.html”, “foo-`x+1`.html”, etc., redirect back to (i think, it’s been a minute) “foo.html”

swizzard14:04:57

so i wrote this:

swizzard14:04:59

(defn redirect? [url]
  (try
    (let [redirs (> 0 (-> (client/get url) :trace-redirects distinct count))]
      (do (println redirs)
          redirs))
    (catch Exception e true)))


(defn filter-redirected
  "Takes a url string and returns it if it doesn't redirect
   :param url: the address of the resource to retrieve
   :type url: String
   :returns: String or nil"
    [xf]
    (fn
      ([] (xf))
      ([res] (xf res))
      ([res url]
       (if (redirect? url)
         (reduced res)
         (xf res url)))))

dominicm14:04:03

@swizzard I may be being a bit thick here, any reason to define your own transducer, and not use (filter redirect?)?

swizzard14:04:35

@dominicm the frustrated lashing out of a man in over his head?

swizzard14:04:02

i also tried (take-while (not redirect?)) but that didn’t seem to help?

dominicm14:04:18

(not redirect?) is always false iirc.

dominicm14:04:38

You may have been looking for:

dominicm14:04:04

but you get the idea

swizzard14:04:46

unless there’s a bug in redirect? (which there could be), the source of the problem might be my use of sequence + (range)

dominicm14:04:14

@swizzard Once you no longer use a custom transducer, it's much easier to check if it's in your redirect? function.

swizzard14:04:34

like i said, the custom was not my first choice

dominicm14:04:56

I think you want (remove redirect?) as your transducer. Try that.

swizzard14:04:24

i don’t want remove

swizzard14:04:49

because if “foo-x.html” is a redirect, “foo-x+1.html” will also be a redirect

swizzard14:04:20

i’ve been playing in the repl, and i think clj-http’s {:follow-redirects false} might be the answer to my prayers

giaosudau14:04:12

Hi everyone, I am just learning clojure for 2 weeks.

(defn get_videos [page-number order-by & query]
  “Get videos”
  (let [order-field-name (first (str/split order-by #“-”))
        rank (second (str/split order-by #“-”))]    
    (if (not-empty query)
      (select entities/videos
              (with entities/artists)
              (limit  limit-by-page)
              (order (keyword order-field-name) (keyword rank))
              (where {(keyword (first query)) (second query)})
              (offset (*  (- page-number 1) limit-by-page)))
      (select entities/videos
              (with entities/artists)
              (limit  limit-by-page)
              (order (keyword order-field-name) (keyword rank))
              (offset (*  (- page-number 1) limit-by-page)))      
      )
    ))
here is the code I wrote for a function to query data from database using korma. Is there anything I can improve on that? Thanks.

timo14:04:52

how would you set up a projects architecture when you want to use reagent for clientside and plain clojure for backend?

dominicm14:04:18

@timok You might want to check out the saapas template 🙂

timo14:04:48

@dominicm that's boot, right? I would prefer to stick with leiningen

timo14:04:54

but thanks

sveri14:04:46

@timok this is a leiningen template for exactly this: https://github.com/sveri/closp. I just put frontend code in src/cljs, backend code in src/clj and shared code in src/cljc

timo14:04:53

looks good, thanks!

timo15:04:52

@sveri why does it say migrate is not a task?

sveri15:04:02

@timok I am currently reworking everything and stepped away from migrations. It will take some time until I make a new release and update the documentation. So, migrations are not supported anymore, sorry. If you need that, take a look at luminus: http://www.luminusweb.net/ which supports them AFAIK.

timo07:04:15

Hi, can you tell me what you are reworking?

sveri08:04:26

A lot of stuff, I removed korma and use plain clojure jdbc. I switched from selmer to hiccup. Switched to a different i18n library and added translation for german / english. Switched to immutant from http-kit, introduced clojure.spec and therefore use clojure 1.9-alpha xx. And some more. Currently I am totally rewriting closp-crud to be able to define entities differently and some more to come. The biggest difference between lumins and closp is that I provide one set of dependencies, I test everything on postgres, for instance, while luminus supports more databases, etc...

timo15:04:59

so I can leave that step in your tutorial?

sveri15:04:37

Yes, you can leave that out

curlyfry15:04:24

Luminus is pretty awesome even if you're not using it. You can get a lot of inspiration by looking at the set of libraries it uses + its architecture + the docs

sveri15:04:29

While I agree with this in general (based my template on luminus more than two years ago) there are still decision which might not fit you. For instance, luminus is based on mount, while there are many that prefer component.

donaldball16:04:02

The disagreement between component and luminous is well-trod ground, but to summarize for new folk: luminous favors implicit, ambient state with a var for each stateful thing and provides fns to manage their lifecycle; component favors explicit state with at most one var for all stateful things and provides fns to manage such a system.

didibus06:04:03

donaldball: Does luminous forces you into that? Or is that just the effect of Mount vs Component? Normally with mount you don't have to be implicit. You can write everything in a way where they take the state. Could you do that with luminous too?

donaldball11:04:09

Beg pardon, I should have said mount above, not luminous. And yeah, there’s nothing that forbids you from writing fns that are, if not pure, at least fns of their args, in a project in which mount manages application state.

didibus16:04:28

Ah ok, ya I thought maybe because of some luminous conventions it forced you more so than just regular mount. Thanks for clarifying.

swizzard18:04:37

can i e.g. write to a db within a transducer?

swizzard18:04:58

or will the multithreadedness give me problems?

noisesmith18:04:10

typically the role of a transducer is to isolate logic from context, so my gut feeling is that putting side effects inside one is a mess (even if it behaves correctly)

ghadi18:04:57

what noisesmith said

swizzard18:04:35

do you have any solutions for doing something like that without having to realize the whole sequence at once?

noisesmith18:04:33

make a transducing context that writes to the db as its data destination, perhaps?

noisesmith18:04:51

that might be silly

swizzard18:04:13

(transduce xf write-to-db coll)?

noisesmith18:04:39

transduce is eager though

noisesmith18:04:05

but you could leverage sequence, or make a new lazy transducing context with the behavior you need (maybe?)

noisesmith18:04:31

this is very speculative, there's probably a smarter and simpler way to do this, sorry

swizzard18:04:48

eh, it’s still helpful

noisesmith18:04:13

seeing how you phrase the question on the #clojure channel, you can use sequence - sequence uses a transducer and is lazy, and you can decide how much of it to consume in your db-writing function

mss19:04:07

is it possible to deref a delay with a timeout?

mss19:04:44

when I run something like (deref (delay true) 100 false), getting an error java.lang.ClassCastException: clojure.lang.Delay cannot be cast to java.util.concurrent.Future

mss19:04:50

is that because the delay is already resolved?

tbaldridge19:04:51

no, it's because you're using a delay

tbaldridge19:04:07

and delays don't accept timeout derefs

mss19:04:47

hmm, say a delay is coded up to take an interminably long amount of time. are they un-timeoutable?

mss19:04:24

thank you for the response btw

tbaldridge20:04:57

correct, since delays run on the thread that derefs them it's hard (impossible?) to code them in a way that they can be timedout.

tbaldridge20:04:05

(future ...) can be timed out however

mss20:04:06

ah that makes sense, thanks for the help