Fork me on GitHub

I want to write a simple web crawler, how to send request with cookie in Clojure?


@vale This is how I get around the issue:

(def data-source
  (when-not *compile-files*
    (make-datasource options)))
*compile-files* is set when AOT is happening and prevents the eval.


is that some magic variable or do i need to set it myself?


Its a special variable set in the environment by Clojure. Have a look at this:


cool, thanks!

😄 4

Is there any way to use to trigger a reload of a namespace whenever a configuration file changes? Say I have a “resources” directory with “selectors.edn” in it, is there a way to automatically reload the namespace that reads that config file in whenever it is changed?


Nevermind, I found a small library that I was able to use to cause a reload of the config namespace on resource file change:


Hi everyone, I wander to know how you are setting up your development environment, particularly if you are using Visual Studio Code. Any input would be greatly appreciated 🙂


I think calva is the way to go with VSC


But personally I'm using Cursive now after being on atom for about a year and a half


I’m trying to pipeline a stream of XML. What I have was working for smaller files, and I was under the impression that I was streaming, but everything locks up when I pass it a very large xml file. Here’s a snippet of my let block

content [( ( filepath))]
          from (a/chan 1 (comp (mapcat :content)
                               (filter #(-> % :tag (= :TagOfInterest)))))
          _ (a/onto-chan from content)
          to (a/chan 10)
          _ (a/pipeline 1 to xf from true ex-handler)


it never get’s into the pipeline transducer xf


I assume you are simply consuming from the to channel in a loop?


also, what does xf look like?


xf just does a computation transform (map (fn [x] (some-transform x))


when you say consume, do you mean from to? If so I’m just piping like this (also snippet from let block)

res-ch (a/promise-chan)
          into-ch (a/into [] to)
          _ (a/pipe into-ch res-ch)


is the xml parsing fully realized before going through the channels?


when it was working on smaller sets I assumed it was, but now that it locks up on larger files I assume I’m missing something


the doc string for says it is lazy


> “Parses the source, which can be an InputStream or Reader, and returns a lazy tree of Element records. Accepts key pairs with XMLInputFactory options, see and xml-input-factory-props for more information. Defaults coalescing true.”


whoops sorry, yeah, what you said


exactly, so not sure why it’s freezing up now :thinking_face:


because you are doing io on the core.async threadpool


I mean, I dunno, I am not sure it should be freezing, but io definitely should not be done on the core.async threadpool, processing xml records which are lazily realized from a file like that is a no go


so I was under the impression that the reading and parsing was not happening on the core.async threadpool. parse returns a lazy sequence, which is then spooled onto the from channel with onto-chan, which should get back pressure because from has a finite buffer, 1 in this case


there are a lot of faulty assumptions there


onto-chan doesn't examine the contents of the collection, so the result of xml is not traversed and not realized by onto-chan


onto-chan is also implement using a go loop, so even if it did, it would be doing it on the core.async thread pool


and xml is a linearized tree structure, so parsing it lazily can get weird


ahh. thanks @hiredman, that helps


I am not sure that would cause it to lock up, because the io from the file, while technically blocking, should be guaranteed to make forward progress and eventually complete


hah! i got it working


^ s/it’s/its


whoops. messed up last code block snippet


calling mapcat on [x] is slightly odd - would you have multiple items in your real use case?


pipeline has a variant, pipeline-blocking that safely performs ops in a threadpool outside the core.async go block pool, but you would still need to consume the channel from pipeline to drive consumption


ah, yeah, sorry. that’s left over from when I was exploring via (def children (mapcat :content)) which it’s nicer to use mapcat when you’re calling children repeatedly. totally unnecessary in this context though, you’re right.


in this case there’s currently no io going on in the transducer, xf, that’s being passed to a/pipeline, just a simple projection


OK, right, I'm just saying don't expect xf to run against the data if you aren't consuming from the pipeline


which you may or may not be doing given what you shared, just thought I'd mention


ah, right you mean from the end of the pipeline? yeah, sorry. super snipped; we’re pulling from to


@hiredman even though I got it working, I am going to take your advice and make a version of onto-chan using thread instead of go


none of what I have seen fully realizes the xml structure (as far as I can tell), it won't matter how the structure is put on to the channel if some go block down the line traverses down the xml tree to some part that hasn't been realized yet and then does io