I have to parse a 2G xml file. Large but not huge by big data standards

I’m parsing it using clojure.data.xml, which I thought would be lazy, but it’s chewing up huge amounts of memory

starting the jvm up with 12G means I end up using all of it on the parse process and it doesnt return

what’s the best way to parse an xml file of this size?

First of all, I'm so sorry for your plight 😉

I can't speak to the Clojure xml libs, but there are certainly streaming apis in Java that are more than performant and memory friendly.

hrm ok

Regarding data.xml, are you using the lazy marked functions? http://clojure.github.io/data.xml/

I’m using this: http://clojure.github.io/data.xml/#clojure.data.xml/parse

how do I tell if something is marked as lazy?

Parses the source, which can be an
InputStream or Reader, and returns a lazy tree of Element records

should be ok. And, btw, thanks for your help here

here is the code loading the file

(defn load-feed
    "Loads the given filename"
    [filename]
    (-> filename
        
        
        clojure.data.xml/parse))

thanks, that helps. What are you doing to cause the memory symptom? Are you printing to repl (that won't be lazy)

ha, yeh, I am actually

Im sitting there in lein repl and running the function

sorry, totally me being stupid

Not at all, it happens.

Try to def the result and watch mem consumption

thanks, now I can just test sending it downstream into firebase

well ...

I am turing it into a zipper

and then parsing it into a representation of lists and sets

but we’ll see if that’s a problem

Let us know. You should be good: (-> "<foo><bar baz=\"42\" /></foo>" x/parse-str :content class) gives back clojure.lang.LazySeq

I admit I'm not super familiar with the peculiarities of the library. (But there's always Java, yippee!)

yeh, I dont want to rewrite this

Im just building my clojure service now and will see if it can stream the data

hrm, testing loading my large XML file in an exploded war is leading to java.security.AccessControlException: access denied ("java.lang.RuntimePermission" "accessClassInPackage.com.sun.xml.internal.stream”)

I have the xml file in my /resources directory. When I build it into a war it ends up in WEB-INF/classes

shouldnt I be able to load it using ?

hi, Im still trying to identify my bottleneck here. I am loading a 2 gig xml file, building a zipper on it, then attempting to take the first 5 elements from it and println a token

just to represent that I am not loading the full 2 gigs into memory anywhere

but somehow I am

https://gist.github.com/nicholasf/e8309e6b2ffea364654f

Reading logging and debugging from https://github.com/Day8/re-frame#logging-and-debugging and Talking To A Server section. With regards to replay, if I log anything sent to dispatch such as [:initalize], which is handled by a http call to setup initial db state, then dispatches a [:populate-db], if I replay I'm going to see two [:populate-db] one caused by log replay and one caused by http call of initialize.

nicholasf, I dont think clojure.data.xml/parse is SAX style

yeh, I just switched to golang and did what I wanted to do in 5 minutes after not thinking about it for 6 months

me just sucking at clojure I guess

the documentation for parse is confusing it does mention something about SAX, but i guess XML gets less attention nowadays given the uncoolness

I spent days understanding how to work with large xml in clojure

http://blog.korny.info/2014/03/08/xml-for-fun-and-profit.html#zippers

I learnt a lot about clojure but reached zero results, really

can parse smaller files

Im pretty sure if Id just broken out a sax parser and used java interop I would have been fine

The transit github made it to #1 on HN right now! Is there a reason why its topical today?

Is there any way to get a projects version while running code inside project.clj?

I know about the System/getProperty trick, but that doesn’t seem to be set at the time I’m running code inside an :alias in project.clj, I think that only applies when you’re evaluating code in the context of your app.

@nicholasf: dealing with laziness can be tricky - but I still far prefer to use a lazy language to parse xml than sax! It really depends on the XML though. Wikipedia is an overly-simple example, but it was handy as it’s huge and easily avaialable.

@danielcompton: what about something like (-> "project.clj" slurp read-string)?

@korny: I think a lot of the documentation was misleading

@sander I was trying to avoid that route

unfortunately Im back working in golang again

@danielcompton: why? the edn structure of project.clj is well-defined

it’s not really edn

@nicholasf: True - or just very terse. I got far more value browsing the code than the docs. And it’s a pain as it’s split over several libraries.

@cfleming: no worries, at this point I'm fighting more with nrepl than cursive. 😉

@thheller: Yeah, for CLJS I think that’s pretty common!

I’m planning to provide direct CLJS REPL integration with no nREPL in sight.

oh nice, that is what I was thinking as nrepl is pretty JVM specific with all the bindings and stuff

which CLJS just doesn't need

but I still want "Run test ... in REPL" for CLJS

Right.

You’ll get it

Along with a test runner.

sweet

And a debugger, at some point.

No promises on that one, though

hehe

Your custom REPL is going to be cljs.repl compatible, right?

would it be possible to get a webview in intellij? that would be sweet too

cfleming kind of yes and no

So it’s possible, but it’s complicated and involves a commercial component. But I’m planning to try it out.

the cljs.repl is modeled after the clojure one (ie. streaming)

It embeds Chromium and presents it in a Swing view.

I don't think that is the best approach and even things like piggieback have to work arround tricks to get it going

yeah figured it was tough to get a native component into swing

Ok. It’s a long time since I looked at this, I’ll try to look soon and refresh my memory.

you can have them side by side, probably to much work to deal with devtools and other stuff

I’d like to sort out the CLJS REPL soon since it’s a pain point for a lot of people.

https://www.teamdev.com/jxbrowser

There’s an OSS one but the price on that is pretty reasonable and people speak highly of it.

interesting

You could do some really nice things with that, though.

In general it’s much easier to make a zero-pain option when it’s all integrated.

yeah, would definitely be cool

very curious what you come up with for CLJS

I was just playing around with my REPL implementation which is now at figwheel level and working quite nicely

Nice, is the code open?

I’d like to take a look.

yes, just a sec

I’d be interested if you have ideas about what an interface to a CLJS REPL should look like. Ideally I’d provide an interface that a REPL implementation would provide an implementation of, so that Cursive could fire up different impls.

Although I’m not sure how much I need to make it generic, since there’s probably not a lot more custom REPLs out there.

That said, there’s already Figwheel, cljs.repl and yours so there’s clearly scope for different takes on tooling.

I split the repl/figwheel'ish parts into its own project

since they bring in some deps I didn't want in shadow-build

https://github.com/thheller/shadow-devtools

Nice, thanks.

I’ll take a look, it sounds very interesting.

run a clojure.main repl

(require 'build) and (build/browser-dev)

that will run the browser repl

Cool, thanks

open test-project/public/index.html in a browser to connect the repl

actual implementation bits are still very much in flux

Ok. It’ll probably be a while before I can look at it seriously, definitely post-conj.

But like I say, it’s something I want to sort out soon.

the interface to the compiler is pretty much final though

none of it deals with actually running stuff

you just feed it strings of cljs code

it gives you back the required actions to run in the client

https://github.com/thheller/shadow-build/blob/cdc570159ec5c8a1aa71a6e12c5151c043ebc7f5/test/shadow/cljs/build_test.clj#L477-L485

that are the compile bits

I outlined how that works in my email

that is unchanged

Yes, I remember

So you get back a JS snippet?

js snippet or information yes

Ok.

eg (require ...) tells you which files need to be loaded

but doesn't have any actual js to eval

Oh ok.

That sounds reasonable.

Ok, I have to get to bed, but I’ll copy all this and look at it when I get a chance.

Thanks for the info!

https://github.com/thheller/shadow-devtools/blob/0efa4feeb8485d2c1738d3692eb1d666db1c9e85/src/cljs/shadow/devtools/browser.cljs#L166-L173

those are the client repl bits

they will look pretty different for the node repl

@cfleming cool cool, sleep well

Thanks, seeya

Any ideas in compojure-api, why does PUT* ignore :body while POST* reads it? Both seem to read :body-params though.

Nevermind.

hi

I am new to clojure, and I got a question I have not being able to find a definitive answer on google

I work with a big java desktop application, where I have managed to embedd jython to provide a scripting framework

it works well, but I hate java scientific libraries, jython cannot use numpy and Co. I got interested in Inchanter instead.

Now I am wondering if I can also embedd clojure into the java app in the same way as I did for jython.

What I do with jython, is on java, I create an interpreter, set paths to different modules I need dynamically

then I put a 2D matrix (a list of lists) into the interpreter

and then I load the script I want and execute a function that will always be in every script by convention, and which takes that 2D matrix I put into the interpreter

the function returns another 2D matrix, which then I can recover from java side and cast to java types

and then some other small things like redirecting jython stdout and stderr to java so that I can print nicely on the java app, etc

Would it be possible with clojure? Somehow I got the feeling it is not designed for this purpose.

@sidrero: I don’t have the final answer on this because I’m not very familiar with the environment, but my understanding is that since Clojure compiles to JVM code, mixing Clojure and Java is much simpler than what you had to do with Jython.

@sidrero: I don’t know how you make a Clojure / Java hybrid project, but Clojure can use Java libraries and viceversa.

makes sense

I guess the only problem is that for jython they provide very clear examples on how to do this, while for clojure the documentation is not there, or at least I have not been able to find it

@sidrero: when I was working at Google, as a 20% project I wanted to figure out whether we could use Clojure to build apps using Google’s infrastructure. I don’t remember building a mixed project, but long story short, I was done within a day, having built sample apps that touched on the various internal databases, deployed them to our staging infrastructure and so on.

the Clojure->Java interop is pretty straight forward and I’ve made many mixed projects (requires some fiddling with the build process though). The other way around (calling Clojure from Java) I’ve not much experience with, but should be possible. Maybe this helps? https://stackoverflow.com/questions/2181774/calling-clojure-from-java

sidrero: how do you build your java apps? what build tool do you use?

also for scientific computing we pretty much decided to use Python and call it from Clojure using IPC (over ZeroMQ usually)

joelkuiper: I think more than the actual code, he needs the build tool chain.

ah I see, sorry!

I have to do an eclipse plugin, because the java app is an eclipse app

sidrero: I think this might help, if you are using Maven: http://alexott.net/en/clojure/ClojureMaven.html

sidrero: and this might help if you wan to have java in a lein (clojure) project: https://github.com/technomancy/leiningen/blob/master/doc/MIXED_PROJECTS.md

but for the scripting, the idea is not to compile the script as in the stackoverflow example, but load the script and interpret in on the fly

so that hte user can do changes to the script and see the changes immediately

eval should do

sidrero: in that case, yeah, just add the clojure jar to your project and call its eval from java.

I see

As joelkuiper said.

sounds promising

sidrero: something like this: https://clojurefun.wordpress.com/2012/12/24/invoking-clojure-code-from-java/

does clojure have a mechanism to load a whole script and then pass it to eval? or I should read the script with java, build a string, then pass to clojure eval

I mean, in jython the only thing I have to do is to import the script

no need to run it manually

sidrero: it’s definitely possible and I would bet the results are much satisfying than Jython (not due to Lisp being great, but due to Clojure targeting Java).

I am willing to try

IFn eval = Clojure.var("clojure.core", "load-file");
eval.invoke("my-script.clj");

sidrero: what about this: http://threecrickets.com/scripturian/ (from https://groups.google.com/forum/#!topic/clojure/slFNKpw1gLk)

smth like that should help

that looks great

Anyway, I have to go now.

thanks pupeno

just thinking out loud here (no idea what your project actually is …) but if it’s some sort of scientific project you can also try and start the iPython notebook as a subprocess and call that over WebSockets 😛 as Python science libraries are still the best out there imho

no, you need to have a server with python etc, and I don't have that

well, yes, but the story is complicated ... not a real option anyway

k

if I do what ul suggests

which is more or less what I want

except that I need to pass a 2D array as argument to the clojure function I want to run

how do I pass this argument, and how do I cast java types to clojure types?

cast them in clojure code

Incanter should have appropriate funcs

alternatively take a look at https://github.com/mikera/core.matrix

and if I would like to hide this complexity from my users? they should see only clojure types in and out

can I cast a java Object[][] to a core.matrix?

probably, or otherwise with a straight forward loop

wouldn’t starting a Clojure REPL be an option?

core.matrix works just fine on Object[][]. And on nested Clojure vectors like [[1 2] [3 4]]

All the different types are handled by protocols, so the translation is pretty seamless

sounds good

But you can convert if you want, something like (coerce :persistent-vector (object-array [1 2 3])) => [1 2 3]

@mikera: I have a project somewhere on the todo to write an APL interpreter using core.matrix (including all the funky symbols) so I’ll take this moment as an opportunity to say thank you for the excellent library!

this scripturian thing looks really cool

And the reverse works, e.g. (coerce :object-array [1 2 3]) => an Object[] array containing the longs 1, 2, 3

Haha no probs. APL was a big inspiration for me!

Should be a fun project to do Instaparse + core.matrix => APL on the JVM

my thoughts exactly!

mikera: regarding matrix, what if I have an Object[][] where sometimes I have numbers, sometimes strings and sometimes dates, would matrix cope with that?

Yup. The following round-trip works just fine: (coerce :persistent-vector (coerce :object-array [[:a "Hello"] [1 nil]]))

Also check out the "dataset" functionality, which allows named columns etc. Useful if you are importing DB tables / CSV files etc

that's great!

thanks for all the suggestions by the way, it is very cool to see such a lively community ... as comparison jython chat is mostly dead and it is difficult to get any help

Question about components: Let's say that I have a bunch of components that access a database, I have the connection set up as a component in the system. That's all fine, but I have all of my database access code as a function in some namespace. I can require this wherever I want, but the purpose of components seems to be to keep things like that from happening (having my database functions wrapped around all my other code like tentacles). Thoughts on how I might clean that up?

Also, kind of related, let's say I have a component with a bunch of workers. These workers get assoc on to the component and may need restarting, or to be removed. Is it a bad idea to change what's in a component after it's assoc'ed on to the system?

@jstew: (:require [my-db-ns :refer [my-get-fn my-upload-fn]]) is too messy?

swizzard: Not really. I've been spoiled by my last project. Everything was pretty self contained, passing data back and forth between core.async channels, and not using too many things from other namespaces.

Let it be known: untyped PHMs are now known as hashbags

cc @mjroghelia

@jstew There's a lot of options there; for example you could have a component that owns the database connection. It may even store the connection in an Atom (as part of, say, a failover strategy). So inside that namespace are functions that access the database, expecting (by convention, as the first parameter) the component ... from which is extracted the active connection. If this really irritates you, you could define a kind of derived component, that is just a map of partial functions. But still, if you were using Java and a DI container, you'd be invoking dbComponent.writeToDatabase(myRecord) and here it's just (write-to-database db-component my-record).

map of partial functions: i.e., {:write-to-database (partial write-to-database db-component)} ... seems like a lot of work.

@jstew: it is valid for components to encapsulate some local state, typically stored in Atoms (or some other mutable ref type). I've used that for things like caches, tracking failover state, even some orderly shutdown code. There is a debate about whether that works best as one-global-atom-to-rule-them-all, or distributed state in Atoms inside components. I favor the latter in my experience, but there's something compelling about being able to see all application state in one place, as a kind of aid to debugging.

@hlship: Thanks! That's exactly the type of input I was hoping for. I'm going to try going with local state encapsulated by components.

let’s give some love to the idea of proper deprecation mechanism in Clojure

http://dev.clojure.org/jira/browse/CLJ-706

^^ upvote this ticket to influence the future for the better

that was the first Clojure ticket I ever worked on :tear:

I have a small question regarding clj-http. How can I emulate a curl post that does this? curl -X POST 'https://website.com/endpoint' -d 'request_body={ ... }'

I have a question for those using Pedestal: Is there a way to prevent a route handler from being called? For example, I am writing an authentication interceptor that will assoc :status 401 to the context if the user is not authenticated but the problem is that the handler still gets called. To reduce load on the db/server, I do not want the handler to be called unless the user is authenticated.

nnbosko: should be a lot of examples on the clj-http readme

@nnbosko: clj-http.client/post "" {:query-params {...}}

Nothing with that case in particular @ghadi

and query-params doesn't do it either @jstew

@nnbosko: Obviously you'll have to change the accept headers if it's json.

It's expecting -d 'request_body={ ... }' which is where the problem is

not -d '{"request_body" : {...}}' but -d 'request_body={"data1" : "val1" ... }'

@nnbosko: I'm assuming you tried (http/post "" {:body "request_body={\"data1\" : \"val1\"})

nnbosko: see https://github.com/dakrone/clj-http#input-coercion Body as string specifically

what pbostrom said

If you really have to go low level you can use clj-http/request and set the body yourself.

Yes @pbostrom

hmm I'll try clj-http/request

Either way you're setting the body the same way as the 2 other examples though, so I would expect the same results.

As long as it's not https, you could use something like charles proxy to see what the difference is between the curl post and clj-http. That might give you more insight into what's going on with your request body.

@nnbosko: curl sets the content-type to application/x-www-form-urlencoded automatically, clj-http doesn't

hello!!!

does anyone know of a good way to set up a local maven proxy?

I'm building docker containers and each one downloads the same jars over again

each time it's built!!!

maybe it's as simple as mounting the .m2 directory (hoping)

We usually build our jars on our machines and have docker build using the uberjar that’s generated from that for local dev.

huh

that's interesting

do you have a build server do the same?

and that probably keeps the container leaner

@ericnormand: we mount the local .m2 directory