Fork me on GitHub
#beginners
<
2015-11-27
>
catonano07:11:48

How do I write a huge file in clojure ? It's about 1.3 Gb

catonano07:11:40

I'm having an error OutOfMemoryError Java heap space  clojure.lang.PersistentVector$TransientVector.editableTail (PersistentVector.java:564)

catonano07:11:10

previously I used data.csv to write csv files. Now I'd like to write a file in a specific format

catonano07:11:38

I copied the data.csv function to write it (with the loop recur and all) but still...

catonano07:11:44

What am I not getting ?

ordnungswidrig08:11:08

@catonano: still around?

catonano08:11:55

@ordnungswidrig: here I am 😉

ordnungswidrig08:11:02

still need help?

ordnungswidrig08:11:55

can you post the complete stacktrace?

ordnungswidrig08:11:01

(as a snipped or gist)

catonano08:11:12

This is all I get user=> (core/main ) OutOfMemoryError Java heap space clojure.lang.PersistentVector$TransientVector.editableTail (PersistentVector.java:564) user=>

catonano08:11:27

I can post the code

ordnungswidrig08:11:46

are you using emacs?

ordnungswidrig08:11:10

you should see the stacktrace in the nrepl buffer

ordnungswidrig08:11:49

or call `cider-visit-error-buffer´

catonano08:11:12

I use Emacs only for editing. To run the thing I open a terminal. I just copied and pasted from the terminal. I am using "system" in order to have hot reloading though

ordnungswidrig08:11:59

So in the repl you can use (pst) to print the last exception stacktrace

ordnungswidrig08:11:32

you might want to (use 'clojure.repl) first.

catonano08:11:02

yes I got it. Just a minute, I put it in a gist

catonano08:11:37

I have to go for a little while. I'll be back soon

roelof08:11:38

Anyone else who can help me with this code : https://www.refheap.com/112123

ordnungswidrig09:11:59

@catonano: can you show a snipped on how you to the writing?

catonano09:11:58

@ordnungswidrig: sure, a minute

ordnungswidrig09:11:37

@roelof: #(+ amount) is a 0-ary function. you might want to use #(+ % amount) or even just (send account + amount) because send will call the f with any additional arguments you specify

catonano09:11:28

This is the vanilla version (not the one producing the stacktrace I provided you with, but the same error)

(defn writelines [destination-path lines]
  (with-open [out-file (io/writer destination-path :append true)]
    (doseq [line lines] (.write out-file line))))

catonano09:11:20

and this is another attempt

(defn write-ttl [^Writer writer lines]
  (loop [lines lines]
    (when-first [line lines]
      (.write writer line)
      (recur (next lines)))))

(defn write-ttl-file [lines]
  (with-open [out-file (io/writer "resources/processed-files/semantic-arpa.ttl")]
    (write-ttl out-file lines)))

(defn main []
  (write-ttl-file  (thread op/path)))

catonano09:11:38

the last version is the one producing the stacktrace I provided you with

roelof09:11:52

@ordnungswidrig: Thanks, finally got it working

roelof09:11:58

Amount is updated

roelof09:11:48

next problem. Make it work with a sort of description so the transactions get saved. I think I need a list of {} for that

ordnungswidrig09:11:02

@catonano: I the first version looks sensible. I does not hold the head of line. How is lines generated? Can you call (last line) without an error?

ordnungswidrig09:11:23

@roelof: sort of description?

catonano09:11:47

@ordnungswidrig: I try, just a minute

roelof09:11:32

@ordnungswidrig: I try to make a little financial toy app and I like to see that the user calls the function like this : (deposit 1000 "Withdraw from the bank on 01-10-2015 ". So later on I could make a report on all the transactions

catonano09:11:44

@ordnungswidrig: no, I can't call (last lines) without the same error showing up

catonano09:11:07

so the culprit is not the code writing the file !

catonano09:11:13

i just arranged some transducers, the I called (into [] my-transducer original-collection)

ordnungswidrig09:11:32

@roelof: i see, you might want to do sth. like (send account conj {:amount 100 :desc „money!“})

catonano09:11:23

I assumed that (into []... produced a lazy thing

catonano09:11:51

I'm using sequence now and it has written more than 2 Gb so far !

roelof10:11:13

@ordnungswidrig: thanks all

roelof10:11:07

now a really beginners question . What is wrong with my let here https://www.refheap.com/112129

catonano11:11:17

@roelof: I attempt a suggestion: the solution should be one of the arguments to your function, together with start and end. Like it is now, the solution gets reset to the empty vector at every iteration because the let is inside the loop

agile_geek11:11:01

@roelof: I would look at the ‘loop’ bindings. Think about what you are binding to start? Also that recur doesn’t look quite right either

roelof14:11:10

For start I need to bind the start and end value together with a empty []

roelof14:11:09

the next iteration when start is not equal to end , start needs to be conj with the empty [] and then increased

roelof14:11:34

after that start, end en the new [] needs to be in the next run

roelof16:11:01

problem solved in the clojure channel

seancorfield18:11:23

Welcome @adam_cameron !

adam_cameron18:11:06

G'day @seancorfield. Guess I kinda have to start putting my money where my mouth is now!

seancorfield18:11:49

No excuses now you’re here simple_smile

seancorfield18:11:25

Nando is here too. You should badger Andy to join as well.

seancorfield18:11:23

And I see Chris just joined too...

adam_cameron18:11:28

@seancorfield: I will, trust me

seancorfield19:11:48

@roelof: I’m converting my friends to Clojure, one parenthesis at a time...

seancorfield19:11:59

…well, maybe a matched pair of parentheses at a time!

adam_cameron19:11:09

Well let's not say "converted" yet 😉. "Assessing". That said, I just wanna learn something that's structured so differently from all the rest of the bumpf I've programmed in

roelof19:11:41

@adam_cameron: which languages did you programmed in if I may know

roelof19:11:56

Me myself come from the ruby world

adam_cameron19:11:24

Mostly CFML. Moved to PHP a year or so ago. And doing web stuff, a chunk of JS too. I've had a bit of a look @ Ruby, but only experimentally.

adam_cameron19:11:40

Only got as far as "G'day World" with Clojure thusfar.

roelof19:11:52

I would suggest the clojure koans , I learned a lot of then

roelof19:11:05

you could even do them online

adam_cameron19:11:17

CFML got a very bad rap in the last decade. A lot of it was justified.

roelof19:11:40

where does CFML stands for ?

adam_cameron19:11:46

Oh... ColdFusion?

adam_cameron19:11:00

The Allaire->Macromedia->Adobe thing

roelof19:11:12

oke, I never did these

adam_cameron19:11:14

I was gonna start looking @ Project Euler

eggsyntax19:11:16

Cold Frontin Markup Language

adam_cameron19:11:02

CFML is pretty niche. <cfand> <cfspent> <cfa> <cflot> <cfof> <cftime> <cflooking> <cflike> <cfthis>

adam_cameron19:11:11

(which - rightly - puts ppl off)

adam_cameron19:11:43

But the koans thing could be good too actually

roelof19:11:31

and if you are ready with those. We have 4clojure for some 200 - 300 exercises from easy to very hard

roelof19:11:45

im now with the easy ones

nando19:11:51

I’ve found Living Clojure to be a very good introductory text. The glide path is designed for beginners, not to steep and yet well paced so you get a good overview of the language’s main features in a relatively short amount of time.

nando19:11:31

The second section of Living Clojure is a “Weekly Training Plan”. I’ve appreciated the structure because I get easily enamoured/distracted with the higher level concepts. Using Clojure is a completely different matter. Hitting walls trying to get through “simple" http://4clojure.com koans is the most instructive thing that has happened to me so far.

nando20:11:16

The other really good resource I’ve found, at least for me, is http://www.purelyfunctional.tv.

roelof20:11:22

nando: you are right. I hit the wall very often on exercises of 4clojure but they learn me also a lot

roelof20:11:19

Some easy ones costs me a day but after that I learned why I do not have to use a function but better use another one

roelof20:11:09

I think clojure is a language you can learn the best by making your hands dirty

seancorfield20:11:28

I mostly learned by doing when I first got started with Clojure. That means some of the oldest Clojure at work is not very idiomatic but we refactor as we go so it all gets cleaned up eventually.

seancorfield20:11:57

And now we have three Clojure engineers working on the code base so that helps establish a better "group think" of what makes readable and idiomatic code.

nando20:11:38

Eric, the guy behind http://PurelyFunctional.tv, is an excellent instructor. It’s easy to get bogged down. He pulls you out of the mud and keeps things light with his video instruction.

seancorfield20:11:29

This week I’ve been removing with-redefs from our tests since we came at things with a very OO / mocking mindset and we’ve survived with that for a while, and now we’re cleaning that up.

nando20:11:59

@seancorfield: It must of been tough learning Clojure without much in the way of resources. Obviously, you were up for it!

seancorfield20:11:32

A lot of the languages I’ve learned over the years have mostly only had the language reference docs available simple_smile

seancorfield20:11:52

And, yes, Eric is a great instructor (and an all round nice guy).

nando20:11:56

Anyway, I find it helpful to bounce back and forth between solving Clojure koans and Eric’s videos, getting stuck, hitting the wall, and then watching someone else explain and code. Keeps me moving forward, slowly ...

nando20:11:13

I’m also working through the om.next tutorial (and wondering what experienced folks think of om.next vs reagent.

roelof20:11:06

nando where can I find that one

roelof20:11:32

it looks something I can use for a toy project I have in mind

roelof20:11:07

at this moment. Im doing the 4clojure challenges and I try some things with different databases

seancorfield20:11:40

I don’t know that anyone has enough experience with Om.Next yet to compare it to Reagent. Based on what I saw at Clojure/conj, I think I’d still prefer Reagent — Om.Next still looks more OO and a little too opinionated for my taste but it’s a big change from Om in several of its core concepts.

nando20:11:44

What I like about om.next from what I understand (purely on a conceptual level) is that it is not necessary to build a complex nested data structure for a complex nested ui.

nando20:11:20

I have a use case where jQuery simply does not work well enough.

nando20:11:22

It should be rendered using React, one way or another. It will still take me some time for my coding skills to match my appetite … tho’.

nando20:11:53

I’ll give reagent another look after I’ve gotten through the om.next tutorial.

nando20:11:52

@seancorfield: By the way, you mentioned in the javascript slack channel (before you left it) that your team had built a proof of concept using reagent and some other library, which I vaguely remember handled data transport or some other aspect of the data. Your comment was that “it was sick” - which I took to mean “really good”. Which library was that? I’d like to look into what you worked out ...

nando21:11:35

@roelof: I’ve also taken more than a day (several evenings) trying to get through an “easy" 4clojure problem! Humbling.

seancorfield21:11:15

@nando: We used Sente which provided core.async channels over WebSockets

nando21:11:55

Ah, ok. Now I remember. This time I will write it down! Thanks

roelof21:11:55

@nando: im now thinking how to flatten a seq without using flatten, that one im not allowed to use

seancorfield21:11:03

That meant that we could just use core.async for pushing messages around the application — both client and server — without worrying about actually making API calls.

nando21:11:12

@roelof - which problem is that?

seancorfield21:11:43

Everything was asynchronous: a component could ask for data by sending a message containing a "tag" (representing where the data should go in the client state) and a "query" (whatever the app understood); the server would read the message, dispatch it based on the contents of the query, that would lead to a message being sent back to the connected client(s) containing the data, which would cause the client state to update (which would trigger a re-render).

roelof21:11:13

no hints please at this moment