This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
- # admin-announcements (7)
- # beginners (37)
- # boot (36)
- # cider (8)
- # cljs-dev (16)
- # clojure (155)
- # clojure-hamburg (1)
- # clojure-russia (2)
- # clojurecup (21)
- # clojurescript (69)
- # clojurex (12)
- # core-async (2)
- # datavis (3)
- # datomic (5)
- # devcards (4)
- # events (2)
- # hoplon (15)
- # lein-figwheel (45)
- # off-topic (22)
- # om (77)
- # re-frame (8)
- # reagent (7)
- # slack-help (1)
@andrewboltachev: just a tip, having an upload folder as part of your uberjar could be troublesome, I wouldn't do that
@andrewboltachev try to find the answer in the Java / JVM ecosystem, it will be the same for Clojure for this kind of topic
@andrewboltachev: https://ring-clojure.github.io/ring/ring.util.response.html#var-file-response could help maybe. Either as-is or by lookign at the source.
#C03RZGPG1 appears to have not seen my query, so I would like to see if I have some input as to where I should look to ask
Now, assuming I have a vector of maps that are all of the same structure, what would be the best approach to processing that vector so that I can write to database? Ignoring the type of database
(Not looking for implementation details but more on logical and high-level thinking)
@trancehime: I know it sounds like @jaen is ducking the question but there are a number of options and it really depends on your use case. For example, is the vector likely to be large, does 'processing' (upsert?) the maps involve considerable latency, does the processing have to be 'synchronous', is it strictly transactional in nature....etc.
Do you need every map in the vector to be 'processed' as one transaction? I.e. all succeed or all fail?
@agile_geek: well, that too, but I was thinking of the somewhat narrower architectural question - do you want to translate keys between db and Clojure, only change it to kebab case, or also change names, are you content with Java dates or want JodaTime. But those are good points as well.
I mean, there will be cases where I'm going to be working with a sizeable chunk of data at once
I have to admit to not having a huge amount of experience about doing this myself in Clojure (although huge amounts in other languages at scale).
@trancehime: is this a one time thing like importing something, or you expect to need to do bulk insert often?
You would use something like doseq (or doall) for the reasons why to use these fn's instead of a 'for' or 'map' see http://stuartsierra.com/2015/08/25/clojure-donts-lazy-effects
I actually use
doseq for a different operation that's basically initializing empty records to be filled in later on
Yes. Although the lazyness of Clojure sequences often isn't discovered in experimentation with the REPL as the P (Print) tends to force realisation of the full sequence.
@trancehime: with postgres if you want to import a lot of data there's
copy - http://www.postgresql.org/docs/current/interactive/sql-copy.html - you can just pass over some CSV as file and it'll import it. It's the fastest way, but might not be right for you, hence I was asking how often would you do that and what for.
purpose is data submission of statistics in some table which will be compared against similar records input by different peoples
MySQL seems to support something similar - https://dev.mysql.com/doc/refman/5.1/en/load-data.html - since I think you mentioned you're usign MySQL
I think you'll find a 'bulk insert' operation in just about all RDBMS and for large datasets it should be your default option.
I mean it's kind of like a setup where the data input is occurring during observation of a given event
Since your sample data spoiled you seem to be doing something like LoL match analysis - the question is would data come as things happen (someone is killed, someone gets the buff and so on), or would you import the match wholesale after it ended?
First of all, the user in this case would do data submission once the individual set/game is over
Secondly, the idea is that multiple people will be submitting data for the same game and then there will be some checks made to see if the submitted data is the same
Now here's the caveat - this all has to happen within a period of time since the data submission
I see. I guess in that case you could have a table for submissions; when a submission happens you do a single insert to create a submission id to associate the data with then serialise your vector as a CSV file with rows having a foreign key to that submission row, and import it with the copy mechanism.
You could just select all valid rows from the db and use copy again to insert them wherever they are needed after you confirmed they are valid.
in Clojure can you have a mutable (transient) nested vector, or should I look at core.matrix?
But what if I have multiple people submitting data for different schedules at once, this means I'll just have a ton of CSV files wouldn't I?
@borkdude: I think I've read somewhere that transience isn't transitive if that's what you mean. That is with
(transient [1 2 [3 4]]) only the outer vector is transient. But I'm not 100% sure.
I tried core.matrix, but it didn't quite do what I expected. When I googled, I saw someone had the same problem with
@borkdude: hmm, what's wrong with
set-selection!? I've been fighting with it today for Advent of Code, curious what you had problems with.
@jaen: I played around with it, like this
(matrix/set-selection! m [[0 0] [4 4]] (matrix/new-matrix 5 5)) (where m is a :double-array matrix), but I keep getting: RuntimeException Inconsistent count of selection arguments
@trancehime: I see. Well yes, but after you copy the file into the database you can just get rid of those temporary files, so I don't think that's much of a problem?
@borkdude: well, the docs aren't entirely clear to be honest. The coords should be exploded I think for starters, that is
(matrix/set-selection! m [0 0] [4 4] (matrix/new-matrix 5 5)) but I'm not sure if that solves everything.
@jaen: "RuntimeException Mismatched shapes in assign!" ... I guess I should read the source. The docs are in fact not clear.
@jaen: I hate to say it, but this is the kind of library where static typing would help 😉
@borkdude: here, have an example from my solution (that works) -
(matrix/set-selection! m [[0 0] [4 4]] (matrix/new-matrix 5 5))
(m/set-selection (m/zero-matrix 10 10) [4 5 6] [4 5 6] (m/fill (m/zero-matrix 3 3) 125)) seems to work
@borkdude: nice, though I must say I'm surprised you could just assign a keyword like that.
if I understood correctly, ndarray stores objects of any type: https://github.com/mikera/core.matrix/wiki/Matrix-implementations
Yeah it is implementation dependent. There is an implementation that allows complex numbers, for example
@mikera: oh, since you're here - how would you logically index a matrix to overwrite negative elements? I couldn't exactly figure this out and ended reshaping to 1D array and back to 2D, but I guess that's inefficient.
Or if you don't need to mutate it, you can broadcast a single-element NDArray to the shape you want
@mikera: I mean, now I have:
(defn start-state  (-> (m/new-matrix :ndarray 10 10) (set-grid! 0 0 9 9 :off)))
Right now I have something like:
but that's probably not too smart.
(defn clamp-to-zero! [matrix] (let [linear (m/reshape matrix [(m/ecount matrix)]) clamped (ms/set-sel! linear (ms/where-slice neg?) 0)] (m/reshape clamped [(m/row-count matrix) (m/column-count matrix)])))
@mikera: seems to work, thanks; I kind of couldn't get selectors to select negative elements in a 2D matrix for some reason.
The selector stuff is a bit tricky for sure. I was was thinking about replacing it with something more like specter
is there a thing like
interleave that always exhausts the supplied colls? i.e. colls don’t have to be same length?
not great (filter identity (interleave (concat [:a 😛 :c] (repeat nil)) [:1 :2 :3 :4 :5]))
hi, all! please remind me if any convenient data structure does already exist in clojure libs for many-to-many relation? naive implementation is to keep two maps of sets and keep them in sync., but i'm sure that somebody have done better
I’m developing a library in the context of an app, how do I make clojure pick the code of the library instead of the installed jar (so I don’t have to re-install it every change I make)?
@pupeno: leiningen has a feature called checkouts. It means you have a checkouts folder where you link to your other project. You still have to install it once and declare it as dependency in your project.clj file: https://github.com/technomancy/leiningen/blob/master/doc/TUTORIAL.md#checkout-dependencies
why is this so terribly slow for a 1000x1000 matrix? it makes my Emacs become unresponsive:
(defn to-string [matrix] (let [row-to-str (fn [row] (apply str (for [e row] (case e :on "*" " "))))] (string/join "\n" (map row-to-str (m/to-nested-vectors matrix)))))
with a StringBuilder and doseq it's better, probably because 1000 vectors don't fit in memory easily? (it surprised me)
I suppose it's rather has to do with the fact that Java strings are immutable, so you're generating a million of throwaway strings and copying the contents each time.
@jaen yeah, just tried it for a smaller set, that worked, but big matrix is indeed too big to display
turning on all lights works:
(count-lights (apply process-line (start-state) (parse-line "turn on 0,0 through 999,999"))) equals a million
this equals a 1000:
(count-lights (apply process-line (start-state) (parse-line "toggle 0,0 through 999,0")))
not really, I'm doing it like this:
(defn swap-value! [matrix x y f] (let [v (m/mget matrix x y) v' (f v)] (m/mset! matrix x y v')))
Can you make a 10x10 or so matrix and check if, say, turning on (1,1) to (5,5) and then toggling (0,0) to (4,4) and see if that equals IIRC 18
this equals 999996:
(count-lights (apply process-line (m/assign! (start-state) :on) (parse-line "turn off 499,499 through 500,500")))
@jaen: I could try to run over your input and see if I have a different answer than you?
On OS X, does anyone know how to get lein to load what I think is a java extension? There's an issue with MIDI on OS X and I've read that this library is required
I've placed it into /Library/Java/Extensions, but I'm getting an error about my classpath in the REPL, and I'm not sure how to instruct lein to load it
@jaen: ah got it right now, I counted the lights that were off, though. Very fast in Scala too btw