Fork me on GitHub
#clojure
<
2015-08-23
>
ghadi02:08:03

honza: rauh: unfortunately that work for an infinite sequence, and will consume all heap. We haven't implemented "locals clearing" in the core.async go compiler

ghadi02:08:14

the workaround in that ticket is +1, but the real fix is to add locals clearing to the go compiler

sveri08:08:51

I find myself repeating the following reduce pattern from time to time: (reduce (fn [a b] (if any-cond (conj a {:any "map}"}) a)) [] some-seq) This means, I am reducing over a seq and then return a seq adding a map or not depending on a condition. "Normally" I'd use map here, but map will add a nil into my seq if the condition is not met. Is there a more idiomatic approach for this?

val_waeselynck08:08:50

@sveri: if any-cond only depends on the current item, you could use remove or filter

val_waeselynck08:08:03

e.g instead of (->> some-seq (reduce (fn [p x] (if (pos? x) (* p x) p) )))

val_waeselynck08:08:33

you'd have (->> some-seq (filter pos?) (reduce *))

val_waeselynck08:08:26

@ghadi: wow, that sucks 😕

sveri08:08:27

@val_waeselynck I see. Somehowe my case is different, I put up a pastebin here: http://pastebin.com/kd4Vi2yP I am reducing over a dynamically ranged seq to extract data from a map that might exist in the map or not

val_waeselynck08:08:32

@sveri: I don't see why map does not work here

sveri08:08:43

@val_waeselynck: because I will get back a seq like this: ({...} nil {....})

val_waeselynck08:08:51

(->> (range 1 11)
  (map (fn [b] (let [id-correct (keyword (str "answer_correct_" b))] (id-correct params))))
  (filter identity)
  (map (fn [ic] {:question_id q-id :user_id user-id :answer_id ic})))

val_waeselynck08:08:05

does this not work?

sveri08:08:47

the thing is, params most probably does not contain the whole range of keys, there might be some missing

sveri08:08:21

so there may be answer_correct_1 and answer_correct_3 but not answer_correct_2

val_waeselynck08:08:30

This problem seems independent to the style issue we've talked about, I only refactored the code here

sveri08:08:14

Hm, I don't think so, the problem the map is not "complete" is the reason I use reduce instead of map

val_waeselynck08:08:56

Sorry, don't get what you mean 😕

sveri08:08:17

Np.. It's hard to explain

sveri08:08:48

thanks for helping anyway

potetm11:08:21

@sveri: You want keep.

potetm11:08:19

For example:

(let [->id (comp keyword (partial str "answer_correct_"))]
  (map
    (partial hash-map :question_id q-id :user_id user-id :answer_id)
    (keep
      (partial get params)
      (map ->id (range 1 11)))))

potetm11:08:15

As a side note, every time I thought I needed reduce to do some form of map/filter, I’ve been wrong. Even if you have to carry state along, you can do so via reductions.

potetm11:08:43

That’s not to say I never use reduce. But I prefer to use it when I’m actually reducing a collection down to one value (e.g. (reduce + 0 coll)). That a) allows you to keep things lazy, and b) tends to make things more declarative.

jamesaoverton14:08:00

@joelkuiper: YeSPARQL is a good idea! If you’re working with linked data in Clojure, this library I wrote might be useful: https://github.com/ontodev/edn-ld

joelkuiper14:08:30

@jamesaoverton: Nice, I’ll check it out simple_smile

joelkuiper14:08:49

if you have any suggestions, do let me know!

rickmoynihan15:08:23

@joelkuiper: - Interesting I've been meaning to write something similar -- though I personally quite like the ideas in honeysql

rickmoynihan15:08:54

@jamesaoverton: edn-ld looks interesting -- I've been developing something similar over the past 18 months... http://grafter.org/

rickmoynihan15:08:17

we've been using it to generate billions of linked data quads

rickmoynihan15:08:26

I'll have to try and find some time to take a look at these libraries

jamesaoverton15:08:48

@rickmoynihan: Yes, I’ve seen Grafter, but haven’t used it yet. EDN-LD is much smaller.

jamesaoverton15:08:50

@rickmoynihan: The Grafter site mentions Google/Open Refine several times, and Pentaho Data Integration (Kettle). Are you working towards a GUI tool?

jamesaoverton15:08:30

@rickmoynihan: Cool. Is any of that public yet?

rickmoynihan15:08:01

not yet - we're currently focussing on tools that introspect pipelines and generate upload forms and webservices etc... with tools for previewing pipeline outputs and viewing their errors inline with appropriate context. We should be opening up some more of these tools soon

rickmoynihan15:08:52

regarding the size of grafter - its actually not that big -- we did some work to split it into several different libraries (with lein-repack) so you can import just the bits you need... Unfortunately we ran into an as yet unidentified bug with repack and protocols - which has prevented us from releasing a modular build of the library

rickmoynihan15:08:10

@jamesaoverton: What is EDN-LD focussing on?

jamesaoverton15:08:13

@rickmoynihan: I use it as a component in different workflows when I want to change the shape of my data.

jamesaoverton15:08:40

I often work with OWL, and have a loose cluster of different tools the I pull together for different tasks.

jamesaoverton15:08:42

Grafter would work for many of those tasks. EDN-LD is refactored out of project-specific code that I wrote before Grafter was public.

rickmoynihan15:08:38

its primary use for us is transforming tabular data into LD - though we have various pieces for reading in RDF (in any RDF serialization) via SPARQL etc...

rickmoynihan15:08:32

Is there a reason you went with Jena over Sesame?

jamesaoverton16:08:20

@rickmoynihan: Not really. Just familiarity, and because it’s already a dependency in some of the projects I use EDN-LD for. Is Sesame better?

joelkuiper16:08:50

Fo me the same, I was already familiar with Jena/Fuseki; never really looked at how it compares with Sasame

rickmoynihan16:08:51

They're both good projects... but Sesame has much cleaner APIs, with much more consistency - and the code is generally of a higher quality. Also Sesame is what almost all of the commercial triple stores build off... so its much easier to migrate to a commercial offering when you need to handle bigger data or need better performance

joelkuiper16:08:05

Yeah a colleague of mine did jena-es (ephemeral time model/even sourcing on top of RDF using Fuskeki) https://github.com/gertvv/jena-es and the API hacks to make that work were not pretty

rickmoynihan16:08:08

however it depends how you access the stores -- if all you care about is a SPARQL endpoint it doesn't matter much

joelkuiper16:08:56

Jena’s SPARQL support seemed to be a bit better last time I checked though (things like property paths), but maybe that changed

rickmoynihan16:08:33

When I started grafter - I actually started by using Jena - as at the time we were using Fuseki - but I had so much friction with the API's I switched to Sesame -- and things became much easier

rickmoynihan16:08:37

@joelkuiper: Jena definitely does a good job at keeping upto speed with the standards... but Sesame supports SPARQL 1.1 and property paths etc fine

rickmoynihan16:08:53

I suspect Jena had the support for those first though

rickmoynihan16:08:01

the code and API quality is the main reason to choose Sesame though

joelkuiper16:08:46

right, well if I run in trouble I’ll consider switching for YeSPARQL, so far so good though simple_smile

rickmoynihan16:08:48

@joelkuiper: it depends on how deep you need to go... there's another project we started at the same time as Grafter - that required much deeper integration into the triple store / SPARQL processing - with Jena it would've been too hard... but sesame's design made it pretty easy... Though we do actually use ARQ to do some SPARQL rewriting, because Sesame only lets you roundtrip SPARQL 1.0 from an AST back into SPARQL; where as ARQ supports 1.1

joelkuiper16:08:55

hmm good to know! Do you know the status of https://jena.apache.org/documentation/notes/sse.html ? it seems a bit lost in time it seems

joelkuiper16:08:02

but it could’ve been a nice idea to integrate with Clojure

rickmoynihan16:08:40

@joelkuiper: yes it works fine... and thats actually what we use (integrated with clojure)

joelkuiper16:08:33

ah cool, I tried outputsAsSSE … but those function’s weren’t implemented in 3.0, so was kinda lost and ignored it at that point

joelkuiper16:08:41

do you have some pointers on how to get started?

rickmoynihan16:08:41

well it depends what you mean by integrated of course...

rickmoynihan16:08:47

Unfortunately the code I wrote is proprietary

rickmoynihan16:08:11

but I can probably find a few examples of interacting with the API

rickmoynihan16:08:38

@joelkuiper: try that to get an SSE

rickmoynihan16:08:39

I wrote a function to build an clojure.zip/zipper out of an SSE

rickmoynihan16:08:20

however SSE's dont cover the whole of SPARQL - as they miss out prefixes - and I think construct queries might be represented as selects at that level of abstraction...

joelkuiper16:08:07

cool, thanks!

sveri17:08:36

@potetm: Thanks, keep is exactly what I am looking for. I knew there must be a function as this is a pretty common usecase I guess

rickmoynihan19:08:17

@joelkuiper: @jamesaoverton: Thinking it might be worth setting up a clojure-rdf Google Group - to share ideas and discuss potential interop/synergies between libraries. We're all part of the Clojure RDF eco-system and individually we probably can't achieve everything we want

joelkuiper19:08:13

@rickmoynihan: I'm all for it 😀

rickmoynihan19:08:08

lol I'm in two minds though - but it would be good to find a place to talk clojure & rdf

bmay20:08:31

i'm getting "IllegalArgumentException No matching field found: close for class java.lang.String" when calling the following:

whacked20:08:36

is with-open treating file-path as a file?

bmay20:08:45

does that mean i'm doing something wrong?

whacked20:08:22

yes. you need to move file-path out of with-open, or just put it directly into the (io/writer ) part

whacked20:08:13

so

(let [file-path (str dir ...)]
  (with-open ...))
or
(with-open [out-file (io/writer (str dir ...))])

bmay20:08:46

ahh so it's trying to close the file-path string which isnt a file?

whacked20:08:02

ye look like it

bmay20:08:06

yeah it works now

jamesaoverton20:08:26

@rickmoynihan, @joelkuiper: It would be good to talk more. I’m not sure how wide the interest would be. Phil Lord does OWL stuff with Clojure, https://github.com/phillord

machty20:08:43

can someone help me distinguish why one works but the other throws?

user=> (let [{:keys [:foo/bar]} {:foo/bar 123}] bar)
123
user=> (let [{:keys [:foo/bar/baz]} {:foo/bar/baz 123}] baz)

CompilerException java.lang.RuntimeException: Can't let qualified name: bar/baz, compiling:(NO_SOURCE_PATH:1:1)

machty21:08:32

is there a way to make this work for keywords with multiple slashes? i don't understand the use case here if more than 1 slash breaks stuff

malcolmsparks21:08:09

@rickmoynihan: please sign me up for any RDF-related group

rickmoynihan21:08:57

@jamesaoverton: yes Phil and I have discussed stuff over email in the past... Agreed there probably won't be a lot of interest - but for those of us actually building/using stuff in this space it makes sense to try and build awareness of our work - and try to build libraries that can complement each other

malcolmsparks21:08:59

In a previous role, I've struggled with Jena and come up against some insane concurrency issues with it

rickmoynihan21:08:08

The grafter users google group has about 17 members already - very low traffic just now -- so an alternative would be to co-opt that as a clojure-rdf list... as it might not make sense for a grafter/edn-rdf/yessparql specific one

malcolmsparks21:08:10

Clojure spoils you when it comes to concurrency, it's much harder when you have to drop back down into Java

rickmoynihan21:08:39

@malcolmsparks: true that simple_smile - the RDF space is a lot better than it once was... but a lot of the tools lack maturity