Fork me on GitHub
#clojure
<
2017-04-14
>
hiredman00:04:28

for one, that isn't the correct syntax for s/or,

hiredman00:04:08

you need to name each alternative, for another I think it has better behavior when used as a generator

tbaldridge02:04:39

@josh.freckleton yeah, that sort of join is basically O(n * m) (I think?), if you index the collections once you can re-use the collections . But if you're handling 10M rows, you're gonna be optimizing this for awhile. It's not exactly something you can hack out quickly.

josh.freckleton02:04:21

How does something like SQL do it?

josh.freckleton02:04:18

I mean, i started with a sql script that was fast, but, specs dictate this has to take place not in db.

tbaldridge02:04:26

lots of computer science, indexing, and very efficient structures. You're storing your data in lists, DBs often store data in hash map like things

josh.freckleton02:04:29

sure, I'm with you there...

tbaldridge02:04:53

@josh.freckleton I;m in the middle of something, I'll write more in about 30min

souenzzo02:04:42

Hey, there is some function that does #(or (string? %)(keyword? %)(symbol? %))?

tbaldridge02:04:50

replying here so we don't clutter main chat so much

tbaldridge02:04:59

So what you have in this code is basically a full table scan in SQL. The join is slightly faster by the fact that you're doing on-demand indexing (the first call to reduce). The problem is, you're throwing away the index after the join is finished.

tbaldridge02:04:51

So in SQL this sort of join is slow (left join over two unindexed tables). But in SQL you can generate indexes, and that's where it gets faster.

josh.freckleton02:04:14

gotcha, ya those tables are indexed in sql

tbaldridge02:04:43

In SQL you'd create a index on your second dataset that says "index :aid". In clojure this looks something like (group-by :aid second-dataset)

josh.freckleton02:04:13

totally, or clojure.set/index

tbaldridge02:04:37

Now the idea is that whenever you add data to the second dataset you also add it to that index. This reduces the performance of inserts, but improves queries. And most of the time since you read way more often than you write, that's just fine.

josh.freckleton02:04:05

I'm with ya, that indexing cost is paid long before the query happens

tbaldridge02:04:55

right. That's the biggest computational difference. The other things have to do with hashmaps and how you access them. Now I'm not saying hashmaps are slow, but the rule is: when you do anything 10M times...it's all slow.

tbaldridge02:04:46

So SQL servers will also store data in very efficient data structures. In Clojure that would be something like using defrecord instead of hashmaps, and perhaps even using (.-aid row) instead of (get row :aid).

josh.freckleton02:04:51

ok, i wasn't familiar with that trade off. i'd guess that defrecord and hashmap, while performance may be different, lookups still probably cost around O(log n), no?

tbaldridge02:04:59

Doing all that in Clojure ^^ isn't easy, but I have a query engine in production that can find 1.8mil results (involving about 5 joins per result) in about 27sec.

tbaldridge02:04:20

Right, and defrecord is basically the cost of looking up a pointer in a location of memory.

tbaldridge02:04:50

Also, if you know aid will always be a integer/long you can type-hint it, and that saves even more memory and a few pointer lookups.

josh.freckleton02:04:24

ok, ya, it's always ints, that's a good point

tbaldridge02:04:44

You might even look into (deftype) which is even more bare-bones, but also saves more space.

josh.freckleton02:04:30

ok, i'll need to look into all of these, but I'm still not clear on my O(?) performance, that's probably my lowest hanging fruit

josh.freckleton02:04:50

so do I sort all rows, and then do like @qqq is saying in the main thread?

josh.freckleton02:04:02

that seems to make sense, but I haven't thought it through all the way yet

tbaldridge02:04:06

Yeah, I think he's right, at first I though you were doing: (for [a as b bs] join a with b), and that's O(m, n), but he's right, you're pre-indexing the collection, so it should be faster.

tbaldridge02:04:24

Do you have access to a profiler? If so those are a big help here as well.

josh.freckleton02:04:08

I haven't had something this perf sensitive, so I'm not familiar, but I think I've read up on some good ones for CLJ

josh.freckleton02:04:21

you mean one of those right?

tbaldridge02:04:38

Top of the line, IMO, is YourKit, so start there (I think there's a free trial).

tbaldridge02:04:56

It isn't cheap, but maybe you can convince your employer to buy a copy for the team

josh.freckleton02:04:26

ok, great thanks!

tbaldridge02:04:00

And one last thing, if you get to the place where you need a ton of data in memory, and can sacrifice immutability, look into off-heap storage. Stuff like ChronicleMap are open source and don't store your data in JVM memory, so you can spend less time in GC collections, and still have access to GB of data.

tbaldridge02:04:10

but that may be way more than you need, lol

souenzzo02:04:45

Maybe #(not (coll? %))? I'm really not sure...

didibus02:04:00

Question: Is there a map data structure, where each map entry can be safely updated. I could put a map in an atom, but that means if I'm updating two different keys at the same time, one will update will need to retry. So I want more fine grained control. Does that exist? Or even make sense?

souenzzo02:04:38

@yonatanel My question is more for a function that test if x was something like a string

tbaldridge02:04:50
replied to a thread:@tbaldridge thanks so much!

replying here so we don't clutter main chat so much

qqq02:04:57

@tbaldridge : why is that join O(m * n) ? Can't you do better, like O(m log m + n log n + size of output) by (1) sorting both lists, and (2) marching down the two sorted lists in sequence (instead of having to look at all possible pairs)

tbaldridge02:04:44

@qqq yeah, I think you're right, I missed the on-the-fly indexing in the start of that function

josh.freckleton02:04:23

@qqq that makes sense to me, so, my sort would be just the stock sort-by on the keys that will be joined on, and the march would be like 2 nested loops , right?

tbaldridge02:04:27

@qqq although I don't recommend sorting...sorting 10M rows is not a fun thing to optimize 🙂

josh.freckleton02:04:43

Hm, I just set up a little test of sorting [{:a (rand-int 9999999)} ...]

100k -> 2 s
1M    -> 23 s
10M  -> still going after a couple minutes...

josh.freckleton02:04:08

this might be crazy, but is there a way to get data from SQL already indexed?

josh.freckleton02:04:41

like, some binary that jvm recognizes?

noisesmith02:04:57

that would require some impossible voodoo on the gc I think

noisesmith02:04:30

getting the data pre-sorted might help though

tbaldridge03:04:44

yes, most indexes store data pre-sorted, so getting an entire table sorted by the index should be fast

borkdude06:04:34

@yonatanel For now I avoided map literals, I didn’t get to the cause yet

matan07:04:26

anyone has an example for honeysql inserts to point at? I got a little lost on hooking it up through clojure.java.jdbc

matan07:04:06

I've built a statement with honeysql, but cannot find the clojure.java.jdbc function that will simply execute it. It looks like insert! expects a table name and row contents, not a sql statement (which I prepare with honeysql). Am I missing much, or is honeysql just practically superfluous when using clojure.java.jdbc other than for sql queries?

matan07:04:10

I am also not sure from the http://clojure.github.io/java.jdbc/ whether statements are automatically "prepared" on their first use, or should be explicitly prepared, if one wishes to use prepared statements, I find the doc a bit ambiguous there. Your advice would be appreciated.

seancorfield07:04:19

@matan you want execute!

seancorfield07:04:56

query is for selects, execute! is for everything else. insert! etc are just shorthand for SQL generation and then calling execute!

seancorfield07:04:46

Also, read the expanded community-maintained docs at http://clojure-doc.org/articles/ecosystem/java_jdbc/home.html (linked from the readme).

seancorfield07:04:25

And, yes, all statements are automatically "prepared" for you.

seancorfield07:04:56

You can provide a parameterized PreparedStatement if you want (instead of the ["sql statement" param1 param2] construct if you want).

seancorfield07:04:26

See prepare-statement for more details.

seancorfield07:04:18

Feel free to submit PRs against the community docs site if you have suggestions for how to improve the docs.

matan07:04:25

Thanks, indeed in the source of db-do-prepared I can see it calles prepare-statement

seancorfield07:04:42

(Although no one does - they just complain 😸 )

seancorfield07:04:18

query also calls it, as do other parts of the library.

matan07:04:27

Thanks, I should look at the community docs indeed! I'll PR clojure.java.jdbc itself, to link there.

seancorfield07:04:48

The repo DOES link to that!!

seancorfield07:04:30

(And you can't submit PRs against control projects -- you have to submit patches on JIRA)

matan07:04:52

Are you sure it points there? I can't find the link

qqq07:04:06

@josh.freckleton , @tbaldridge : I have never implemented this algorithm myself, but I think the details goes something like this suppose we have A :: list of objects of type a B :: list of objects of type b f :: a -> c // index function on a g :: b -> c // index function on b then, let X = intersection (f (A), g(B)) // takes O(m log m + n log n ) time then, for every x in X: run your op on f^{-1)(x), g^{-1}(x) f^{-1} and g^{-1} should be relatively cheap as you've indexed your data

josh.freckleton11:04:38

this is inherently a little different from your "walk both at the same time", right? But the implementation is very straightforward, and I'm curious to understand: so, for my understanding of big O, - indexing presorted list costs O(m) + O(n) - intersection costs O(m log m + n log n) - lookup * 2 + merge costs O(log m + log n + merge) where merge is.... m + n? so in total, this specific algo would be O(m log m + n log n + m + n)? and then if I walked both at the same time, like two nested loops, I could probably eliminate the lookup * 2 + merge portion, so my cost would probably be just: - index both, presorted - O(m+n) - walk them simultaneously - O(m+n) so perhaps this one is better?

qqq12:04:19

@josh.freckleton : you're right, if they're both pre-sorted, than you can "walk both at the same time" in O(m + n + size-of-output) time

qqq12:04:12

@josh.freckleton : I'm currently hacking on something similar, but my rows are hashed, not sorted -- so I thought there had to be a cost for sorting -- but if they're already sorted, yeah, what you're suggesting is even better

qqq12:04:10

also, I don't think it's two nested loops, I think it's just one loop; assume the keys for A are 2, 4, 6, 8, 10, 12, 14, 16 ; and the keys for B are 3, 6, 9, 12, 15

qqq12:04:40

it'd be something like: ` (let [as [2 4 6 8 10 12 15 16] bs [3 6 9 12 15 18]] (loop [as (rest as) bs (rest bs), fa (first as) fb (first bs)] (cond (= fa fb) (do processing; (recur (rest as) (rest bs) (first as) (first bs)) (< fa fb) (recur (rest as) bs (first as) fb)) :else (recur as (rest bs) fa (first bs)))) -- or something like that where if cur-a = cur-b, we process otherwise, we increment the smaller one by taking the next value from the corresponding list thus "walking both lists at the same time"

josh.freckleton12:04:39

oh nice, i'd thought of doing the same in two loops, but this is clearly better 🙂

qqq12:04:29

I think we have to do it in one loop -- because if we have two loops, the and the outer loops M times and the inner loops N times then we get M * N, instead of M + N right?

qqq12:04:56

the reason this one is only M+N is that each element of As is only consumed once, and each element of Bs is only consumed once

matan07:04:18

Oh right found it

seancorfield07:04:06

Ok. 1pm here. Bedtime.

matan09:04:28

Any built-in shorthand for defining a function that returns nil? I have lots of functions that have no return value semantics, finishing up with nil feels repetitive. Or should I change the way I think?

leonoel09:04:41

@matan if the return value is not important, why not simply ignore it ?

matan09:04:03

@loenoel I do. Which way would be more idiomatic?

leonoel09:04:23

clojure functions always return something, there isn't an equivalent to void, except for interop

matan10:04:17

thanks, so ignoring when appropriate is the idiomatic way then

leonoel10:04:32

I think so

mbjarland10:04:25

I have a (probably idiotic) question about the .. member access macro. I'm working with mapdb and the following works:

(.keySerializer (.treeMap db "some-map-name") Serializer/STRING)
whereas the following doesn't:
(.. (.treeMap db "some-map-name")
    (.keySerializer Serializer/STRING))
IllegalArgumentException No matching method found: .keySerializer for class org.mapdb.DB$TreeMapMaker
could somebody put me out of my misery and explain which part of my coffee cup i missed this morning...

bronsa10:04:15

(.. db (treeMap "some-map-name") (keySerializer Serializer/STRING))

bronsa10:04:18

this is what you want

mbjarland10:04:23

trying to tie together the nippy clojure serialization framework with the apache mapdb embedded database engine to get a "there is no database and we get to stay in our data oriented clojure world" scenario

mbjarland10:04:45

ah, no dots to begin with and start with the db

bronsa10:04:47

(.. foo (bar) (baz)) expands to (.baz (.bar foo))

mbjarland10:04:02

thank you sir

bronsa10:04:29

think of .. as roughly the same as -> but using methods rather than functions

mbjarland10:04:30

and that works like a charm, still fairly new to clojure, thanks

bronsa10:04:47

these days .. is not used that much as using -> is more generic and as just as readable

bronsa11:04:12

so I'd write that as (-> db (.treeMap "some-map") (.keySerializer Serializer/STRING))

mbjarland11:04:05

ok duly noted, these are the kinds of things that you don't get from reading the doc strings

bronsa11:04:29

the advantage of using -> instead of .. is that if you want to thread a function call after a method you can

bronsa11:04:26

yeah, this comes with experience and reading other people's code

bronsa11:04:32

or asking in here :)

mbjarland11:04:02

fun constellation, I'm implementing interfaces and overriding classes from mapdb, written in kotlin, in clojure...a blessedly java free experience

mbjarland11:04:23

got another question, guess it was inevitable given my problem above : ). So I was implementing an interface in my first attempt, now I would like to override the following class from java (kotlin):

public class SerializerClass extends GroupSerializerObjectArray<Class<?>> {
ignoring the class hierarchy for a while, what is the correct way to subclass the above and override some methods from clojure?

mbjarland11:04:00

ok found an example, proxy seems like a/the way

mbjarland11:04:26

is proxy performant?

bronsa11:04:52

proxy/gen-class are the only two constructs in clojure that can extend classes

mbjarland11:04:35

@bronsa yeah, kind of where I ended up, and I have no interest in exposing this internal implementation detail which I assume is what gen-class is for...which leaves us with proxy

mbjarland11:04:31

is proxy dynamic as in every method call is somehow intercepted etc? guess what I'm asking is if there is a call overhead to proxy, I could potentially be calling this a very large number of times

bronsa11:04:04

mbjarland: proxy does have an overhead, yes

mbjarland16:04:10

thanks! missed this thread in my slack ui

mbjarland11:04:46

if there is I might opt for some java code in between

matan11:04:25

@mbjarland would a few extra instructions on the CPU make a difference?

mbjarland11:04:13

@matan normally no, but the reason we are using mapdb here is for performance and millions of reads, so in this scenario, maybe

mbjarland11:04:42

I'll do some testing...thanks everybody for the pointers

matan11:04:59

@mbjarland good to know about mapdb I guess 🙂

matan11:04:48

A question about argument passing. I have this protocol method implementation in my code: (log [this {:keys [text is-user time session-id]}] ; function body omitted now if the wrong kind of map were to be passed here, all keys would just be nil, whereas I'd appreciate some more safety. Of course I can :pre my way though this, but how would you idiomatically go about it?

quan12:04:51

you can add some schema for validation https://github.com/plumatic/schema

matan16:04:31

@U050PGQ9J thanks, I was hoping for some macro from clojure.core or something. Assuming users of my library won't make mistakes shouldn't be part of clojure philosophy I think 😉

matan16:04:26

Maybe I'll just use java-like types! but I'm gonna take a longggg look at those two libraries you suggested

matan16:04:06

@U050PGQ9J wow great stuff to read through, guess it's not only my problem. eagerly reading those two resources...

quan17:04:45

clojure.spec will be included into core in clojure 1.9

quan17:04:33

there are some guides here fyr https://clojure.org/guides/spec

lmergen12:04:35

what’s the fastest serialization/deserialization format for clojure, i assume transit ?

lmergen12:04:13

i might actually benchmark the speed difference between cheshire and transit, cannot find anything on the internet about it..

lmergen12:04:14

hmmm looks like data.json is actually the winner.. ?

Alex Miller (Clojure team)12:04:20

Depends what on the other end

lmergen12:04:45

it’s mostly internal message queueing, but data will need to be store persistently

Alex Miller (Clojure team)12:04:50

You haven't mentioned any binary formats like fressian, nippy, etc

lmergen12:04:30

i’m trying to balance between readability and performance — the latter one needs benchmarks, though

lmergen12:04:41

contains all the info i need, albeit it a bit dated

elena.poot13:04:48

I was probably doing something wrong, as it was early on in my use of clojure, but if you're going to use cheshire, test it on large sets of data if that's something you'll need to do. I had to stop using it as it was truncating somewhere around 440K. As I said, probably my fault, but run some tests on your own data.

tbaldridge13:04:22

@lmergen I'd also try formats that retain types (like transit) before trying to use JSON. Dealing with date/float/url conversions to and from strings is a big pain.

elena.poot13:04:47

Agreed, I've since switched to EDN

tbaldridge13:04:59

And isn't a computer going to read the data most of the time? Might want to optimze for that.

tbaldridge13:04:09

yeah, EDN works, but it's really slow.

lmergen13:04:21

@tbaldridge that actually is a good point, and transit is readable (enough)

elena.poot13:04:11

I actually like being able to read the data, and it's helped with my debugging. And with EDN, it's the actual data, not something the other end has to transform, which is nice.

lmergen13:04:16

(context: this is for processing in onyx, and storing inside an event store)

raymcdermott14:04:37

quick question on special forms …. using *' in reduce

raymcdermott14:04:25

the form with *' does not compile in CLJS

raymcdermott14:04:09

what’s going on with the the and the ' ?

raymcdermott14:04:11

(oh and the * form does compile in CLJS and gives the correct (non-overflowed) sequence

raymcdermott14:04:40

I’m obviously missing something basic here 🙂

lsenjov14:04:30

IIRC *' is multiplication without checking for overflow

lsenjov14:04:30

Ah, standard * is:

Returns the product of nums. (*) returns 1. Does not auto-promote
longs, will throw on overflow. See also: *'

lsenjov14:04:43

*' must auto-promote

lsenjov14:04:54

Which may not exist in cljs

raymcdermott14:04:01

yes I see the docs now

raymcdermott14:04:43

promotes to BigInt with *'

raymcdermott14:04:56

closed due to complexity / effort

cgrand14:04:56

that’s no fun the original input is lost...

cgrand14:04:23

it was (#(str %4/3) 12)

cgrand14:04:24

you can also specify arguments in base 2: %2r11 instead of %3

bronsa14:04:15

that's a new one to me

cgrand15:04:07

and %-1.41 is %&

cgrand14:04:35

going to fix tools.reader?

benbot15:04:39

Can someone give me a quick spiel about clojure.spec? I’ve been trying to grok the docs but i’m still not sure what it is

xiongtx17:04:19

Have you seen the Rationale and Overview doc? https://clojure.org/about/spec It’s basically a contract system. You specify what you data/parameters/return vals are supposed to look like, and clojure.spec checks that for you at runtime Check out the #beginners and #clojure-spec channels

lmergen15:04:05

@benbot go to #clojure-spec 🙂

jsdiaz1917:04:27

how can i connet two terminals that receive and send messages?

jsdiaz1917:04:38

i am new on clojure

mbjarland18:04:10

anybody know of how you would go about performing polyglot compilation of clojure with other non-java languages (groovy, kotlin, scala) using leiningen?

tjtolton18:04:24

So, in the CORS standard, browser's always do a pre-flight OPTIONS call on any http route, right? so if there is a route (GET "/api/client/:client-id/record/:record-id"), a browser would actually make 2 calls to a backend service running on the same machine: curl OPTIONS and curl GET

tjtolton18:04:49

so the problem i'm running into is that we have to define a corresponding OPTIONS route for every other route I define

tjtolton18:04:08

is there some way to mitigate that? I feel like this is a pretty common domain concern

mbjarland18:04:01

in my experience that is not a true statement, that browsers always do an OPTIONS call…

matan18:04:19

@mbjarland compiling clojure and java on the same project with leiningen should give you some inspiration, there's an art for weaving the compilation stages

mbjarland18:04:47

clojure and java I don’t have an issue with, I was looking for groovy-clojure, kotlin-clojure, scala-clojure

mbjarland18:04:59

I am, after all, trying to avoid java at all costs : )

mbjarland18:04:03

kinda gave up for now, seems to be few and far between hits on joint compiling anything but java

matan18:04:11

I think you might want to clump them together building each one with its own tool 🙂 sharing classes through directories

mbjarland18:04:39

@matan, yeah, boring but I hear you

matan18:04:39

Really I saw no good integrations last I saw

matan18:04:51

boring is better than interesting here 😉

mbjarland18:04:10

maybe I’ll go back to building with gradle 😉

matan18:04:28

if gradle knows how to build all of them languages you mentioned..

mbjarland18:04:33

it does, not sure about cross language dependencies, I know it can do mutual between say groovy-java, but I’ll try not to ocd, I’ll just eat it and write some java

matan18:04:54

🙂 do share if you get something even mildly elegant!

mbjarland18:04:03

hah…sure will do

matan18:04:15

basically each language grown tool (sbt, lein, etc) is best at its own juice, but then things like getting a uniform java version, across-layers cleaning and artefact passing needs its own orchestration

matan18:04:29

frankly, even building scala with the scala tool itself (sbt) can make one pluck out their own hair in frustration on occasion.

mobileink18:04:04

boot does that kinda thing quite well.

matan18:04:16

Looks quite promising! at least for a clojurian!

matan18:04:04

I guess if I had to interleave different language-specific build tools by hand, using boot I would get all the flexibility I needed.

mobileink18:04:14

yeah, it's amazing. but i hate to steal leinengen's thunder, it has its place too.

matan18:04:01

hopefully it is easy to weave external build tools with boot, and pass around artefacts between them through clojure code running under boot.

mobileink18:04:37

do take some time to grok filesets, tho. there is a bit of a learning curve. the folks on #boot are very helpful.

mbjarland19:04:14

gradle does a beautiful job with groovy, scala, kotlin etc…and I just got it to work (unidirectional) between groovy and clojure (clojure calling groovy). But agreed, would not want to build a large clojure project with the somewhat thin clojure support in gradle

matan19:04:23

which speaks to the point that each language-specific tool is needed beyond the hello-world project complexity level per language

matan19:04:03

anyway, the guys at the sbt-dev gitter channel are somewhat cooperative, to help with any kind of integration

mbjarland19:04:09

tried and failed to get bidirectional dependencies to work…does work between groovy and java, but not clojure groovy. Going to give up this particular experiment I think : )

mbjarland19:04:04

anyway, in the far-out event that this would be useful for somebody: https://bitbucket.org/mbjarland/clojure-groovy-gradle

mobileink19:04:35

@mbjarland what do you mean by bidir deps? ( i did a fair amt of work with groovy once and it drove me to the brink of insanity).

mbjarland19:04:15

bidirectional as in groovy can call java and java can call groovy…i.e. transparent source tree, you call what you want in any direction

mbjarland19:04:40

: ) and by that criteria I should be institutionalised pronto

mobileink19:04:52

so it failed when you tried groovy <-> clojure?

mbjarland19:04:03

well yes, but I kind of assumed it would

mbjarland19:04:09

unidirectional is simple

mbjarland19:04:22

bidirectional I think required some gradle woodoo which I did not apply

mobileink19:04:04

yeah, you'd have to reeeaallly need that to go thru the pain

mbjarland19:04:16

not even sure if gradle does this for any other non-java hosted languages, I know it does do between non-java and java

mbjarland19:04:01

with groovy-java there is no pain, it just works, but somebody sacrificed some lambs somewhere along the way

mobileink19:04:12

worked a lot with gradle for appengine. not bad (except for the groovy part), but really just more of the same build tool stuff. boot is in a different league alrogether.

mbjarland19:04:45

: ) yes, I have even gone as far as to write a product which is partially based on gradle plugins

mbjarland19:04:43

and I spent considerable time with groovy, but I am glad to hear that boot is that good…looking forward to grokking yet another awesome tool

mobileink19:04:50

i'm so sorry. i hope you were not oermanently scarred.

mbjarland19:04:07

: ) I don’t actually dislike groovy

mbjarland19:04:22

now java just makes me sad

mobileink19:04:41

boot has its pain points, but it's not really a build tool. it's more of a general job control language that happens to work beuatifully for sw building.

mobileink19:04:01

sovmuch of language preference is what you learn first and are most used to. i still like assembler, love C. ;)

mbjarland19:04:01

oh and you can now write gradle scripts in kotlin if that makes you less sad : ) it does give you completion and type checking in IDEs which for build logic might not be a bad idea

mobileink19:04:19

it's such a firehose, this tech stuff.

mbjarland19:04:19

yeah…kotlin is not half bad though, if you don’t mind the static typing

mobileink19:04:23

so much stuff i want to check out, esp. when i already know a bit (groovy, gradle, kotlin, etc.) so little time.

mbjarland19:04:03

true, and in the other direction for me, boot, a gazillion clojure frameworks, clojure script, reagent, etc

mbjarland19:04:22

now I’m going to write some horrible java code and try to call clojure from it…think I need a drink first

mobileink19:04:44

yeah, i have to admit i spent a lot of time grokking boot. someday we'll get some decent docs.

danp18:04:13

Hey all, I've got some Java interop stuff that I'm not sure I'm doing the most effectively. I'm using Clojure to talk to Oracle Data Integrator's SDK. I'm having some success, but at the moment my code isn't the most intelligible!

danp18:04:25

This specific line:

OdiProject proj = ((IOdiProjectFinder) mgr.getFinder(OdiProject.class)).findByCode("NEW_PROJECT");

danp18:04:01

I have the approx equivalent:

(.findByCode (cast IOdiProjectFinder (.getFinder (.getTransactionalEntityManager odi-instance) OdiProject)) "NEWPROJ")

danp18:04:24

I used cast, as my limited Java knowledge reminded me that's what was happening in the Groovy code, but is that the right way to translate to Clojure?

danp18:04:24

I've read that reify is for implementing Java interfaces, so would it be suitable for this, or am I misunderstanding?

noisesmith18:04:19

cast guarantees that you throw an exception if the arg is not an instance of the requested type

noisesmith18:04:00

reify, proxy, defrecord, and deftype are all viable ways to extend interfaces, they each have their use cases (more rarely, also gen-class) https://cemerick.com/2011/07/05/flowchart-for-choosing-the-right-clojure-type-definition-form/

mbjarland18:04:44

@matan …guess it’s time to share : )

mbjarland18:04:58

I got at least groovy-clojure polyglot to work with gradle

danp18:04:03

Excellent, thanks @noisesmith will have a read now.

mbjarland18:04:10

@matan built an executable jar with clojure calling a groovy class…I strongly suspect it is not bidirectional though

noisesmith18:04:43

@danp also your code can look more like the original if you use -> syntax to chain the method call forms

danp18:04:06

@noisesmith, so there's nothing particularly wrong with cast? I'd not seen it anywhere, so was unsure whether it was particularly idiomatic. From the flowchart, it also looks like reify is suitable the way I'm currently approaching it.

noisesmith18:04:36

(-> odi-instance (.getTransactionalEntityManager) (.getFinder OdiProject) #_(cast IOdiProjectFinder) (.findByCode "NEWPROJ")) ; I commented cast as I don’t think its needed

danp18:04:14

Oh, thanks on the ->... I'd experimented with doto elsewhere in the code, but not thought of the ->

noisesmith18:04:16

all cast literally does is throw an exception if the type isn’t an instance of the one asked for, but this can have a side effect that clojure knows which type you expect the compiled form to see at runtime which can be good

danp18:04:14

got it - thank you.

waffletower19:04:26

I am curious about future support for automagical generation of specs. I am not really focussed on the use case suggested by my example below, but more about the mechanics of using macros to generate specs:

(defmacro validity-wrapper [symbol spec]
  `(do
     (defn ~symbol [x#]
       (s/valid? ~spec x#))
     (s/fdef ~symbol
             :args (s/cat (keyword (name 'x#)) ~spec)
             :ret boolean?)))

waffletower19:04:17

In the above example, the presence of a list (keyword (name 'x#)) breaks validation of of s/cat:

Assert failed: cat expects k1 p1 k2 p2..., where ks are keywords
   (c/and (even? (count key-pred-forms)) (every? keyword? keys))

waffletower19:04:44

Would such nested evaluations be supported by clojure spec in the future?

tbaldridge19:04:36

@waffletower create the symbol outside of your syntax quoting, and you'll be fine

tbaldridge19:04:10

(let [x (gensym "x") x-kw (keyword (name x))] ... your code ...)

waffletower19:04:01

that’s fine by me, many thanks

mbjarland20:04:18

ok I take it back, I do have a cursive question

waffletower20:04:24

@tbaldridge it would be fine to use (let [x (symbol "x")... instead in such a context, right?

waffletower20:04:12

Changing my variable parameter to sym first 🙂

mbjarland20:04:18

cursive question: each time I restart the repl it ends up in what I assume is the user namespace, even if I load an entire clojure file in the repl (Shift-Apple-L in osx) the repl still stays in the user namespace. I have to do a (in-ns 'myns) every time I restart the repl…which considering I’m tinkering with java is often. Expected behaviour or cursive issue?

Alex Miller (Clojure team)20:04:58

I think that’s the expected behavior

Alex Miller (Clojure team)20:04:19

but I think there is a different command to do the in-ns for you

mbjarland20:04:13

@alexmiller well there is a “evaluate forms in REPL namespace” setting which is unticked

mbjarland20:04:38

@alexmiller any idea what the command name is?

mbjarland20:04:50

not in my kb layout

Alex Miller (Clojure team)20:04:57

that’s for “Switch REPL NS to current file”

mbjarland20:04:14

: ) unmapped, thanks

Alex Miller (Clojure team)20:04:23

maybe I set that to match cider or something

mbjarland20:04:41

works like a charm…thanks that just saved me from the brink of insanity

Alex Miller (Clojure team)20:04:27

I’ll send you my bill

mbjarland20:04:19

I pay in scotch

Alex Miller (Clojure team)20:04:43

now we’re talking - Laphroaig Quarter Cask? :)

mbjarland20:04:02

sure, if you’re going that hairy I also have the Aardbeg Supernova

mbjarland20:04:51

which is close to undrinkable due to insane smokiness : )

seancorfield20:04:15

Sounds right up my street tho’… For our first wedding anniversary my wife bought me a bottle of Ardbeg 25 yr old reserve. And I certainly do enjoy a Laphroaig 15 yr old (but not the 10).

mbjarland20:04:44

well I do like the supernova, but you kind of have to be in the mood for a brawl

mbjarland21:04:12

Malört…it is what you make absinth from

mbjarland21:04:05

Mal in swedish is moth and ört is herb, in olden days they believed the herb involved could repel moths…but yes, you can also die from it..wormwood I believe in English

mbjarland21:04:45

ah, would have helped if I read your article

mbjarland21:04:27

and of course, had to be a swede involved in this insanity

mbjarland21:04:27

wow I decidedly have to buy a bottle next time I’m on that side of the pond

mobileink21:04:27

well if you're swedish i think it's obligatory

mobileink21:04:58

look me up when you hit chicago (i don't think you can get it anywhere else). it really does taste like gasoline/kerosine - well, haven't tried either of those but they must taste like malort. but: it's a great medicinal the day after. ;)

mars0i21:04:48

Hey! @seancorfield I thought you were going to send it #off-topic . Oh no! My image of Clojurians is crumbling.

mobileink22:04:58

oh sh**, forgot which channel i was on. sorry!

seancorfield22:04:23

I figured it was “mostly harmless” in a thread… but, yeah, it went on a bit 🙂

mars0i22:04:53

Or is scotch a new plugin?

sveri07:04:17

Just looked up the price for the Ardbeg and I finally find out what a marriage is good for 😄

Tim22:04:22

I would like this:

(let [elem 'b]
  '(not elem))
to actually be
(let [elem 'b]
  '(not b))

Tim22:04:33

I can’t figure out how to do this

Tim22:04:29

(I’ve tried a few ways, including macro-style templating)

bfabry22:04:51

@tmtwd I think you want (let [elem 'b] `('not b))

mars0i22:04:41

why the first '~ ?

Tim22:04:56

yup that works 🙂

Tim22:04:59

thank you

bfabry22:04:00

@mars0i otherwise not will become clojure.core/not

mars0i22:04:19

Ah , ok. Thanks.

mars0i22:04:55

I can use that in a macro I wrote.

bfabry22:04:45

~ says "evaluate the next form" and ' says "quote the next form"

bfabry22:04:59

so ~'not says "evaluate (quote not)"

mars0i22:04:24

I've seen it an actually used it here and there, but didn't understand 100%.

benzap22:04:55

Hey guys, i'm trying to use hiccup to generate a template, but it appears that element attributes need to remain static, or any symbols get picked up as 'strings' ie. `(let [class-local-var "class-foo class-bar"] (hiccup.page/html5 [:body {:class class-local-var}])` --> ...<body class="class-local-var">..... Any way to break out of this behaviour, and force it to evaluate the local var?

bfabry22:04:22

using hiccup.compiler/compile-html would probably work

bfabry22:04:57

I never realised hiccup was so macro focused

bfabry22:04:50

ah, for performance, that makes a little sense

benzap22:04:43

any way I can get it to evaluate without replacing hiccup.page/html5?

benzap22:04:56

I feel like this should just evaluate instead of grab the name

benzap22:04:17

or since it's an attribute, I guess the main area is hiccup.compiler/render-attribute

weavejester22:04:40

@benzap the macros in Hiccup are for pre-compilation. It should work with symbol bindings.

weavejester22:04:45

user=> (let [class-local-var "class-foo class-bar"]
  #_=>    (hiccup.page/html5 [:body {:class class-local-var}]))
"<!DOCTYPE html>\n<html><body class=\"class-foo class-bar\"></body></html>"

benzap22:04:43

this prints out ...<div class="test class-local-var">...

weavejester22:04:52

For Hiccup 1.0, presumably?

benzap22:04:06

ya, should I try hiccup 2?

benzap22:04:12

[hiccup "1.0.5"]

weavejester22:04:33

Yeah, looks like a bug. It’s partially fixed in Hiccup 2.0.0-alpha, but also blows up under some circumstances, so that needs to be fixed.

weavejester22:04:03

I’ll take a look at it, but it might be a week before I can fix it as I’m a little swamped right now.

benzap22:04:22

cool cool, glad I could help. I have a workaround in place for now

benzap22:04:43

do you want me to make a github issue just to track it?

weavejester22:04:57

Yes, please open an issue 🙂

Tim23:04:42

is there a reason that I can get this to work:

(doseq [e '(r w)]
  (println (swap-with-not-sublists li2 e)))
((w | s (not r)) (s | c) ((not r) | c) c)
(((not w) | s r) (s | c) (r | c) c)
=> nil
but when I try to adopt this to reduce it doesn’t:
(reduce swap-with-not-sublists li2 '(r w))
=> ((w | s (not r)) (s | c) ((not r) | c) c)

Tim23:04:01

nvm, some problem with my code I missed

j-po23:04:05

I'm trying to convert nested structures of vectors and hashmaps to nested structures of Java arrays and hashmaps, but I'm running into a mysterious array type mismatch error. Here's my code:

(defn- java-y [form]
  (walk/postwalk #(cond (map? %) (java.util.HashMap. %)
                        (vector? %) (into-array java.util.HashMap %)
                        :else %)
                 (walk/stringify-keys form)))

j-po23:04:44

(We were originally using clojure.java.data/to-java, but couldn't figure out how it was supposed to work.)

j-po23:04:07

Or, rather, (vector? %) (into-array Object %)

hiredman23:04:52

do you have the actual exception?

hiredman23:04:28

if I recall, postwalk traverses through maps as a seq of map entries, and clojure map entries are likely also vectors

j-po23:04:53

#error { :cause "array element type mismatch" :via [{:type java.lang.IllegalArgumentException :message "array element type mismatch" :at [java.lang.reflect.Array set "Array.java" -2]}] :trace [[java.lang.reflect.Array set "Array.java" -2] [clojure.lang.RT seqToTypedArray "RT.java" 1733] [clojure.core$into_array invokeStatic "core.clj" 3345] [clojure.core$into_array invoke "core.clj" 3334] [org.panlex.api$java_y$fn__1734 invoke "api.clj" 72] [clojure.walk$walk invokeStatic "walk.clj" 45] [clojure.walk$postwalk invokeStatic "walk.clj" 52] [clojure.walk$postwalk invoke "walk.clj" 52] [clojure.core$partial$fn__4759 invoke "core.clj" 2515] [clojure.core$map$fn__4785 invoke "core.clj" 2646] [clojure.lang.LazySeq sval "LazySeq.java" 40] [clojure.lang.LazySeq seq "LazySeq.java" 49] [clojure.lang.RT seq "RT.java" 521] [clojure.core$seq__4357 invokeStatic "core.clj" 137] [clojure.core.protocols$seq_reduce invokeStatic "protocols.clj" 24] [clojure.core.protocols$fn__6738 invokeStatic "protocols.clj" 75] [clojure.core.protocols$fn__6738 invoke "protocols.clj" 75] [clojure.core.protocols$fn__6684$G__6679__6697 invoke "protocols.clj" 13] [clojure.core$reduce invokeStatic "core.clj" 6545] [clojure.core$into invokeStatic "core.clj" 6610] [clojure.walk$walk invokeStatic "walk.clj" 49] [clojure.walk$postwalk invokeStatic "walk.clj" 52] [clojure.walk$postwalk invoke "walk.clj" 52] ...

Tim23:04:07

far 
=> ((w | s (not r)) (s | c) ((not r) | c) c)
(def bar '((w | s (not r)) (s | c) ((not r) | c) c))
=> #'math-stats.probability/bar
(= bar far)
=> true
(swap-with-not-sublists bar 'w)
=> (((not w) | s (not r)) (s | c) ((not r) | c) c)
(swap-with-not-sublists far 'w)
=> ((w | s (not r)) (s | c) ((not r) | c) c)
somehow bar and far are getting different results when passed into the same function, even though they are equal? anyone know how this could be ?

j-po23:04:15

@hiredman I also thought it might be that map entries are represented as vectors. Not sure how to get around that, though

hiredman23:04:28

use prewalk

j-po23:04:39

(without making further assumptions about the structure of the data, which would suck)

hiredman23:04:51

I guess that wouldn't change it

hiredman23:04:21

(and (vector? ...) (not (map-entry? ...)))

hiredman23:04:27

map-entry? I don't think exists

hiredman23:04:58

I am sure @nathanmarz will tell you all about how to use specter to do it

j-po23:04:57

@seancorfield , could clojure.java.data/to-java be helpful for this?