Fork me on GitHub
#clojure
<
2016-03-20
>
curtis.summers00:03:09

@lmergen what database are you using? Here's an async library for postgresql: https://github.com/alaisi/postgres.async

cky03:03:08

@virmundi: I’ve used it for a re-frame project of mine.

virmundi04:03:31

What did you think of it? I'm getting tired of looking at http://draw.io images to know what my data entities should be.

roberto04:03:49

not sure what http://draw.io has to do with schema

virmundi04:03:51

I'm documenting entities using the Extended Entity Relation model (simplified). I use http://draw.io to house and make the diagrams.

roberto04:03:31

not sure why you would do that tho

roberto04:03:06

i’ve never drawn an entity relation diagram (only in school)

virmundi04:03:10

I have fairly complex hierarchy of maps (I guess). I need to know what keys go where. Modeling helps make sure I can get all the data. Normally in Java I'd have an class structure to look at too. Clojure lacks a clear system like that. schema looked like it could help.

roberto04:03:18

ok, it is not common to have those types of hierarchies in clojure

roberto04:03:43

i normally use schema as a way to document a function (inputs and outputs), but only after that function is stable and I need some clarity

virmundi04:03:37

How would you have a Profile with an email, password and an address, where an address has all the parts of an address?

roberto04:03:25

probably have a schema for address, and include that in the profile schema

virmundi04:03:05

That's what I figured. It's structures like that I document.

roberto04:03:25

I don’t document the structures, only the functions. And the only time I need to look at what a schema looks like is when I’m trying to use the function and need specifics as to the shape of its inputs and/or outputs.

roberto04:03:41

I never look at the schemas in isolation

roberto04:03:45

have never had that need

virmundi04:03:22

fair enough.

andfadeev13:03:52

Hi, I've stucked with a problem I want my custom user.clj loaded in repl on startup, I have project_root/dev/user.clj

(ns user)

(def foo 2)

(defn dev
  "Load and switch to the 'dev' namespace."
  []
  (require 'dev)
  (in-ns 'dev))
and in project.clj
:profiles { :dev {:dependencies [[org.clojure/tools.namespace "0.2.11"]]
                  :source-paths ["dev"]}}
but on repl start
user> foo
CompilerException java.lang.RuntimeException: Unable to resolve symbol: foo in this context
and it seems that dev dir is not on my classpath, checked with
(println (seq (.getURLs (java.lang.ClassLoader/getSystemClassLoader))))
I tried reloaded template (https://github.com/stuartsierra/reloaded)
lein new reloaded my-app
and user.clj is loaded on repl startup there, but still can't figure out problem with my project...

lmergen13:03:31

@curtis.summers: well actually I'm using PostgreSQL through Korma, and am looking for the best way to wrap these long-running, blocking queries in an async way

lmergen14:03:00

but I feel like my best approach, if it's really important, to just use the async postgresql client api

masnun14:03:48

@andfadeev: I put user.clj in project root and it worked

lmergen14:03:25

i'm really confused by all these people having strong opinions on which concurrency mechanism to use... core.async is a pretty decent async library, which appears to be the de-facto standard, but then i see the Funcool people abandoning core.async and saying that the only proper way to deal with promises is not to use core.async, but rather java8's native promises/future mechanism... i'm so confused what to do!

lmergen14:03:12

I read this on their site: > Clojure comes with a builtin promise abstraction but it is designed only for blocking operations, and in async environments the blocking operations are completely discouraged. so, if I understand it correctly, if I want to make a computation that is doing blocking I/O, I should use core.async, but if it is just an expensive computation, I should use java8's promises?

lmergen14:03:30

imho, the right way to me sounds like core.async should just use java8's promise mechanism if it's available ?

jonahbenton14:03:26

hey @andfadeev you are starting a repl with lein repl ?

andfadeev14:03:01

i've tried lein repl and cider-jack-in from emacs

andfadeev14:03:25

and dev directory is not in classpath(

jonahbenton14:03:13

can you share your entire project.clj ?

mhjort15:03:36

@lmergen core.async is library which implement CSP style of concurrency. And then Promises/Futures is another way of implementing concurrency.

mhjort15:03:07

So I think they are two different abstractions for concurrency

mhjort15:03:18

Implementation wise core.async uses small thread pool and it shares threads by parking processes. Therefore it works efficiently when used with non-blocking I/O calls

mhjort15:03:40

I don't know Java8 promises that well but Clojure's own promise implementation is ok if you do something simple (for example start a background task and wait for the result)

mhjort15:03:43

However, with complex scenarios where you need to synchronize multiple parallel tasks core.async gives you more tools

lmergen15:03:08

yeah well, relatively speaking, i'm still doing simple things

lmergen15:03:32

i am a big fan of CSP-style concurrency, so I think I'm going to stick with that

jonahbenton15:03:36

hey @lmergen this sounds like it's getting too deep in the solution space weeds. What problem are you trying to solve?

lmergen15:03:55

that's a good question

fommil15:03:59

+1 for CSP, it's why I'm learning clojure

lmergen15:03:58

at a high level, i'm "just" implementing an http api server which does some CRUD and sometimes a bit longer operations in the background

lmergen15:03:00

coming from other languages that usually have asynchronous operations baked into the language, doing things synchronously feels like a step back

lmergen15:03:07

so, right now, I'm exploring the possibilities I have for making my webservice a bit more async -- not that I'm going to be implementing this tomorrow, but just for a high-level idea of what kind of design I should keep in mind

lmergen15:03:50

I feel like using an asynchronous database wrapper for PostgreSQL will be a first major step in making this happen

lmergen15:03:14

so it's not that I'm trying to solve a specific problem, but more just trying to understand the state of the ecosystem and what design principles to keep in mind making my Clojure applications more async

jonahbenton15:03:27

gotcha- can i ask- when you're talking about async baked into the language, are you talking about js?

lmergen15:03:18

well, that is a good example, but I'm coming from a strong Erlang background

lmergen15:03:02

but yes, node.js is a good example in that it defines a clear way to deal with this (callback hell) and every library uses it

lmergen15:03:53

it seems that the Clojure community, however, is mostly blocking, and a lot of the most frequently used libraries don't have async support

jonahbenton15:03:58

gotcha- so, in terms of io, java's history for its first 10-15 years was around blocking io. non-blocking io in javaland has come into its own in the last 5+ years, but many battle tested libraries, database drivers, etc, still use blocking io

thiagofm15:03:59

Is there any good book or anything related to CSP, to know the problems, solutions and so on

thiagofm15:03:18

Just wrestling with core.async whenever I have to use it doesn't seem like a nice approach

dominicm15:03:33

You might enjoy the CSP paper. I hear it's very good.

jonahbenton15:03:22

separately, java has always had a good threading story, but they're os threads

lmergen15:03:15

johanatan: yeah I figured it would be like that. seems like new projects like pedestal seem to promote an async-first design ,though

jonahbenton15:03:01

there was a failed foray into green threads many years back; they also are just making a return

lmergen15:03:07

in my case, I might drop Korma in favor of a pure DSL that generates SQL from abstract structures, and then execute the query manually using Clojure's asynchronous postgresql library

mhjort15:03:18

I think this tutorial is pretty good: http://www.braveclojure.com/core-async/

lmergen15:03:37

yeah it's pretty good, it's pretty much an explanation of CSP

lmergen15:03:45

and how it is implemented in Clojure

mhjort15:03:46

I'm not aware of any book that's specifically about core.async.

lmergen15:03:12

but, as a community as a whole, it feels like more and more libraries are starting to embrace core.async, eh?

mhjort15:03:42

I think in Clojurescript side that is already happening

thiagofm15:03:59

It's happening because you can use it instead of using callbacks 😛

mhjort15:03:09

core.async works well both with JVM and Javascript backend

jonahbenton15:03:30

yeah, core.async is widely used because it comes from cognitect and works across hosts, but there are many, many libraries

jonahbenton15:03:37

there isn't a consistency in approach on the jvm- unlike erlang- because several different models have been dominant over the years, and libraries can survive for decades

jonahbenton15:03:09

zach tellman has a widely used async library called aleph that is clojure-only: https://github.com/ztellman/aleph

mhjort15:03:11

In general I really enjoyd this book https://www.goodreads.com/book/show/18467564-seven-concurrency-models-in-seven-weeks. Besides low level concurrency models, it covers promises/futures, CSP and Erlang actor model

thiagofm15:03:51

@mhjort: looks like a great read

jonahbenton15:03:49

anyway, dropping korma and working closer to the wire probably makes sense

dominicm15:03:02

@ztellman's work always impresses me. I think I'll be getting a copy of that book.

lmergen15:03:20

@jonahbenton: thanks, but that's exactly my problem at the moment.. there are so many competing solutions for the same problem in Clojure, I'm not sure what to choose

lmergen15:03:41

I think the safest bet is to just use core.async by default, unless i have a very good reason not to

jonahbenton16:03:24

yes- there is no good reason not to. it can support both io- and cpu-based concurrency

jonahbenton16:03:55

i think the downside for some folks is that core.async's surface area has expanded dramatically

jonahbenton16:03:25

so someone coming into async without much background can get overwhelmed

lmergen16:03:44

@johanatan: makes sense, thanks for the info

johanatan17:03:07

@lmergen: Clojure has future too (which is pretty much the default async story in Scala

lmergen17:03:32

yeah i know, there are too many options! simple_smile

johanatan17:03:41

Also agents :)

lmergen17:03:50

for now, I think I'll stick with go for CPU-intensive tasks, and thread for IO-intensive tasks

lmergen17:03:04

and yes, there's also Akka

lmergen17:03:06

and whatever

johanatan17:03:58

Clojure agents aren't really analogous to Akka actors though. If you understand the underlying mechanisms behind each of these, then the choice will make itself

johanatan17:03:09

In each scenario

lmergen17:03:08

sometimes having the choice between so many primitives can be a disadvantage, though

curtis.summers17:03:44

@lmergen: I tend to run long-running background jobs in a separate worker process that takes items off a queue.

lmergen17:03:23

yes, I use a service oriented architecture, we're putting jobs in Amazon's SQS (= a queue)

lmergen17:03:06

so it all depends upon the task at hand

amacdougall17:03:46

Curious if this is possible: I have a higher-order function which relies on outer (i.e. closure) bindings. Is there a way of running the function while supplying arbitrary values for those bindings? Example:

(defn outer []
  (let [outer-a 1
        outer-b 2]
    (fn inner []
      [outer-a outer-b])))

(let [f (outer)]
  ; I realize that with-bindings does not actually do the job here
  (with-bindings [outer-a 10
                  outer-b 20]
    (f)))
As you might guess, this is for testing. I'm open to other approaches for inserting arbitrary data here! (To be exact, I'm trying to test the inner, or "renderer", function of a re-frame component in ClojureScript. The renderers rely on a subscription channel set up in the initial (outer) component invocation.)

curtis.summers17:03:36

@lmergen: Also, if you drop korma for straight SQL, you might also look at HugSQL (http://www.hugsql.org), which has protocols for the underlying db library, and there is a postgresql async adapter for it: http://www.hugsql.org/#adapter-community

johanatan17:03:13

@lmergen: I don't see it as a disadvantage at all. Each primitive has unique semantics

johanatan17:03:25

And/or syntax

lmergen17:03:57

@curtis.summers: I was actually looking at http://funcool.github.io/suricatta/latest, which had the "core.async is not good" quote which got me confused

lmergen17:03:25

also, I'm taking advantage of SQL composability, which HugSQL seems to miss

jonahbenton17:03:30

@amacdougall: is there a reason not to produce a function that just takes parameters and has defaults set from the initial definition? do you not have control over outer?

jonahbenton17:03:41

@lmergen looks like that lib recommends using agents for async

lmergen17:03:12

yes, one dedicated thread per db connection

curtis.summers17:03:27

@lmergen: As of version 0.4.x, HugSQL has Clojure Expressions and Snippet support that give you composability: http://www.hugsql.org/#using-expressions Of course, HugSQL is primarily an "SQL first" workflow, so if you prefer to keep things entirely in a "Clojure first" workflow, then you HoneySQL or Suricatta support that.

lmergen17:03:00

@curtis.summers: yeah i was aware of this, but this would pretty soon make my whole application a giant mess

lmergen17:03:05

so it's not a good fit for me

jonahbenton17:03:52

@lmergen: do you need to be able to cancel your long running queries?

curtis.summers17:03:50

@lmergen: Yeah, I get it. Some prefer to build everything in Clojure data structures. I've found DSLs to be cumbersome and/or leaky for my purposes. Good to have options on both sides, I say.

jonahbenton17:03:11

do you have a means to control how many are in flight at once?

lmergen17:03:19

@johanatan: i prefer a thread-pool like approach, but am actually used to having the environment I work in (be it Erlang, Go, or Haskell) handle that for me

johanatan17:03:43

future is what you want then

lmergen17:03:14

.. which is in core.async, right?

lmergen17:03:39

ah, so that's the problem

johanatan17:03:59

Core.async is CSP

johanatan17:03:09

Why's that a problem?

lmergen17:03:27

nah problem is not the right word

lmergen17:03:48

i was just confused, so core.async is actually core.csp

lmergen17:03:07

and there are other functions related to asynchronous programming in core simple_smile

jonahbenton17:03:22

yup- two slightly different implementationsof a similar model. future returns a dereference-able object; thread is part of core.async and returns a channel

lmergen17:03:51

but what if the framework i work in (for example, pedestal) works with core.async? is there middleware to convert a future to a core.async channel?

amacdougall17:03:13

@jonahbenton: In re-frame, a component is expected to be a function which returns Hiccup, or to be a function which returns a function which returns Hiccup. In the latter case, the returned function must have the same params as the outer one: the outer function does setup, usually establishing a "subscription" to the data atom.

(defn animals-view []
  (let [birds (subscribe :birds)]
    (fn animals-view-renderer []
      (into [:div]
            (map render-individual-bird @birds)))))
In the above example, re-frame will run the outer function once, and "install" the inner function as a renderer, which will be run every time that subscription data changes. ...You know, I suppose I could write it like this:
(defn animals-view [& [birds]]
  (let [birds-sub (subscribe :birds)]
    (fn animals-view-renderer [& [birds]]
      (into [:div]
            (map render-individual-bird (or @birds-sub @birds))))))
...though at that point, I'm doing weird stuff to my design to enable testing / devcards. I think it's not going to pay off.

amacdougall17:03:33

Uh, sorry for the wall of rubber-ducking.

jonahbenton17:03:09

@lmergen thread should be fine for your use case. no need to cancel, and you can let core.async handle the pool, and you get the composability of channels

johanatan17:03:54

@lmergen: promise-chan

johanatan17:03:26

If you need it

jonahbenton17:03:49

@amacdougall: i see. but you need to change some of the setup/configuration outside of a data subscription? (i haven't used re-frame, may not have the terminology right)

amacdougall17:03:47

Well, really I just want to test the inner function by supplying it some arbitrary data instead of having to create the central datastore, subscribe to it, then reset it after the test. However, doing that whole process is a fairly reasonable approach as well.

amacdougall17:03:16

This is pretty much what the author suggests anyway, I notice: https://github.com/Day8/re-frame/wiki/Testing#components---part-2a

amacdougall17:03:35

So, I'll do that! If I think testing these UI components is even worthwhile. I have a hunch it's not. ¯\(ツ)

amacdougall17:03:47

Test behaviors, not UI.

amacdougall17:03:52

The main reason I wanted to get this working is for devcards, a CLJS thing where you can live-edit a component without needing to hook it up to real data. But re-frame makes it fairly easy to load any arbitrary app state in a flash, so it's no big deal.

jonahbenton17:03:49

right, gotcha. i mean, if it makes sense from an app design perspective for the specific values to change in flight independent of initial configuration, you can make them dynamic vars, and then use with-bindings, but it sounds like that's not a use case that exists outside of testing

jonahbenton17:03:04

and perhaps not even then

amacdougall17:03:13

My app is not gonna be all that large in the first place! It's just very easy to overthink my tooling. I think back to when I was editing "taglib definition files" in Java, compiling, turning Tomcat off, turning it back on... and I was perfectly happy, right? I just didn't know any better.

jonahbenton17:03:45

heh, painful memories. simple_smile

amacdougall17:03:44

Anyway, this helped solidify my thoughts. Thanks!

jonahbenton17:03:10

@lmergen the question, or challenge, around middleware to convert from a future to a core.async object is interesting, and i think points to one of the areas where Clojure's insistence on being a hosted language makes it more difficult for new folks coming into the language. you are forced to be aware of the underlying jvm Thread abstraction, the fact that it corresponds to a real OS thread and the unfortunate fact of the still pervasive presence of blocking IO in jvm world. core.async is much bigger than it would otherwise be, and has some impedance mismatches with other async abstractions, because of this.

shanekilkelly18:03:21

@jonahbenton: funny your comment should be right there when I opened slack simple_smile I ran into the same problems (and came to roughly the same conclusions) a few weeks ago when I was researching how to build pervasively non-blocking services in clojure

shanekilkelly18:03:03

and all the core.async based solutions seemed to end up being really bad re-inventions of the Actor model from Erlang, but without any of the performance, robustness, or any of the debugging/monitoring tools

johanatan18:03:04

@jonahbenton: converting from core.async to promise/future is possible via promesa (just 'resolve' the callback with the result from a channel when it becomes available)

johanatan18:03:16

and promise-chan is another piece of the puzzle. there's a ton of ways to do it actually. it certainly is a good idea to understand the underlying facilities being relied upon however

jonahbenton18:03:15

hey @shanekilkelly sure, though i wouldn't put the blame on core.async specifically; the jvm doesn't offer an appropriate virtualization abstraction to build the equivalent of actors.

shanekilkelly18:03:26

Surely the underlying facilities are incidental complexity though? I mean, we just want ways to ensure our CPUs are as hot as possible, is ThreadpoolExecutorServiceFactoryBean really bringing anything useful to the table?

shanekilkelly18:03:17

and yeah, it’s not the fault of core-async alone, not at all. it’s just there’s no cohesive, overarching narrative on how to do async, non-blocking programming in clojure

shanekilkelly18:03:12

I mean, it’s 2016, surely we can’t just be relying on early-2000’s java knowledge to get this kind of stuff done? simple_smile

johanatan18:03:14

i don't buy the "JVM limits us" argument. CSP/cooperative multithreading and green/lightweight threads are both accomplished outside the JVM's purview (or rather within single JVM concepts)

jonahbenton18:03:42

yeah. did you wind up settling on a particular set of tools? sounds like aleph + co were closest?

shanekilkelly18:03:53

aleph does look promising

johanatan18:03:08

and the key to erlang is apparently cheap, lightweight "processes". I can think of ways to get those within a JVM context

shanekilkelly18:03:17

true, that is what erlang does, but the difference is that in Erlang you can be sure that everything you touch will understand that paradigm. In Clojure we’re left to fend for ourselves, to figure out how to make aleph, channels, promises, jdbc, play well together and not block

jonahbenton18:03:26

@johanatan: right- you can sort of get something like them, but the mechanical differences are very significant.

lmergen18:03:29

@shanekilkelly: i think you kind of correctly phrased the point i was trying to make, and i indeed think it is correct that the jvm is to blame.

shanekilkelly18:03:29

I’d say it’s more like JVM culture is to blame, honestly. there’s nothing to technically stop us from doing that stuff well, but we’re wedded to the “yay java” mindset, and so we’re stuck with blocking java apis mixed with sort-of-non-blocking async code

johanatan18:03:51

well yes standardization is good. nobody expects non-Akka Scala code to work seamlessly with with Akka-heavy Scala code though.

lmergen18:03:56

but in all other languages that excel at concurrency, there is just one pervasive concurrency approach that is enforced everywhere. clojure, in this aspect, is really lacking, and is not opinionated enough.

johanatan18:03:01

well yea JVM culture should be burned to the ground

johanatan18:03:08

Factories, beans and XML along with it

shanekilkelly18:03:24

i think we’re all in furious agreement here simple_smile

lmergen18:03:37

haha that's great for a change

jonahbenton18:03:48

well, there are technical obstacles. the jvm is really a spec for a JIT. a JIT is a very different thing from BEAM

shanekilkelly18:03:07

eh, yeah, and it kinda sucks. I really like clojure-the-language, but have come to really dislike clojure-the-implementation. The java stuff just pokes through too much

lmergen18:03:20

"furious agreement"... i'm going to remember that one lol

johanatan18:03:38

has anyone researched feasibility of hosting Clojure on BEAM?

shanekilkelly18:03:45

i’ve personally started to put more energy into Elixir and Erlang, as they seem to have much more promise for large-scale computing.

shanekilkelly18:03:56

ooh, good question.

lmergen18:03:59

yes i am a big fan of Elixir also

jonahbenton18:03:07

not introducing disagreement, but what's interesting about the "java stuff" is that it's definitely a significant reason clojure has seen the adoption and success that it has

lmergen18:03:13

well rather, the erlang vm

jonahbenton18:03:27

there are a zillion lisps with no libraries

shanekilkelly18:03:56

@jonahbenton: you raise a good point, it certainly helped with initial adoption. but I fear it may hinder further adoption.

lmergen18:03:03

@jonahbenton: yep this in fact the only reason i would be able to use clojure. because of jvm.

jonahbenton18:03:56

but agree that if you don't come into it with a significant java background, it has many bizarre, mysterious and unfortunate sharp edges

shanekilkelly18:03:10

i kinda feel the clojure world is divided roughly into two camps, those with a professional/financial/emotional investment in Java/JVM, and those without. The latter camp (including myself) seem to struggle with the degree to which Clojure is wedded to JVM culture

johanatan18:03:43

@jonahbenton: what about the success of ClojureScript? simple_smile

jonahbenton18:03:55

yeah- i think this is due to the "hosting" philosophy

jonahbenton18:03:24

@johanatan: exactly- cljs is a pretty similar language, hosted in the js vm

jonahbenton18:03:37

which to a certain degree shows the success of the hosting approach

shanekilkelly18:03:47

clojurescript is interesting, simple_smile it seems to not have as many interop problems because the existing JS culture is not as heavy on 90’s style object frameworks

shanekilkelly18:03:01

the interop between the two worlds is a lot smoother

jonahbenton18:03:03

building exactly the same abstractions, the same language, on both the jvm and js vm would be an impossilbe task

lmergen18:03:31

especially considering the lack of threads in the js vm :)

johanatan18:03:33

and yet core.async works on both

jonahbenton18:03:37

yeah, and numbers

bronsa18:03:42

@shanekilkelly: I don't think that's a correct depiction of clojure users

shanekilkelly18:03:16

@bronsa: it’s just my perception from hanging around the Clojure community for about five years.

shanekilkelly18:03:32

I’m sure there are some who would disagree with my assesment simple_smile and that’s fine

bronsa18:03:43

most clojure devs I know are ok with the JVM

johanatan18:03:15

JVM and JVM/Java culture are different entities though. I'm ok with the former-- not so much the latter

shanekilkelly18:03:16

out of curiousity, did they have a pre-existing investment in the JVM?

bronsa18:03:21

in fact, I feel that clojure devs that are not ok with the JVM are a striking minority

jonahbenton18:03:43

minority, but probably growing

bronsa18:03:46

@shanekilkelly: some did, some didn't

lmergen18:03:11

@bronsa: that might be out of ignorance, though

jonahbenton18:03:13

i've been working with the jvm for 20 years, but i see the pain on a daily basis

bronsa18:03:26

I've started using clojure some 6 years ago myself without any prior knowledge of JVM ecosystem, never found it an issue

shanekilkelly18:03:39

'cos I meet a lot of clojure newcomers who a pretty much instantly put off by the underlying Java stuff. to someone coming to clojure from python or ruby it feels like a hugely bloated platform filled with legacy libraries they have no interest in, so the “oh it’s on the JVM and thus it’s good” arguement doesn’t wash.

jonahbenton18:03:05

yeah, some folks have avoided the jvm over their careers, the way some folks avoid windows

jonahbenton18:03:16

or avoid php or...

bronsa18:03:20

@shanekilkelly: really? who's forcing anybody to use those legacy libraries?

shanekilkelly18:03:32

well, for instance jdbc

shanekilkelly18:03:44

if you want to talk to databases from clojure it can’t be avoided

lmergen18:03:50

actually, i think Clojure does a pretty decent job at hiding the jvm stuff... it can always be better, but i think it's not too bad

bronsa18:03:10

there are tons of actively maintained and good clojure libraries on top of jdbc

hiredman18:03:56

I have some clojure scripts that run postgres's psql and read and write to its stdio

shanekilkelly18:03:06

take this for example: https://twitter.com/shanekilkelly/status/703299796476563457 if you want to talk to postgres from clojure, you must get your hands dirty with java object manipulation

shanekilkelly18:03:40

that’s the kind of thing us experienced clojure devs can deal with, but is very off-putting for someone coming from (for example) python and psycopg

hiredman18:03:02

if you write something better, people will use it

bronsa18:03:12

@shanekilkelly: must? surely it's your choice to use jdbc

jonahbenton18:03:51

@shanekilkelly: curious about that code- what was the use case?

shanekilkelly18:03:35

use case: simply reading data from postgres, nothing fancy and any other language can do it without that degree of boilerplate.

juhoteperi18:03:53

Those interfaces are in fact not for interop but they are defined by the java.jdbc clojure lib

juhoteperi18:03:18

But yeah, would be nice to have lib defining the implementations for Postgres types

hiredman18:03:28

they aren't suitable for defaults either

juhoteperi18:03:47

Yeah those vector and map implementations are "dangerous"

shanekilkelly18:03:52

it’s just one example of the kind of stuff that’s considered normal in Java/JVM land, but seems insane to anyone coming from other languages

bronsa18:03:06

@shanekilkelly: those languages had to write drivers/libraries form scratch. We get that for free from the JVM ecosystem and we get the choice to use them or not.

hiredman18:03:32

I think you are just used to similar things in other languages

fommil18:03:43

what's the idiomatic way of dealing with a non-200 response in an HTTP call in clojure?

shanekilkelly18:03:45

and yet that boilerplate is part of luminus, the closest thing clojure has to a web framework. not only are we lumped with ~100 lines of boilerplate to do a simple thing, the boilerplate isn’t even safe

mpenet18:03:53

About beam, it might get a tracing jit via a llvm impl in the future

fommil18:03:02

especially if it's in an http-kit callback

hiredman18:03:35

every time I look at a rails project, all I see is weird boiler plate that seems to be cargo culted from one project to another

hiredman18:03:52

which is not to call out rails

hiredman18:03:13

I am just saying, it is easy to ignore the familiar

bronsa18:03:19

@shanekilkelly: I still don't see how any of what you're saying can be a criticism of clojure being hosted on the JVM -- it feels like what you're actually complaining about is lack of "user-friendlier" libraries?

hiredman18:03:34

people who regularly use rails likely don't even notice all that stuff

johanatan18:03:49

@fommil: disable throw-on-failure and check the status?

shanekilkelly18:03:54

@hiredman: that’s kinda my point here though, this same lump of boilerplate ends up copy-pasted into every project that touches postgres, it’s not good practice at all, but for some reason it’s considered normal.

bronsa18:03:11

just release a library?

hiredman18:03:14

shane: I don't think that is true

johanatan18:03:28

ya, if that's true, then release it in a library

hiredman18:03:31

I have used postgres without any of that

shanekilkelly18:03:04

@hiredman: how are you handling arrays/json/etc? curious to see how it’s handled elsewhere

shanekilkelly18:03:28

because i know for a fact that jdbc won’t handle those cases without being told how to do so.

hiredman18:03:35

it depends, generally I don't handle those things, and I store relational data in postgres

shanekilkelly18:03:59

fair enough, but not encountering the problem is not the same as the problem not being there simple_smile

hiredman18:03:10

the few times I have used arrays it bit me because of issues about creating indices on arrays

shanekilkelly18:03:30

ah, yeah, indices on array columns can be a bit horrible

hiredman18:03:43

if even possible

hiredman18:03:29

that code means every time you pull json out of the db you decode it, and if you want to just return it to a browser you re-encode it again

shanekilkelly18:03:35

but without that code the process will simply crash

lmergen18:03:44

seems like clojure's postgres async library does a decent job, though?

dominicm18:03:18

re the postgres protocol-related stuff, it could easily go into a library and be imported in.

hiredman18:03:58

I doubt the process just crashes (but it has been a while)

shanekilkelly18:03:32

yeah, it does simple_smile it’s not like it just gives you the field is text. it fails to understand how to decode it and dies

hiredman18:03:10

well, then write something better

dominicm18:03:16

I'm not sure that's the best response.

hiredman18:03:35

what other response is there in open source?

shanekilkelly18:03:31

There are open-source communities which, frankly, do a much better job of this stuff. contrast this with how things are handled in the Elixir community. They care about ergonomics, and frankly it shows.

hiredman18:03:16

the postgres jdbc driver is provided to you free, the clojure jdbc wrapper is provided to you free, expecting them to change, or people to provide an alternative to you free is really weird

hiredman18:03:09

I am not super wild about the java.jdbc library my self, it works fine, so meh

shanekilkelly18:03:40

I think we’ve crossed wires somewhere here. I don’t expect or want anything from anyone for free. but it would be nice for us to collectively acknowledge that there are issues here, which are caused directly by the close reliance on the jvm ecosystem

hiredman18:03:05

my issues are often the results of wrapping jvm libraries, my experience has been much better when I got rid of wrapper libraries and the like and depended more directly on the jvm ecosystem

dominicm18:03:15

Expectation isn't it so much. I guess your all-or-nothing statement of "go write a better jdbc" was a little blunt from my perspective. I think the conversation is more about considering the different possibilities (one of which is, write jdbc from the ground up). The other might be to produce a more beautiful layer, or tidy up some weird edge cases.

hiredman18:03:59

like, the reason you see big piles of the protocol extends like that, is people came in, and rejected the jvm dependence, and layered shim under shim to get away from it

lmergen19:03:24

well isnt that what programming languages are all about? moving away from low-level construct and defining abstractions that better fit their framework/appeoach in solving problems?

hiredman19:03:44

because you are using the clojure java.jdbc library, which has all these clojure protocols shimmed in, the way you change how you interact with jdbc stuff is by fiddling with protocols, which is a global thing, and actually against the recommend best practices for protocols

hiredman19:03:26

(general advice is in order to extend a protocol to a type safely you need to own either the type or the protocol, for extending jdbc protocols to clojrue vectors, you own neither)

hiredman19:03:03

but there is no guarantee that the build up you find on the internet is any good

dominicm19:03:47

That's a good point, and the cause of a lot of my frustration with jdbc, that's an excellent observation about the problem under it.

hiredman19:03:21

it reminds me of the dijkstra quote

hiredman19:03:25

"Being abstract is something profoundly different from being vague … The purpose of abstraction is not to be vague, but to create a new semantic level in which one can be absolutely precise."

bronsa19:03:03

@hiredman: that's (only extending protocols/types you own) a good practice for library code, I find that it doesn't make much sense when writing app code

hiredman19:03:13

libraries you find very very often facilitate being vague

hiredman19:03:49

bronsa: yeah, but it means putting that code in a library is not good idea (as many people have told shanekilkelly to do)

hiredman19:03:01

I should say, libraries are often selected for their ability to be vague

hiredman19:03:59

the writer of a library is being precise, but generally the programmer that selects it for use, isn't selecting it for precision, but for ease of use, which is often a function of how much it hides so you can ignore for at least right now (but will bite you later)

hans20:03:17

I have a strange problem with agents and promises that I somehow can't figure out: I have two independent agents, and one asks the other to do some work for them and return the result by the way of a promise. But somehow, the two agents seem to be scheduled to the same thread and thus the operation blocks forever. Do I need to force them to different threads? Why?

(let [x (agent :x)
      y (agent :y)]
  (send-off x (fn [this]
                (let [result (promise)]
                  (println "agent x, asking y to do something")
                  (send-off y
                            (fn [this]
                              (println "agent y, delivering promise")
                              (deliver result "hello")
                              this))
                  (println "agent x got result" @result))
                this)))

hiredman20:03:57

agent sends inside an agent are held until the action completes

hiredman20:03:31

there is a function to release pending sends, but I would suggest you look at an alternate means of doing whatever you are doing instead

hans20:03:28

where is that behaviour described? what is a better way than agents to serialize access to a single resource?

hiredman20:03:09

a lock serializes access

hiredman20:03:26

agents are an async reference type

hans20:03:12

so, you're suggesting that i should use reach out to the java lock api?

hiredman20:03:00

or using clojure's locking macro, or use a queue or something, depending on what you are doing

hiredman20:03:34

if you insist on using agents the way to do it is to manually pass continuations around

hans20:03:16

i don't insist on anything, i'm just looking for the right synchronization primitive to use.

hiredman20:03:31

(send x (fn [value] (let [a agent] (send x (fn [v] (send a + v)))))) or something

hans20:03:39

release-pending-sends seems to work fine..

hiredman20:03:43

that should be 'agent'

hiredman20:03:31

if all you care about is serializing access to a resource, then that is a lock

hans20:03:22

I'm using the agent to communicate state about the serialized activity to the outside, so i think my current scheme is just fine

hiredman20:03:38

the fact that you want return values from stuff, means you don't want the async nature of agents

hans20:03:55

i'll remember the locking macro next time.

hiredman20:03:00

square peg in a round hole

hans20:03:06

i sometimes want to wait, and sometimes i don't want to.

hans20:03:12

thank you for your help.

hans20:03:39

I'm looking at the documentation of the locking macro now, and it reads: "Executes exprs in an implicit do, while holding the monitor of x." What is "the monitor of x"? Any pointers?

hiredman20:03:02

the jvm lets any given object be treated as a lock, the jvm instructions for holding and release the lock on an object are monitorenter and monitorexit

hiredman20:03:59

there can be some draw backs to using an arbitrary object as a lock instead of one of the lock objects from java util concurrent, but for many cases they are fine

hiredman20:03:27

if you are familiar with java's synchronized methods, the objects monitor is what hey synchronize on

hans20:03:04

Well, one of the cases where I used agents as a serialization device can be simplified by locking nicely.

hans20:03:36

I'm not familiar with java, which is probably my problem here simple_smile

hiredman20:03:22

depending on what you are doing, you can also just create a threadpool with a single thread and run things on that threadpool to serialize the actions, that works best if you don't have an associated value that all the actions need

hiredman20:03:37

(sort of like how a lot of gui frameworks work)

thiagofm21:03:35

@hans what is the challenge now? simple_smile

hans21:03:02

thiagofm: well, it is still the same for what we've discussed... the place that i've simplified using locking is somewhere else

adamkowalski23:03:51

how do you guys typically deal with having a server and client app. do you have two repls open, one with figwheel running for the client, and a standard nrepl for the server portion?

adamkowalski23:03:14

or is there some way to have both your clj and cljs stuff all working together in one repl

richiardiandrea23:03:04

@adamkowalski: if you use cider you will have two buffers and you can split, but no, you cannot have everything works in one repl to my knowledge