This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2016-03-20
Channels
- # beginners (7)
- # boot (39)
- # braid-chat (3)
- # braveandtrue (1)
- # cider (27)
- # cljsjs (15)
- # cljsrn (6)
- # clojars (18)
- # clojure (307)
- # clojure-art (1)
- # clojure-brasil (1)
- # clojure-italy (2)
- # clojure-poland (3)
- # clojure-russia (61)
- # clojure-sdn (2)
- # clojure-taiwan (4)
- # clojure-uk (5)
- # clojurebridge (7)
- # clojurescript (19)
- # core-async (1)
- # core-matrix (1)
- # cursive (35)
- # datomic (3)
- # emacs (51)
- # euroclojure (3)
- # hoplon (20)
- # jobs (1)
- # keechma (1)
- # mount (3)
- # off-topic (2)
- # om (177)
- # onyx (96)
- # parinfer (4)
- # pedestal (4)
- # re-frame (19)
- # reagent (7)
- # untangled (5)
@lmergen what database are you using? Here's an async library for postgresql: https://github.com/alaisi/postgres.async
does anyone have experience with https://github.com/plumatic/schema
What did you think of it? I'm getting tired of looking at http://draw.io images to know what my data entities should be.
not sure what http://draw.io has to do with schema
I'm documenting entities using the Extended Entity Relation model (simplified). I use http://draw.io to house and make the diagrams.
I have fairly complex hierarchy of maps (I guess). I need to know what keys go where. Modeling helps make sure I can get all the data. Normally in Java I'd have an class structure to look at too. Clojure lacks a clear system like that. schema looked like it could help.
i normally use schema as a way to document a function (inputs and outputs), but only after that function is stable and I need some clarity
How would you have a Profile with an email, password and an address, where an address has all the parts of an address?
I don’t document the structures, only the functions. And the only time I need to look at what a schema looks like is when I’m trying to use the function and need specifics as to the shape of its inputs and/or outputs.
Hi, I've stucked with a problem I want my custom user.clj loaded in repl on startup, I have project_root/dev/user.clj
(ns user)
(def foo 2)
(defn dev
"Load and switch to the 'dev' namespace."
[]
(require 'dev)
(in-ns 'dev))
and in project.clj
:profiles { :dev {:dependencies [[org.clojure/tools.namespace "0.2.11"]]
:source-paths ["dev"]}}
but on repl start
user> foo
CompilerException java.lang.RuntimeException: Unable to resolve symbol: foo in this context
and it seems that dev dir is not on my classpath, checked with
(println (seq (.getURLs (java.lang.ClassLoader/getSystemClassLoader))))
I tried reloaded template (https://github.com/stuartsierra/reloaded)
lein new reloaded my-app
and user.clj is loaded on repl startup there, but still can't figure out problem with my project...@curtis.summers: well actually I'm using PostgreSQL through Korma, and am looking for the best way to wrap these long-running, blocking queries in an async way
but I feel like my best approach, if it's really important, to just use the async postgresql client api
@andfadeev: I put user.clj
in project root and it worked
i'm really confused by all these people having strong opinions on which concurrency mechanism to use... core.async
is a pretty decent async library, which appears to be the de-facto standard, but then i see the Funcool people abandoning core.async
and saying that the only proper way to deal with promises is not to use core.async
, but rather java8's native promises/future mechanism... i'm so confused what to do!
I read this on their site:
> Clojure comes with a builtin promise abstraction but it is designed only for blocking operations, and in async environments the blocking operations are completely discouraged.
so, if I understand it correctly, if I want to make a computation that is doing blocking I/O, I should use core.async
, but if it is just an expensive computation, I should use java8's promises?
imho, the right way to me sounds like core.async
should just use java8's promise mechanism if it's available ?
hey @andfadeev you are starting a repl with lein repl ?
can you share your entire project.clj ?
@lmergen core.async is library which implement CSP style of concurrency. And then Promises/Futures is another way of implementing concurrency.
Implementation wise core.async uses small thread pool and it shares threads by parking processes. Therefore it works efficiently when used with non-blocking I/O calls
I don't know Java8 promises that well but Clojure's own promise implementation is ok if you do something simple (for example start a background task and wait for the result)
However, with complex scenarios where you need to synchronize multiple parallel tasks core.async gives you more tools
hey @lmergen this sounds like it's getting too deep in the solution space weeds. What problem are you trying to solve?
at a high level, i'm "just" implementing an http api server which does some CRUD and sometimes a bit longer operations in the background
coming from other languages that usually have asynchronous operations baked into the language, doing things synchronously feels like a step back
so, right now, I'm exploring the possibilities I have for making my webservice a bit more async -- not that I'm going to be implementing this tomorrow, but just for a high-level idea of what kind of design I should keep in mind
I feel like using an asynchronous database wrapper for PostgreSQL will be a first major step in making this happen
so it's not that I'm trying to solve a specific problem, but more just trying to understand the state of the ecosystem and what design principles to keep in mind making my Clojure applications more async
gotcha- can i ask- when you're talking about async baked into the language, are you talking about js?
but yes, node.js is a good example in that it defines a clear way to deal with this (callback hell) and every library uses it
it seems that the Clojure community, however, is mostly blocking, and a lot of the most frequently used libraries don't have async support
gotcha- so, in terms of io, java's history for its first 10-15 years was around blocking io. non-blocking io in javaland has come into its own in the last 5+ years, but many battle tested libraries, database drivers, etc, still use blocking io
Is there any good book or anything related to CSP, to know the problems, solutions and so on
Just wrestling with core.async whenever I have to use it doesn't seem like a nice approach
separately, java has always had a good threading story, but they're os threads
johanatan: yeah I figured it would be like that. seems like new projects like pedestal seem to promote an async-first design ,though
there was a failed foray into green threads many years back; they also are just making a return
in my case, I might drop Korma in favor of a pure DSL that generates SQL from abstract structures, and then execute the query manually using Clojure's asynchronous postgresql library
I think this tutorial is pretty good: http://www.braveclojure.com/core-async/
but, as a community as a whole, it feels like more and more libraries are starting to embrace core.async
, eh?
yeah, core.async is widely used because it comes from cognitect and works across hosts, but there are many, many libraries
there isn't a consistency in approach on the jvm- unlike erlang- because several different models have been dominant over the years, and libraries can survive for decades
zach tellman has a widely used async library called aleph that is clojure-only: https://github.com/ztellman/aleph
In general I really enjoyd this book https://www.goodreads.com/book/show/18467564-seven-concurrency-models-in-seven-weeks. Besides low level concurrency models, it covers promises/futures, CSP and Erlang actor model
and he is working on a book: http://elementsofclojure.com/
anyway, dropping korma and working closer to the wire probably makes sense
@jonahbenton: thanks, but that's exactly my problem at the moment.. there are so many competing solutions for the same problem in Clojure, I'm not sure what to choose
I think the safest bet is to just use core.async
by default, unless i have a very good reason not to
yes- there is no good reason not to. it can support both io- and cpu-based concurrency
i think the downside for some folks is that core.async's surface area has expanded dramatically
so someone coming into async without much background can get overwhelmed
@johanatan: makes sense, thanks for the info
@lmergen: Clojure has future
too (which is pretty much the default async story in Scala
for now, I think I'll stick with go
for CPU-intensive tasks, and thread
for IO-intensive tasks
Clojure agents aren't really analogous to Akka actors though. If you understand the underlying mechanisms behind each of these, then the choice will make itself
@lmergen: I tend to run long-running background jobs in a separate worker process that takes items off a queue.
yes, I use a service oriented architecture, we're putting jobs in Amazon's SQS (= a queue)
Curious if this is possible: I have a higher-order function which relies on outer (i.e. closure) bindings. Is there a way of running the function while supplying arbitrary values for those bindings? Example:
(defn outer []
(let [outer-a 1
outer-b 2]
(fn inner []
[outer-a outer-b])))
(let [f (outer)]
; I realize that with-bindings does not actually do the job here
(with-bindings [outer-a 10
outer-b 20]
(f)))
As you might guess, this is for testing. I'm open to other approaches for inserting arbitrary data here!
(To be exact, I'm trying to test the inner, or "renderer", function of a re-frame
component in ClojureScript. The renderers rely on a subscription channel set up in the initial (outer) component invocation.)@lmergen: Also, if you drop korma for straight SQL, you might also look at HugSQL (http://www.hugsql.org), which has protocols for the underlying db library, and there is a postgresql async adapter for it: http://www.hugsql.org/#adapter-community
@lmergen: I don't see it as a disadvantage at all. Each primitive has unique semantics
@curtis.summers: I was actually looking at http://funcool.github.io/suricatta/latest, which had the "core.async is not good" quote which got me confused
@amacdougall: is there a reason not to produce a function that just takes parameters and has defaults set from the initial definition? do you not have control over outer?
@lmergen looks like that lib recommends using agents for async
@lmergen: As of version 0.4.x, HugSQL has Clojure Expressions and Snippet support that give you composability: http://www.hugsql.org/#using-expressions Of course, HugSQL is primarily an "SQL first" workflow, so if you prefer to keep things entirely in a "Clojure first" workflow, then you HoneySQL or Suricatta support that.
@curtis.summers: yeah i was aware of this, but this would pretty soon make my whole application a giant mess
@lmergen: do you need to be able to cancel your long running queries?
@lmergen: Yeah, I get it. Some prefer to build everything in Clojure data structures. I've found DSLs to be cumbersome and/or leaky for my purposes. Good to have options on both sides, I say.
do you have a means to control how many are in flight at once?
@johanatan: i prefer a thread-pool like approach, but am actually used to having the environment I work in (be it Erlang, Go, or Haskell) handle that for me
yup- two slightly different implementationsof a similar model. future returns a dereference-able object; thread is part of core.async and returns a channel
but what if the framework i work in (for example, pedestal) works with core.async
? is there middleware to convert a future
to a core.async
channel?
@jonahbenton: In re-frame
, a component is expected to be a function which returns Hiccup, or to be a function which returns a function which returns Hiccup. In the latter case, the returned function must have the same params as the outer one: the outer function does setup, usually establishing a "subscription" to the data atom.
(defn animals-view []
(let [birds (subscribe :birds)]
(fn animals-view-renderer []
(into [:div]
(map render-individual-bird @birds)))))
In the above example, re-frame
will run the outer function once, and "install" the inner function as a renderer, which will be run every time that subscription data changes.
...You know, I suppose I could write it like this:
(defn animals-view [& [birds]]
(let [birds-sub (subscribe :birds)]
(fn animals-view-renderer [& [birds]]
(into [:div]
(map render-individual-bird (or @birds-sub @birds))))))
...though at that point, I'm doing weird stuff to my design to enable testing / devcards. I think it's not going to pay off.Uh, sorry for the wall of rubber-ducking.
@lmergen thread should be fine for your use case. no need to cancel, and you can let core.async handle the pool, and you get the composability of channels
@amacdougall: i see. but you need to change some of the setup/configuration outside of a data subscription? (i haven't used re-frame, may not have the terminology right)
Well, really I just want to test the inner function by supplying it some arbitrary data instead of having to create the central datastore, subscribe to it, then reset it after the test. However, doing that whole process is a fairly reasonable approach as well.
This is pretty much what the author suggests anyway, I notice: https://github.com/Day8/re-frame/wiki/Testing#components---part-2a
So, I'll do that! If I think testing these UI components is even worthwhile. I have a hunch it's not. ¯\(ツ)/¯
heh, right
Test behaviors, not UI.
The main reason I wanted to get this working is for devcards, a CLJS thing where you can live-edit a component without needing to hook it up to real data. But re-frame
makes it fairly easy to load any arbitrary app state in a flash, so it's no big deal.
right, gotcha. i mean, if it makes sense from an app design perspective for the specific values to change in flight independent of initial configuration, you can make them dynamic vars, and then use with-bindings, but it sounds like that's not a use case that exists outside of testing
and perhaps not even then
My app is not gonna be all that large in the first place! It's just very easy to overthink my tooling. I think back to when I was editing "taglib definition files" in Java, compiling, turning Tomcat off, turning it back on... and I was perfectly happy, right? I just didn't know any better.
heh, painful memories.
Anyway, this helped solidify my thoughts. Thanks!
@lmergen the question, or challenge, around middleware to convert from a future to a core.async object is interesting, and i think points to one of the areas where Clojure's insistence on being a hosted language makes it more difficult for new folks coming into the language. you are forced to be aware of the underlying jvm Thread abstraction, the fact that it corresponds to a real OS thread and the unfortunate fact of the still pervasive presence of blocking IO in jvm world. core.async is much bigger than it would otherwise be, and has some impedance mismatches with other async abstractions, because of this.
@jonahbenton: funny your comment should be right there when I opened slack I ran into the same problems (and came to roughly the same conclusions) a few weeks ago when I was researching how to build pervasively non-blocking services in clojure
and all the core.async based solutions seemed to end up being really bad re-inventions of the Actor model from Erlang, but without any of the performance, robustness, or any of the debugging/monitoring tools
@jonahbenton: converting from core.async to promise/future is possible via promesa (just 'resolve' the callback with the result from a channel when it becomes available)
and promise-chan
is another piece of the puzzle. there's a ton of ways to do it actually. it certainly is a good idea to understand the underlying facilities being relied upon however
hey @shanekilkelly sure, though i wouldn't put the blame on core.async specifically; the jvm doesn't offer an appropriate virtualization abstraction to build the equivalent of actors.
Surely the underlying facilities are incidental complexity though? I mean, we just want ways to ensure our CPUs are as hot as possible, is ThreadpoolExecutorServiceFactoryBean really bringing anything useful to the table?
and yeah, it’s not the fault of core-async alone, not at all. it’s just there’s no cohesive, overarching narrative on how to do async, non-blocking programming in clojure
I mean, it’s 2016, surely we can’t just be relying on early-2000’s java knowledge to get this kind of stuff done?
i don't buy the "JVM limits us" argument. CSP/cooperative multithreading and green/lightweight threads are both accomplished outside the JVM's purview (or rather within single JVM concepts)
yeah. did you wind up settling on a particular set of tools? sounds like aleph + co were closest?
aleph does look promising
and the key to erlang is apparently cheap, lightweight "processes". I can think of ways to get those within a JVM context
true, that is what erlang does, but the difference is that in Erlang you can be sure that everything you touch will understand that paradigm. In Clojure we’re left to fend for ourselves, to figure out how to make aleph, channels, promises, jdbc, play well together and not block
@johanatan: right- you can sort of get something like them, but the mechanical differences are very significant.
@shanekilkelly: i think you kind of correctly phrased the point i was trying to make, and i indeed think it is correct that the jvm is to blame.
I’d say it’s more like JVM culture is to blame, honestly. there’s nothing to technically stop us from doing that stuff well, but we’re wedded to the “yay java” mindset, and so we’re stuck with blocking java apis mixed with sort-of-non-blocking async code
well yes standardization is good. nobody expects non-Akka Scala code to work seamlessly with with Akka-heavy Scala code though.
but in all other languages that excel at concurrency, there is just one pervasive concurrency approach that is enforced everywhere. clojure, in this aspect, is really lacking, and is not opinionated enough.
i think we’re all in furious agreement here
well, there are technical obstacles. the jvm is really a spec for a JIT. a JIT is a very different thing from BEAM
eh, yeah, and it kinda sucks. I really like clojure-the-language, but have come to really dislike clojure-the-implementation. The java stuff just pokes through too much
i’ve personally started to put more energy into Elixir and Erlang, as they seem to have much more promise for large-scale computing.
ooh, good question.
not introducing disagreement, but what's interesting about the "java stuff" is that it's definitely a significant reason clojure has seen the adoption and success that it has
there are a zillion lisps with no libraries
@jonahbenton: you raise a good point, it certainly helped with initial adoption. but I fear it may hinder further adoption.
@jonahbenton: yep this in fact the only reason i would be able to use clojure. because of jvm.
but agree that if you don't come into it with a significant java background, it has many bizarre, mysterious and unfortunate sharp edges
i kinda feel the clojure world is divided roughly into two camps, those with a professional/financial/emotional investment in Java/JVM, and those without. The latter camp (including myself) seem to struggle with the degree to which Clojure is wedded to JVM culture
@jonahbenton: what about the success of ClojureScript?
yeah- i think this is due to the "hosting" philosophy
@johanatan: exactly- cljs is a pretty similar language, hosted in the js vm
which to a certain degree shows the success of the hosting approach
clojurescript is interesting, it seems to not have as many interop problems because the existing JS culture is not as heavy on 90’s style object frameworks
the interop between the two worlds is a lot smoother
building exactly the same abstractions, the same language, on both the jvm and js vm would be an impossilbe task
yeah, and numbers
@shanekilkelly: I don't think that's a correct depiction of clojure users
@bronsa: it’s just my perception from hanging around the Clojure community for about five years.
I’m sure there are some who would disagree with my assesment and that’s fine
JVM and JVM/Java culture are different entities though. I'm ok with the former-- not so much the latter
out of curiousity, did they have a pre-existing investment in the JVM?
in fact, I feel that clojure devs that are not ok with the JVM are a striking minority
minority, but probably growing
@shanekilkelly: some did, some didn't
i've been working with the jvm for 20 years, but i see the pain on a daily basis
I've started using clojure some 6 years ago myself without any prior knowledge of JVM ecosystem, never found it an issue
'cos I meet a lot of clojure newcomers who a pretty much instantly put off by the underlying Java stuff. to someone coming to clojure from python or ruby it feels like a hugely bloated platform filled with legacy libraries they have no interest in, so the “oh it’s on the JVM and thus it’s good” arguement doesn’t wash.
yeah, some folks have avoided the jvm over their careers, the way some folks avoid windows
or avoid php or...
@shanekilkelly: really? who's forcing anybody to use those legacy libraries?
well, for instance jdbc
if you want to talk to databases from clojure it can’t be avoided
actually, i think Clojure does a pretty decent job at hiding the jvm stuff... it can always be better, but i think it's not too bad
I have some clojure scripts that run postgres's psql and read and write to its stdio
take this for example: https://twitter.com/shanekilkelly/status/703299796476563457 if you want to talk to postgres from clojure, you must get your hands dirty with java object manipulation
that’s the kind of thing us experienced clojure devs can deal with, but is very off-putting for someone coming from (for example) python and psycopg
@shanekilkelly: must? surely it's your choice to use jdbc
@shanekilkelly: curious about that code- what was the use case?
use case: simply reading data from postgres, nothing fancy and any other language can do it without that degree of boilerplate.
Those interfaces are in fact not for interop but they are defined by the java.jdbc clojure lib
But yeah, would be nice to have lib defining the implementations for Postgres types
Yeah those vector and map implementations are "dangerous"
it’s just one example of the kind of stuff that’s considered normal in Java/JVM land, but seems insane to anyone coming from other languages
@shanekilkelly: those languages had to write drivers/libraries form scratch. We get that for free from the JVM ecosystem and we get the choice to use them or not.
what's the idiomatic way of dealing with a non-200 response in an HTTP call in clojure?
and yet that boilerplate is part of luminus, the closest thing clojure has to a web framework. not only are we lumped with ~100 lines of boilerplate to do a simple thing, the boilerplate isn’t even safe
every time I look at a rails project, all I see is weird boiler plate that seems to be cargo culted from one project to another
@shanekilkelly: I still don't see how any of what you're saying can be a criticism of clojure being hosted on the JVM -- it feels like what you're actually complaining about is lack of "user-friendlier" libraries?
@hiredman: that’s kinda my point here though, this same lump of boilerplate ends up copy-pasted into every project that touches postgres, it’s not good practice at all, but for some reason it’s considered normal.
@hiredman: how are you handling arrays/json/etc? curious to see how it’s handled elsewhere
because i know for a fact that jdbc won’t handle those cases without being told how to do so.
it depends, generally I don't handle those things, and I store relational data in postgres
fair enough, but not encountering the problem is not the same as the problem not being there
the few times I have used arrays it bit me because of issues about creating indices on arrays
ah, yeah, indices on array columns can be a bit horrible
that code means every time you pull json out of the db you decode it, and if you want to just return it to a browser you re-encode it again
but without that code the process will simply crash
re the postgres protocol-related stuff, it could easily go into a library and be imported in.
yeah, it does it’s not like it just gives you the field is text. it fails to understand how to decode it and dies
There are open-source communities which, frankly, do a much better job of this stuff. contrast this with how things are handled in the Elixir community. They care about ergonomics, and frankly it shows.
the postgres jdbc driver is provided to you free, the clojure jdbc wrapper is provided to you free, expecting them to change, or people to provide an alternative to you free is really weird
I think we’ve crossed wires somewhere here. I don’t expect or want anything from anyone for free. but it would be nice for us to collectively acknowledge that there are issues here, which are caused directly by the close reliance on the jvm ecosystem
my issues are often the results of wrapping jvm libraries, my experience has been much better when I got rid of wrapper libraries and the like and depended more directly on the jvm ecosystem
Expectation isn't it so much. I guess your all-or-nothing statement of "go write a better jdbc" was a little blunt from my perspective. I think the conversation is more about considering the different possibilities (one of which is, write jdbc from the ground up). The other might be to produce a more beautiful layer, or tidy up some weird edge cases.
like, the reason you see big piles of the protocol extends like that, is people came in, and rejected the jvm dependence, and layered shim under shim to get away from it
well isnt that what programming languages are all about? moving away from low-level construct and defining abstractions that better fit their framework/appeoach in solving problems?
because you are using the clojure java.jdbc library, which has all these clojure protocols shimmed in, the way you change how you interact with jdbc stuff is by fiddling with protocols, which is a global thing, and actually against the recommend best practices for protocols
(general advice is in order to extend a protocol to a type safely you need to own either the type or the protocol, for extending jdbc protocols to clojrue vectors, you own neither)
That's a good point, and the cause of a lot of my frustration with jdbc, that's an excellent observation about the problem under it.
"Being abstract is something profoundly different from being vague … The purpose of abstraction is not to be vague, but to create a new semantic level in which one can be absolutely precise."
@hiredman: that's (only extending protocols/types you own) a good practice for library code, I find that it doesn't make much sense when writing app code
bronsa: yeah, but it means putting that code in a library is not good idea (as many people have told shanekilkelly to do)
the writer of a library is being precise, but generally the programmer that selects it for use, isn't selecting it for precision, but for ease of use, which is often a function of how much it hides so you can ignore for at least right now (but will bite you later)
I have a strange problem with agents and promises that I somehow can't figure out: I have two independent agents, and one asks the other to do some work for them and return the result by the way of a promise. But somehow, the two agents seem to be scheduled to the same thread and thus the operation blocks forever. Do I need to force them to different threads? Why?
(let [x (agent :x)
y (agent :y)]
(send-off x (fn [this]
(let [result (promise)]
(println "agent x, asking y to do something")
(send-off y
(fn [this]
(println "agent y, delivering promise")
(deliver result "hello")
this))
(println "agent x got result" @result))
this)))
there is a function to release pending sends, but I would suggest you look at an alternate means of doing whatever you are doing instead
where is that behaviour described? what is a better way than agents to serialize access to a single resource?
or using clojure's locking macro, or use a queue or something, depending on what you are doing
if you insist on using agents the way to do it is to manually pass continuations around
i don't insist on anything, i'm just looking for the right synchronization primitive to use.
I'm using the agent to communicate state about the serialized activity to the outside, so i think my current scheme is just fine
the fact that you want return values from stuff, means you don't want the async nature of agents
I'm looking at the documentation of the locking macro now, and it reads: "Executes exprs in an implicit do, while holding the monitor of x." What is "the monitor of x"? Any pointers?
the jvm lets any given object be treated as a lock, the jvm instructions for holding and release the lock on an object are monitorenter and monitorexit
there can be some draw backs to using an arbitrary object as a lock instead of one of the lock objects from java util concurrent, but for many cases they are fine
if you are familiar with java's synchronized methods, the objects monitor is what hey synchronize on
Well, one of the cases where I used agents as a serialization device can be simplified by locking
nicely.
depending on what you are doing, you can also just create a threadpool with a single thread and run things on that threadpool to serialize the actions, that works best if you don't have an associated value that all the actions need
thiagofm: well, it is still the same for what we've discussed... the place that i've simplified using locking
is somewhere else
how do you guys typically deal with having a server and client app. do you have two repls open, one with figwheel running for the client, and a standard nrepl for the server portion?
or is there some way to have both your clj and cljs stuff all working together in one repl
@adamkowalski: if you use cider you will have two buffers and you can split, but no, you cannot have everything works in one repl to my knowledge