Fork me on GitHub
#clojure
<
2020-05-14
>
Ravi Kumar02:05:26

Hi, I am Ravi, working on Clojure for last 6 months. I have a project where I am using compojure, using toucan-db to connect to pgsql database. Now I am trying to connect to another pgsql server, to store log data. But not able find a way to connect to both databases from the same api function. How to achieve it in clojure using db-connection function in toucan db? Any example would help.

didibus02:05:29

(binding [*db-connection* <other-db-conn>]
  (db/...))

didibus02:05:07

Ousp, sorry its just binding

seancorfield03:05:02

This is why dynamic vars used for global state are such a terrible idea.

Ravi Kumar03:05:22

I think the library is trying to be helpful by having a default value, and allowing users to change to different db configuration when needed. Till the concept is clear, it is not very obvious. Coming from C++ background, I am still getting accustomed to Clojure way of doing things.

seancorfield04:05:48

Dynamic variables are "convenient" but come with a lot of restrictions on how you can use code based on them. They are global mutable state which is just a terrible idea. They were used a lot in the early days of Clojure a decade ago, but most libraries have moved away from them now (with one or two notable exceptions) and that sort of architecture is considered an anti-pattern by most Clojurians.

seancorfield04:05:30

I wouldn't use any library that is based on that principle (and libraries that I have maintained have all moved away from it).

seancorfield04:05:54

My background also includes C++ -- eight years on X3J16 (and three years as secretary) as well as producing a compiler front end that was used for static source code analysis. Clojure will take a bit of getting used to after C++ 🙂

Ravi Kumar07:05:51

Got it. Thank you.

myguidingstar08:05:57

@UTLP63467 toucan has many design flaws, see their github issues for more. Have you looked at other "ORM" approaches like https://github.com/ReilySiegel/EQLizr https://github.com/exoscale/seql or https://walkable.gitlab.io

Ravi Kumar08:05:35

I will check them. Thanks

g04:05:09

can anybody recommend any libraries that make it easier to test applications with kafka?

j07:05:11

I don't know what's your exact requirements, but jackdaw is a great lib to work with Kafka, with testing support

Darin Douglass11:05:45

^ what he said. Jackdaw is great.

g04:05:28

specifically, in cleaning up state between tests

g04:05:46

just curious if anything already exists

ujjwalt04:05:42

What are some of the best ways to build an asset pipeline in clojure web apps like building and serving css and images

dpsutton14:05:15

https://hub.docker.com/_/clojure on docker hub, these images are listed as "Docker Official Images". Are these official in any way?

😮 4
lukasz14:05:20

AFAIR not really, best example of that is Elasticsearch - the "official" Docker image is not in fact maintained by Elastic (or wasn't I don't know what's the status now) - the official images were provided by Elastic's own public registry. If you look at the Clojure image's readme - it links to https://github.com/Quantisan/docker-clojure and not as I would expect http://github.com/clojure/docker (or something like that)

lukasz14:05:35

It's just a bit shady marketing from Docker ;-)

4
dpsutton14:05:38

awesome. thank you

dominicm15:05:13

Official to docker, not official to Clojure.

dominicm15:05:19

This is a common question around Docker Hub, e.g. the mongo docker image isn't necessarily officially by mongodb corp either.

jsa-aerial15:05:20

So, does 'official to docker' actually have any significance?

paul a15:05:14

from https://docs.docker.com/docker-hub/official_images/ > Docker, Inc. sponsors a dedicated team that is responsible for reviewing and publishing all content in the Official Images. This team works in collaboration with upstream software maintainers, security experts, and the broader Docker community. > While it is preferable to have upstream software authors maintaining their corresponding Official Images, this is not a strict requirement. Creating and maintaining images for Official Images is a public process. It takes place openly on GitHub where participation is encouraged. Anyone can provide feedback, contribute code, suggest process changes, or even propose a new Official Image.

lukasz15:05:25

IMHO, no

👍 4
ghadi15:05:13

it's slightly better than "random unverified stuff off github"

ghadi15:05:13

I generally do my artifact builds outside of docker, then put the jar inside a normal JDK image

👍 4
ghadi15:05:33

It's Just a Jar™

Adrian Smith16:05:38

It'd be cool if there was some better Clojure merch out there, using assets like: https://github.com/tallesl/Rich-Hickey-fanclub/tree/master/cartoon but all official and above board

seancorfield16:05:37

It would be nice to have the parentheses stickers on there (but they're not Clojure-specific).

💯 4
lilactown18:05:22

do transducers get applied to channels on put or take?

Lennart Buit18:05:43

Testing in my repl it seems to be on put:

(def chan (async/chan 1 (map (fn [v] (println "Effect") v))))
=> #'dev/chan
(async/>!! chan 1)
Effect
=> true

leonoel18:05:28

it depends

leonoel18:05:34

if the channel buffer is full, the transducer will be run on the following take

Lennart Buit18:05:09

huh yeah, interesting

ghadi18:05:49

transducers transform the process of putting into a channel

ghadi18:05:10

specifically the transformed process is "add a message to a channel's buffer"

ghadi18:05:15

so whenever that happens

lilactown18:05:52

yeah, it’s making sense as “before put completes” the transducer will be applied

lilactown18:05:24

so if the buffer is full, the put will block until there’s space and then will apply the transducer before unblocking

jmromrell19:05:18

If I create a new namespace and navigate to it, I appear to lose access to clojure.core. Is there a way I can enable use of clojure.core within the newly created namespace?

(let [new-ns (create-ns (gensym "temp-"))]
  (in-ns (ns-name new-ns)))
(+ 1 1)
Returns:
Syntax error compiling at (C:\Users\<my-username>\AppData\Local\Temp\form-init12166700272800114096.clj:3:1).
Unable to resolve symbol: + in this context

hiredman19:05:33

You don't need to call create ns there, in-ns implicitly creates namespaces

hiredman19:05:53

Symbols are resolved via the current namespace

hiredman19:05:36

Namespaces by default don't have any refers or aliases

seancorfield19:05:04

user=> (in-ns 'foo.bar)
#object[clojure.lang.Namespace 0x308a6984 "foo.bar"]
foo.bar=> (clojure.core/refer-clojure)
nil
foo.bar=> (+ 1 1)
2
foo.bar=> 

hiredman19:05:10

When the ns macro creates a namespace it implicitly refers clojure.core

jmromrell19:05:16

It looks like it works if I intern 'require into the new namespace and then (require '[clojure.core :refer :all])

jmromrell19:05:24

refer-clojure looks like exactly what I was looking for!

jmromrell19:05:26

Thank you both 🙂

seancorfield19:05:59

Note that you need clojure.core/refer-clojure rather than just refer-clojure because the latter won't exist as a symbol in the new ns.

seancorfield19:05:37

Is there a reason why you're creating a new namespace with a random name? @jmromrell

seancorfield20:05:33

Another possible option?

user=> (eval (list 'ns (gensym "temp-")))
nil
temp-149=> (+ 1 1)
2
temp-149=> 

jmromrell20:05:49

I'm dynamically running some clj I read in from a database. I might be running more than one such clj file at a time. As such, I want to evaluate each in their own namespaces to avoid possible collisions.

jmromrell20:05:16

I'm additionally hoping, though am not sure if this is the case, that calling remove-ns on the namespace when I'm done will allow any state left behind by the evaluation to get garbage collected.

kenny20:05:23

Do folks use reducers in a long running production service? The implicit parallelism seems like it’s asking for trouble.

rickmoynihan21:05:42

There’s no implicit parallelism in reducers; it’s very explicit, as you only get it if you opt in to it by calling r/fold

kenny21:05:35

I meant implicit threadpool with no control over how various things can utilize that pool.

rickmoynihan22:05:23

If you need control over the pool; then yeah you can’t use them; but it depends what you’re doing. Yes fold jobs submitted to that pool will contend with each other; but that’s the case anyway. The main difference is I guess there’s no control over the fairness policy… so a large fold job will contend and may block a smaller one; but throughput will be as high possible; just not responsiveness. If you want some kind of backpressure; just put a channel or a java.util.concurrent queue in front of it.

kenny22:05:53

Yes, my concern was with the "fairness". It seems very easy for a multi-purpose service to have certain jobs take over the pool. Not sure how those backpressure mechanisms would help with reducers & fairness.

rickmoynihan22:05:25

well it would help by failing fast; rather blocking indefinitely

rickmoynihan22:05:34

though yes that might not be acceptable

kenny22:05:28

Oh, I see. As with most things, this seems very use-case dependent 🙂 With that approach, you'd also need to know the size of the job upfront, which may not be realistic.

rickmoynihan22:05:16

parallelism always is; that’s why clojure provides many ways to achieve it a la carte.

kenny22:05:21

Out of curiosity, are you using reducers + parallelism in a long running, production service?

rickmoynihan22:05:58

No, not yet… though it’s mainly because few of the things I deal with are ameneable to fold. Or if they I’ve been getting the perf I need out of a sequential approach; or have been achieving parallelism in a different way.

kenny22:05:47

Curious how many folks are actually using it in production

kenny22:05:13

My feeling is that most needs require a "use-case" specific solution.

rickmoynihan22:05:39

However there is an area of one our systems where I might one day choose to use them in production… Essentially we have a job scheduling process that builds large downloads. Those downloads are already async things; i.e. using core.async, we either serve users a cached download we’ve already built; or we generate them. If we need to generate it on demand, our users have to wait and fall into a holding process / wait screen etc for it to build. These jobs can sometimes take an hour or more to generate; and they can contend with each other… Part of the processing to build those downloads might benefit from using r/fold; and if the jobs on the forkjoinpool contend with each other it won’t matter; as users are already waiting. Does that make sense?

kenny22:05:42

Yes! That sounds like the perfect application. All jobs equal and a service that does 1 thing.

rickmoynihan22:05:07

well the service itself is actually monolithic; i.e. the download generation is done in the same process as many other aspects of the app. No need to scale out across machines/services when you don’t have to handle millions of requests - you can do a huge amount in one JVM with a lot less developer friction. Internally though the processes are essentially separate services provided to the app that communicate through various means e.g. core.async channels being the main one.

kenny22:05:35

And you're not concerned about this download generation eating up all the resources the many other tasks your monolith performs?

rickmoynihan22:05:12

Not all resources no, as the number of threads servicing those channels is configured. If you mean am I concerned about r/fold s potential addition swamping the system. We’d need to investigate it; but I suspect it would be fine, because again though the fjpool is assigned to all cores, the O/S will still switch in other threads that need to do work. And the r/fold won’t be over the whole download process; just bits of it; which may even themselves effectively be chunked.

rickmoynihan22:05:03

anyway it’s largely hypothetical at this point. r/fold may not even be faster than reduce

rickmoynihan22:05:02

well it would if the data was in the right form; i.e. vectors or hashmaps… but converting into that form might itself blow the perf advantage.

kenny22:05:39

Interesting. My concern was with r/fold swamping the system. I'd also be concerned about the potential increase of latency for other jobs.

rickmoynihan23:05:50

Yes latency for other requests is certainly a factor… but infrequent and relatively small spikes in latency can be tolerated. As I said earlier if your service isn’t under high load or SLAs, then you can get away with monolith deployments. It’s a trade off for sure… I should add that it would be pretty trivial to move this subsystem to another machine or cluster behind a load balancer if we needed to. Config is done via integrant; so it would essentially just be a matter of starting a bunch of processes with a subset of keys/routes, and putting a LB infront.

rickmoynihan23:05:15

I think this is one unsung benefit of integrant actually; it makes it very easy to move from a monolith to micro services, or use a monolith in dev contexts, but deploy as micro services.

rickmoynihan23:05:51

Though I should add I’ve not had the need to do that, yet.

rickmoynihan23:05:57

Actually, we do in dev and test envs start two services that are normally independent processes in the same process. It is in dev very convenient to have the same REPL into two or more separate services.

borkdude21:05:04

It's technically possible to use a data reader tag in the very same namespace in which the associated reader function was defined - right? Does this happen a lot or is it reasonable to assume that you can require a namespace with reader functions to add it to *data-readers* yourself using set!? The context is this issue: https://github.com/borkdude/babashka/issues/419 I want to implement data_readers for babashka, but I don't want to incur any startup penalty by scanning the classpath if people aren't going to use any data readers.

rickmoynihan21:05:39

A colleague of mine (Andrew Mcveigh) wrote what I think is quite a beautiful example of a clojure macro for generating SPARQL 1.1 property paths: https://github.com/Swirrl/grafter/blob/master/src/grafter_2/rdf4j/sparql/path.clj It show cases a number of quite advanced macro tricks, so I thought folk here might find it instructive or interesting of how far you can take macros if you choose to. Obviously the first rule of macro club still applies. However in this case property paths are usually static things (so there’s seldom a need to dynamically build them), and if a pure data syntax had been chosen it would require quoting and syntax quote splicing etc, as URI’s in this world are by convention bound to vars or symbols, e.g. rdfs:label is a var named that way by convention; to match RDF’s cURI syntax. I should note also that if a data syntax was also required it would be pretty trivial to provide that in addition to this; by essentially lifting the spec parsers into shared functions, and removing the &env processing from that side of the interface, and I guess finding a data syntax for the pre and postfix unary operators. Anyway things it shows are: 1. How you can use spec to parse macro syntax which provides an infix notation almost identical to actual property paths. 2. How macros can provide compile time checking of DSL syntax 3. How a macro can be just a thin layer over a function/prefix API 4. How a macro can implement special syntax rules; for example the symbol `foaf:name` is usually just a var binding a URI; but how within the scope of the macro you can use the symbols `foaf:friend+` to represent the path of one or more “friends of”; or `-foaf:friend` to represent the inverted path, or `!foaf:friend` to represent every predicate except `foaf:friend`. 5. Finally also how you can do some pretty neat compiler magic in macros.  For example in 4 you might say “but what if the symbol in my scope happened to be prefixed or suffixed with a +, - or !“.  Well the answer is the macro checks the scope for bindings that might be ambiguous; and raises a compile time error if they are; (which lets you rename or rebind the symbol locally to a non coliding name). The tests port the examples from the W3C property path spec into the DSL format as an example: https://github.com/swirrl/grafter/blob/master/test/grafter_2/rdf4j/sparql/path_test.clj#L173-L220

👍 4