Fork me on GitHub
#beginners
<
2020-08-18
>
Timofey Sitnikov11:08:08

Good morning Clojurians, when running https://github.com/dharrigan/nextjdbc-integrant , I am getting a Reflection warning, https://github.com/dharrigan/nextjdbc-integrant/blob/8741d2f8498ba549188310099582a9ec13dd63dd/src/simple/db.clj#L58 - reference to field close can't be resolved., how can I resolve it? Below is the code execution.

[I] /home/sporty/clojure/nextjdbc-integrant~> clj
Clojure 1.10.1
user=> (require   '[simple.core :as core])
nil
user=> (simple.core/-main {})
Reflection warning, simple/db.clj:58:3 - reference to field close can't be resolved.
Reflection warning, simple/migration.clj:15:14 - call to method locations on org.flywaydb.core.api.configuration.FluentConfiguration can't be resolved (argument types: unknown).
Reflection warning, simple/migration.clj:19:3 - reference to field migrate can't be resolved.
[2020-08-18 07:12:13,926] --- [INFO ][o.f.c.i.l.VersionPrinter ] - Flyway Community Edition 6.1.0 by Redgate
[2020-08-18 07:12:13,934] --- [INFO ][c.z.h.HikariDataSource   ] - HikariPool-1 - Starting...
[2020-08-18 07:12:14,203] --- [INFO ][c.z.h.HikariDataSource   ] - HikariPool-1 - Start completed.
[2020-08-18 07:12:14,229] --- [INFO ][o.f.c.i.d.DatabaseFactory] - Database: jdbc: (PostgreSQL 12.2)
[2020-08-18 07:12:14,269] --- [INFO ][o.f.c.i.c.DbValidate     ] - Successfully validated 2 migrations (execution time 00:00.017s)
[2020-08-18 07:12:14,290] --- [INFO ][.s.JdbcTableSchemaHistory] - Creating Schema History table "public"."flyway_schema_history" ...
[2020-08-18 07:12:14,346] --- [INFO ][o.f.c.i.command.DbMigrate] - Current version of schema "public": << Empty Schema >>
[2020-08-18 07:12:14,354] --- [INFO ][o.f.c.i.command.DbMigrate] - Migrating schema "public" to version 1.0.0.000 - initial version
[2020-08-18 07:12:14,376] --- [INFO ][o.f.c.i.command.DbMigrate] - Migrating schema "public" to version 1.0.0.100 - create simple
[2020-08-18 07:12:14,398] --- [INFO ][o.f.c.i.command.DbMigrate] - Successfully applied 2 migrations to schema "public" (execution time 00:00.061s)
[2020-08-18 07:12:14,419] --- [INFO ][simple.operations        ] - Initialised.

dharrigan11:08:09

@timofey.sitnikov I replied to your question on the github issue. Have a look 🙂

Timofey Sitnikov12:08:45

@U11EL3P9U, why not? I tried to use component, which i think is what @U04V70XH6 prefers, how can I choose? Why did you move to clip?

Timofey Sitnikov12:08:28

OK, so the first commit is less than a year ago, there must be some awesome new idea.

Timofey Sitnikov12:08:44

Also, they do say that Clip is based on existing working ideas, but hasn’t yet been used in production. @U11EL3P9U, do you use it in production?

dharrigan12:08:18

Yes. Very successfully. In a site that deals with millions of data point ingestions per day.

Timofey Sitnikov14:08:58

@U11EL3P9U, would you be OK if I asked author to consider removing that comment on the README, it probably scares people off.

dharrigan14:08:16

Sure 🙂 You can ping him in the #juxt channel.

seancorfield16:08:25

@timofey.sitnikov I believe you can fix that reflection warning by changing line 58 to

(.close ^HikariDataSource @datasource)

3
seancorfield16:08:29

(Also, it's only a warning so, in theory, you can just ignore it)

seancorfield16:08:20

The two warnings in migration.clj would also need type hints to avoid reflection but I'd have to do some digging into FlyWay to figure out what.

seancorfield16:08:15

@U11EL3P9U Looking at that Star Trek repo with Clip, it looks like you don't bother trying to shut the connection pool down?

dharrigan16:08:36

Nope. It dies when the app dies

dharrigan16:08:46

I should mention that in my "production" apps, I do have this:

dharrigan16:08:55

:stop (txi.higgins.db/connection-pool-stop this)}

dharrigan16:08:27

so when Clip shuts down, the stop fn will be invoked. this is a reference to the value passed back on the start

dharrigan16:08:01

as an example here

rmxm11:08:38

Hey quick question, i have a function that returns nil or value. What is the most elegant way of trying a list/vector of values until it produces result? map & filter?

mloughlin11:08:44

Returns the first logical true value of (pred x) for any x in coll,
else nil.  One common idiom is to use a set as pred, for example
this will return :fred if :fred is in the sequence, otherwise nil:
(some #{:fred} coll)

practicalli-johnny12:08:38

Is there a function or simple expression to get a key from a hash-map, given a value (assuming all values are guaranteed unique). Or does it make more sense to reverse the map using clojure.set/map-invert and then the values become keys / keys become values and I use the hash-map with get as usual. The hash-maps are used as a dictionary lookup and have no other purpose and will be no more than 100 key/value pairs. Thank you.

noisesmith12:08:56

depending on usage patterns, map-invert, or a bimap implementation make sense

Jack Arrington12:08:22

Could someone give me examples of where laziness is helpful in Clojure? Links, explanations, or anecdotes are all welcome 🙂. Basically, switching back and forth between lazy/non-lazy operations has so far seemed a bit clumsy to me. I'm constantly asking myself, "Wait, does this function return the data structure (typically a vector) that I actually want in the end, or is it a lazy seq?" I also find myself calling into [] a lot, which seems kind of verbose. It seems clumsier than, say, Haskell, where I don't have to think about laziness much if I don't want to because everything is lazy. That said, I've been impressed enough by other parts of Clojure's design that I'm willing to consider that I just don't "get it" yet. What are practical examples of where I would benefit from lazy evaluation? Being able to manipulate infinite lists and objects bigger than memory is neat and all, but not something I use all that often.

gibb07:08:35

Hi Jack! Are you looking for practical examples as in what business problems you'd solve with laziness?

Jack Arrington11:08:57

Sure, or just anywhere laziness would provide significant benefit or make implementing something easier or more elegant

gibb13:08:37

A generic case is doing "stuff" with all your users in your applications database, for example sending an email notification or just counting how many there are. You could have a lazy way of providing access to all the user db records that is fully decoupled from the procedures you'd like to run on them.

gibb13:08:35

Some of the abstractions you could create there is a page size for the SQL SELECT for example that is appropriate to your database, available memory and so on. And all the functions you'd want to run on your records would not have to have any specific knowledge of appropriate SQL db behavior.

Jack Arrington14:08:17

If I'm understanding correctly, you're talking about wrapping SQL (or whatever else) inside a lazy-seq macro right?

Jack Arrington14:08:00

I'm less confused about that, and more about why all/most of the operations on vectors, lists, and maps are lazy by default. I don't always see the point in lazifying my concrete data structure and then re-concretizing it (those are both probably made up words 🙂) with into when I actually want to use it

Jack Arrington14:08:21

I suppose I could try to write everything to accept a seq to begin with, rather than relying on a concretion. But for the particular use case I'm trying Clojure out on, vectors are pretty much a necessity - lots of random access, splicing elements in and out of vectors at arbitrary points, etc. This gets a little frustrating/confusing when there are options like, say, drop which returns a seq, or subvec which returns a vector. Part of me feels like I should be using drop (writing to the abstraction, not the implementation), but then I lose the performance benefits of subvec, and have the added verbosity of into

gibb15:08:05

I see! I guess I sort of find it intuitive that map and filter etc return lazy seqs because if not they would force the evaluation of things before they were needed. I think I just see eager evaluation as the "odd" thing. Maybe not the best viewpoint to explain it from 😃

Jack Arrington20:08:23

I think I am starting to understand it better now. Just taking some time to sink in

noisesmith12:08:11

laziness simplifies code by turning a series of side effects / a stateful generator, into a value

noisesmith12:08:54

because it only calculates as needed, a loop of side effects just to calculate a value can be replaced with a simple traversal

Jack Arrington12:08:34

Hm, I am trying to understand what you mean by this. Say I called map or reduce on a list type in an imperative language, or a strictly-evaluating functional one like OCaml. How does that have any more side effects than the lazy, seq-ified version in Clojure?

noisesmith13:08:06

it's about driving the production of results via the consuming code, but yes you are right, standard imperative fp has a lot of that benefit already

noisesmith13:08:56

I'm thinking about concrete examples of things that are easier to do with laziness, beyond the classic fibonacci / ackermann

noisesmith13:08:31

the benefit with those is that you no longer need logic inside the generator to limit the production - that can be a separate and modular aspect of the consumer

noisesmith13:08:00

on the other hand, many clojure features could be seen as attempts to replace lazy-seqs in places where they don't quite fit (eg. core.async where side effects and coordination matter, transducers where throughput / performance matters) lazy-seqs are a good default thing to turn to in a first iteration, and sometimes need to be replaced with something else, they help keep things unentangled in most cases

noisesmith13:08:33

consider this code, between a side effecting source (wrapped by get-data), which provides a lazy stream of input) and a work dispatch system

(-> (get-data)
    (filter actionable?)
    (map assign-worker-id))
what comes out is data describing a series of tasks, which can then be used to drive a strict (side effecting) process

noisesmith13:08:02

I'm not quite happy with this as an example though - get-data producing a lazy-seq can mean that confirming message receipt to a remote host, or handling host availability, can get tangled into lazy code (probably a bad idea), let's assume that get-data is well designed and provides a nice lazy seq while preventing that complexity from leaking into the rest of the code

noisesmith12:08:05

this is easier to reason about, so errors are easier to see and fix

noisesmith12:08:53

even compared to generators, it's simpler because the logic around holding / releasing the generator is replaced by something much simpler to think about: creating a piece of data, and letting it go

noisesmith12:08:49

counter-indications are cases where the thing you are doing is not a calculation that would produce the same result (or interchangable result) twice - side effects and laziness can increase rather than decrease complexity

noisesmith12:08:11

but if all you are doing is avoiding doing expensive CPU work that would give the same answer twice, lazy-seqs can help a lot

noisesmith12:08:48

also, a lazy-seq can turn a recursive algorithm that would consume stack, into one that streams results to a consumer on demand without going deeper on the stack

noisesmith12:08:30

(the gotcha here is that mixing lazy and imperative collection operations can undo that behavior - it's good to remember which "world" you are in)

noisesmith12:08:58

also, the lazy operations can act as algorithmic middleware, I can compose lazy functions to make an algorithm in a modular way

noisesmith12:08:46

@mail985 apologies, I put more attention into my attempt at an answer than I did to your question. You know a bunch of this already if you have haskell experience.

Jack Arrington12:08:37

My Haskell experience is pretty basic (though enough that it turned me on to FP), so no worries @noisesmith

Valentin Podkovyrov16:08:17

Hello. Howto resize image?

noisesmith16:08:29

are you using an image library? clj or cljs?

noisesmith16:08:56

I've had good results from java awt BufferedImage, maybe I can find the repo...

Valentin Podkovyrov16:08:50

I use clj. BufferedImage has methods for resize?

noisesmith16:08:33

warning: I wrote this code when I was a junior engineer, but it works https://github.com/caribou/lichen/blob/master/src/lichen/image.clj#L96

3
noisesmith16:08:16

this library is made for auto-resizing and caching images for a cms, the resize function should be usable on its own, with a few of the helpers as needed

noisesmith16:08:36

there might be a mainstream clojure lib too, but I wrote this one (so I know it exists) and I know it works

noisesmith16:08:37

NB: one gotcha is many JVM options in Linux distros are "headless" - they contain no GUI code. The AWT libs are part of the GUI stuff, so the headless VM won't work.

seancorfield16:08:19

This is what we use very heavily at work https://github.com/josephwilk/image-resizer -- and we deal with thousands of images every day.

💯 6
noisesmith16:08:55

very cool - that's probably better than my old code I wrote as a junior dev 😄

seancorfield16:08:36

And we have the Twelve Monkeys image library added too, to support more formats:

com.twelvemonkeys.imageio/imageio-jpeg {:mvn/version "3.6"}
  com.twelvemonkeys.imageio/imageio-tiff {:mvn/version "3.6"}

seancorfield16:08:54

Those add to what the imagez library can process.

josh_tackett19:08:20

Hey anyone know how to write a clojure regex to match “4X4” ? I tried #“\d+[A-Z]\d+” but it’s matching all words also

walterl19:08:54

How are you using it?

dpsutton19:08:18

(map (partial re-find #"\d[A-Z]\d") ["bob" "4X4" "33X45" "word"])
(nil "4X4" "3X4" nil)

dpsutton19:08:22

it doesn't seem to for me?

josh_tackett19:08:49

(clojure.string/replace “CUTTING BOARD TEX 4X4” #“[\d[A-Z]\d]+” “”)

josh_tackett19:08:05

this gives me and ” ”

josh_tackett20:08:40

@U11BV7MTK that’s so strange, must be the difference between replace and re-find

dpsutton20:08:06

(str/replace "CUTTING BOARD TEX 4X4" #"\d[A-Z]\d" "")

dpsutton20:08:16

why do you have the [ in your regex?

josh_tackett20:08:28

because I’m not sure what I’m doing haha

dpsutton20:08:41

lol fair enough. drop those. add them in if you need.

dpsutton20:08:36

and add them in when/if needed

Lennart Buit20:08:23

Just as context, square brackets denote character groups, so [aef] matches a single a, e or f. The regex you had was a bit of a strange repeated character group, I guess it would match a number, capital A-Z or a number, meaning that after replacement you were left with just spaces

Lennart Buit20:08:48

So, like already mentioned, drop those, but I hope this context helps you understand regex just a bit better :)