Need some help

I am running selenium with Clojure using taxi library

I need to find a way to click this element:

@anthgur: ideas?

@jtackett: I’m writing selenium tests in clojure too

what is taxi?

https://github.com/semperos/clj-webdriver/wiki/Taxi-API-Documentation#execute-script

@tmtwd: ^^^

oh yeah

I saw that

I figured it out btw:

`(click dr {:tag :a, :class "actionsButton"}) `

I’m just doing directly java interop

ah alright

ya this is way easier

really?

I should try it

although there is a core-reviews channel there seems to be nobody there, so can someone give some tips regarding this codewars code: http://www.codewars.com/kata/reviews/564062eec7329a74c1000017/groups/56bea05bfd9652465700010e ?

I would like some tips of "what is the best way to convert an integer to string and navigate on each number"

the solution works however I have the feeling that I might be doing it too imperative

@esnunes: can you post that somewhere where login isn't required?

In java.jdbc is there an option to add timeout for jdbc calls?

@kul: timeout is an available key after your sql statement

https://github.com/clojure/java.jdbc/blob/990928cb005b0665f8c9c90dadf3040f4295d321/src/main/clojure/clojure/java/jdbc.clj#L444-L446

How do I list directories when in a jar? io/resource returns a URL, so can't call file-seq on it.

@jethroksy is it only available for prepared stmt ?

What about query insert! Update!

nvm found this lib: https://github.com/xsc/cpath-clj

@dm3 sorry I thought login wasn't required

https://www.refheap.com/114760

@kul: not sure what you mean

would your insert/update query statements not be a SQL statement?

(insert! con :table {:aa 1})

Where do i set timeout

doesn't look like the timeout option is available there

internally it would be creating a prepared stmnt, not sure why options are not exposed.

hey @kul what problem are you trying to solve with the timeout?

i mean that in the sense that the implementation of that feature varies across db platforms, and the semantics are poorly defined, particularly with DML

Just trying to avoid problem when a db becomes highly congested

Causing querying threads to stall

Rejecting queries is more acceptable than a unresponsive application

gotcha. so if queries are expensive, one approach is tuning at the db level, another would be to utilize a results cache. another approach could be to keep tabs on query response time and if it starts to climb, then simply don't send the query

Isn't jdbc spec standard?

I mean the behaviour can vary but at least the api is there

it's a standard for communicating from the client side

but different databases attend to that particular request differently

the implication of .setQueryTimeout is that the database stops executing the query

but some don't do that

I just want postgres

Not sure if i can install pgbouncer on rds

All the approaches are costly to do correctly than a simple timeout

it looks like functionality around this is still in flux on the pg jdbc driver. Report of a bug fix in December to thread safety issues: https://jdbc.postgresql.org/documentation/changelog.html

Changelog talks about making method threadsafe

the other issue is that sending work and then asking the database to stop in the middle is still subjecting the database to the load.

I am assuming some support is already there

yeah- it may be that for the common case it works fine, i haven't used pg since much earlier in the 9 series

if the semantics of your queries can be captured in key-value maps, clojure core.cache is pretty easy to use.

Problem is with update queries

ah

Db reads are pretty robust

your updates are timing out?

Yep

Not timing out

😆

sorry- what's happening with updates?

the thing about cancelling a transactional update is that it then causes a rollback

which can be very expensive

Makes sense

because it impacts other transactions also in flight, both reads and writes

if you absolutely have to get those writes to the transactional store, then aside from tuning, you may need more IO throughput

You mean better hardware for db?

but another option if the semantics are more flexible is to maintain state in another location, like a redis store

and then propagate data from redis to pg

maybe some postgresql tunning, changing a little bit the postgresql defaults can speedup inserts more than 3 or 4x

@niwinz i am listening

yes, if you are on rds, you can use provisioned iops

@jonahbenton that seems costly to implement

I mean the redis thing

do you have multiple applications writing to pg or just one?

Just one

put synchronous_commit= off and wal_writer_delay=200ms

put high value for checkpoint_segments, (32-128)

and increment shared_buffers

Thanks bookmarked! Will try these out

the default in 9.4 is 128MB, you increase depending on the ram available

512MB can increase also writes and reads

Great but i am still concerned about those threads waiting on SQL statements in my app

I still think timeout at global level on java.jdbc could be useful

yeah- the issue is that the semantics aren't clear

if you want async behavior where your app can respond to users while not waiting for background work to be done

can do that with agents or core.async

Its a crud app, no async

Strong consistency is required, i would not chose rdbms otherwise

Hurmm

A core.async pipeline for db ops

what is the nature of the updates? why are they so expensive?

It updates multiple tables

Ah that reminds me of adding more indexes

yeah- though indexes increase cost at write time

is there row-level write contention?

But back pressure still need to be handled gracefully

Not really

Every user has its own rows to update

so users are able to wait for their operations to complete?

Yes they must

Its a simple crud app

Where data is in sql

Nothing fancy

What do you think about db operations handled via a core.async thread?

well, core async doesn't like blocking io

the thread pool is limited

A couple of (thread ..) for db ops and (go ..) and wait for result completion?

well- how many concurrent users do you have?

I was testing with 500 when many threads stalled making it unresponsive

promise with a timeout?

and the bottleneck was database writes?

I guess

Jetty can take 100000 req per sec for a simple no db endpoint

yeah. easy work is easy

25 40 k at least for a small vm at aws

They suck

The vm i mean

Ok so java.jdbc has no timeout is a problem? Do we agree on this?

you mean for writes?

when you load tested, did you do it in step fashion to figure out at what stage the system stopped responding?

if the bottleneck there is db, then application architecture isn't going to matter. but if db is doing fine, then can look at changing app architecture to ensure responsiveness. the fundamental issue when talking to a database with a live thread is that most drivers use blocking io. so you are going to need a thread per concurrent action

using timeouts, if actually implemented in the underlying driver, can free up a thread, but a) the work requested of the db doesn't get done b) more load is put on the database

if you figure out that at your budget and provisioning you can support 20 concurrent actions, then you can just maintain a 20-connection db pool. when a user submits work, they try- with a timeout- to retrieve a connection. if they can't get one, you tell them they have to try again later

I am.already using c3p0

15 max default connections

Try with a timeout ? You mean a future ?

http://www.mchange.com/projects/c3p0/#checkoutTimeout

set that to 500ms or something

Ok that property makes a lot of sense but it still.does not avoid stalled threads

did you take a thread dump?

Yes

ah, so what were they blocked on?

Most were waiting.on java.jdbc stack trace

Some on query most on updates

very good

so you need better write throughput

🙂

the above mentioned settings, plus http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_BestPractices.html#CHAP_BestPractices.PostgreSQL will reduce write activity

FWIW, at World Singles, most of our connection pools have min/max pool size of 20/90 and 40 helpers (threads)… and we have this in c3p0:

;; expire excess connections after 10 minutes of inactivity:
               (.setMaxIdleTimeExcessConnections (* 10 60))
               ;; expire connections after 61 minutes of inactivity:
               (.setMaxIdleTime (* 61 60))
               ;; idle connection testing
               (.setAutomaticTestTable "c3p0connection")
               (.setTestConnectionOnCheckin true)
               (.setIdleConnectionTestPeriod (* 11 60))
               ;; unreturned (app leakage) get zapped after an hour:
               (.setUnreturnedConnectionTimeout (* 59 60))

but you may just need more iops

@seancorfield: what db are you using?

MySQL… well, Percona… SSD-backed servers.

What sort of request load are you looking to support?

the magic of ssd

I totally agree but it going to break at some point

Btw i have enough inputs to work with

Thanks all

We have about 450 connections on the master, 50 on the slave; about 3.5k reads/sec on the master; 250 writes/sec on the master; 4k reads/sec on the slave… and the DBs are not even breaking a sweat...

(but, yeah, fairly low write loads for us)

nice when you have lots of headroom. easy to sleep at night

@seancorfield do not want to be pushy,just want to ask if exposing prepared stmt options to common calls like query etc makes sense to you?

You can already create a prepared-statement and pass it as the first element of the sql-and-params vector to query...

Per the query docstring: "The second argument is a vector containing a SQL string or PreparedStatement, followed by any parameters it needs. See db-query-with-resultset for details."

It’s possible that prepare-statement could expose more options directly? http://clojure.github.io/java.jdbc/#clojure.java.jdbc/prepare-statement

It becomes a bit difficult to adapt framework like yesql etc for this

I guess

I guess dedicated helper threads for doing SQL ops make sense

It would work even when the driver or db does not implement termination

Any suggestion for an easy to work with threadpooling on Clojure ?

@kul agents with a custom executor pool may work for you. you can assign users to individual agents with a hash function. they preserve order of operation semantics, you can wait for results, and recover on errors

As long as we're talking about SQL, I just ran into a fun one... for non-alphanumeric characters, Clojure's compare and Postgres's ORDER BY do not appear to give the same results. Normally I'd just trust Postgres, but I want my unit tests to make sure I'm applying the right sorting. Has anyone else run into this, and have any suggestions on how to deal with it?

hey @amacdougall that's a character set issue.

pg is probably using sql-ascii?

java may be utf-8

Oh, interesting. It seems like I'd want to tell Clojure to adopt PG's sorting mechanism, though honestly it doesn't matter as long as alphanumeric strings are sorted the same way in both, which is likely.

I'm generating test data using Prismatic Schema, which seems to prefer generating usernames/etc like "//+;.&". I appreciate the fuzz-testing approach, so I don't want to try to limit the test data to alphanumeric.

The Clojure docs on compare say: > Strings, Symbols, Keywords: lexicographic order (aka dictionary order) by their representation as sequences of UTF-16 code units. This is alphabetical order (case-sensitive) for strings restricted to the ASCII subset.

That sounds like they'd work the same as long as they're all ASCII, which they do seem to be...

Of course "alphabetical order" is undefined for punctuation. I'd assume they'd all sort by ASCII code, which should be the same in UTF-8... but that mention of UTF-16 really muddies the waters. I honestly have no idea what "sequences of UTF-16 code units" means. I would assume that it boils down to UTF-8 for characters which don't need double-byte representation, which should be the same as ASCII for ASCII characters...

But I'm making several assumptions to get there, and clearly things are being sorted differently.

do you have an example of a group of chars being sorted differently?

Sure. Here's a list of strings as PG sorted them:

(" " ";" "/+" "\"" ")" "]" "1797gR}F#j" "1ORfdr" "<1+UE'
1'VA" "20wDClW" "2H01a" "49RF2`]" ".4K5o;N{" "*4q" "4X yCjCk+(" "'>+7" "\"`7s
m:{" "7WOW?/gq[c" "8k-L}3" "9qm.aeM6O1" "a!S!@!" "b2" "Bq`\"^G" "C:sLAo" "d" 
"dF" "-\"Do" "DPo" "#E" "&eb")

And here are the compare results from Clojure:

(-27 12 13 -7 -52 44 -24 -11 10 -24 -2 6 4 -10 13 5 -21 -1 -1 -40 -1 32 -1 -33 -1 55 -23 33 -3)

As you can see, they're all over the map. That's the result of running (fn [[a b]] (compare a b)) on a (partition 2 1) of the list of strings.

i.e. a sliding 2-item window.

Ah, I think I see what's up...

PG is ignoring leading non-alphanumeric characters. That honestly doesn't seem like the way I'd do it.

But if I am willing to just accept that, then I can at least test that my sorting was applied by doing the same thing in my Clojure unit test.

...and this in turn has to do with the database's "collation" setting—a vague term that probably has a very precise meaning in PG-land. It's language/locale specific. There's a way of disabling it and doing an ASCII sort if I really want.

At least I know what's up. Now I can choose the least bad way of dealing with it. ¯\(ツ)/¯

Most likely I will use a custom comparator which applies the same prejudices as Postgres.

yes: https://docs.oracle.com/javase/tutorial/i18n/text/locale.html

Thanks for the help!

Sometimes I just need a little nudge out of my mental track.

you bet

How do you guys handle integration test during dev to prod (aot) transitions? Is someone testing at a prod Repl for instance?

hey @richiardiandrea what kind of application?

Websocket based backend

@jonahbenton integration meaning testing the effect of a websocket call from the client, queued events and all that

right- but in a test environment, or production environment? IOW, you're talking about deploy-time validation?

in production?

Yep

most of my apps have involved manual validation- at a prod repl or from the production client- but there were plans to encode system-level tests in a custom client, which would run from privileged infrastructure

Yes that was my idea as well, so it was not that random :) thanks @jonahbenton