Fork me on GitHub
#off-topic
<
2017-06-03
>
tasdikrahman10:06:53

how should I get the whole path that I should follow to go from source to destination (in latitude and longitude). Any API which would let me do so?

john14:06:45

@tasdikrahman Dijkstra, A* and Contraction Hierarchies. See GraphHopper: https://github.com/graphhopper/graphhopper

qqq19:06:02

at which point, should I switch from "develop directly against production db" to "have a production db and a dev db" ?

noisesmith19:06:14

before starting your project

qqq19:06:43

are you serious?

qqq19:06:55

it's not a good idea to just develop until it's basically working, then worry about prod vs test db ?

qqq19:06:17

some would argue this is premature optimization // not necessairly me, I don't know yet

noisesmith19:06:23

develop directly against dev db, and eventually make a prod db

noisesmith19:06:30

but never develop directly against prod

qqq19:06:50

alright, so prodjuction is beanstalk + aws + rds

qqq19:06:05

what if, during dev, I want a way to interact with the production rds

noisesmith19:06:10

if you need to be sure that works early on, put dev there too

qqq19:06:17

mimicking all of aws seems way too much work

noisesmith19:06:29

don't, just make a dev environment on aws

noisesmith19:06:55

what you shouldn't ever be doing is having a "prod db" that you apply untested code to

qqq19:06:39

okay, so there's dev-rds and prod-rds // even when developing locallhy, I'm ioperating with dev-rds

qqq19:06:47

then, right before I deploy to beanstalk, I switch to prod-rds

noisesmith19:06:43

right - and I like a "staging" for verification too

qqq20:06:20

can you explain what you mean by staging? I'm literarlly going from "building hobby projects" to "oh shit, I'jm taking credit card payments now, so I need to be serious"

mobileink20:06:42

you must have a qa env that faithfully mimics prod, but never uses prod data. never go directly from dev to prod; stage to qa first, then promote to prod. so you need at least 3 environments.

qqq20:06:35

why not just qa and prod ?

qqq20:06:38

why have a dev at all?

mobileink20:06:44

treat qa the same way you treat prod. dev is for farting around, trying crazy ideas.

mobileink20:06:43

or: treat your environments the same way you treat your code.

mobileink20:06:30

a good qa env will tell you if you've added something in dev that won't fly in prod. you don't want to discover that after you've pushed to prod.

mobileink20:06:40

or rather, you don't want your users to discover that for you.

qqq20:06:03

in theory, I agree with you

qqq20:06:21

in practice, I'm probably goign to dev on prod, do something stupid, get hit with chargebacks, then decide, okay, time to invest in fixing this

noisesmith20:06:58

qqq: one vital reason to have a qa staging environment is to test the very process of a "release"

noisesmith20:06:09

in dev you fart around, do things to the db you might have forgotten etc.

noisesmith20:06:28

but if you push up to the qa environment and forget the migration, things will fail

noisesmith20:06:46

that's your reminder to write a proper migration and run it when deploying to prod

qqq20:06:36

how do you populate qa / dev databases ?

qqq20:06:40

do you snapshot the prod?

mobileink20:06:17

qqq: NO! never! treat all prod data as sensitive, never ever let it leak. otoh, you could write some code to xform prod data in a way that is guaranteed to to be immune to disclosure analysis - impossible to recover the original data - but that is not easy.

noisesmith20:06:43

qqq: it's not just chargebacks, your reputation is not immutable, and it's finite. People leave services and don't come back if they think the devs are just fucking around

qqq20:06:24

yeah, but it appears SV has bought into the whole "move fast and break things" movement

noisesmith20:06:28

@qqq yeah, a snapshot is a good place to start with staging (but without sensitive customer details if such things exist)

qqq20:06:50

does AWS make this easy?

qqq20:06:02

if I launch an RDS with Aurora, MySQL, or Postgres, is there an easy way to "copy this prod over to dev" ?

noisesmith20:06:53

with postgres I honestly usually use pg_dump directly, because I've always used it

noisesmith20:06:58

I don't have experience with the others

qqq20:06:44

alright, I'm convinced this is not too hard to do, and may be even scriped, i.e. dump prod, restore to dev

qqq20:06:51

is there any step by step guide you can recommend?

noisesmith20:06:15

I just learned this stuff at work where we had a wiki...

mobileink20:06:17
replied to a thread:do you snapshot the prod?

qqq: NO! never! treat all prod data as sensitive, never ever let it leak. otoh, you could write some code to xform prod data in a way that is guaranteed to to be immune to disclosure analysis - impossible to recover the original data - but that is not easy.

mobileink20:06:03

would you do business with a bank that used snapshots of prod data for qa?

qqq20:06:29

eh, probably better than mt goz

qqq20:06:11

Scylla / Charybdis = lose important data vs leak important data ?

mobileink20:06:34

it's all schemas. your prod dbs have schemas. use the schemas for qa. esp. now that we have spec generators.

mobileink20:06:17

leaking = lawsuits.

mobileink20:06:19

btw, you'd be amazed at what a sophisticated attacker can do with seemingly innocuous data. where i work we have PhDs whose job is to do "disclojure analysis".

qqq20:06:38

though I'm already suspecting, that if you're willing to pay, there are places where you can type in someone's name and it'll dig through gov records to find brand of first car, elementary/middle/highschool attended, list of previous addresses, parent's name, etc ...

mobileink21:06:58

you don't have to pay. it's called "google".

dominicm21:06:26

@mobileink I've been thinking about this lately (re the spec thing). The problem is that spec doesn't necessarily hold the right proportions that the real data does, or the relationships that is really there.

mobileink21:06:41

dominicm: you mean re: using spec to generate qa data?

mobileink21:06:28

dominicm: yeah, i haven't looked into it too deeply. would probably work for simple stuff. for more complicated stuff - well, that's a business opportunity, heh.

dominicm21:06:43

Yes indeed. I'd be really interested in seeing it!

dominicm21:06:19

I'd love to have prod-like quantities data for debugging performance issues

mobileink21:06:00

oh man, wish i had the time! not hard to imagine a spec-based tool like that; everybody would buy it.

mobileink21:06:24

cognitect is probably working on it.

dominicm21:06:55

@stathissideris is working on https://github.com/stathissideris/spec-provider which does statistical analysis to infer the spec. But I'm not sure it could do occurrence count into the resulting generators.

mobileink21:06:11

dominicm: damn, that's got some real possibilities!

dominicm21:06:03

@mobileink @stathissideris reveals to me that he's got some stats things he's working on, so you can tweak it manually if you feel the need to, and they're put in with comments.

mobileink21:06:02

dominicm: unfortunately i am the lone clojure soldier where i work.

john21:06:45

"disclojure analysis" - somebody's been programming clojure too long 😉

mobileink21:06:18

haha. disclosure, not disclojure.

qqq21:06:46

until a website hits 1M dailya ctive users, does the choice of mysql vs postgresql matter ?

nikki21:06:10

this person makes art with clj

emperorcezar22:06:47

Is it more likely, in a clojure shop, for things to be done "right" the first time, or is it just a given in our industry that things are rushed out the door?

noisesmith22:06:33

I've definitely seen things rushed - but we can get closer to right when rushing than some competitors for sure

emperorcezar22:06:39

I feel maybe I've been doing startups too long. We're re-doing something because it was rushed out the door the first time, and now it seems we're cutting corners and rushing this one also.

nikki22:06:43

as it turns out there is no "right" ;O

nikki22:06:12

if u rush in a future-compat way with being more "right" tho that's nice

joelsanchez22:06:09

@nikki What's that from?