Fork me on GitHub
#clojure
<
2020-07-24
>
chrisulloa00:07:50

interestingly it works if you wrap (first is) with (boolean (first is)) in the recur, no idea why though.

daniel.spaniel00:07:04

is there offical clojure way to parse numbers with either euro format "1.222,33" or american style "1,222.33" ?

kenny00:07:27

@christian.gonzalez Yep. I'm curious why the above does not work though. Digging through the source does not make it clear. I did find https://stackoverflow.com/questions/26198603/clojure-llegalargumentexception-in-a-clojure-recur-type-mismatch but it doesn't dig into the why.

andy.fingerhut00:07:23

The Clojure compiler tries to use primitives for loop 'variables' for better performance. If it does not for a particular loop variable, that one will be any JVM object, which can in many cases lead to allocating a new JVM object for that loop variable on each iteration.

andy.fingerhut00:07:42

Is that the 'why' part you are wondering about? Or some other aspect of the behavior?

kenny01:07:11

Why does this work? Note the only change is pos? -> true? on the loop first binding.

(defn test-2
  []
  (loop [x? (true? "")
         is (map some? [1 nil 2 nil])]
    (when (not-empty is)
      (recur
        (first is)
        (rest is)))))

seancorfield02:07:50

pos? is defined as inline and clojure.lang.Numbers/isPos returns boolean (primitive) but true? is defined in Clojure to return Boolean (object).

seancorfield02:07:09

user=> (source pos?)
(defn pos?
  "Returns true if num is greater than zero, else false"
  {
   :inline (fn [num] `(. clojure.lang.Numbers (isPos ~num)))
   :added "1.0"}
  [num] (. clojure.lang.Numbers (isPos num)))
nil
user=> (source true?)
(defn true?
  "Returns true if x is the value true, false otherwise."
  {:tag Boolean,
   :added "1.0"
   :static true}
  [x] (clojure.lang.Util/identical x true))
nil
user=>

kenny02:07:31

Ah. So then how about this one?

(defn test-2
  []
  (loop [x? (pos? 1)
         is (map some? [1 nil 2 nil])]
    (when (not-empty is)
      (recur
        ^boolean (first is)
        (rest is)))))
Syntax error (IllegalArgumentException) compiling fn* at (src/foo.clj:3:1).
 recur arg for primitive local: x_QMARK_ is not matching primitive, had: boolean, needed: boolean

kenny02:07:24

Also that unnecessary newline in pos? 😭

seancorfield02:07:31

Because you can't type hint an expression with a primitive.

seancorfield02:07:03

That actually "tags" the expression with clojure.core/boolean 🙂

seancorfield02:07:25

user=> ^boolean []
[]
user=> (meta *1)
{:tag #object[clojure.core$boolean 0xa567e72 "clojure.core$boolean@a567e72"]}
user=>

kenny02:07:38

I see. So the only way to get a loop to work if the initial binding is a primitive is to use a predicate that returns a primitive?

seancorfield02:07:03

(boolean (first is)) works?

kenny02:07:10

Hmm…

(defn test-2
  []
  (loop [x? (pos? 1)
         is (map some? [1 nil 2 nil])]
    (when (not-empty is)
      (recur
        (boolean 1)
        (rest is)))))
Syntax error (IllegalArgumentException) compiling fn* at (src/foo.clj:3:1).
 recur arg for primitive local: x_QMARK_ is not matching primitive, had: Object, needed: boolean

kenny02:07:34

boolean calls to

static public boolean booleanCast(Object x){
	if(x instanceof Boolean)
		return ((Boolean) x).booleanValue();
	return x != null;
}
which appears to return a primitive.

seancorfield02:07:13

user=> (loop [x (pos? 1)] (when-not x (recur (boolean nil))))
nil
user=> (loop [x (pos? 1)] (when-not x (recur (first [false]))))
Syntax error (IllegalArgumentException) compiling fn* at (REPL:1:1).
 recur arg for primitive local: x is not matching primitive, had: Object, needed: boolean
user=> (loop [x (pos? 1)] (when-not x (recur (boolean (first [false])))))
nil
user=>

seancorfield02:07:21

Seems to work for me.

kenny02:07:53

Pass 1 instead of (first …

seancorfield02:07:50

Right, but you're not passing 1, you're passing (first is) for which it does work.

kenny02:07:58

But yes, (boolean (first ... does work.

kenny02:07:10

Why not (boolean 1)?

seancorfield02:07:35

Look at the code again and think about it being inlined.

kenny03:07:10

Inline expands to (. clojure.lang.RT (booleanCast 1)) which will hit this branch return x != null; in booleanCast. Why is that returning an object? Forgive me if it’s blatantly obvious, I’m a bit tired but this problem has been nagging at me all evening!

seancorfield03:07:21

user=> (loop [x (pos? 1)] (when-not x (recur (not= 1 nil))))
Syntax error (IllegalArgumentException) compiling fn* at (REPL:1:1).
 recur arg for primitive local: x is not matching primitive, had: java.lang.Boolean, needed: boolean
user=> (loop [x (pos? 1)] (when-not x (recur (.booleanValue (Boolean. 1)))))
nil
user=>
I'm assuming that inlining a function call on a literal value is going to expand parts of it and so you'd get the equivalent of the first one since 1 isn't an instance of Boolean

seancorfield03:07:15

(although it's happening at a Java level I assume -- I'd have to go digging into the compiler in depth for that)

seancorfield03:07:37

I'm just not terribly surprised there are weird edge cases like this...

kenny03:07:35

I’m not surprised either… My colleague hit this and I just could not give him a reason why. In all the years I’ve been doing Clojure I haven’t hit this one yet.

seancorfield03:07:04

inline is weird. Primitive deduction/auto-hinting locals is weird. The combination doubly-so.

kenny03:07:05

I still don’t totally follow how it’d get to not= in Clojure though

seancorfield03:07:36

I was just simulating the code from the Java method definition.

seancorfield03:07:00

By way of illustration of what I suspect is happening (since it fits the observed behavior).

kenny03:07:48

But the actual code appears to be returning the primitive unless the JVM itself is doing something.

kenny03:07:44

static public boolean booleanCast does that mean booleanCast will always return a primitive? I’d think so

seancorfield03:07:11

If it is inlining the bytecode, and potentially optimizing based on knowing it is passed a literal, it's entirely possible it "loses" the primitive return type of the function being inlined in that case.

seancorfield03:07:14

I'm thinking back to when I used to write compilers and virtual machines (late '80s/early '90s) and that was the sort of optimization we would do, including pre-evaluating parts of inlined functions when the compiler knows the argument is a literal.

kenny03:07:10

Oh, if it’s doing more than what meets the eye here then that seems like a plausible explanation. I’m not very familiar with how inline works other than it being “like a macro.”

seancorfield03:07:59

Like I say, I'd have to dig deep into the Clojure compiler sources to be sure of exactly what's going on in this case -- but it fits with what I'd expect, based on what I've seen done. I would say that it's a bug that the primitive return type is lost in that pre-evaluation tho'...

kenny03:07:05

I thought that’s where we were landing 🙂 I’m about ready to fall asleep though. Will keep looking into this tomorrow morning. Very curious what is happening!

seancorfield03:07:17

You should post some of the smaller fragments I showed above into a post on http://ask.clojure.org and see if it gets accepted as a bug.

kenny03:07:52

Great idea! I’ll do that tomorrow.

kenny14:07:19

@seancorfield You said “Because you can’t type hint an expression with a primitive” yesterday. Why can’t you type hint an expression with a primitive?

kenny15:07:18

This one is odd…

(loop [x (pos? 1)]
  (when-not x
    (recur (pos? x))))
Syntax error (IllegalArgumentException) compiling fn* at (src/foo.clj:4:1).
 recur arg for primitive local: x is not matching primitive, had: Object, needed: boolean

seancorfield16:07:43

I showed you why earlier in this thread. When you use boolean or long or double in a "type hint" you are referring to the Clojure.core function

seancorfield16:07:59

When you have ^x y that expands to ^{:tag x} y and both x and y are evaluated.

seancorfield16:07:07

^Boolean is fine because Boolean evaluates to java.lang.Boolean -- the class -- but ^boolean evaluates to ^{:tag clojure.core/boolean} and Java primitives aren't classes so it's pretty much impossible to get a reference to them in this case. It's why locals (`let`, loop) are auto-hinted as primitives, because you can't do it yourself.

kenny20:07:34

Auto-hinted when the compiler can. So when it can't you have to apply the hint elsewhere?

emccue00:07:57

Not as far as I know - java covers that pretty well though

emccue00:07:14

so just use the java

emccue00:07:33

if you need to support clojurescript too you'll have to look at npm and see whats available

daniel.spaniel00:07:44

got it , makes sense @emccue .. and thank goodness I dont need clojurescript

emccue00:07:12

yeah, lmk if you need help with any of the java interop stuff

emccue00:07:33

its pretty simple but it did take some learning for me before i was okay with just jumping to that

daniel.spaniel00:07:17

i will try it now and see if i get it ( thanks for offering help )

daniel.spaniel00:07:31

thanks again @emccue I got it working .. for both usa and euro format parsing

jonathanj07:07:16

I’m exploring the idea of an object-capability structure in an application I’m writing, I’m hoping for some inspiration from anyone who has either done this or thought about it. The obvious transcription from a typical OO implementation is protocols+records but it feels a bit strange and overly difficult in Clojure, maybe some sort of (perform :cap/whatever user-identity ,,,) multimethod could work more naturally but it feels a bit foreign too.

seancorfield07:07:51

@jonathanj My first reaction would be "Why? What problem are you actually trying to solve here?"

jonathanj07:07:51

@seancorfield Fair question, so I guess my reasoning is that previous systems I’ve written or worked on (not all necessarily in Clojure) do this kind of authorization checking at the fringes, e.g. the HTTP API, but once you’re past that the whole system is this soft gooey centre where Bad Things can happen accidentally because the rest of the system (the SQL DB, state atoms, etc.) are just code that other code is fully privileged to do with as it pleases. Maybe one solution to that is to sprinkle permission checks around but that often seems to have patchy coverage.

gerritjvv07:07:49

One issue could find with this is: There is perceived security vs real security. Let me explain.

gerritjvv07:07:35

If your aim is to protect each method from some bad code infecting the system, then your solution would not provide real security.

gerritjvv07:07:08

in worst case the bad code would just write/rewrite its own bytecode.

gerritjvv07:07:32

If you're trying to prevent a programming mistake by calling code that a user should never access via a legitimate api call, then it could work.

jonathanj07:07:33

Yes the former is probably pointless to try prevent, if you have a malicious programmer then there are probably a hundred ways to circumvent this. My concern is more a matter of making it simple to avoid obviously bad behaviour without the complexity of having to consider the cascading effects of your intent at each point.

gerritjvv07:07:10

cool. The java way would be to sprinkle annotations on each method. Which is just metadata in the end, and requires a processor to interpret it.

gerritjvv07:07:32

one idea would be to,annotate functions with a permission.

gerritjvv07:07:36

in clojure via metadata.

gerritjvv07:07:25

and then run the functions in some kind of processor that would check user permissions etc.

gerritjvv07:07:37

e.g (with-user user (call stack ) )

jonathanj07:07:08

The function would presumably have to check its own metadata, right?

cursork07:07:45

Or just some context to each function and an assertion?

(defn whatevs
  [ctx ...]
  (has-perm! ctx :updating-thingimys))

gerritjvv07:07:17

(sec/defn whatevs ^...perms... [] code.. )

gerritjvv07:07:11

where sec/defn is your own macro checking for e.g *user* or something. that could work

jonathanj07:07:12

Right. I tried to do this with just defn and realised that the metadata is attached to the var, so trying to write some kind of (perform fn ctx …) wrapper wasn’t working the way I wanted it to.

jonathanj07:07:34

So that basically boils down to what @cursork said, except the macro implicitly implements the assertion.

cursork07:07:16

But the macro still needs to be fed what the right perm is... Might as well accept the one line of code for making everything explicit

gerritjvv07:07:21

a harder way would be to have the (with-user call any function) intercept all functions with the metadata associated and add the checks as bytecode

jonathanj08:07:59

@cursork I actually ended up with pretty much that implementation yesterday but it seems like a macro might make it slightly harder to forget to perform the assertion. Just because sec/defn is the first thing you write and then a spec could verify the data, vs calling an assert as the third or fourth step of the implementation. :man-shrugging:

jonathanj08:07:52

I went looking for a precondition library that didn’t use AssertionError but didn’t come up with much.

gerritjvv08:07:08

👍 JVM world Exceptions are standard for this, and imo not bad.

cursork08:07:50

Well.. we do have pre and post-conditions in Clojure. So it can be:

(defn x [ctx ...] {:pre [(has-perm ctx :xyz)]} #_some-code)

gerritjvv08:07:43

+1 and you could macro it up if you wanted to. I prefer explicit context but ctx could come form a dynamic var.

jonathanj08:07:20

Thanks for the discussion, helped me get out of my own head a bit and find some inspiration. 🙏

👍 6
borkdude10:07:05

I have a weird problem probably related to AOT. In a project I'm factoring out the front-end build and hence I don't need the dependency on ClojureScript anymore. When I remove that dep, I suddently can't require taoensso.carmine anymore:

boot.user=> (require '[taoensso.carmine.commands] :reload-all)
       java.lang.ClassNotFoundException: clojure.tools.reader.reader_types.Reader
         java.lang.NoClassDefFoundError: clojure/tools/reader/reader_types/Reader
clojure.lang.Compiler$CompilerException: Syntax error compiling at (taoensso/carmine/commands.clj:267:1).
But the class is there!
boot.user=> clojure.tools.reader.reader_types.Reader
clojure.tools.reader.reader_types.Reader
When I add the dependency on ClojureScript back into the project, the problem disappears...

gerritjvv10:07:20

weird, different classloaders issue maybe??

borkdude10:07:10

even if I include the exact same deps as clojurescript but not cljs itself, I get this problem

gerritjvv10:07:17

some ideas: :0, check if cljs pulls in a dep from somewhere else you're not seeing, also try to remove all generated classes when switching deps. (find . -iname "*.class") or something.

borkdude10:07:19

When I use carmine as the only dep it works. It must be some combination of deps... I'll try bisecting. Removing .class files is not a problem here, since I'm using boot

👍 3
borkdude11:07:51

But weirdly enough that doesn't have a dep on tools.reader

gerritjvv11:07:28

strange, it does have this:

gerritjvv11:07:00

[[org.clojure/clojure "1.8.0"]                 [org.clojure/clojurescript "0.0-2156"]
[org.clojure/tools.cli "0.3.1"]]

borkdude11:07:18

yeah, we had an exclusion on the clojurescript part

borkdude11:07:35

it's pretty bad when a cross-platform dep has a dependency on clojurescript

👍 3
gerritjvv11:07:11

well at least you found it 🙂.

borkdude11:07:22

thanks for the mental support 🙂

👍 3
sandbags11:07:52

I've narrowed it down but am still baffled that I'm losing 8.5s somewhere… If -verbose:gc does what I expect then I am losing about 0.1s to GC. The outer function store-tuple is recorded at ~10s and consists of a call to two functions insert-tuple-returning-id (617ms) and insert-elements (806ms) so about 1.5s of clock time between them. What am I missing?

sandbags12:07:24

Okay so I wasn't seeing the with-transcation and clearly this takes time. If I remove that the time drops to 19s but is easier to explain. That said, 19s to insert something like 5.5k records doesn't sound right either.

sandbags12:07:14

Actually rather puzzling, I spend 10s inserting the 1,000 tuple records and the same 10s inserting the 4,500 odd element records

gerritjvv12:07:49

I was going to mention it 🙂 .there's allot more going on than just your functions.

👍 3
sandbags12:07:41

This is the timings with transactions removed and all indices (except the foreign key between tuples and elements) also removed. The logged time for jdbc-insert-tuple and jdbc-insert-elements are for profile/p wrapping calls to jdbc/execute-one! with SQL for inserting a tuple and 5 elements respectively.

sandbags12:07:57

9s to insert 1,000 records (these records are not much, the tuple record is literally two id's and an integer) doesn't sound right. I'm passing the result of a call to next.jdbc/get-datasource as my data source. I am assuming that means it uses one connection and isn't reconnecting to the db on each insert. That's my next assumption to test.

sandbags12:07:52

Oh ho ho, this does not appear to be a safe assumption

sandbags12:07:52

MySQL server connection count goes up by 2,000 on each test run

sandbags12:07:26

I have completely misunderstood what a next.jdbc data source is

sandbags12:07:13

So, using with-open and a single connection insert time for 1000 tuples (about 6000 records) drops to 2.9s!

👍 3
gerritjvv14:07:23

btw. inserting records like this is sloooow in any language.

gerritjvv14:07:41

depending on your usecase you'll win by doing batch inserts

gerritjvv14:07:49

somethings I've done in the past are:

gerritjvv14:07:12

1. write to csv and use load in file <-- extremely fast

gerritjvv14:07:29

2. use psql or mysql tools to do the load file and just call it from clojure/python etc.

gerritjvv14:07:09

3. use kafka/kinesis to write fast out of my app, and have another process read big chunks of records and insert into a db.

sandbags15:07:08

Thanks. This will all be sitting behind an API eventually and the degenerate case of inserting a million tuples is probably unlikely. But I need inserting a few thousand to be plausibly performant. Of course this is all running on an out of the box MySQL config on my laptop, probably could be better.

👍 3
valerauko12:07:20

i just realized that nubank also acquired the elixir company this january. can i start hoping for an official beam clojure?

Kevin12:07:02

There seems to be a lot of development in https://github.com/clojerl/clojerl 🙂

Kevin12:07:11

Not "official" though

gerritjvv12:07:18

nice!. I'm playing around with it now.

Alex Miller (Clojure team)12:07:46

The Elixir assets were given to the community and Jose formed a new company around Elixir called Dashbit

valerauko13:07:13

i heard about the acquisition and his new company on the changelog. i was wondering what was the rationale between acquiring two prominent open source programming language companies so quickly

3
wombawomba13:07:58

Is there a clean way to access the ‘original var’ inside a with-redefs? I’m wrapping a function and want to replace all calls to the original function with my wrapper fn within a given scope (but still have the wrapper fn call the original fn).

wombawomba13:07:35

I guess I’ll just do something like

(let [f some.ns/the-original-fn]
  (defn my-wrapper-fn [& args]
    (do-stuff)
    (apply f args)))

Darin Douglass13:07:57

heh, was just typing that up :thumbsup:

Darin Douglass14:07:23

well using with-redefs instead of re-`defn`ing it

wombawomba14:07:09

what if I’m redefing a macro though?

Darin Douglass14:07:07

that won't work b/c you can't assign a macro

Darin Douglass14:07:45

user> (let [original-and and])
Syntax error compiling at (*cider-repl(clj)*:95185:17).
Can't take value of a macro: #'clojure.core/and
user> 

wombawomba14:07:55

yeah, that was my thought

wombawomba14:07:17

but I suppose it’s not really a problem because it seems you can’t with-redef macros either

Ben Sless16:07:30

Any recommendation / best practices for checking if an infinite loop inside a thread terminated abnormally? The two I can come up with with are polling with a timeout and sharing an error channel between the task thread and the calling thread. Any other options I'm missing? (beside whole buy-ins like suprev)? Which one would you say is preferable?

dpkp03:07:33

can you wrap the infinite loop in a future and use realized? to check whether it has terminated? https://clojuredocs.org/clojure.core/future

Ben Sless05:07:57

The infinite loop is already wrapped in a async/thread. I can check if it terminated with poll!. I was wondering if there's a better way

dpsutton16:07:43

I think there’s an executor that can restart things automatically for you. Can also use an uncaughtexeptionhandler for info about it

dpsutton16:07:10

newSingleThreadExecutor(ThreadFactory threadFactory) and provide a threadfactory that creates threads doing your infinite loop?

Ben Sless16:07:56

I'm actually not necessarily interested in restarting in the case of failure. I might want to just let it fail and return the exception as data

dpsutton16:07:28

sounds like async/thread might work a treat for that then

Ben Sless16:07:56

Same problems I enumerated previously. I tried to make the question slightly generic, but in my particular use case I'm already using async/thread. The problem with an infinite loop running in it being that <!!ing from the returned channel will block indefinitely unless there's an error. So I need to poll, but polling can be too quick, and then we start bringing timeouts into the picture, or error channels

Ben Sless17:07:55

Another option with slightly less manual work will be async/merge ing them then blocking take from there, but it just reduces the problem to a single instance instead of multiple ones, doesn't address the underlying issue.

ghadi17:07:50

Does anyone here use an AWS Lambda for signing Maven artifacts?

ghadi17:07:45

(or some other mitigation to isolate the GPG key from the CI system)

euccastro18:07:18

I'm trying to import classes in a package[1] which has a CamelCase name. after adding the dependency to my Leiningen project and restarting the REPL (upon which the jar and pom were downloaded as expected), clojure gives me a ClassNotFoundException when I try and (import '(org.HdrHistogram Histogram)) [1] https://github.com/HdrHistogram/HdrHistogram/blob/5071aec2809d7d2f0564c1e81ce73c6d9e470731/src/main/java/org/HdrHistogram/Histogram.java#L8

noisesmith18:07:16

small thing aside from your issue - ' is optional with import due to the way it's parsed, I suggest not using it

thanks2 3
noisesmith18:07:04

user=> (import '( OutputStream))
java.io.OutputStream
user=> (import (java.nio ByteBuffer))
java.nio.ByteBuffer

Alex Miller (Clojure team)18:07:07

as descibed, those sound like correct things to do

Alex Miller (Clojure team)18:07:05

lein classpath should show you your classpath - verify the dependency jar is in there

👍 3
Alex Miller (Clojure team)18:07:32

should be at ~/.m2/repository/groupId/artifactId/artifactId-version.jar

Alex Miller (Clojure team)18:07:24

wherever that is, jar tf <that.jar> | grep Histogram to check the class is in the jar

Alex Miller (Clojure team)18:07:59

stepping away, but those would be my first checks

euccastro18:07:44

thanks! strangely enough, the jar is there and the class is in it, but the path to the jar is not in the lein classpath for this project (although, as I said, it got downloaded only after adding it to that project). I'll keep digging...

seancorfield18:07:21

@euccastro Try lein clean and then lein classpath, just in case Leiningen has gotten stale.

euccastro18:07:21

@seancorfield thanks! I did that and lein classpath | grep togram still outputs nothing

noisesmith18:07:05

@euccastro can you share your :dependencies [] from your project.clj?

hiredman18:07:57

you should add a dependency that doesn't exist next to your hdrhistogram dependency and see if you get an error

euccastro18:07:19

duh-- sorry, my bad; I had added the dependency to :plugins not :dependencies :man-facepalming:

noisesmith18:07:37

that would do it

euccastro18:07:39

got befuddled by the fact that it got downloaded anyway 😳

euccastro18:07:50

sorry for the noise, and thanks all for the help!

murtaza5219:07:46

I am trying to parse a given date string - "5/9/18" or "12/23/18" to a java.util.Date for inserting into datomic, what will be a good way to do that ?

seancorfield19:07:45

@murtaza52

user=> (java.time.LocalDate/parse "5/9/18" (java.time.format.DateTimeFormatter/ofPattern "M/d/yy"))
#object[java.time.LocalDate 0x2a28bf25 "2018-05-09"]
user=> (java.time.LocalDate/parse "12/23/18" (java.time.format.DateTimeFormatter/ofPattern "M/d/yy"))
#object[java.time.LocalDate 0x6c54bc3 "2018-12-23"]
user=> 
Does Datomic support Java Time types?

dpsutton19:07:02

or alternatively

user=> (import (java.text SimpleDateFormat))
java.text.SimpleDateFormat
user=> (let [format (SimpleDateFormat. "yyyy-MM-dd")] (.parse format "2009-12-31"))
#inst "2009-12-31T06:00:00.000-00:00"
user=>

seancorfield19:07:47

Yeah but I try to encourage folks to use Java Time unless they can't 🙂

dpsutton19:07:57

good point 🙂

seancorfield19:07:20

Admittedly, it's a lot more code if you need to ultimately end up with a java.util.Date:

user=> (-> "5/9/18" (java.time.LocalDate/parse (java.time.format.DateTimeFormatter/ofPattern "M/d/yy")) (.atStartOfDay (java.time.ZoneId/systemDefault)) (.toInstant) (java.util.Date/from))
#inst "2018-05-09T07:00:00.000-00:00"
user=> (-> "12/23/18" (java.time.LocalDate/parse (java.time.format.DateTimeFormatter/ofPattern "M/d/yy")) (.atStartOfDay (java.time.ZoneId/systemDefault)) (.toInstant) (java.util.Date/from))
#inst "2018-12-23T08:00:00.000-00:00"
user=> 

seancorfield19:07:50

But it underscores that you do need to think about timezones at some point 🙂

murtaza5219:07:16

@U11BV7MTK I tried that but the day is one day off -

(.parse (SimpleDateFormat. "MM/dd/yy") "5/10/18")
;; => #inst "2018-05-09T21:00:00.000-00:00"

seancorfield19:07:33

Exactly my point! Timezones!

dpsutton19:07:10

some dbs support a Date object without time which is probably what you want otherwise you'll always be chasing timezone stuff

murtaza5219:07:04

I am still missing the point, how is the timezone coming in the above picture, why is the day one day off

seancorfield19:07:20

It isn't one day off. It's three hours adjusted from UTC.

seancorfield19:07:31

What timezone are you in @murtaza52?

murtaza5219:07:56

East Africa Timezone - EAT

seancorfield19:07:39

That same "simple" code for me

user=> (.parse (java.text.SimpleDateFormat. "MM/dd/yy") "5/10/18")
#inst "2018-05-10T07:00:00.000-00:00"
user=> 
because I'm in Pacific time.

dpsutton19:07:41

the trouble comes from the fact that right "now" has two different days depending on where in the world you are

seancorfield19:07:54

So I'm -7 and you're +3.

seancorfield19:07:31

Current time in East Africa Time ‎(UTC+3)‎ 10:53 PM

seancorfield19:07:52

Current time in Pacific Daylight Time ‎(UTC-7)‎ 12:53 PM

seancorfield19:07:07

Current time in Coordinated Universal Time ‎(UTC)‎ 7:54 PM

seancorfield19:07:31

This is why Joda Time was created and then Java Time (the successor to Joda Time, which was then added to Java 8 -- and most of java.util.Date is deprecated).

seancorfield19:07:34

(this is also why it's so important to have all your servers and all your databases and all your JVMs set to the same timezone, preferably UTC)

👍 3
seancorfield19:07:18

Does that help @murtaza52? (time is hard because of this sort of thing)

murtaza5220:07:08

I would understand that it returns different dates if I did (now), however I would expect that when I parse it just returns me that date in my current timzone, but whyc change the date itself ?

murtaza5220:07:28

@seancorfield this still guves the day off -

(-> "5/9/18"
    (java.time.LocalDate/parse
     (java.time.format.DateTimeFormatter/ofPattern "M/d/yy"))
    (.atStartOfDay
     (java.time.ZoneId/systemDefault))
    (.toInstant)
    (java.util.Date/from))
;; => #inst "2018-05-08T21:00:00.000-00:00"

seancorfield20:07:35

As I said, that is correct because it is adjusted to your timezone. But if you want it in UTC, use (java.time.ZoneId/of "UTC") instead of systemDefault.

dmillett20:07:33

And JVM/OS timezone file updates might also provide a hiccup or two. I'm a fan of UTC whenever possible

seancorfield20:07:40

Here's my default (UTC-7):

user=> (-> "5/9/18" (java.time.LocalDate/parse (java.time.format.DateTimeFormatter/ofPattern "M/d/yy")) (.atStartOfDay (java.time.ZoneId/systemDefault)) (.toInstant) (java.util.Date/from))
#inst "2018-05-09T07:00:00.000-00:00"
user=> (-> "12/23/18" (java.time.LocalDate/parse (java.time.format.DateTimeFormatter/ofPattern "M/d/yy")) (.atStartOfDay (java.time.ZoneId/systemDefault)) (.toInstant) (java.util.Date/from))
#inst "2018-12-23T08:00:00.000-00:00"
and here it is in UTC:
user=> (-> "12/23/18" (java.time.LocalDate/parse (java.time.format.DateTimeFormatter/ofPattern "M/d/yy")) (.atStartOfDay (java.time.ZoneId/of "UTC")) (.toInstant) (java.util.Date/from))
#inst "2018-12-23T00:00:00.000-00:00"
user=> (-> "5/9/18" (java.time.LocalDate/parse (java.time.format.DateTimeFormatter/ofPattern "M/d/yy")) (.atStartOfDay (java.time.ZoneId/of "UTC")) (.toInstant) (java.util.Date/from))
#inst "2018-05-09T00:00:00.000-00:00"

seancorfield20:07:34

Note the time portion in the default version: 07 in May, 08 in December. Due to daylight savings time changes, in addition to my timezone.

seancorfield20:07:15

@murtaza52 And here are those two dates in your timezone:

user=> (-> "12/23/18" (java.time.LocalDate/parse (java.time.format.DateTimeFormatter/ofPattern "M/d/yy")) (.atStartOfDay (java.time.ZoneId/of "Africa/Nairobi")) (.toInstant) (java.util.Date/from))
#inst "2018-12-22T21:00:00.000-00:00"
user=> (-> "5/9/18" (java.time.LocalDate/parse (java.time.format.DateTimeFormatter/ofPattern "M/d/yy")) (.atStartOfDay (java.time.ZoneId/of "Africa/Nairobi")) (.toInstant) (java.util.Date/from))
#inst "2018-05-08T21:00:00.000-00:00"
(took me a while to figure out a zone ID that corresponded to EAT)

murtaza5220:07:48

ok these dates are for events that happened in a timezone in US, it could be any of the US timezones, and I dont have that info available, however it happened on that particular date, and when I store it in the DB that is what I want to reflect, how should I parse them ?

seancorfield20:07:02

You have to know the timezone to get it right. Do you know the lat/lon of the event location? You can deduce timezone from lat/lon.

seancorfield20:07:16

(or the city/state -- either way you'll need some service to lookup the TZ at the date of the event since the offset can change due to daylight savings time -- hence my 7/8 hour offset being different in May and December)

seancorfield20:07:46

Some parts of the US observe DST and some don't. Some states have parts in one timezone and parts in another timezone (or observe DST differently in different parts of the state!).

murtaza5220:07:43

(-> "5/9/18"
    (java.time.LocalDate/parse
     (java.time.format.DateTimeFormatter/ofPattern "M/d/yy"))
    (.atStartOfDay
     (java.time.ZoneId/of "UTC"))
    (.toInstant)
    (java.util.Date/from))
;; => #inst "2018-05-09T00:00:00.000-00:00"
So is UTC the solution, if I use UTC then I assume it will not ad any timezone offsets, and thus the date represented in the string be preserved ?

seancorfield20:07:45

If you really only care about the date, you could probably get away with sticking noon as a time on the end and parse that since that will preserve the date within all of the US timezones -- but that will be a different date to where you are in EAT...

murtaza5220:07:28

will making it a UTC timzeone have similar effect ? (check the code above)

seancorfield20:07:12

UTC is UTC everywhere. But be careful that your database doesn't adjust the timezone when you're storing/reading dates.

seancorfield20:07:40

(hence my admonition that server, database, and JVM should all be set to the same timezone, preferably UTC)

murtaza5220:07:26

thanks will check that

murtaza5220:07:25

JVM ? does it also have a timzeone setting ?

seancorfield20:07:57

Yup, you can specify it via a JVM option at startup.

seancorfield20:07:31

You can have your JVM and your database on the same server and have all three set to different timezones.

seancorfield20:07:43

(but you want to ensure they are all set to the same timezone 🙂 )

seancorfield20:07:39

This is always an issue for us when our data center on the East Coast (US) stands up a brand new server for us -- they always leave it on Eastern time, even though we always specify we want it set to UTC, and we file a ticket, and they fix it 🙂

murtaza5220:07:06

cool appreciate the info !

andy.fingerhut21:07:54

I know you guys are discussing serious stuff that matters to software developers, but it reminds me of this song: https://www.youtube.com/watch?v=9FzCWLOHUes

3
🎶 3
murtaza5208:07:32

@seancorfield this is what I came up with -

(require '[java-time :as jt])

(defn java-date->zoned-date-time
  [d]
  (jt/zoned-date-time (jt/local-date-time jud "UTC") "UTC"))

(defn string->java-date
  [s]
  (jt/java-date (jt/zoned-date-time (jt/local-date "M/d/yy" s)
                                    (jt/local-time 12 0 0)
                                    "UTC")))
(comment
  (string->java-date "5/8/18")
  (java-date->zoned-date-time (string->java-date "5/8/18")))

murtaza5208:07:54

do I still need to set all my servers/ db / jvm to utc, bcoz in the above I am explicitly using UTC time when reading from a string->zoned-date-time->java-date before storing in db, and then I read it back as UTC time to zoned-date-time.

murtaza5208:07:56

another question, does it make more sense to just using java.time.Instant and then store everything as milliseconds (bigdec) in the db, rather than java.util.Date

murtaza5209:07:00

this is the second approach I was talking about -

(defn string->epoch-milli
  [s]
  (bigdec
   (.toEpochMilli
    (jt/instant
     (jt/zoned-date-time (jt/local-date "M/d/yy" s)
                         (jt/local-time 12 0 0)
                         "UTC")))))

(defn epoch-milli->zoned-date-time
  [v]
  (jt/zoned-date-time (jt/instant v)
                      "UTC"))

(comment
  (string->epoch-milli "5/8/18")
  (epoch-milli->zoned-date-time (string->epoch-milli "5/8/18")))

murtaza5209:07:37

Now I dont have to worry about time zones on servers / dbs /jvm (my assumption). Also dont have to deal with java.util.Date. I just save the bigdec in my db. Is there any disadvantage to this approach, do I loose some db functionality when I use this (Ia ssume I can still do all sort of query comparisons using the bigdec num)

seancorfield18:07:36

There are some people who store date/time as plain milliseconds and do all their conversions in their app code. I think that's a waste of perfectly good functionality in SQL and in the database. Setting everything to UTC and working with date/time in UTC is a widely accepted approach and it's what I would recommend. I don't know what problems you'll run into with storing date/time as milliseconds but I suspect you'll end up with a lot of unnecessary boilerplate code to convert to/from actual dates -- and you'll only need to miss one conversion in one place to introduce a bug in one of your queries...

cjmurphy17:07:32

@murtaza52 A java.util.Date is in UTC. And is just a number of milliseconds past a certain point. UTC does not suffer from daylight savings problems as some TZs do. When you print the java.util.Date the JVM finds your TZ (if you haven't set the TZ on the JVM) and uses that. That leads some people to think that java.util.Date somehow contains the TZ attribute of their country, which it does not.

murtaza5204:08:10

@U0D5RN0S1 thanks for the clarification