Fork me on GitHub
#clojure
<
2018-12-29
>
macrobartfast02:12:37

what's the most current/usable dep for interacting with the filesystem now?

macrobartfast02:12:48

seems to be several out there.

seancorfield03:12:08

@macrobartfast It depends what you want to do...?

macrobartfast03:12:41

I should have said that! I'm trying to list the directories in a directory.

seancorfield03:12:49

Are you looking for something list this? https://github.com/clj-commons/fs

seancorfield03:12:39

Or you could just use Java interop -- which may well be the easiest path, depending on what you need.

seancorfield03:12:52

(.listFiles (io/file "path/to/folder")) should work I think...

seancorfield03:12:41

Yeah... example

[email protected]:~/clojure$ clj
Clojure 1.10.0
user=> (require '[ :as io])
nil
user=> (.listFiles (io/file "/tmp"))
#object["[Ljava.io.File;" 0x13d186db "[Ljava.io.File;@13d186db"]
user=> (seq *1)
(#object[java.io.File 0x15723761 "/tmp/hsperfdata_root"] #object[java.io.File 0x312afbc7 "/tmp/hsperfdata_seanc"])
user=> (doseq [f (.listFiles (io/file "/mnt/c/users/seanc"))] (println f)
)
#object[java.io.File 0x2ce45a7b /mnt/c/users/seanc/.atom]
#object[java.io.File 0x153d4abb /mnt/c/users/seanc/.cisco]
#object[java.io.File 0x6d4c273c /mnt/c/users/seanc/.ssh]
#object[java.io.File 0x5a67e962 /mnt/c/users/seanc/3D Objects]
#object[java.io.File 0x545e57d7 /mnt/c/users/seanc/AppData]
#object[java.io.File 0x2bc9a775 /mnt/c/users/seanc/Application Data]
#object[java.io.File 0x27b000f7 /mnt/c/users/seanc/Contacts]
#object[java.io.File 0x42f9c19a /mnt/c/users/seanc/Cookies]
#object[java.io.File 0x64bfd6fd /mnt/c/users/seanc/Desktop]
#object[java.io.File 0x2ab2710 /mnt/c/users/seanc/Documents]
#object[java.io.File 0x253b380a /mnt/c/users/seanc/Downloads]
#object[java.io.File 0x29c2c826 /mnt/c/users/seanc/Favorites]
#object[java.io.File 0x3350ebdd /mnt/c/users/seanc/IntelGraphicsProfiles]
...

macrobartfast03:12:06

that's probably what I'm looking for... I was using Raynes's original project.

seancorfield03:12:02

clj-commons will maintain it from now on. I'm not sure that it got many updates between Raynes' last version and what's in that new organization?

seancorfield03:12:57

We use it at work for some compression stuff but elsewhere, when we're just interacting with the file system, we use Java interop.

macrobartfast03:12:23

I'm getting objects back too, but I'm wondering if I should be getting strings... I'll have to see if objects will work in my case.

seancorfield03:12:19

File objects. That's what io/file returns. You're expected to work with those.

seancorfield03:12:45

Call .getAbsolutePath or .getCanonicalPath to get a string 🙂

seancorfield03:12:56

Or just .getName.

macrobartfast03:12:19

yay! that helps. thanks! 😀

seancorfield03:12:13

The joys of a hosted language 🙂

ahungry04:12:59

Are there any packages (or would it ever be a useful concept) to combine core typed + spec? For pre-runtime analysis of things like ADT and runtime instrumentation of actual pre/post conditions?

seancorfield04:12:08

Do you mean "Is there any value to using core.typed on my code and clojure.spec?"... Sure, they serve completely different purposes.

seancorfield04:12:26

Could core.typed annotations be derived from specs? Maybe, in some situations.

ahungry04:12:21

That's what i'm thinking - at least for the general shape of data, and having some macro based system like ghostwheel provides for spec that would handle both cases seems like it could be pretty useful

ahungry04:12:44

in common lisp, I did something similar with their (well, sbcl's) declare ftype + deftypes: https://github.com/ahungry/cl-typed

ahungry04:12:04

and using an easy macro syntax, so the type defs were very haskell'ish

seancorfield05:12:34

I think that core.typed will eventually be a useful addition but it is too restrictive today, IMO.

seancorfield05:12:09

We've been back and forth between core.typed and Schema several times, and now we use spec heavily at work.

ahungry05:12:49

isn't it similar to typescript, in that it allows optional/gradual typing? so you could just choose not to type some things?

seancorfield05:12:53

core.typed was very painful to use. See the CircleCI post about why they abandoned it. Ambrose is taking a step back and looking at radical improvements.

seancorfield05:12:06

No, core.typed is not like TypeScript.

ahungry05:12:55

Ahh nice, I did take a quick look at that circleci post awhile ago (before i was using clojure frequently). Oh, I'll have to spend some time with it and find how it varies (spec is pretty great, i'm excited to see what new things come of that in the future as well

seancorfield05:12:26

core.typed is a lot more invasive. You have to annotate everything in a namespace -- you can't just annotate one function. It also often requires you to annotate code within functions and/or refactor code to satisfy the type checker.

ahungry05:12:23

Oh, I see what you mean. if I have something like this:

=(t/ann add-one [t/Int :-> t/Int])
=(defn add-one [x]                                                                                                   
=  (+ 1 x))
=
=(t/ann add-two [t/Int :-> t/Int])
=(defn add-two [n]                                                                                                   
=  (add-one (add-one n)))

ahungry05:12:34

if I didn't have the annotation on add-two, it fails the checker

ahungry05:12:49

(excuse the horrendous formatting)

ahungry05:12:31

Still, I wonder if tihs isn't just a cumbersome syntax construct that causes most the grief, writing a signature like this:

(defun add-three (Int n → Int)
   (add-two (add-one n)))

ahungry05:12:55

if that were a macro that generated a defn + the t/ann, seems like it would be nice and concise

ahungry05:12:01

(and have it set spec int? type stuff as well)

emccue05:12:12

personally I still don't understand spec

ahungry05:12:35

from what I see, its more akin to schema validation (like jsonSchema or AJV in js world)

emccue05:12:39

not its purpose, but the actual "how do I use this thing"

ahungry05:12:43

in small little portions

ahungry05:12:37

the generative testing is cool - I found a good (huge) macro a couple months ago that will automatically inject each spec def into your unit test suite so you can run generate testing as part of the default test runner system

emccue05:12:15

Its just...

emccue05:12:22

i dont know how to put it

emccue05:12:37

I can't read this and use it

emccue05:12:48

Predicates

emccue05:12:54

check, got it

ahungry05:12:19

the syntax (imho) sucks, thats why I want to write some useful macro to make the types look like C types

ahungry05:12:28

and just have it 'dwim' for both spec and core typed

emccue05:12:30

but once it gets into the "Registry" I just get baffled

emccue05:12:18

It's definitely something stupid on my part, but I register a spec in the registry under a key, they use namespaced ones to avoid colissions cool cool

emccue05:12:34

and that translates to a check...somewhere

emccue05:12:06

immediately after that it explains conform and explain

emccue05:12:56

which seem to just be fancy ways of calling the predicate function

emccue05:12:27

I understand the entity maps, but I have no clue why each key needs to be defined as its own thing with its own spec

ahungry05:12:59

well, each 'thing' is a tpye

ahungry05:12:04

type* (or, think of it like one)

ahungry05:12:23

so, (s/def ::acctid int?) - you just made a new primitive type

ahungry05:12:33

and you could have an acctid in two totally unrelated composite types

ahungry05:12:47

maybe a map of bank data, maybe a map of twitter accounts

ahungry05:12:26

I think the idea is that names are important - so this acctid is now a label (just like 'string' vs 'int' is more useful than some weird _3182 symbol to refer to them)

emccue05:12:26

but in the s/keys call, doesn't the name of the "type" matter?

emccue05:12:47

like I cant have a spec that defines an email

ahungry05:12:58

so, ::person is a composite type (a map) and to meet the minimum definition of a 'person" the data must match, in that it has to have a first name + last name + email

emccue05:12:05

I need to have a spec that is a keyword that points to a spec in a global registry

ahungry05:12:07

its combining labelling (a map with these keys) and typing

ahungry05:12:12

keys with values that match the defined types

ahungry05:12:26

btw, this works and was pretty trivial to write a macro for: `

ahungry05:12:30

(defmacro defun [name sig & args]                                                                                   
   `(do                                                                                                              
      (t/ann ~name [[email protected](take-nth 2 (butlast sig)) :-> ~(last sig)])                                                   
      (defn ~name [[email protected](take-nth 2 (rest sig))]                                                                        
        [email protected])))

 (defun add-nums (t/Int a t/Int b t/Int)                                                                             
   (+ a b))

ahungry05:12:56

still ugly, Ill need to auto-namespace the types and keep them plain symbols i think (then roll in spec 🙂 )

emccue05:12:24

wait, how in the hfil is ::person a map?

ahungry05:12:47

s/keys = has to be a "something" with those keys

ahungry05:12:52

map is just (usually ) the default

ahungry05:12:10

if you dont like the double dots, maybe you wanna use with a defrecord, you can also use unreq for unqualified required

emccue05:12:32

I wouldn't ask to peek under the hood of s/keys unless I was confused

emccue05:12:59

okay lets just laser focus here

emccue05:12:17

(s/keys :req [::first-name])

ahungry05:12:34

thats one I was playing with the other day while writing a blog article

ahungry05:12:55

even though defrecord isn't idiomatic, I made a 'user model' - its a record, not a map, but to be a 'valid' one it has to have name, age, id, language

ahungry05:12:01

as key labels that match the spec defs

emccue05:12:11

(s/def ::first-name string?)

emccue05:12:44

this puts string? into a global registry under the key :my.ns/first-name

emccue05:12:04

then (s/keys :req [::first-name])

emccue05:12:51

this says "I am a spec that requires a key :first-name which satisfies the spec fn registered under :my.ns/first-name

emccue05:12:44

Why in the heck does making a spec that checks a key in a map require I have registered a function in the global registry under the correct name?

ahungry05:12:07

good question

emccue05:12:14

like (s/keys :req {:first-name string?}) is what i would have expected

emccue05:12:01

and then support for subbing string? with ::first-name

emccue05:12:06

also, how do I instrument everything?

emccue05:12:35

like "Okay im in development time, always check my specs and check to make sure I dont mess up specs in my dependencies"

emccue05:12:43

because the docs are confusing on that point

emccue05:12:22

...kinda just telling me how to instrument a single function

emccue05:12:47

also, why the heck in heck above does spec give a way to stub things for testing

emccue06:12:27

Nothing is bad I just feel like im missing something

ahungry06:12:01

when I was looking, it seemed I had to have an stest/instrument next to each fdef I made, or do something like query the ns for all symbols and map stest/instrument across them

emccue06:12:06

also why does instrument take a symbol and not a var?

emccue06:12:30

actually no I can see how having a symbol would make whatever it is spec does to work better with reloading during dev

emccue06:12:37

its 1am so ill stop polluting everyones notifications now

seancorfield06:12:46

@emccue I think you're seriously overthinking spec (if that helps) 🙂

seancorfield06:12:58

Specs are mostly just predicates

seancorfield06:12:33

s/keys defines a predicate that is a conjunction of predicates on keys.

ahungry06:12:36

How can I quasiquote without using a backtick?

ahungry06:12:53

(= pp 'pp)` - that isn't equal, so (quote does not seem to help me here

ahungry06:12:11

(I want to enable stest/instrument based on a macro expansion

ahungry06:12:29

(stest/instrument (quote ~name)) -

ahungry06:12:01

nevermind, stest/instrument can just be called with no arg and enable full instrumentation (basically exactly what I want during a new defun)

rdsr07:12:21

is there a way I can add a local dir (of jars) to deps.edn ?

rdsr07:12:59

Also, adding a single local jar throws an error like so

Error resolving /Users/rratti/code/calcite-play: 1 problem was encountered while building the effective model for org.apache.calcite:calcite-core:1.18.0
[ERROR] Failed to determine Java version for profile java8 @ org.apache.calcite:calcite:1.18.0
My hunch is that it might be a problem with the local jar?

alexmiller13:12:47

When you use a local jar, it will read the pom inside the jar to determine its transitive deps - that’s what is failing here

alexmiller13:12:52

I haven’t encountered this particular error before

rdsr16:12:32

Thanks Alex. For adding a whole dir of jars, I guess the only way is the add every jar one by one?

alexmiller19:12:18

Currently, yes

flyboarder17:12:28

Any clues? Reflection warning, <file> - call to java.lang.ProcessBuilder ctor can't be resolved.

flyboarder17:12:58

here is the function

flyboarder17:12:34

(defn- my-proc [version args]
  (ProcessBuilder. ["java"]))

andy.fingerhut17:12:10

A reflection warning in Clojure is only a warning that calls to that Java method will be very slow, as compared to calls to Java methods that do not give reflection warnings.

andy.fingerhut17:12:22

If it is in setup or init code, run once or only a few times, I wouldn't worry about it.

andy.fingerhut17:12:44

If it is in an inner loop of your code run billions of times, then it is worth fixing.

flyboarder17:12:52

@andy.fingerhut I plan on compiling to a native image with graalvm - I need to resolve these reflections

andy.fingerhut17:12:10

ah, I was missing that context. Thanks.

👍 8
eraserhd17:12:37

@flyboarder you'll need to bind ["java"] to a name and type hint that name. I presume it's telling you that there are multiple constructors for ProcessBuilder and it doesn't know which one to use.

andy.fingerhut17:12:05

Looking at the Java docs for ProcessBuilder, there are 2 constructors, one which takes a java.util.List, and one which takes a String..., meaning Java varargs, which in Clojure must be an explicit Object array.

andy.fingerhut17:12:56

Maybe try: (ProcessBuilder. (object-array ["java"])) ?

phill17:12:30

I wonder why it needs a hint in this case. The literal string vector does satisfy List and does not satisfy String[]

andy.fingerhut17:12:42

Depending on how long you want to dig into it, you could put debug print statements inside the Clojure compiler reflection code and try to suss it out.

flyboarder17:12:28

@eraserhd could you provide an example? what should the type hint be?

flyboarder17:12:46

@andy.fingerhut the object-array did not work 😞

eraserhd17:12:48

Off the top of my head, (let [args ^java.util.List] (ProcessBuilder. args)) ?

eraserhd17:12:51

er, (let [^java.util.List args ["java"]] (ProcessBuilder. args))

parrot 4
flyboarder17:12:14

@eraserhd yes that did it!!

flyboarder17:12:36

thank you all!

schmidt7318:12:12

clojurians, i'm trying to build functions on top of my database to work with users. for example I have a function get-user which takes a username and returns their row in the users table. my problem is this: I want to be able to swap out the database that get-user uses to grab users (for mocking, testing, etc). one solution would be to have each function take an extra db param as argument so get-user would have the following type get-user : db * username -> user-row. the other is to have the db as some global parameter that I can swap out. both of these solutions don't suffice in my mind. the first has the problem of having to pass the same db to every function call, whereas the second requires that all of my function calls access the same db (i might want them to access different ones).

Eccentric J18:12:00

https://github.com/tolitius/mount is a good solution for that, allows you to treat db as a global object but is easy to redefine the state in your tests

schmidt7318:12:23

thank you, i've heard of mount before.

schmidt7318:12:34

is it considered the best way to do it?

schmidt7318:12:44

i'm very averse to global state 😐

Eccentric J18:12:21

Component is also an alternative, but yes this seems to be the recommended way to handle app state since each piece you define with mount can be stopped, restarted, and reset in a deterministic way. If you create a luminus template like lein new luminus test-app +postgres +site you can see mount and hugsql in action for how it handles test environments and separating db functions from db state.

haus19:12:58

in the case where the db is a param to your function, you could also have a partial fn for your application with db already bound to it

cjsauer20:12:49

The problem with a global db is that now all of those functions are impure. db as a param seems more idiomatic imo, and matches the “database as a value” paradigm common in Clojure. And as @U3XNU4Y57 mentioned, it’s easy to partially apply those functions with an initialized db value. Another way to think about it is in the context of OOP; it’s common to see db.getUsers(...), in which the db object is the (implicit) first parameter (`this`). Edit: I suppose this only makes sense if the database tech in question supports immutability though...

cjsauer20:12:48

That said, Mount is a great tool for setting that db value up initially (w/ potential dependency injection) and tracking its state in the face of REPL reloads. It can then easily be passed in to the (pure) user functions.

schmidt7318:12:14

in haskell i'd use a monad, so that get user would have the following type get-user : username -> db user and then i could call rundb : db * db a -> a to get my value out in the context of a certain database.

schmidt7318:12:53

i like the haskell way, but we are in clojure land now. my other idea was to write a function that takes a database and returns a map of user database functions specialized to that db. this seems silly though and not idiomatic clojure. what is the best way to accomplish my goal?

Lennart Buit19:12:19

Not to say that this is a solution to your problem — But if you were to mark a global definition as dynamic, you can use binding to redefine it in tests.

Lennart Buit19:12:20

So you can define some global constant: (def ^:dynamic *some-global-constant* 12), and make up your mind later and use: (binding [*some-global-constant* 42] …) to locally redefine the global constant. But that is not the elegant solution in this case I think.

lilactown19:12:47

@henri.schmidt you can implement a similar pattern to the Haskell monadic approach in Clojure, if you like

lilactown19:12:52

there are many ways to do it, but one could be to define a simple data structure to describe the database function you want to be executed:

;; get-user
;; example output: [:db/user "someuser342"]
(defn get-user [uname]
  [:db/user uname])

(defn run-db! [descriptor]
  ... )

lilactown19:12:52

if you can make the descriptors of your db commands simple enough, a multimethod for the run-db! function is quite nice:

(defmulti run-db! first)

(defmethod run-db! :default [_]
  (throw (RuntimeException. "Unknown database function.")))

(defmethod run-db! :db/user 
  [[_ uname]]
  ;; fetch user row
  )

cjsauer20:12:16

Would the above pattern be useful for a database like Datomic, where an immutable db value can be obtained efficiently, and then just passed as an argument? Is the monad pattern mainly useful in contexts where the db is mutable/stateful?

lilactown20:12:03

I think it could be useful no matter which db

lilactown20:12:20

but how useful, is hard to gauge

lilactown20:12:10

it’s essentially the “command pattern” in data

schmidt7322:12:47

thank you everyone for the advice. i think i'll use mount for the db regardless and then have all the functions take the db as a parameter. should i manually be passing this db parameter around to every function call? id kind of like to define the db at the top level and have it pass down.

flyboarder23:12:07

@henri.schmidt I think the correct answer is to use both ways of doing it. Thats what I do and you get the best of both worlds assuming you have a sane way to reason about when you use each style

flyboarder23:12:07

for example I have this for a database reference a (def ^:dynamic *db* (db-connection))

flyboarder23:12:57

Then I make my api:

(defn get-user
  ([user] (get-user *db* user))
  ([db user] (.getUser db user)))

flyboarder23:12:16

This way I have a default, which is dynamic ( so optional binding overrides) and I also get explicit db to use whenever you need