Fork me on GitHub
#clojure
<
2023-02-02
>
Guild Navigator15:02:41

How can I skip the fist N rows of a file using clojure.data.csv/read-csv ?

Alex Miller (Clojure team)15:02:50

you can't directly, but you could arrange to have a reader/string that drops n lines

Alex Miller (Clojure team)15:02:30

if the file is "short", you could slurp it, then split-lines, drop N, then string/join back to a string

Alex Miller (Clojure team)15:02:23

seems easily slurp-able, if performance is not critical

Guild Navigator15:02:40

a few extra ms is just fine

Guild Navigator15:02:26

I think I got it:

(def raw-data (clojure.string/join (drop 4 (clojure.string/split-lines (slurp "foo.csv")))))

Guild Navigator15:02:25

So my files are weird and are UTF-16 little endian. Slurp is not detecting the encoding, or at least pprint isnt printing it right

Guild Navigator15:02:20

But it's not guaranteed to be UTF-16 LE — it might be UTF-8 or ISO-8859-1. Any way to detect the encoding to pass to slurp?

Alex Miller (Clojure team)15:02:34

you can pass an encoding to slurp

Alex Miller (Clojure team)15:02:56

(slurp "foo.csv" :encoding "UTF-16") I think

Alex Miller (Clojure team)15:02:35

if you don't know the encoding, not sure what exists in the JDK to do that detection (wouldn't surprise me if there is something, but don't know)

Guild Navigator15:02:27

I found [clj-det-enc "1.0.0"] but it operates on a stream and I have a string at this point

Guild Navigator15:02:55

so i created this (defn detect-encoding [stream] (det/detect stream)) but i dont know how to pass a string to it (sorry i'm on week 2 with Clojure)

Guild Navigator15:02:17

okay so i have to read the file twice - once to detect the encoding, and then again with slurp?

Alex Miller (Clojure team)15:02:46

yeah, I assume detect is not actually going to read the whole file?

Guild Navigator15:02:37

yeah my guess is if it's written well it only reads the first few bytes

Guild Navigator15:02:27

ok so i have this:

(defn detect-encoding [file] ((det/detect ( file))))
but I get an exception here:
(p/pprint (detect-encoding "foo.csv"))
> Execution error (ClassCastException) at whatsnew-clj.core/detect-encoding (core.clj:14). > class java.lang.String cannot be cast to class clojure.lang.IFn (java.lang.String is in module java.base of loader 'bootstrap'; clojure.lang.IFn is in unnamed module of loader 'app')

Alex Miller (Clojure team)16:02:14

seems like you have an extra set of parens in there?

Alex Miller (Clojure team)16:02:25

just in general, you should like questioningly any time you have more than 1 ( stacked up

Guild Navigator16:02:31

dont think so — i mean it compiles 😉

Guild Navigator16:02:25

it prints "UTF-16LE" so i know it works 😉

Guild Navigator16:02:37

ok very nice now:

(def filename "foo.csv")

(def encoding (detect-encoding filename))

;; drop the first 4 lines of the file (we don't need the header info)
(def raw-data
  (clojure.string/join
   (drop 4
         (clojure.string/split-lines
          (slurp filename :encoding encoding)))))

(defn -main
  [& args]
  (p/pprint raw-data))

Michael W15:02:54

Does anyone know of a clojure library that can generate api client code from an openapi spec? I tried the openapi-generator but it's not very nice to try to tweak the output, and in a large spec it's a bit of a nightmare.

Michael W15:02:29

I'll have a look thanks.

Michael W20:02:03

Martian is pretty amazing, I was able to use it to integrate 4 different apis into a new app in record time. Thanks again for the recommendation.

Eugen15:02:36

hi @alexmiller, I checked this https://clojure.atlassian.net/browse/TLOG-28 and it seems that all that it's needed is to have the new slf4j 2.0 API on the classpath. clojure tools-logging uses factory# (org.slf4j.LoggerFactory/getILoggerFactory) which was migrated to use ServiceLoader: https://github.com/qos-ch/slf4j/blob/a5540ad51066b4b15132fdf27ead630519541d35/slf4j-api/src/main/java/org/slf4j/LoggerFactory.java#L107 (I don't have access to comment on the jira issue and I don't know a tools-logging specific channel ). IMO nothing should be done for tools-logging to work with slf4j 2.0. Clients just need to have slf4j 2.0 dependencies on the classpath. I do recommend using 2.0 for dev profile as 1.7x will not be maintained.

Eugen15:02:56

thanks, I just did. I think nothing needs to be done on TLOG-28.

Alex Miller (Clojure team)15:02:41

tools.logging does pull in slf4j 1.7.32 as a test dependency right now, maybe that would need to be changed?

Eugen15:02:07

yes, for future proffing development.

Eugen15:02:14

should I open a PR for that?

Alex Miller (Clojure team)15:02:39

we don't use PRs on contrib projects, but I can put it in my list for tomorrow

Eugen15:02:01

ok, thanks

Alex Miller (Clojure team)15:02:14

I went ahead and just did that now

Alex Miller (Clojure team)15:02:27

not going to release as, no change really

Eugen15:02:47

no problem, does not affect users in any way. only devs

Joel21:02:09

Is it possible to redef a variable “by reference”? Something to the effect of: (def foo 1) (let [x (var foo)] (with-redefs [@x 3] foo)) --> 3

hiredman21:02:42

user=> (def foo 1) (let [x #'foo] (with-redefs-fn  {x 3} (fn [] foo)))
#'user/foo
3
user=>

💯 2
hiredman21:02:13

something to keep in mind, if you are using with-redefs it is usually a mistake

Joel21:02:08

just to enable testing, seems like using protocols as the alternative can get cumbersome quick.

hiredman21:02:15

nah, with-redefs is worse

hiredman21:02:04

with-redefs captures the value of the var then sets the var to a new value, then runs your code, then in a finally sets the value of the var back to the old value

hiredman21:02:00

and that kind of mutating is problematic when combined with things like multithreading and laziness

hiredman21:02:29

not really, I am pretty confident in my opinion on this, having worked on a large clojure code base before with-redefs was added to clojure.core, where we had written our own version of the macro

hiredman21:02:40

that was used extensively for tests in that code base

hiredman21:02:59

and now I pass things as arguments and use protocols sometimes and it is way better

hiredman21:02:01

back in the before time, when all vars where dynamically bindable, people would use that for testing

Joel21:02:09

What I am wanting to do at a high-level is be able to refer to individual vars within my function, but I want to embellish those functions before they are used. I can pass them all in (“embellished”) but that seems tedious to “get a handle” to each. I could pass each one indiv… I don’t know, I just didn’t like where that was headed.

hiredman21:02:23

and it was terrible (even worse problems with multithreading)

Joel21:02:59

I trend toward agreeing with you though honestly, just thought coding was going to get too tedious without it.

Joel21:02:07

I’m sort of exploring so plan on trying at least a couple of tacts. multi-method might work well.

hiredman21:02:40

they might, my big issue with multimethods is extending them is also global, so you can't just create a little stub inside a particular test, like you can using reify for protocols

2
hiredman21:02:42

if you haven't experimented with metadata extended protocols, I would check those out, I find that they get rid of a lot of what people dislike about using protocols

8
dpsutton21:02:21

big plus 1 on the protocols. at metabase we’re very multimethod heavy. i made a protocol that would delegate to the multimethod during regular operation and allows me to pass in easy implementations in testing. huge fan. Previously you’d have to write things like (defmethod ddl/persist-table ::my-fake-driver-that-throws …) and instead can now just spin up whatever reify IPersist that you need as values in the test.

Joel21:02:44

this is almost a footnote in the clojure docs, any better write-up?

Joel21:02:49

although i suppose it’s fairly straightforward.

dpsutton21:02:28

my example: • protocol and an implementation that dispatches to a multimethod based on the db driver type (multimethods are ddl.i/unpersist! and ddl.i/persist!https://github.com/metabase/metabase/blob/master/src/metabase/task/persist_refresh.clj#L49-L66 • tests which assert methods are called, throws when persisting the second table, persist! isn’t called, only unpersist!, etc ◦ https://github.com/metabase/metabase/blob/master/test/metabase/task/persist_refresh_test.clj#L133

2
Max00:02:20

To add a little more color, I would say that the “never use with-redefs” viewpoint is definitely not a consensus in the Clojure community. I’m not saying this to say that the “never with-redefs” people don’t have strong points, but I’ve seen it go badly both ways. Personally I tend to try to avoid with-redefs when at all possible by using DI, but occasionally the overhead isn’t worth it. Multithreaded concerns aren’t usually a big deal in tests since Kaocha doesn’t support multithreaded test runs anyways

💯 2
didibus06:02:12

I'm a big with-redef fan, have had 0 issues with them for over 4 years. But our code bases are made up of many micro-services, so none of them is like some enormous monolith monster, and also we never parallelized our tests.

didibus06:02:08

That said, mocking out a global var seems a bit weird to me. Normally I use with-redef to mock a function. If I'm testing a function that calls another which does IO, I'll with-redef the function that does IO. And in general, I inject all dependencies, and wouldn't access them through a global. So maybe that reduces my blast radius for with-redef.

anovick21:02:38

Could somebody explain to me what exactly is Fulcro library/framework? Is it like Django / Spring / Rails / Next.js ? Clojure on backend + frontend? in the past I've dabbled with Reagent for front-end and saw a lot of options for back-end, but it was always standalone libraries that gave you a lot of freedom to craft your own thing. I've been wondering how polished of a stack can be made in Clojure that can really do a lot of things out of the box or with some minor adjustments. In particular, I'm a big fan of state machines for UI development and noticed that Fulcro supports that kind of thing. Would love to hear what you guys think of it if you've used it! :hugging_face:

seancorfield22:02:21

If no one can help here, there is a #C68M60S4F channel...

seancorfield22:02:22

The Clojure Way ™️ is composition of libraries rather than frameworks...

Eric Dvorsak09:02:16

Fulcro is still a library, it has some batteries included for a lot of things, but it's not quite a fully fledged framework like Django or rails. Fulcro rad is a step in that direction but you are still left with a lot of choices and things to decided and make on your own. If you have some time to grasp the concepts of Fulcro I'd stay it's really worth it. I personally started a Fulcro/pathom3 project and really appreciate the experience. I'd say it's harder to get going than reagent/reframe, because it does much more and there's less doc/examples in the wild, but it also solve more problems.