Fork me on GitHub
#clojure
<
2022-12-02
>
macrobartfast05:12:23

I muddle along into Hiccup Structures Of Unusual Size, the product of converting HTML to Hiccup, I often run into the classic ‘Map literal must contain an even number of forms’ error. This is because the conversion can do odd things. I visually scan the code for the error or grep along for likely suspects which can take a while as the Hiccup can be several hundred lines long or more. Are there any handy tricks to finding out which of the maps is throwing the error? Seeing the point in the code that is throwing an error would be even better (as I sometimes get other kinds of errors). I currently use Emacs/Cider, if it matters.

phronmophobic05:12:09

I'm trying to figure out how you're getting that error. Are you printing the hiccup out and trying to read it back in?

macrobartfast05:12:35

Ah! I should have mentioned that. I’m evaluating a form with Cider whose sole purpose is to return a hiccup form. so…

(defn datatable []
;; a bunch of Hiccup here
)

phronmophobic05:12:03

Can you fix the html->hiccup converter or are you stuck with some hiccup from somewhere?

macrobartfast05:12:48

That would be a good thing to do… I use http://html2hiccup.buttercloud.com and maybe I’ll need to build my own tool at some point. But I’m guessing the problems will be hard to eliminate, as I drop arbitrary HTML I find on the web (for shame!) into it and the input is probably janky at times.

phronmophobic05:12:19

When I get that error, there's a :line and :column number that shows up in the exception's :via

{:type clojure.lang.LispReader$ReaderException
   :message java.lang.RuntimeException: Map literal must contain an even number of forms
   :data #:clojure.error{:line 417, :column 38}
   :at [clojure.lang.LispReader read LispReader.java 314]} 

phronmophobic05:12:31

My guess is that it's producing a keyword or symbol with a space in it (probably a keyword).

macrobartfast05:12:44

omg. did I not read the error message carefully? stand by.

phronmophobic05:12:01

the first issue on that github repo might also be the cause, https://github.com/buttercloud/html2hiccup/issues/1

phronmophobic05:12:24

I might try just try a different converter, maybe https://github.com/green-coder/html-to-hiccup

macrobartfast05:12:56

I’ll try the other converter. You made me realize that I need to look at the converter itself. And reminded me that I should look for Github issues on the tools I use. My error output doesn’t show the line number, I don’t think… just

Show: Project-Only All
  Hide: Clojure Java REPL Tooling Duplicates  (9 frames hidden)

3. Unhandled clojure.lang.ExceptionInfo

2. Caused by clojure.lang.LispReader$ReaderException

1. Caused by java.lang.RuntimeException
   Map literal must contain an even number of forms

                 Util.java:  221  clojure.lang.Util/runtimeException
           LispReader.java: 1357  clojure.lang.LispReader$MapReader/invoke
           LispReader.java:  285  clojure.lang.LispReader/read
           LispReader.java: 1398  clojure.lang.LispReader/readDelimitedList
           LispReader.java: 1347  clojure.lang.LispReader$VectorReader/invoke
           LispReader.java:  285  clojure.lang.LispReader/read
           LispReader.java: 1398  clojure.lang.LispReader/readDelimitedList
           LispReader.java: 1347  clojure.lang.LispReader$VectorReader/invoke
           LispReader.java:  285  clojure.lang.LispReader/read
           LispReader.java: 1398  clojure.lang.LispReader/readDelimitedList
           LispReader.java: 1347  clojure.lang.LispReader$VectorReader/invoke
           LispReader.java:  285  clojure.lang.LispReader/read
           LispReader.java: 1398  clojure.lang.LispReader/readDelimitedList
           LispReader.java: 1347  clojure.lang.LispReader$VectorReader/invoke
           LispReader.java:  285  clojure.lang.LispReader/read
           LispReader.java: 1398  clojure.lang.LispReader/readDelimitedList
           LispReader.java: 1347  clojure.lang.LispReader$VectorReader/invoke
           LispReader.java:  285  clojure.lang.LispReader/read
           LispReader.java: 1398  clojure.lang.LispReader/readDelimitedList
           LispReader.java: 1347  clojure.lang.LispReader$VectorReader/invoke
           LispReader.java:  285  clojure.lang.LispReader/read
           LispReader.java: 1398  clojure.lang.LispReader/readDelimitedList
           LispReader.java: 1347  clojure.lang.LispReader$VectorReader/invoke
           LispReader.java:  285  clojure.lang.LispReader/read
           LispReader.java: 1398  clojure.lang.LispReader/readDelimitedLis
… unless I’m missing it. So maybe something to ask #C0617A8PQ about.

phronmophobic05:12:11

what do you see if you print (:via (Throwable->map *e))

macrobartfast05:12:16

Well, I’ll be darned.

macrobartfast05:12:32

[{:type clojure.lang.ExceptionInfo,
  :data #:clojure.error{:phase :read-source},
  :at
  [clojure.main$repl$read_eval_print__9206$fn__9207
   invoke
   "main.clj"
   433]}
 {:type clojure.lang.LispReader$ReaderException,
  :message
  "java.lang.RuntimeException: Map literal must contain an even number of forms",
  :data #:clojure.error{:line 14, :column 36},
  :at [clojure.lang.LispReader read "LispReader.java" 314]}
 {:type java.lang.RuntimeException,
  :message "Map literal must contain an even number of forms",
  :at [clojure.lang.Util runtimeException "Util.java" 221]}]

macrobartfast05:12:57

Now that is something good to know about.

macrobartfast05:12:18

:star-struck: Now I can roast marshmallows under the glowing stars instead of hunting through maps all night long.

🔥 3
seancorfield05:12:52

CIDER can be less than helpful when dealing with errors 😕

phronmophobic05:12:19

I turned off cider's stacktrace popup thing

phronmophobic05:12:41

I just (pprint *e) when I need to look at the stack trace 🤷

macrobartfast05:12:45

The fact that (:via (Throwable->map *e))did so much suggests that Cider errors could, too… maybe there’s something that can be tweaked for this.

seancorfield05:12:49

I like Portal for this sort of thing, since it's good at displaying data and doesn't try to be "clever" about printing things 🙂

phronmophobic05:12:13

yea, Exceptions contain lots of useful information, but it's not fully utilized by all the available tools yet (I'm not complaining. I'm very grateful for tools like cider).

macrobartfast05:12:47

Too late. I already reported you to the cider people.

😱 1
macrobartfast06:12:27

Better find another career.

macrobartfast06:12:48

I absolutely need to get going with Portal as I’m dealing with huge web data structures now and wading into working with db’s.

seancorfield06:12:04

I go back and forth on using CIDER vs plain nREPL with VS Code, mainly because CIDER messes up stacktraces...

seancorfield06:12:39

...but since I stopped even looking at the REPL window and started using Portal instead, I no longer care what CIDER does to stacktraces 😛 🙂

macrobartfast06:12:58

Do you use Portal in a browser?

seancorfield06:12:27

Portal for VS Code uses a native panel in VS Code which is nicer.

seancorfield06:12:47

(I switched away from Emacs because I wanted a better integrated UI)

macrobartfast06:12:23

I now have VS Code open so that I can do certain things in it and copy the result into Emacs. Emacs is getting suspicious and wants to know where I’ve been when I do that. It’s a bad sign for me and Emacs.

macrobartfast06:12:26

Portal installed! Standalone version via Chrome WPA. That alone was a rather mind opening experience. https://github.com/djblue/portal/blob/master/doc/guides/standalone.md

seancorfield06:12:35

What I like about Portal integrated into VS Code is that I can control Portal navigation and viewing via hot keys without even changing focus to the Portal window.

seancorfield06:12:01

Between Joyride and Calva REPL snippets, it's a very scriptable environment.

macrobartfast06:12:33

Slinking around, whispering this into my ear… on the eve of my 92nd anniversary with Emacs. Looking up Joyride now.

seancorfield06:12:53

Hey, I used Emacs on and off since the 17.x days!

aaron5117:12:21

We made & use https://urbandictionary.github.io/html2hiccup/ which parses with hickory and does some transforms & simplifications

Jacob Emcken07:01:59

Also checkout https://html2hiccup.dev/ which is pretty neat

pez06:12:45

I have an atom which I name with a leading bang, !state, to indicate that it is a ref thing. I name functions that update this atom with trailing bang, so from the call sites it looks like (set-day! [d] ...). I hesitate for functions that deref this. Should I go for a trailing bang there as well, you think? So, something like (p/let [input (input!) ...] ...). I'm making a small template for using #C03DPCLCV9N for #C0GLTDB2T stuff, so would like for it to be reasonably idiomatic.

👀 2
pez06:12:51

The bang makes it look like input! mutates something, doesn't it? Could mitigate that someone by naming it get-input!, but some people tell me to skip that get for getters, and there is still that sense of it mutating something, anyway...

grav07:12:41

Any reason for the getters and setters? Would it make sense to just use swap! and @ from call-sites?

pez08:12:02

It woudln't. 😃 For set-day! quite a few things happen. For this particular getter it could work, but I want this to work for a beginner of Clojure, and not be introducing deref and things.

👍 1
pez08:12:36

I'm getting there... 😃

🙌 1
pez08:12:51

So, the template would look something like so:

(ns aoc2022-1
  (:require [util-aoc :as aoc]))

;; Start by loading `util_aoc.cljs` in the REPL
;; (ctrl+alt+c enter)
;; Then load this file the same way.
;; Then evaluate the top level forms in the
;; Rich comment form `(comment ...)` below

(comment
  (aoc/set-day! 1)
  
  ;; See `util_aoc.cljs` for caveats re this `def`
  (def real-input (aoc/fetched-input))

  (def test-input [1000
                   2000
                   3000
                   nil
                   4000
                   nil
                   5000
                   6000
                   nil
                   7000
                   8000
                   9000
                   nil
                   10000])

  (defn part-1 [input]
    ;; Your solution here
    )
  ...
  (aoc/update-attempt! (part-1 test-input))
  ...
  :rcf)
And it is the naming for (aoc/fetched-input) I want to get right.

Joshua Suskalo15:12:53

One thing I'd maybe question here is for input and other getters they should be passed the data they're getting stuff from, so maybe just raise the deref to the calling code if it's not too much trouble? That would make obvious where it's being dereferenced, plus allow you to reduce race conditions by allowing you to do a single deref and use the data for multiple things.

Nundrum16:12:52

I'm trying to scrape some data from my pfsense router using clj-http. The router is protected by CSRF. Thus far I can perform a login (using CSRF) without a problem, and then fetch pages from the router. But some of the data requires a POST first, and those are failing with a 403. More in thread.

Nundrum16:12:46

The only thing I can see from the browser that might cause this is that the crsf field is sent twice

Nundrum17:12:39

I have the token in a var named crsf and tried passing it like this:

:form-params {:__csrf_magic [csrf csrf] .....}
That doesn't work. Is there some other way to send that form field twice? Naturally I can't put the key into the map twice.

hiredman17:12:25

Are you sure that what the browser is showing you is form encoded?

hiredman17:12:49

I would check the content type header

lukasz17:12:45

Are these supposed to be form fields or headers? Also you can control how array fields are serialized: https://github.com/dakrone/clj-http#query-string-parameters

p-himik17:12:33

In any case, different form data fields can have the same name. You can't accurately model it with just a map, but there should be a way to create a proper form data object that your HTTP client understands. E.g. JS itself has js/FormData. No clue about clj-http though.

hiredman17:12:29

sure, and I don't know, but my expectation is a browser would show it as multiple fields with the same name, not the same field with multiple values

hiredman17:12:53

that looks vaguely like a pretty way a browser would display json

hiredman17:12:13

a top level def also seems like a bad place for a crsf token, because I would expect that to change with each request

hiredman17:12:40

I guess it does say "Form data" at the top, so maybe I am completely off base there

hiredman17:12:29

if you search for Multipart in the clj-http readme there are some examples that will be useful

Nundrum17:12:43

The content-type is application/x-www-form-urlencoded; charset=UTF-8 and the raw data is __csrf_magic=sid:709a278c6a9cdafe00e2d91a691c00546629a759,1525927520&__csrf_magic=sid%3A709a278c6a9cdafe00e2d91a691c00546629a759%2C1525927520&viewtype=default&filter=&sorttype=bytes&states=100&getactivity=yes

Nundrum20:12:25

Ah ha! Mucking with curl showed the CSRF token was changing. Firefox wasn't showing that, or somehow I was very confused by what it was showing me. Anyway, after getting clj-http to follow redirects on a POST, I was able to get the 2nd token. Let there be data 😄 Thanks everyone!

🎉 1
Nundrum18:12:47

Is there a convenient way to render a captured HTML page as text? Like w3m --dump would do?

p-himik18:12:35

How is it relevant to #clojure though? Are you looking for a Clojure library?

Nundrum19:12:28

Yeah, I'm trying to do it from within Clojure. Using clj-http to capture the page. Mainly for debugging purposes. Looking at a wall of HTML isn't fun. Seeing a partially rendered version would be helpful.

Nundrum19:12:51

Looking at a bunch of Hiccup is a little better, but not much.

p-himik19:12:42

If it's just for a one-off debugging, I'd use w3m --dump. :) Otherwise, I'd use some Java library, like jsoup.

elliot21:12:39

I’ve been using jsoup for parsing

elliot21:12:44

and a library called jericho can do some rendering

elliot21:12:01

; up in the ns
(:import net.htmlparser.jericho.Source
           java.io.StringReader
           net.htmlparser.jericho.LoggerProvider) 


(defn jericho [html cols]
  (let [r (-> html
              (StringReader.))
        src (Source. r)
        _ (.setLogger src (.getLogger LoggerProvider/DISABLED "foo"))
        r (-> src
              (.getRenderer)
              (.setMaxLineLength cols)
              (.setIncludeHyperlinkURLs false)
              (.toString))]
    (.toString r)))

elliot21:12:35

net.htmlparser.jericho/jericho-html {:mvn/version "3.4"}

phill00:12:09

enlive parses HTML (with JSoup or another option) and emits a data structure like clojure.xml's. Then you can use clojure.core/xml-seq and filter by text node. This is a 92% solution - but you can improve it, if necessary, by using clojure.zip/xml-zip.

🎉 1
Omar Bassam20:12:17

I started doing Advent of Code in Clojure and I am using deps.edn . I was wondering if there's a way to programatically generate aliases in the deps.edn file. I tried the following the following clj -M:day01 in my terminal using the following deps.edn file:

{:paths ["src" "resources"]
 :deps {}
 :aliases
  (reduce (fn [acc i]
            (assoc acc
                   (keyword (str "day" (when (< i 10) 0) i))
                   {:main-opts ["-m" (str "aoc2022.day" (when (< i 10) 0) i)]}))
          {} (range 1 31))}
but I get the following error that I don't quite understand for some reason:
Error building classpath. contains? not supported on type: clojure.lang.PersistentList
java.lang.IllegalArgumentException: contains? not supported on type: clojure.lang.PersistentList
	at clojure.lang.RT.contains(RT.java:853)
	at clojure.core$contains_QMARK_.invokeStatic(core.clj:1506)
	at clojure.tools.deps.alpha.script.make_classpath2$check_aliases$fn__1615.invoke(make_classpath2.clj:61)
	at clojure.core$complement$fn__5737.invoke(core.clj:1455)
	at clojure.core$filter$fn__5962.invoke(core.clj:2838)
	at clojure.lang.LazySeq.sval(LazySeq.java:42)
	at clojure.lang.LazySeq.seq(LazySeq.java:51)
	at clojure.lang.RT.seq(RT.java:535)
	at clojure.core$seq__5467.invokeStatic(core.clj:139)
	at clojure.tools.deps.alpha.script.make_classpath2$check_aliases.invokeStatic(make_classpath2.clj:58)
	at clojure.tools.deps.alpha.script.make_classpath2$run_core.invokeStatic(make_classpath2.clj:114)
	at clojure.tools.deps.alpha.script.make_classpath2$run.invokeStatic(make_classpath2.clj:174)
	at clojure.tools.deps.alpha.script.make_classpath2$_main.invokeStatic(make_classpath2.clj:228)
	at clojure.tools.deps.alpha.script.make_classpath2$_main.doInvoke(make_classpath2.clj:195)
	at clojure.lang.RestFn.applyTo(RestFn.java:137)
	at clojure.lang.Var.applyTo(Var.java:705)
	at clojure.core$apply.invokeStatic(core.clj:667)
	at clojure.main$main_opt.invokeStatic(main.clj:514)
	at clojure.main$main_opt.invoke(main.clj:510)
	at clojure.main$main.invokeStatic(main.clj:664)
	at clojure.main$main.doInvoke(main.clj:616)
	at clojure.lang.RestFn.applyTo(RestFn.java:137)
	at clojure.lang.Var.applyTo(Var.java:705)
	at clojure.main.main(main.java:40)
Is there's other ways to actually generate aliases programatically in the deps.edn file. I would like to have the alias :day01 for example with the :main-opts as follows ["-m" "aoc2022.day01"] and similarly for the next 30 days for Advent of Code.

p-himik20:12:59

Instead of using one alias per main namespace, I'd use a single alias with a single entry point and make it accept one argument, which would be the day.

1
p-himik20:12:36

Your deps.edn doesn't work because EDN is not code - it's data. In your case, :aliases ends up being a list where the first item is a symbol reduce, the second item is another list with fn, and so on.

Omar Bassam20:12:00

Yeah thank you for the Explanation. I guess I get it now I'll try your suggestion and see how it goes.

👍 1
Omar Bassam21:12:37

I've added the following code to a single entry point in a my core namespace that is bound to my :dev alias and i now simply run cllj -M:dev 1 in the terminal and it works like a Charm.

(defn -main
  ""
  [day-str]
  (let [day (parse-long day-str)
        main-fn (read-string (str "aoc2022.day" (when (< day 10) 0) day "/-main"))]
    (println "AOC 2022 Day " day)
    (require (symbol (str "aoc2022.day" (when (< day 10) 0) day)))
    ((resolve main-fn))))

🎉 1
p-himik21:12:11

You can also use requiring-resolve. And instead of using str+`when`, you can use format:

user=> (format "aoc2022.day%02d/-main" 2)
"aoc2022.day02/-main"

p-himik21:12:02

%02d means "take the argument, treat it as an integer, pad it to the length of two, with zeroes on the left side".

Omar Bassam21:12:48

interesting. I didn't know that Clojure had a Common Lisp like format. I did replace the when part as you suggested. However, I don't understand how to use the requiring-resolve function. is it instead of the require function or instead of the resolve?

p-himik21:12:48

That's Java's format though. But there's also cl-format, yes. requiring-resolve is used instead of require+`resolve`.

Omar Bassam21:12:59

Yeah. I got it to work now. the function is only llike 2 lines.

(defn -main [day-str]
  (let [day (parse-long day-str)]
    (println "AOC 2022 Day " day)
    ((requiring-resolve (symbol (format "aoc2022.day%02d/-main" day))))))

Omar Bassam21:12:41

Thank you @U2FRKM4TW for your patience.🙏

👍 1
Mathieu Pasquet21:12:48

I know it's possible for macros to execute functions (or they'd be pretty useless...), but is it possible to pass a function to a macro? For example

(defmacro eg [fn] (fn 1))
If I try run that (`(eg inc)`), I just get nil and I don't understand why.

p-himik21:12:14

fn is bound to a symbol there since symbols passed to a macro aren't resolved. You gotta resolve it before calling it.

phronmophobic21:12:42

I tend to use the following pattern when writing macros since it makes them easier to test, but I think it also helps show what's happening in this case:

(defn eg* [fn]
  (fn 1))
(defmacro eg [fn]
  (eg* fn))

(eg* 'inc) ;; nil

phronmophobic21:12:49

calling the eg macro will pass the unevaluated forms as the arguments

p-himik21:12:55

Shouldn't the macro return a quoted list instead of calling the function?

phronmophobic21:12:37

right, I think that's the intention, but just trying to show what's happening

phronmophobic21:12:17

(eg* 'inc) will call ('inc 1) which evaluates nil

hiredman21:12:37

it is possible to pass a function to macro, but it is almost never something you actually want to do

hiredman21:12:41

https://gist.github.com/hiredman/b2096b04927a9c9db480c39806dac297 for example builds a function object that is the continuation of expanding the macro and passes it back to itself