Fork me on GitHub
#data-science
<
2024-03-25
>
wcohen17:03:05

Hi all --very basic Clay question (copying the organization of a basic scicloj notebook to kick the tires on it, with a dev ns that generates an HTML): • My namespace code appears, but no code below it is either shown or evaluated (either defs of local things to input or analyze, or defns of analysis functions). The only stuff below the ns declaration are some random commented out code lines for WIP functions. • I've got clay, kind, and kindly all in my require. • I did cargo cult copy the following just to see if this enabled making the code appear, to no avail.

^:kindly/hide-code
(def md (comp kindly/hide-code kind/md)
Is there somewhere else that's now a common VSCode/Calva default that would normally hide stuff? I haven't otherwise set any kindly advice anywhere. My goal is to see all my definitions in a notebook and work interactively with it

Daniel Slutsky17:03:55

Hi. Do you see any error at the REPL?

Daniel Slutsky17:03:06

Would you like to share your deps.edn, clay.edn (if any), and namespace? And what actions do you do in VSCode to render the page?

wcohen17:03:18

clay.edn is nonexistent in a clay.clj I run

(clay/make! {:format [:html]
             :base-source-path "src/boston/"
             :source-path ["parcel_analysis.clj"]
             :base-target-path "docs"
             :clean-up-target-dir true})
and my deps is still pretty boring and from clj-new, basically:
{:paths ["src" "resources"]
 :deps {org.clojure/clojure {:mvn/version "1.12.0-alpha5"}
        org.clojure/data.csv {:mvn/version "1.1.0"}
        ovid/ovid {:local/root "../ovid"}
        Factual/geo {:local/root "../geo"}
        aurelius/aurelius {:local/root "../aurelius"}
        org.locationtech.jts.io/jts-io-common {:mvn/version "1.20.0-SNAPSHOT"}
        org.locationtech.jts/jts-core {:mvn/version "1.20.0-SNAPSHOT"}
        org.wololo/jts2geojson {:mvn/version "0.18.1"}
        scicloj/tablecloth {:mvn/version "7.021"}
        org.scicloj/clay {:mvn/version "2-alpha87"}
        org.babashka/http-client {:mvn/version "0.3.11"}}
 :aliases
 {:run-m {:main-opts ["-m" "foo.parcel-analysis"]}
  :run-x {:ns-default foo.parcel-analysis
          :exec-fn greet
          :exec-args {:name "Clojure"}}
  :build {:deps {io.github.clojure/tools.build
                 {:mvn/version "0.9.6"}} 
          :ns-default build}
  :test {:extra-paths ["test"]
         :extra-deps {org.clojure/test.check {:mvn/version "1.1.1"}
                      io.github.cognitect-labs/test-runner
                      {:git/tag "v0.5.1" :git/sha "dfb30dd"}}}}
 :mvn/repos {"jts-snapshots" {:url ""
                              :releases {:enabled false}
                              :snapshots {:enabled true
                                          :update :always
                                          :checksum :warn}}
             "releases" {:url ""},
             "snapshots" {:url ""}}}
the ns def:
(ns foo.parcel-analysis
  (:require [clojure.string :as clojure.string]
            [clojure.tools.logging :as log]
            [ovid.io :as oio]
            [geo.crs :as gcrs]
            [geo.jts :as gjts]
            [geo.io :as gio]
            [ovid.feature :as of]
            [aurelius.jts :as ajts]
            [tech.v3.dataset.io.csv :as csv]
            [tech.v3.dataset :as ds]
            [portal.api :as p]
            [clojure.java.io :as io]
            [tablecloth.api :as tc]
            [babashka.http-client :as http]
            [charred.api :as charred]
            [charred.bulk :as bulk]
            [scicloj.kindly.v4.kind :as kind]
            [scicloj.kindly.v4.api :as kindly])
  (:import (org.locationtech.jts.geom GeometryFactory
                                      Geometry
                                      LineString
                                      PrecisionModel)
           (org.locationtech.jts.algorithm MinimumAreaRectangle MinimumDiameter)
           (org.locationtech.jts.operation.linemerge LineMerger)
           (org.locationtech.jts.operation.buffer BufferParameters
                                                  OffsetCurveBuilder)))

wcohen17:03:47

all just a bunch of GIS kitchen sink stuff for now as I get to playing around

Daniel Slutsky17:03:08

Thanks! That is very helpful. I'll try to reproduce in my environment.

wcohen17:03:30

I'm sorry I'm dumping a bunch of junk at you

Daniel Slutsky18:03:01

It is great help to get real-world examples, especially those which break unexpectedly.

wcohen18:03:22

The only thing I can currently imagine is that many of these def'd things are geojsons weighing hundreds of MB -- there's nothing specific to display, but maybe it's crossing some kind of threshold

👍 1
wcohen18:03:01

(also as you can see I'm basically trying to wean myself off of the previous transducer based stuff and work toward using TMD for gis stuff, as discussed in zulip)

🆒 1
Daniel Slutsky18:03:12

(I'm renaming the namespace foo.parcel-analysis to boston.parcel-analysis, assuming it is intended to match the file path.)

wcohen18:03:29

yep, sorry

wcohen18:03:32

was trying to keep things generic

wcohen18:03:35

missed one

wcohen18:03:47

but you know where i live!

🙃 1
Daniel Slutsky18:03:47

(and I'll use the Clojars versions of Ovid and Geo, assuming this is not a relevant difference)

wcohen18:03:54

totally irrelevant

wcohen18:03:35

you might have to use a github sha, i never posted ovid to clojars

wcohen18:03:43

or maybe i did but it's vestigial

wcohen18:03:17

it's all going to be irrelevant if i'm putting a tmd dataset as a base case, so i'm going to need a new base gis library

Daniel Slutsky18:03:20

Ok, so a page with the ns definition is rendered.

Daniel Slutsky18:03:37

Is it what you see too?

wcohen18:03:51

hold on -- making a new empty ns to try to diagnose alongside

wcohen18:03:21

oh my goodness. it's totally my fault. i'm so used to having to put all my pieces of info as defs to avoid massive repl output that i'm realizing that i wasn't actually sticking anything to stdout

wcohen18:03:22

i'm so sorry.

wcohen18:03:07

oh wait, no that's not the problem. let me simplify:

👍 1
wcohen18:03:47

(ns boston.parcel-analysis
  (:require [clojure.string :as clojure.string]
            [clojure.tools.logging :as log]
            [ :as oio]
            [geo.crs :as gcrs]
            [geo.jts :as gjts]
            [ :as gio]
            [ovid.feature :as of]
            [aurelius.jts :as ajts]
            [tech.v3.dataset.io.csv :as csv]
            [tech.v3.dataset :as ds]
            [ :as io]
            [tablecloth.api :as tc]
            [babashka.http-client :as http]
            [charred.api :as charred]
            [charred.bulk :as bulk]
            [scicloj.kindly.v4.kind :as kind]
            [scicloj.kindly.v4.api :as kindly])
  (:import (org.locationtech.jts.geom GeometryFactory
                                      Geometry
                                      LineString
                                      PrecisionModel)
           (org.locationtech.jts.algorithm MinimumAreaRectangle MinimumDiameter)
           (org.locationtech.jts.operation.linemerge LineMerger)
           (org.locationtech.jts.operation.buffer BufferParameters
                                                  OffsetCurveBuilder)))

(def t (gcrs/create-transform 4326 2249))

(+ 1 1)
with (added quarto in just to be safe)
(clay/make! {:format [:quarto :html]
             :base-source-path "src/boston/"
             :source-path ["parcel_analysis.clj"]
             :base-target-path "docs"
             :clean-up-target-dir true})
yields

wcohen18:03:04

(sorry, as you can see i made a parcel_analysis2.clj to avoid deleting my other junk but that shouldn't matter)

wcohen18:03:17

deps have nothing to do with it

wcohen18:03:22

it's a vs code or windows or something problem

Daniel Slutsky18:03:35

I get the following:

Daniel Slutsky18:03:47

which is probably what you hoped to see.

wcohen18:03:05

are you emacs, intellij, or calva?

wcohen18:03:15

i guess that shouldn't matter

Daniel Slutsky18:03:19

Emacs, but it should not matter.

Daniel Slutsky18:03:34

I haven't tested it in Windows so much.

wcohen18:03:41

right, sorry -- was also poking around with portal so i forget what's dev environment dependent

Daniel Slutsky18:03:43

That could make a difference

wcohen18:03:50

lovely. will check mac tonight

wcohen18:03:58

sorry for the bother!

Daniel Slutsky18:03:13

This is amazing help @U2B2YS5V0!

Daniel Slutsky18:03:32

Any user surprise is the best insight I could hope for.

Daniel Slutsky18:03:26

You are encouraging me to test it on Windows, hopefully later this week. I was not so careful in making file management generic enough, etc.

Daniel Slutsky18:03:01

Do you use https://learn.microsoft.com/en-us/windows/wsl/about? I think I know another person with WSL who seems to be fine.

wcohen18:03:11

no wsl unfortunately. it's a new world for me too

wcohen18:03:32

i haven't had to do windows only in like a decade. the more i think about it it's GOTTA be that

Daniel Slutsky18:03:05

It should be solvable, but probably not today.

wcohen18:03:26

no hurry at all. i can just do it on my mac for now and poke around on my windows installation as time allows

Daniel Slutsky18:03:53

(leaving the keyboard for a while)

wcohen18:03:09

thank YOU! i know it's way late for you

Daniel Slutsky18:03:37

perfect timing actually 🙏

wcohen20:03:38

Update -- also doesn't work on mac. I thought maybe it was local dependencies but removing those didn't seem to matter. Back to thinking it's something about a kindly advice setting hiding all the code.

wcohen20:03:02

(At first I was thinking I could just switch to clerk in the mean time, but that idea of a leaflet map is just so enticing...)

Daniel Slutsky20:03:05

Do you see any errors in the browser console?

wcohen20:03:19

is there a place i can override the hide settings to like, an empty set?

wcohen20:03:41

ie :kindly/options {:kinds-that-hide-code #{:kind/md :kind/hiccup}}

wcohen20:03:33

ah -- found it in config

wcohen20:03:38

(didn't seem to do anything)

Daniel Slutsky20:03:49

:thinking_face:

wcohen20:03:19

will keep poking. what NS actually generates this stuff in clay?

wcohen20:03:06

(also -- not a big deal, but just as a note, the pom.xml for clay has https://github.com/org.scicloj/clay as the url, which means that the link from clojars to the repo is broken)

Daniel Slutsky20:03:21

The 'make' ns is the main entry point. I suggest I'll share a minimal example with you, and we start debugging from there.

wcohen20:03:52

again, no hurry!! i'll just keep plodding along with clerk re work output for now

Daniel Slutsky20:03:30

mkdir test1

cd test1

mkdir notebooks

cat > notebooks/scratch.clj
(ns scratch)
(+ 1 2)
^D

clj -Sdeps '{:deps {org.scicloj/clay {:mvn/version "2-alpha87"}}}'

(require '[scicloj.clay.v2.api :as clay])

(clay/make! {:source-path "notebooks/scratch.clj"})

Daniel Slutsky20:03:38

Could you try this?

wcohen20:03:41

it works! okay. i'll build up from this and see where i get stuck

Daniel Slutsky20:03:25

nice, let us think what might be the difference. maybe the next step could be something similar to the code we tried yesterday, but from the shell, without any editor involved.

Daniel Slutsky21:03:53

If you use VSCode Calva, when you start the REPL (Jack In), the corresponding Clojure command appears in the Terminal window. In my environment, it looks like:

pushd /path/to/your/project ; clojure -Sdeps '{:deps {nrepl/nrepl {:mvn/version,"1.0.0"},cider/cider-nrepl {:mvn/version,"0.28.5"}}}' -M -m nrepl.cmdline --middleware "[cider.nrepl/cider-middleware]" ; popd
Maybe we'll learn something from what we see in your Calva environment. In Emacs CIDER, there is something similar in the *Messages* buffer. In my environment, it looks like:
[nREPL] Starting server via /usr/local/bin/clojure  -J-XX:-OmitStackTraceInFastThrow -Sdeps '{:deps {nrepl/nrepl {:mvn/version "0.9.0"} cider/cider-nrepl {:mvn/version "0.28.5"} refactor-nrepl/refactor-nrepl {:mvn/version "3.5.2"}} :aliases {:cider/nrepl {:main-opts ["-m" "nrepl.cmdline" "--middleware" "[refactor-nrepl.middleware/wrap-refactor,cider.nrepl/cider-middleware]"]}}}' -M:cider/nrepl

wcohen15:04:55

okey dokey. i still have no idea why the old clj-new project i shoved clay into wasn't working, but i successfully just made a smaller clay-only project with a notebook alongside, and am just shifting my data analysis in there. i feel like it's something about the project setup or namespace organization or something, rather than having a "notebook" where i separately work through things. though i have no evidence to back that up.

wcohen15:04:20

but it's working at least. if it breaks at some point i will let you know what conclusively did it. it works in vs code / emacs too

Daniel Slutsky16:04:05

many thanks for these efforts, @U2B2YS5V0.