Hello, I'm new to boot. Is there a way to start a repl in a script's environment?

i usually boot repl in the script's directory, and then (load-file "script.boot") at the repl

alternatively, in the script, you can call (boot (repl))

Thanks, I'll try those

Nice, that was easy.

Oh man, I've always wanted to simply add a library to a running lein repl session. I need to gush about how easy it was for myself as a boot noob to do just that! 😄!

that parrot emoji is incredible

Yeah it's a good one. This might not be the right place for this kind of question, but: I'm trying to write a script that uses enlive to make some content pages. I went for boot since it's more of a one-off script, and simple so I can learn boot. I'm having an issue with enlive though. my 'project directory' has a directory with 2 files:

-rw-r--r--  1 escherize  staff  219 Dec 20 15:16 page.html
-rwxr-xr-x  1 escherize  staff  278 Dec 20 15:31 sausage-maker

@escherize: what are your :source-paths and :resource-paths?

oh boy. probably should have a look at those. 🙉

i was asking because you have multiple files named "page.html"

can you do boot show -f please?

that will show you a tree of the files in the fileset

so you can see how your source and resource paths are interpreted and merged

I see.

boot show -f
Implicit target dir is deprecated, please use the target task instead.
Set BOOT_EMIT_TARGET=no to disable implicit target dir.

BOOT_EMIT_TARGET=no boot show -f

oh i forgot, you're not making a build script, you're making a shebang script?

that's right

oh sorry, disregard that

np

in your case there's no fileset involved

Is there something around core/add-resource that I should do

can you paste your script?

sure

#!/usr/bin/env boot

(set-env! :dependencies '[[enlive "1.1.6"]
                          [markdown-clj "0.9.82"]])
(require '[net.cgrand.enlive-html :as html])
(require '[markdown.core :as md])

(html/deftemplate main-template "page.html" [])

(defn -main [& args]
  (main-template))

i'm not so familiar with enlive, it's been a while since i used it

Yeah, that's my issue too.

it uses some weird classpath loading

what is supposed to happen?

which works magically with lein :<

this version would just read in page.html

ah ok

ah i think i know

can you paste the output of tree . please?

sure:

$ tree .
.
├── fake_page.html
├── page.html
├── sausage-maker
└── target

and your script is sausage-maker?

also lol

yah haha

i see maybe the problem

i think lein adds src to the classpath by default

a better tree for you is:

tree .
.
├── page.html
└── sausage-maker

ok graet

$ ./sausage-maker
    clojure.lang.ExceptionInfo: HTML resource not found.
    data: {:file "boot.user6845050520989036543.clj", :line 17}
java.lang.NullPointerException: HTML resource not found.
           net.cgrand.tagsoup/parser                       tagsoup.clj:  30
   net.cgrand.enlive-html/eval205/fn                   enlive_html.clj:  89
                                 ...
net.cgrand.enlive-html/html-resource                   enlive_html.clj:  69
net.cgrand.enlive-html/html-resource                   enlive_html.clj:  72
                        boot.user/fn  boot.user6845050520989036543.clj:  17
                                 ...
                  boot.main/-main/fn                          main.clj: 177
                     boot.main/-main                          main.clj: 177
                                 ...
                    boot.App.runBoot                          App.java: 390
                       boot.App.main                          App.java: 467
                                 ...
                    boot.Loader.main                       Loader.java: 253

so you can add the cwd to the classpath

i think that's what you want to do

ah cool, i think so too

one sec

(set-env! :directories #(conj % "."))

if you put that before the deftemplate, does that help?

yeah!!

Thanks Micha!

sweet!

where can i learn more about set-env!

https://github.com/boot-clj/boot/wiki/Boot-Environment

verywell. Thanks again

time to make some sausage hah

haha ceom by if you run into any issues

*come by

love the parrot

cops should have that on top of their cars

i'd have to pull over, i'd be laughing too hard to get away

@micha: You suggested appending "." to the Boot :directories for @escherize but would specifying :source-paths as #{"."} have worked as well?

@seancorfield: probably, but wouldn't have been appropriate since a fileset wasn't needed

we used to have an option for adding CWD as a source path, but realized people's homedirs can be pretty huge/deep... and it would lock up instantly

Got it. So the fileset aspect is the decider there, thank you.

right, no problemo

A question came up on boot-expectations (from mimmo costanza) that was essentially what's the difference between user-dirs and input-dirs, as a consideration for a task?

It seemed to me that input-dirs could be a larger set, including folders that a previous task has written to?

there are two "roles" that files in a build generally have

input or output

or both, or neither

So you could have a task that generated tests and you'd want input-dirs to pick up those, rather than user-dirs since the generated tests are user-supplied folders?

input are files that are potentially source files for some later task to operate on, eg. compile them, process them, etc

yes exactly as you say

output files are files that can potentially be included in the "final artifact"

On the filesets page, I was surprised that assets are not considered inputs...

eg. packaged in a jar, emitted to the target dir, etc

well there are 4 permutations of #{input, output}

So if you want to preprocess a file to generate CSS (for example), that file has to be a source, not an asset?

(and the generated CSS file is an asset)

well the original source file must be source or resourse

*resource

because those are the ones with the "input" flag set

when you emit the css file, i would emit as "resource"

because that has both input and output flags set

so it can be input for some later task, but it will also be included in a jar or a war file

or spit into the target dir

asset paths contain files that will never be the input of any task

but will be included in the output of the pipeline

i've never found a good use for it myself

The table of input/output is fairly clear -- and input-dirs follows on from that -- but the I don't think that page is all that clear on user-dirs...

but it was one of the permutations so for completeness it's there

ah, the user-dirs contain files that were created by the user

as opposed to files that were created by tasks

those could be input files or output files

For web apps I can see CSS and images being assets rather than resources... but I can see the value of them also being available to later tasks.

boot needs to keep track of such things though, for various reasons

but you probably don't need to know or care about user-dirs

OK, so using input-dirs in a task is normally the right thing to do?

yeah the :asset-paths are sometimes maybe helpful if you have thousands of images

yeah if you're querying over the fileset to get files i would normally use input-files or output-files

because then you can filter them, etc

usually your task is interested in a subset of all input or output files

like ones matching *.css perhaps

If you output the fileset, you get series of "random" temp paths into the cache: is there an easy way to find out what real files / folders those refer to (in the user space)? Clearly Boot knows the mapping...

...mostly just curiosity on my part

like how do you mean the "real" files?

like the paths in boot's temporary directories?

or the original files you have in your project?

one thing that's a little strange is that boot doesn't operate on your files

The mappings between real user folders and the temporary directories (where those exist).

it doesn't ever see the files you edit in your editor

Right

boot copies those into temp dirs

so the fileset contains TmpFile records

those have a :file key

you can use boot.core/tmp-file to access it

that gives you a java.io.File representing the path to the actual file in the tempdir

for example here: https://github.com/boot-clj/boot/blob/master/boot/core/src/boot/task/built_in.clj#L620-L622

I'll try asking a different way...

that gets the java.io.File objects for all the pom.xml files in the fileset that are input files

Let's say we start with user dirs A, B, and C, and we tell Boot that A and B are resources, and C is target. Boot creates temporary directories for A, B, and C, yes? And copies (links) the original content into those temporary directories right?

ah, i see now

not exactly

the fileset has a directory for each of the types, source/resource/asset

it has one set of these for user file, and one set for non-user files

when you set :source-paths or :resource-paths etc

that attaches a filesystem event watcher to the dirs in those sets

so the dirs you specify in :resource-paths are all attached to filesystem watchers

the watchers copy files into the fileset user-dir for resource

asynchronously

So... the watchers know the mapping

yep, it's set up here:

https://github.com/boot-clj/boot/blob/master/boot/core/src/boot/core.clj#L576-L582

it's a little convoluted because it's all stateful code

i guess more than a little

If I get to the point where I think I have a really good handle on it, I may suggest some edits to the filesets wiki page to clarify the relationships (since reading that caused me to ask these questions).

thanks! feel free to edit at will

working on making a lib boot'iful, it does not seem I can have overlapping :source-paths. I do have a set of cljc tests that use two different apps for testing: one under clj, another one under cljs:

(set-env! :source-paths #(conj % "test" "test/clj")) boot says: (!) :source-paths must not overlap. https://github.com/boot-clj/boot/blob/6afed5f095867870c1be9b34163aa6f4ac80a225/boot/core/src/boot/core.clj#L617

I can move clj and cljs apps that are used for testing outside of test, but wanted to see if I am missing something (in my first couple hours of boot)

@tolitius: when you add a path everything under that path is also added

including sub directories

so :source-paths #{“test”} already includes test/clj and test/cljs

@flyboarder: yep, thanks. I do understand that, but the tests require artifacts from test apps as (require [tapp.foo.bar]), and making them do (require [clj.tapp.foo.bar]) / (require [cljs.tapp.foo.bar]) does not seem clean. converting "it" from lein. there I add both "test" and ("test/clj" or "test/cljs") depending on the platform

and there are other things in test which you need i presume?

Howdy, I'm back with my sausage-makering activities (using a boot script). Tried running it on a fresh boot install (not my machine) and getting:

clojure.lang.ExceptionInfo: Could not locate markdown/common__init.class or markdown/common.clj on classpath:
    data: {:file "markdown/core.clj", :line 1}
java.io.FileNotFoundException: Could not locate markdown/common__init.class or markdown/common.clj on classpath:

#!/usr/bin/env boot

(set-env! :dependencies '[[enlive "1.1.6"] [markdown-clj "0.9.82"]])
(set-env! :directories #(conj % "."))
(set-env! :directories #(conj % "./content-sites"))
(require '[net.cgrand.enlive-html :as html])
(require '[markdown.core :as md])
(require '[clojure.string :as str])
(use '[])

(println (md/md-to-html-string "# Hi."))

nevermind, I blew away ~/.boot on the offending machine and we're good to go

thanks boot for powering our content marketing strats @ http://fetchh.io

@micha testing the above I noticed the "Implicit target dir is deprecated, ..." warning is printed even when the target task is used — is that expected?

@martinklepsch: Yes, you need to set the environment variable.

export BOOT_EMIT_TARGET=no

2.5.2 is released, boot -u to update

https://github.com/boot-clj/boot/blob/master/CHANGES.md#252

@micha: you rock

likewise!

@magomimmo: I'm reading your comment on boot-expectations but when I've used adzerk-oss/boot-test (on which boot-expectations is based in many ways), I didn't see the "tests all loaded namespaces" problem, as I recall: it only tested namespaces in my source code.

I don't know whether that was due to my project setup or something else.

@seancorfield: take a look at https://github.com/magomimmo/modern-cljs/blob/master/doc/second-edition/tutorial-16.md#test-namespaces

Yes, I read that.

@seancorfield: try this as well: https://github.com/crisptrutski/boot-cljs-test/issues/23 and this https://github.com/adzerk-oss/boot-test/issues/18

@seancorfield: I used reader conditions…but I don’t think this causes the problem

Sounds to me like it's related to cljs and cljc files, from reading those issues.

@seancorfield: at least not for the all-ns*

@seancorfield: and what about duplication?

@seancorfield: I’m going to eat...

I don't work with cljs and I don't use cljc (because of that) so I haven't run into any of those problems so far.

I also don't experience the "all namespaces" problem shown in the Jay Fields article you linked to (which was from 2010 -- so maybe that's not even an issue any more?).

`boot.user=> (doc clojure.test/run-all-tests) ------------------------- clojure.test/run-all-tests ([] [re]) Runs all tests in all namespaces; prints results. Optional argument is a regular expression; only namespaces with names matching the regular expression (with re-matches) will be tested. nil boot.user=>`

Just ran boot-test against a small (pure clj) project here with several dependencies and the only namespaces it tries to test are the two that are actually in my source code. So I don't see the behavior that Jay Fields showed in his article.

https://github.com/clojure/clojure/blob/a1eb4062e3ade2478d4b57aff06c29604d9c7a46/src/clj/clojure/test.clj#L779

I know what the source code does.

I'm just saying that in real world projects, I'm not seeing the behavior you claim. If you can create a small, self-contained, test repo that demonstrates a problem with boot-expectations, I'd be perfectly happy to look at the issue.

so there is something that I do not understand too. what does return to you (all-ns)

I don't see the problem with clojure.test in the projects we use at work (with lein test right now, but I'm not seeing it on test projects with boot test either), and I don't see the problem with boot expectations on a large project at work: I instrumented boot-expectations to print all the input namespaces it is requiring and don't see it trying to require extra namespaces and I don't see it trying to test extra namespaces.

I go to see what lein test does

Instead of reading the source, you could try to create a small, self-contained test projects that shows the problem...

I never had the same problem with lein test too

the jaycfield sample is the shorter sample. you can use it at the repl

I'm not very interested in doing something at the REPL since we're talking about how boot test and boot expectations work, not how the underlying code works

Both boot test and boot expectations support include / exclude namespaces by regex (so this all seems like a moot point anyway).

hm there are PRs for boot-test i didn't see

@magomimmo: this is the PR that fixes one of the issues, right?

oops i mean https://github.com/adzerk-oss/boot-test/pull/19/files

yes, distinct should resolve the duplication issue

ok i'll push a release

great!

wow, you did in a nano second….

thanks so much

you tried 1.0.7 already?

@seancorfield @misha contrary to the comment above, boot test only supports include / exclude by namespaces, unless I’ve missed something

whoops, @micha

but regex sounds darn rootin tootin’

thanks for boot expectations @seancorfield, very nice job there

there is the --filters option btw

I cannot claim credit for most of boot-expectations

you can make a predicate to determine which namespaces are included

that runs a bit later though, after loading the vars

ah, i see

If distinct solves the problem with duplicates, I can add that easily to boot-expectations too. I'll open an issue to remind me for when I'm on my desktop system.

i’m still not sure if there was user error leading to those duplicates

distinct is a good safety net though

@micha: give me 10 minutes….I’m eating...

@micha: here is the only commit I make to the fileset in boot-cljs-test, which I believe caused the second copy of cljs files to appear in the fileset

https://github.com/crisptrutski/boot-cljs-test/blob/master/src/crisptrutski/boot_cljs_test.clj#L67

hm, so there are multiple files in the fileset that contain souce code for the same namespaces?

@micha: yes 1.0.7 solved the duplication issue

@micha: and when used alone, without cljs-test task, I do not see all the other namespaces as well

@crisptrutski: maybe the show -f task before and after cljs-test would show something?

you can use boot.core/rm to remove files from the fileset before calling commit!, or boot.core/mv if you want to move a file

that is if you're performing a transformation from say foo.cljs -> bar.cljs, and you don't want foo to be on the classpath

in my case I was actually not intending to do anything, but create a cljs file. so the duplicate clj files were totally out of leftfield

good idea with show -f, I’ll pepper that through the pipeline and see where they get added, potentially during boot-cljs

boot test-cljs test still runs all namespaces (on cljs only). boot test test-cljs as well

there are also the boot.core/mv-asset, mv-resource, and mv-source functions that will maybe be something you can use

So I was right that the problem you saw is related to cljs testing after all? https://clojurians.slack.com/archives/boot/p1450643591005205

@seancorfield: i think it’s a problem with doo + filesets

yeah only seeing issue with cljs also

if i recall correctly, using doo with leiningen doesn’t have the problem

@seancorfield: at the moment, with the distinct form the issue is confined to test-cljs. yes it seems that you were right

maybe it's not being run in a pod?

if it's running in a pod it should be isolated, so namespaces it requires or loads won't be seen by the rest of the boot program

you need to be at least one pod down?

i mean if the problem is that doo is loading namespaces that then are confusing other tasks

it must be because doo isn't running in a pod

if you do (require 'foo) in a pod the other pods won't have foo loaded

leiningen might avoid the problem by using multiple jvms there?

nah, the issue is the other way around - when you finally run doo, which is the test runner, it runs tests for namespaces from your deps. those deps were always loaded, things like clojure.string

ah

the doo lein plugin does boot-inspired magic - code generates an inner leiningen project

pod jealousy

haha

@bensu is going to see this any minute now and chastise my bad characterisation

if you need to you could add meta to namespaces in the boot task

because the task has all the information about which namespaces need testing

so you could to alter-meta! on them to tag them

there is something that interact with cljc extension

doo takes a fixed list of namespaces, like boot-test does, using that at the moment. and should support regexes itself soon. just the “default” case needs better defaults

it's awesome @crisptrutski

ok, have a big file of stuff debugged, posted here: https://www.refheap.com/112956

it hasn't been easy to get all the cljs stuff working reliably

so big 👍 for that

if I run boot add-source-paths -t test/cljc test it correctly run test on all the project namespaces, I mean the ones I defined my self

and here’s the source for the test suite underneath that, if you haven’t been sucked down the rabbit hole to it yet: https://github.com/crisptrutski/boot-cljs-test/tree/portable-testing/example

but if I run boot add-source-paths -t test/cljc test-cljs test the test task runs any other cljc namespace, even the ones that are from a dependency

wonder if it isn’t just the output.out giving the duplicates.

@crisptrutski: it’s not about duplicates anymore (thanks to distinct, it’s about causing test.cljs running the test on a lot of namespaces from the dependencies and the test on any cljc source file from the deps as well (but only that)

@crisptrutski: I should try that branch….give me few minutes.

@crisptrutski: with the portable branch now test behaviour is correct. it only runs the namespaces defined by me, not the ones from any portable namespace coming from a dep

try running boot test-basic in my example

that still has the issue

the other tasks are using my whacky wrap-task decorator

wrap-task is great!

@crisptrutski: test-cljs still runs almost any cljs namespace (on you example and on my repo as well

you’re using the underlying tasks directly? i’m just talking about the test tasks defined in that build.boot

test-all, auto-test, etc

ok, i’ve confirmed the lein/boot split with doo

can checkout https://github.com/bensu/doo/tree/master/example and run lein doo phantom test once and boot add-tests test-cljs -e phantom respectively

sadly, that means the issue is really cljs.test/run-all-tests in CLJS itself

test-all it’s ok and auto-test too. But, as you said, it’s slow

we could fix that, maybe keeping a pod around

but think this issue might be worth fixing at the root, not with a bandaid

sadly, that means the issue is really cljs.test/run-all-tests in CLJS itself…it’s what I was trying to say from the beginning

perhaps a namespace blacklist could be generated from the deps? you could amortize that cost and do it asynchronously

a bandaid though

to solve the (all-ns) jaycfields did this: `(defmethod report :begin-test-ns [m] (with-test-out (if (some #(:test (meta %)) (vals (ns-interns (:ns m)))) (println "\nTesting" (ns-name (:ns m))))))`

whitelist is fast enough, i think the actual perf hit was wrap-task, recreating boot-cljs-test was ending up without cljs analysis cache etc

i’m looking at how cljs-test chooses namespaces to test, comes down to a value called ::ana/namespaces in the analyzer cache.. going to see what that value looks like resp. between these lein and doo examples

@crisptrutski: good luck!

ok, i take it back. they both have the problem. the lein suite just had the namespaces listed after all

wow, back from the deep water with a fish!

i’m off to sleep now, but i’ll just add include, exclude options like boot-expectations this week. it’ll be fast if they’re actual options and not bolted on with wrap-task

haha that's a good expression @magomimmo

for boot-test, as mentioned above -f could be used, but it’s a bit annoying to build the predicate

maybe boot-test wants to add some option sugar or predicate builders as sugar for this case?

not sure how you feel about just breaking semver and changing include exclude to regex

perhaps some prepackaged predicates included?

that works too

we could bump the major version, or add new options

--include-regex

breaking semver i meant a major bump - not breaking semver itself

or actually we can do it by changing the type of the arg in boot

make the arg of type #{edn}

or something

i’m going to sleep as well, good night

i guess that's not a good solution

good night!

what i like about the breaking change is that the various tasks would be somewhat standardised

and i think it would still work in most cases

because not planning on adding literal exclusions, think they’re wimpy against literal inclusions

i.e. #"foo.bar.baz" would still match the foo.bar.baz namespace

sure, as long as symbols and strings get treated as (re-pattern (str “^” ns “$”))

you could even forget hygeine

guess that sucks for cli usage

(re-pattern (str "^\\Q" ns "\\E$")) and have hygeine too!

oh i see lol

nevermind

if you wanted backwards compat you’d need some kind of unusual signifier to actually use regexes from cli

i think we can hack it to be compatible in most cases

yeah

maybe the regexes need to be complete anyway

then done

also we can use the ^:! meta on the option

byo-wildcards

to disable type checking

then we could accept symbols and regexes from the repl

sounds good

it's probably okay to just take regexes on the command line maybe

i think in almost all cases it will work the same

yeah, as long a regexes can never be partial matches

created issue for this: https://github.com/adzerk-oss/boot-test/issues/21

happy to take a stab at a PR later this week

might also take a stab at your “deferred errors via fileset” idea

anyhoo - night

see you!

thanks for munching on our bugbears tonight

i think if we bump major version we can change the cli behavior

because that's not heavily invested in by users

but we want to avoid changing the way it works for clojure

because people have invested in build.boot files and stuff that we don't want them to have to change

but breaking the cli i think is minimally annoying to users

so if we can still accomodate the old style in build.boot it would be totally acceptable i think

LMK what you decide to do re: regex include / exclude — it would obviously be desirable for boot-test and boot-expectations to behave much the same way 😸

And at this point it doesn’t sound like adding distinct is particularly important to boot-expectations? (but it wouldn’t hurt)

i made a PR to boot-expectations in case you decide to do that

Merged. 1.0.1 released.