Fork me on GitHub
#shadow-cljs
<
2019-09-19
>
vinurs01:09:04

hello, anyone meet this error when shadow-cljs release app

vinurs01:09:21

when just shadow-cljs compile app it works ok

alexandergunnarson01:09:10

Are you doing some sort of Closure-unsafe object access? Like (.-field something) as opposed to (goog.object/get something "field")?

thheller07:09:59

this is outdated advice. for code it is absolutely fine to use (.-field something). you just need to "tag" it either directly or indirectly as js to tell the compiler to not rename it. so either (.-field ^js something) or `(fn [^js something] ...). see https://code.thheller.com/blog/shadow-cljs/2017/11/06/improved-externs-inference.html

👍 4
alexandergunnarson06:09:40

Oh, nice! My awareness of this was coming purely from the vanilla CLJS compiler. Don’t profess to know much about shadow-cljs yet 🙂

vinurs01:09:20

(ns app.core
  (:require [reagent.core :as r :refer [atom]]
            ["react-native" :as RN :refer [AppRegistry]]))

(defn app-root []
  [:> RN/View {:style {:flex-direction "column"
                       :margin 40
                       :align-items "center"
                       :background-color "white"}}
   [:> RN/Text {:style {:font-size 30
                        :font-weight "100"
                        :margin-bottom 20
                        :text-align "center"}}
    "Hi Shadow!"]])

(defn init []
  (.registerComponent AppRegistry
                      "YsjApp"
                      #(r/reactify-component app-root)))

vinurs01:09:25

this is all the code

vinurs01:09:13

oh, because i use the [reagent “0.9.0-rc1”]

vinurs01:09:19

0.8.0 works ok

danielgrosse06:09:27

I don't know if this belongs here, but I have a problem with shadow-cljs and cider. I jackin with clj&cljs and everything works as expected. except when I want to add dependencies via cljr. Then I get a message The refactor-nrepl middleware isn’t available! Did you remember to install it? From the code of clj-refactor it tests the availability of (cljr--op-supported-p "find-symbol"). But I thought, CIDER is injecting all the relevant parts in its jackin call. So anyone has a clue how I get this running?

thheller07:09:56

no clue, probably just need to add the refactor-nrepl dependency manually?

pyr06:09:56

I have a working shadow-cljs

pyr06:09:36

project which depends on 3rd-party npm modules, so far so good

pyr06:09:14

I was wondering whether from that project I could somehow distribute and package it into a form that would be consumable by lein based projects

thheller07:09:44

@pyr you best bet is to create cljsjs packages that provide those npm modules. shadow-cljs would continue using npm directly but others would fall back to cljsjs

thheller07:09:24

otherwise you can add instructions on how to do it via webpack but that'll require manual work on the consumer side

mccraigmccraig10:09:02

@thheller so i've got an interesting lsof output from my shadow build which is failing with Too many open files - https://gist.github.com/mccraigmccraig/230d72c25b3a36d31c57fc5369ad0aab

mccraigmccraig10:09:27

there are a tonne of vnode: FD unavailable entries... don't know what they are yet

mccraigmccraig10:09:22

hmm. although that might just lsof reading structures which are in the middle of initialisation

thheller11:09:54

@mccraigmccraig I don't know what that is. you were using lein right? could you try without that?

mccraigmccraig11:09:19

no, we're not using lein - just the shadow-cljs cli (invoked as a child_process.spawn from gulp)

thheller11:09:38

hmm how come you have so many java dependencies then? I see cassandra and stuff?

mccraigmccraig11:09:20

yeah, that's gotten transitively from our model module - it's has both clj and cljs stuff in it

mccraigmccraig11:09:58

it's just pulled in as a jar in the shadow build though

thheller11:09:02

well it doesn't matter too much. what is strange is that the clojure and clojurescript jars are open so many times

thheller11:09:29

but I'm guessing thats the classloader, nothing I can do about that

mccraigmccraig11:09:53

i've now upped my ulimit -n to 262144, which seems ridiculously overspecified, and i'm still seeing the error

thheller11:09:16

what exactly do you do in the build?

thheller11:09:26

do you have side effecting macros that work on files maybe?

mccraigmccraig11:09:11

nothing unusual - there are macros, but no side-effecting macros

thheller11:09:50

and you are just using a plain setup? no extra special build hooks or so?

mccraigmccraig11:09:31

we call shadow-cljs from gulp, but it's just a plain setup once running shadow-cljs

thheller11:09:45

could you remove that part?

mccraigmccraig11:09:39

yep, trying that now

thheller11:09:46

or maybe look at the java process with visualvm. maybe try a heap dump

thheller11:09:08

I don't really know. any obvious file leak would show up for me too but there aren't any

mccraigmccraig11:09:01

yep, still get the issue with the shadow-cljs command issued manually

mccraigmccraig11:09:49

our command is this

shadow-cljs --config-merge '{:output-dir "target/yapster-web/js" :compiler-options {:closure-defines {er-webui.env/app-deployment-platform "web" er-webui.env/api-url "" er-webui.env/expected-api-version "2.6.0" er-webui.env/client-build-version "2.6.0.0" er-webui.env/sentry-release "yapster-2.6.0.0" er-webui.env/log-level "info"}}}' release yapdev

thheller11:09:02

can you run with --verbose please?

thheller11:09:08

see when exactly it starts going wrong?

mccraigmccraig11:09:15

it's as if the ulimit on the build process is getting lowered

thheller11:09:35

my linux knowledge is limited so I don't know how to control that at all

gerred11:09:15

have you tried setting it in limits.conf?

mccraigmccraig11:09:21

i'm on macos ... the maxfiles ulimit on my shell process is massive - 262144

gerred11:09:32

if shadow-cljs is using a subshell, it could be getting reset on a per-subshell basis.

thheller12:09:22

dunno what subshell means but I think not. shadow-cljs is a npm script which then starts the java process

mccraigmccraig12:09:50

doesn't seem to be @gerred - if i create a bash or zsh subshell it's still got ulimit -n of 262144

thheller12:09:26

@mccraigmccraig you can try setting :compiler-options {:parallel-build false} in your build config

gerred12:09:34

subshell would be the Java process shelling out is all that means. 🙂

thheller12:09:42

that'll make things slower but doesn't use as many threads

gerred12:09:44

or any os.exec action in node

mccraigmccraig12:09:15

yeah, although i'm pretty sure java doesn't reset max-open-files

mccraigmccraig12:09:02

but if i could find out how to query the ulimits on another process on macos i could find out... so easy on linux, haven't yet found out how on macos

gerred12:09:32

spawnSync will inherit from the ulimit in the shell it's run from so you're good there

gerred12:09:57

and that's consistent with what @mccraigmccraig is seeing

gerred12:09:18

actually that could be tested real quick...

mccraigmccraig12:09:20

:compiler-options {:parallel-build false} didn't change anything

thheller12:09:42

which java version?

gerred12:09:12

that's not at all what I expected to happen. going to run that in a docker container now..

mccraigmccraig12:09:50

so spawnSync is maybe setting a ulimit

gerred12:09:15

and trying one more thing..

thheller12:09:27

try spawnSync("ulimit", ["-n"], {stdio: "inherit"})

gerred12:09:36

oh good point, missed that.

thheller12:09:41

thats how shadow-cljs calls things

gerred12:09:10

➜ spawnsynctest node index.js 24576

gerred12:09:27

so looks like it's only inheriting the actual pipes.

gerred12:09:56

which makes sense from the docs since it's inheriting the pipes, not the current shell :thinking_face: but i'm still not expecting to get separate ulimits

gerred12:09:44

ok trying something..

mccraigmccraig12:09:38

however... for me the output of

const { spawnSync } = require("child_process");
spawnSync("ulimit", ["-n"], {stdio: "inherit"});
is 262144

mccraigmccraig12:09:48

which is inherited

gerred12:09:50

so maybe that's a mac shell thing.

gerred12:09:05

if you go to my original (console.log version with the default stdio) is it the same?

Dan12:09:09

Does shadow-cljs support default exports? Like export default { title: 'foo'};

Dan12:09:24

I am struggling to see even a basic example online

Dan12:09:55

Something like (defn ^:export ?

gerred12:09:17

out of curiosity (then I gotta jump) try running it inside a docker container?

thheller12:09:37

@studio in which context? it doesn't support outputting ESM but for commonjs you can just do (def ^:export default ...)

mccraigmccraig12:09:11

@gerred yeah, i'm trying it on a deb container on a linux kernel now

Dan12:09:48

I am trying to get Storybook's new format working which is based on ES6 modules:

storiesOf('atoms/Button', module)
  .add('text', () => <Button>Hello</Button>)
  .add('emoji', () => <Button>😀😎👍💯</Button>);
is now:
export default { title: 'atoms/Button' };
export const text = () => <Button>Hello</Button>;
export const emoji = () => <Button>😀😎👍💯</Button>;

Dan12:09:00

But seem to be stuck at the first hurdle 😞

thheller12:09:45

and you are using :npm-module I presume?

thheller12:09:37

(def ^:export default #js {:title "atoms/Button"}) (defn ^:export text [] (reagent/as-element [:button "Hello"]))

thheller12:09:02

it might be that storybook is actually trying to parse the ESM however in which case this wouldn't work since we are outputting commonjs

thheller12:09:31

it also doens't work in "strict ESM" mode

Dan12:09:52

Yep :stories {:target :npm-module

Dan12:09:25

Where has the reagent stuff come from?

thheller12:09:47

are you not using reagent?

Dan12:09:18

We will be in the future, I am just trying to get the most basic setup going atm

thheller12:09:48

(:require [reagent.core :as reagent])

Dan12:09:56

Stupid question, why would I need to use that? This is just basic JS at this point? export const text etc

Dan12:09:12

We already use their React

(:require
    ["@storybook/react" :refer (storiesOf)]
    ["@storybook/addon-actions" :refer (action)]))

thheller12:09:10

from the example code I presume you need to return a react element

thheller12:09:26

which is what () => <Button>Hello</Button>; does

Dan12:09:58

Hmm, sorry, I pasted old code from your original demo. It's now:

import React from 'react';
import { Button } from '@storybook/react/demo';

export default { title: 'Button' };

export const withText = () => <Button>Hello Button</Button>;

Dan12:09:40

So in this scenario am I better to rely on Reagent or the React dependency in package.json?

thheller12:09:17

always package.json for JS dependencies

thheller12:09:33

but you need both, reagent and react

thheller12:09:52

(ns foo.bar
  (:require
    [reagent.core :as reagent]
    ["react" :as react]
    ["@storybook/react/demo" :refer (Button)]))

(def ^:export default #js {:title "Button"})

(defn ^:export withText []
  (reagent/as-element [:> Button "Hello Button"]))

thheller12:09:08

I can make an example without reagent but you are likely going to use that anways?

thheller12:09:35

(ns foo.bar
  (:require
    ["react" :as react]
    ["@storybook/react/demo" :refer (Button)]))

(def ^:export default #js {:title "Button"})

(defn ^:export withText []
  (react/createElement Button nil "Hello Button"))

Dan12:09:37

The end goal of the app will be using Reagent and probably Re-frame.

Dan12:09:23

I guess either way I would need to reference react or reagent so better if the Story file matched the real cljs file I guess

thheller12:09:03

not sure what you mean by that

thheller12:09:08

the examples I pasted should work

thheller12:09:20

if they don't storybook might be trying to process the JS files directly

thheller12:09:52

you might have more fun with https://github.com/nubank/workspaces since its all cljs

gerred12:09:25

i've been needing to check out workspaces.

Dan12:09:01

So this works

(ns frontend.ui.component.button-stories
  (:require
    [reagent.core :as reagent]
    ["react" :as react]))

(def ^:export default #js {:title "Button"})

(defn ^:export withText []
  (reagent/as-element [:button "Hello"]))

Dan12:09:03

But is also tries to process the default export as a component.

Dan13:09:24

I guess because we explicity named it

thheller13:09:41

no we didn't

Dan13:09:12

We created a global variable with the name of default - is how I understood it

thheller13:09:10

it is not a global variable no

thheller13:09:32

and export default creates an .export property just like what we did

Dan13:09:38

Ok, I must have misunderstood this https://cljs.info/cheatsheet - apologies

thheller13:09:15

well yeah this is specific to the npm-module target

thheller13:09:24

in general this creates a namespaced global var yes

thheller13:09:33

so we'd have a foo.bar.default

thheller13:09:47

but the :npm-module is turning that into module.exports.default = ...

Dan13:09:26

Ah, ok that makes sense

Dan13:09:13

So if I change the title, for example, then the hot reload in Storybook breaks can't redefine non-configurable property "default" Storybook can't be trying to access the files directly if they worked on initial load?

thheller13:09:46

I don't have the slightest clue how storybook does anything

thheller13:09:58

my total experience with it is like 30min making the old example

Dan13:09:35

Thanks for the help! Looks like Storybook might not be a good fit - too many abstractions and dependencies when used in a Clojure project 😞

vinurs14:09:53

2.8.53 error in process filter: The shadow-cljs ClojureScript REPL is not available

thheller14:09:33

> process filter

thheller14:09:34

waht is that?

thheller14:09:52

and what is that entire message? it is not produced by shadow-cljs

dpsutton14:09:24

Sounds like emacs?

vinurs14:09:12

wait a minute, maybe update broken

vinurs15:09:33

the same error

vinurs15:09:41

[nREPL] Establishing direct connection to localhost:53629 ... [nREPL] Direct connection to localhost:53629 established error in process filter: user-error: The shadow-cljs ClojureScript REPL is not available error in process filter: The shadow-cljs ClojureScript REPL is not available

vinurs15:09:54

in emacs i use cider-connect-cljs

thheller15:09:24

this is not a shadow-cljs error so I can't say what it means

mccraigmccraig15:09:13

ok @thheller - found something interesting - downgrading shadow-cljs from 2.8.52 down to the version we were previously using, 2.8.39, seems to fix our too-many-open-files issue

thheller15:09:57

are you sure thats not just because it wiped the cache?

thheller15:09:22

might be back if you run a second time?

mccraigmccraig15:09:10

nope, i've already run it half a dozen times

mccraigmccraig15:09:49

previously it was borking 2-3 runs after a cache clear

mccraigmccraig15:09:23

it's not a macos only issue either - one of my devs is on an ubuntu laptop, and he encountered the issue too

thheller15:09:32

no clue. I mostly worked on REPL issues and some of the react-native stuff

thheller15:09:50

you are just running shadow-cljs release ... right? there is no server-mode instance running prior?

mccraigmccraig15:09:45

this is just shadow-cljs release ... - users have also seen the issue with watch builds

thheller15:09:20

yes but with release no server-mode is started. so half of the stuff isn't even running in that case

thheller15:09:29

just trying to narrow down the issue

thheller15:09:44

still don't have the slightest clue what might be happening

thheller15:09:36

if you have too much time you can try running each try each version to see when it started failing

thheller15:09:49

there aren't many commits per release so should be easy to identify

thheller15:09:10

can't find anything going over commit history that would be related to this

mccraigmccraig15:09:37

i'll do a binary chop - i certainly don't want to be stuck on an old release!

thheller15:09:20

but it won't even run in release mode

mccraigmccraig16:09:14

@thheller it looks like it's [email protected] where the issue first arises

mccraigmccraig16:09:06

a simple git revert of that commit causes conflicts, which are beyond my ability to fix in a couple of minutes, so i haven't tested out the hypothesis that it's that commit which is to blame

thheller18:09:47

@mccraigmccraig if you happen to run into this while watch is running could you try shadow-cljs clj-repl (separate terminal) and dump @shadow.build.macros/active-macros-ref?

thheller21:09:28

@mccraigmccraig try [email protected]. I added some caching so the macro stuff doesn't touch so many files. maybe that helps?

mccraigmccraig23:09:52

that seems to fix the problem @thheller - i ran about 6 builds consecutively without issue - i've switched our dev branch to [email protected], so i should get some more reports from my other devs tomorrow

parrot 4
thheller08:09:51

who knew that checking for a last modified timestamp could be that expensive 😛

mccraigmccraig08:09:53

ohhh... so was it the URLConnection which is holding files for file: URLs open then ?

thheller08:09:37

no, that was fixed a while ago. the change was only that it caches those lookups for files in .jar

thheller08:09:44

guess the JVM is doing something weird with them

mccraigmccraig09:09:48

got a first report in from a non-me dev now @thheller - 2.8.54 is confirmed to resolve the too-many-open-files issue!

mccraigmccraig09:09:10

thank you for the excellent support

thheller09:09:43

maybe also upgrade to .55 for the emacs users. I accidentally broke the cider detection mechanism

mccraigmccraig09:09:16

oh, cool - we're mostly CIDER users, so i'll do that

thheller09:09:21

thanks for all the testing. wouldn't have guess the cause with that.

mccraigmccraig09:09:50

btw - what drives you on shadow ? is it part of your job, or a side-project ?

thheller10:09:48

started out as something I needed for work myself. now fun side-project, maybe my job soon when I do consulting and stuff around it 😉

mccraigmccraig10:09:01

well i'm bought into it - if/when you are thinking about it, let's talk about how we (yapster) can contribute to your ongoing work on shadow

thheller11:09:12

for now there is https://www.patreon.com/thheller maybe github sponsors when it becomes available. code and documentation contributions are also very welcome but not sure which parts of the code are easy to contribute to 😛