hello, anyone meet this error when shadow-cljs release app

when just shadow-cljs compile app it works ok

Are you doing some sort of Closure-unsafe object access? Like (.-field something) as opposed to (goog.object/get something "field")?

(ns app.core
  (:require [reagent.core :as r :refer [atom]]
            ["react-native" :as RN :refer [AppRegistry]]))

(defn app-root []
  [:> RN/View {:style {:flex-direction "column"
                       :margin 40
                       :align-items "center"
                       :background-color "white"}}
   [:> RN/Text {:style {:font-size 30
                        :font-weight "100"
                        :margin-bottom 20
                        :text-align "center"}}
    "Hi Shadow!"]])

(defn init []
  (.registerComponent AppRegistry
                      "YsjApp"
                      #(r/reactify-component app-root)))

this is all the code

oh, because i use the [reagent “0.9.0-rc1”]

0.8.0 works ok

I don't know if this belongs here, but I have a problem with shadow-cljs and cider. I jackin with clj&cljs and everything works as expected. except when I want to add dependencies via cljr. Then I get a message The refactor-nrepl middleware isn’t available! Did you remember to install it? From the code of clj-refactor it tests the availability of (cljr--op-supported-p "find-symbol"). But I thought, CIDER is injecting all the relevant parts in its jackin call. So anyone has a clue how I get this running?

Hi!

I have a working shadow-cljs

project which depends on 3rd-party npm modules, so far so good

I was wondering whether from that project I could somehow distribute and package it into a form that would be consumable by lein based projects

@pyr you best bet is to create cljsjs packages that provide those npm modules. shadow-cljs would continue using npm directly but others would fall back to cljsjs

otherwise you can add instructions on how to do it via webpack but that'll require manual work on the consumer side

@thheller thanks!

@thheller so i've got an interesting lsof output from my shadow build which is failing with Too many open files - https://gist.github.com/mccraigmccraig/230d72c25b3a36d31c57fc5369ad0aab

there are a tonne of vnode: FD unavailable entries... don't know what they are yet

hmm. although that might just lsof reading structures which are in the middle of initialisation

@mccraigmccraig I don't know what that is. you were using lein right? could you try without that?

no, we're not using lein - just the shadow-cljs cli (invoked as a child_process.spawn from gulp)

hmm how come you have so many java dependencies then? I see cassandra and stuff?

yeah, that's gotten transitively from our model module - it's has both clj and cljs stuff in it

it's just pulled in as a jar in the shadow build though

well it doesn't matter too much. what is strange is that the clojure and clojurescript jars are open so many times

yeah

but I'm guessing thats the classloader, nothing I can do about that

i've now upped my ulimit -n to 262144, which seems ridiculously overspecified, and i'm still seeing the error

what exactly do you do in the build?

do you have side effecting macros that work on files maybe?

nothing unusual - there are macros, but no side-effecting macros

and you are just using a plain setup? no extra special build hooks or so?

we call shadow-cljs from gulp, but it's just a plain setup once running shadow-cljs

could you remove that part?

yep, trying that now

or maybe look at the java process with visualvm. maybe try a heap dump

I don't really know. any obvious file leak would show up for me too but there aren't any

yep, still get the issue with the shadow-cljs command issued manually

our command is this

shadow-cljs --config-merge '{:output-dir "target/yapster-web/js" :compiler-options {:closure-defines {er-webui.env/app-deployment-platform "web" er-webui.env/api-url "" er-webui.env/expected-api-version "2.6.0" er-webui.env/client-build-version "2.6.0.0" er-webui.env/sentry-release "yapster-2.6.0.0" er-webui.env/log-level "info"}}}' release yapdev

can you run with --verbose please?

see when exactly it starts going wrong?

https://gist.github.com/b020ebbbe95013e0490bf812a25c17e2

it's as if the ulimit on the build process is getting lowered

my linux knowledge is limited so I don't know how to control that at all

have you tried setting it in limits.conf?

i'm on macos ... the maxfiles ulimit on my shell process is massive - 262144

if shadow-cljs is using a subshell, it could be getting reset on a per-subshell basis.

dunno what subshell means but I think not. shadow-cljs is a npm script which then starts the java process

doesn't seem to be @gerred - if i create a bash or zsh subshell it's still got ulimit -n of 262144

hmm

@mccraigmccraig you can try setting :compiler-options {:parallel-build false} in your build config

subshell would be the Java process shelling out is all that means. 🙂

that'll make things slower but doesn't use as many threads

or any os.exec action in node

yeah, although i'm pretty sure java doesn't reset max-open-files

I'm using https://nodejs.org/api/child_process.html#child_process_child_process_spawnsync_command_args_options

but if i could find out how to query the ulimits on another process on macos i could find out... so easy on linux, haven't yet found out how on macos

spawnSync will inherit from the ulimit in the shell it's run from so you're good there

i think

and that's consistent with what @mccraigmccraig is seeing

actually that could be tested real quick...

1s.

dunno how it would leak anything. cache is definitely closed after is is read https://github.com/thheller/shadow-cljs/blob/10553565f796281ce5fdf1b569c0c7ea2f4dabe9/src/main/shadow/build/cache.clj#L35

:compiler-options {:parallel-build false} didn't change anything

😮

which java version?

that's not at all what I expected to happen. going to run that in a docker container now..

uh, wat ?

so spawnSync is maybe setting a ulimit

and trying one more thing..

try spawnSync("ulimit", ["-n"], {stdio: "inherit"})

oh good point, missed that.

thats how shadow-cljs calls things

➜ spawnsynctest node index.js 24576

same deal

so looks like it's only inheriting the actual pipes.

which makes sense from the docs since it's inheriting the pipes, not the current shell :thinking_face: but i'm still not expecting to get separate ulimits

ok trying something..

however... for me the output of

const { spawnSync } = require("child_process");
spawnSync("ulimit", ["-n"], {stdio: "inherit"});

which is inherited

so maybe that's a mac shell thing.

if you go to my original (console.log version with the default stdio) is it the same?

yes, same

Does shadow-cljs support default exports? Like export default { title: 'foo'};

I am struggling to see even a basic example online

@mccraigmccraig hmmm thought.

Something like (defn ^:export ?

out of curiosity (then I gotta jump) try running it inside a docker container?

@studio in which context? it doesn't support outputting ESM but for commonjs you can just do (def ^:export default ...)

@gerred yeah, i'm trying it on a deb container on a linux kernel now

I am trying to get Storybook's new format working which is based on ES6 modules:

storiesOf('atoms/Button', module)
  .add('text', () => <Button>Hello</Button>)
  .add('emoji', () => <Button>😀😎👍💯</Button>);

export default { title: 'atoms/Button' };
export const text = () => <Button>Hello</Button>;
export const emoji = () => <Button>😀😎👍💯</Button>;

But seem to be stuck at the first hurdle 😞

and you are using :npm-module I presume?

(def ^:export default #js {:title "atoms/Button"}) (defn ^:export text [] (reagent/as-element [:button "Hello"]))

it might be that storybook is actually trying to parse the ESM however in which case this wouldn't work since we are outputting commonjs

it also doens't work in "strict ESM" mode

Yep :stories {:target :npm-module

Where has the reagent stuff come from?

are you not using reagent?

We will be in the future, I am just trying to get the most basic setup going atm

(:require [reagent.core :as reagent])

Stupid question, why would I need to use that? This is just basic JS at this point? export const text etc

We already use their React

(:require
    ["@storybook/react" :refer (storiesOf)]
    ["@storybook/addon-actions" :refer (action)]))

from the example code I presume you need to return a react element

which is what () => <Button>Hello</Button>; does

Hmm, sorry, I pasted old code from your original demo. It's now:

import React from 'react';
import { Button } from '@storybook/react/demo';

export default { title: 'Button' };

export const withText = () => <Button>Hello Button</Button>;

So in this scenario am I better to rely on Reagent or the React dependency in package.json?

always package.json for JS dependencies

but you need both, reagent and react

(ns foo.bar
  (:require
    [reagent.core :as reagent]
    ["react" :as react]
    ["@storybook/react/demo" :refer (Button)]))

(def ^:export default #js {:title "Button"})

(defn ^:export withText []
  (reagent/as-element [:> Button "Hello Button"]))

I can make an example without reagent but you are likely going to use that anways?

(ns foo.bar
  (:require
    ["react" :as react]
    ["@storybook/react/demo" :refer (Button)]))

(def ^:export default #js {:title "Button"})

(defn ^:export withText []
  (react/createElement Button nil "Hello Button"))

The end goal of the app will be using Reagent and probably Re-frame.

I guess either way I would need to reference react or reagent so better if the Story file matched the real cljs file I guess

not sure what you mean by that

the examples I pasted should work

if they don't storybook might be trying to process the JS files directly

you might have more fun with https://github.com/nubank/workspaces since its all cljs

i've been needing to check out workspaces.

So this works

(ns frontend.ui.component.button-stories
  (:require
    [reagent.core :as reagent]
    ["react" :as react]))

(def ^:export default #js {:title "Button"})

(defn ^:export withText []
  (reagent/as-element [:button "Hello"]))

But is also tries to process the default export as a component.

I guess because we explicity named it

no we didn't

We created a global variable with the name of default - is how I understood it

it is not a global variable no

and export default creates an .export property just like what we did

Ok, I must have misunderstood this https://cljs.info/cheatsheet - apologies

well yeah this is specific to the npm-module target

in general this creates a namespaced global var yes

so we'd have a foo.bar.default

but the :npm-module is turning that into module.exports.default = ...

Ah, ok that makes sense

So if I change the title, for example, then the hot reload in Storybook breaks can't redefine non-configurable property "default" Storybook can't be trying to access the files directly if they worked on initial load?

I don't have the slightest clue how storybook does anything

my total experience with it is like 30min making the old example

Thanks for the help! Looks like Storybook might not be a good fit - too many abstractions and dependencies when used in a Clojure project 😞

2.8.53 error in process filter: The shadow-cljs ClojureScript REPL is not available

> process filter

waht is that?

and what is that entire message? it is not produced by shadow-cljs

Sounds like emacs?

wait a minute, maybe update broken

the same error

[nREPL] Establishing direct connection to localhost:53629 ... [nREPL] Direct connection to localhost:53629 established error in process filter: user-error: The shadow-cljs ClojureScript REPL is not available error in process filter: The shadow-cljs ClojureScript REPL is not available

in emacs i use cider-connect-cljs

this is not a shadow-cljs error so I can't say what it means

ok @thheller - found something interesting - downgrading shadow-cljs from 2.8.52 down to the version we were previously using, 2.8.39, seems to fix our too-many-open-files issue

are you sure thats not just because it wiped the cache?

might be back if you run a second time?

nope, i've already run it half a dozen times

previously it was borking 2-3 runs after a cache clear

it's not a macos only issue either - one of my devs is on an ubuntu laptop, and he encountered the issue too

no clue. I mostly worked on REPL issues and some of the react-native stuff

you are just running shadow-cljs release ... right? there is no server-mode instance running prior?

this is just shadow-cljs release ... - users have also seen the issue with watch builds

yes but with release no server-mode is started. so half of the stuff isn't even running in that case

just trying to narrow down the issue

still don't have the slightest clue what might be happening

if you have too much time you can try running each try each version to see when it started failing

there aren't many commits per release so should be easy to identify

can't find anything going over commit history that would be related to this

i'll do a binary chop - i certainly don't want to be stuck on an old release!

https://github.com/thheller/shadow-cljs/commit/a21c3fcc42c48aef1d3734fe4c88936d51775e43 thats the only thing thats potentially opening/closing many files

but it won't even run in release mode

@thheller it looks like it's [email protected] where the issue first arises

which includes the commit you referenced earlier https://github.com/thheller/shadow-cljs/commit/a21c3fcc42c48aef1d3734fe4c88936d51775e43

a simple git revert of that commit causes conflicts, which are beyond my ability to fix in a couple of minutes, so i haven't tested out the hypothesis that it's that commit which is to blame

@mccraigmccraig if you happen to run into this while watch is running could you try shadow-cljs clj-repl (separate terminal) and dump @shadow.build.macros/active-macros-ref?

yep @thheller i can do that

@mccraigmccraig try [email protected]. I added some caching so the macro stuff doesn't touch so many files. maybe that helps?

that seems to fix the problem @thheller - i ran about 6 builds consecutively without issue - i've switched our dev branch to [email protected], so i should get some more reports from my other devs tomorrow