Clojurians Log v2

Workaround for now:

#?(:cljs
    (:require [goog.string.StringBuffer]
              [cljs.pprint :refer [pprint]]))


(defn pprint-str [x]
  (let [sb (goog.string.StringBuffer.)]
    (binding [*print-newline* true
              *print-fn* (fn [s] (.append sb s))]
      (pprint x))
    (str sb)))

yeah clojure bindings are not Electric-compatible, you need to wrap

it may be possible to do override in pure css, but yes hacks are acceptable IMO

i will note the internationalization requirement

electric-forms is not in as good state as we want it to be because we had to switch workstreams for a few months – mostly we direct our roadmap based on what our customers need

Ok, thanks for a fast answer

This should work today. You can customize “Submit” and “Cancel”.

(ns ...
  (:require [hyperfiddle.electric-forms5 :refer [Form! Input! SubmitButton! DiscardButton!]]))

(Form! {::message "hello"}
  (e/fn [{:keys [::message]}]
    (e/amb
      (Input! ::message message)
      (SubmitButton! :label "Submit")
      (DiscardButton! :label "Cancel"))))

Yeah for dev I tried to create a docker container for dev mode, manually curled the token to make it build, but couldn't yet open the server then ran out of time (or rather, got excited by the claude 4 release).

This is what bitdefender has in the history. It includes attempts I made to clear the xattr macOS file metadata and putting the files into different directories etc.

GT-JS Acsogenixx.284.CEC83A89 resident access to the file has been denied /Users/baruchberger/electric3-starter-app/resources/public/electric...

does it say why it flagged the file?

do other clojurescript projects build?

The why would be that acsogenixx detection. More information than this is not given sadly.

Yes other projects build, it’s the first time I run into trouble like this.

Can you please upload the flagged file to https://www.virustotal.com/gui/home/upload and see what it says

to determine if this is a false positive based on how many engines flag it

GT-JS means generic trojan - javascript, it is not a specific signature match

also can you upload it here as an attachment, i am going to compare to results on my machine

what's weird is that our js asset does not contain auth code or servers or anything weird it's just another cljs app, so my first question is - maybe your system is compromised?

https://www.virustotal.com/gui/file/fcdaf26fdccd63b6cb70b83409ec24c868b3aab9482c8864ac69c8c297ac86c0?nocache=1

I also tried to send the file here but slack won't let me.

due to virus detection?

I don't know, could be big slack server restrictions, I would try to just send a text snippet but can't view the text in any program that lets me copy 😅

I can see it in the mac finder preview though, just seems like a regular shadow-cljs built main file.

drag drop it into slack as a file to use the attachment feature

I did, but that blocked it.

I guess my filesystem won't let it.

how about a gist

oh bitdefender is blocking you from viewing the content

Yes

I can view it in the finder however, I could transcribe it with ocr but it's 55kb and a small window.

The virus total thing has a behaviour tab which is interesting. It says it's creating files? C:\Windows\ServiceProfiles\LocalService\AppData\Local\FontCache\Fonts\Download-1.tmp

it does not look good that's for sure

but i don't see how electric-starter-app can be compromised, the file came from shadow. the most likely thing based on available evidence is that something on your system appended malicious js to the end

i still need to see if a main.js from my system produces the same alert

Okay, I'm trying to OCR transcribe the whole file

somehow finder of macOS has a preview window I can scroll in

lol

I thought you said you were on a mac mini btw?

oh, that came from the scanner tool's sandbox test machine

Yes

Hmm, I transcribed the whole thing with chatGPT which was laborious, but uploading that reconstructed file doesn't make any alarm bells go off with totalvirus.

Can you reproduce? Because I just tried on my laptop where I was playing with electric on earlier and that also has it.

That's the file.

Seems like it's a false positive. But why does that not happen for other bundles. 😕

i have not attempted to repro, but i did review the file and agree i do not see malicious code ... but how does it do that stuff in the test sandbox?

is this the actual file or an OCR?

This is the actual file (download and see it fail the virustotal). I also don't know why that does that stuff in the test sandbox.

If I were to be infected and this only happens for my machines then this would be a super advanced threat, and somewhat strange to be targeting windows from a macos machine.

the windows targetting is in the secondary payload, the trojan bootstraps by just connecting to a control server and downloading the next stage

Ah that would make sense.

should I spin up a cloud machine to see if I can reproduce?

nah

there doesn't seem to be a threat here, i will continue to investigate when i have time (likely backlog until we get another report as we are traveling to reclojure tomorrow)

i am just "confused" 🙂

Alright, thanks for having a look anyway. Safe travels to ReClojure!

ty for the reports and diagnosing

sure thing

could there be a unix permissions issue? e.g. you accidentally cloned with root or something?

Things I've tried: • removing .cpcache .shadow-cljs and trying again • doing all with sudo • seeing if there's anything special about my filesystem, can't find anything, other files in the resources dir open

try fresh clone in new directory

I've also tried that, but I'll try it with nothing renamed and (meaning electric-starter-app -> something else)

(base) baruchberger@192 ~ % git clone 
Cloning into 'electric3-starter-app'...
remote: Enumerating objects: 305, done.
remote: Counting objects: 100% (197/197), done.
remote: Compressing objects: 100% (128/128), done.
remote: Total 305 (delta 81), reused 92 (delta 36), pack-reused 108 (from 1)
Receiving objects: 100% (305/305), 262.36 KiB | 18.74 MiB/s, done.
Resolving deltas: 100% (108/108), done.
(base) baruchberger@192 ~ % cd electric3-starter-app 
(base) baruchberger@192 electric3-starter-app % 
(base) baruchberger@192 electric3-starter-app % clj -A:dev -X dev/-main                                      
Downloading: com/hyperfiddle/electric/v3-alpha-SNAPSHOT/maven-metadata.xml from clojars
Downloading: com/hyperfiddle/hyperfiddle-contrib/v0-alpha-SNAPSHOT/maven-metadata.xml from clojars
Downloading: com/hyperfiddle/hyperfiddle-contrib/v0-alpha-SNAPSHOT/maven-metadata.xml from clojars
Downloading: com/hyperfiddle/electric/v3-alpha-SNAPSHOT/maven-metadata.xml from clojars
INFO  dev: Starting Electric compiler and server...
INFO  io.undertow: starting server: Undertow - 2.3.10.Final
INFO  org.xnio: XNIO version 3.8.8.Final
INFO  org.xnio.nio: XNIO NIO Implementation Version 3.8.8.Final
INFO  org.jboss.threads: JBoss Threads version 3.5.0.Final
shadow-cljs - server version: 2.26.2 running at 
shadow-cljs - nREPL server started on port 54108
[:dev] Configuring build.
[:dev] Compiling ...
Please sign up or login to activate:  
Compiling ...
:hyperfiddle.electric.shadow-cljs.hooks3/recompile-clj hyperfiddle.electric3
:hyperfiddle.electric.shadow-cljs.hooks3/recompile-clj hyperfiddle.electric-css3
:hyperfiddle.electric.shadow-cljs.hooks3/recompile-clj hyperfiddle.electric-dom3-props
:hyperfiddle.electric.shadow-cljs.hooks3/recompile-clj hyperfiddle.electric-dom3
:hyperfiddle.electric.shadow-cljs.hooks3/recompile-clj electric-starter-app.main
[:dev] Build completed. (200 files, 199 compiled, 0 warnings, 12,61s)
INFO  dev: 👉 
WARN  org.eclipse.jetty.server.HttpChannel: handleException /js/main.js java.io.FileNotFoundException: /Users/baruchberger/electric3-starter-app/resources/public/electric_starter_app/js/main.js (Operation not permitted)
WARN  org.eclipse.jetty.server.HttpChannel: handleException /js/main.js java.io.FileNotFoundException: /Users/baruchberger/electric3-starter-app/resources/public/electric_starter_app/js/main.js (Operation not permitted)

(base) baruchberger@192 electric3-starter-app % cat resources/public/electric_starter_app/index.css
@import url('');

* {
    box-sizing: border-box;
}

body {
    font-family: 'Open Sans', Arial, Verdana, sans-serif;
    background-color: rgb(248 250 252);
}
(base) baruchberger@192 electric3-starter-app % cat resources/public/electric_starter_app/js/main.js 
cat: resources/public/electric_starter_app/js/main.js: Operation not permitted

it looks like a system permission issue to me, MacOS has some weird security stuff happening in certain folders that i've seen cause issues like this

Hmm yes, but why only for the main file.

(base) baruchberger@192 electric3-starter-app % head -n 5 resources/public/electric_starter_app/js/cljs-runtime/electric_starter_app.main.js
goog.provide('electric_starter_app.main');
electric_starter_app.main.Main = (function() {
var electric_starter_app$main$Main = null;
var electric_starter_app$main$Main__0 = (function (){
return cljs.core.PersistentHashMap.fromArrays([1],[hyperfiddle.electric.impl.runtime3.ctor(new cljs.core.Keyword("electric-starter-app.main","Main","electric-starter-app.main/Main",1906897109),(0))]);

ls -al ?

are you in a container?

(base) baruchberger@192 electric3-starter-app % ls -al resources/public/electric_starter_app/js 
total 128
drwxr-xr-x@   5 baruchberger  staff    160 May 21 00:05 .
drwxr-xr-x    6 baruchberger  staff    192 May 21 00:05 ..
drwxr-xr-x@ 403 baruchberger  staff  12896 May 21 00:05 cljs-runtime
-rw-r--r--@   1 baruchberger  staff  55103 May 21 00:05 main.js
-rw-r--r--@   1 baruchberger  staff   6037 May 21 00:05 manifest.edn

did it ever work previously?

Hmm, I've used the starter app a couple of times on my macbook a week or 2 back.

This is on my mac mini, I believe I've run electric 2 on it a long time ago. Not sure.

was the first electric thing I did, a week or 2/3 back, streaming LLM responses into a https://milkdown.dev/docs/guide/using-crepe editor. Challenging to get to know both electric and electric JS interop, but pretty results quick.

It seems this is caused by the licensing strategy electric v3 has. Interestingly grep can inspect the file, but cat can't. I think the code responsible for writing the main.js file might be in the dev hook?

but you said it works on your other computer?

Yes

But that was with a slightly older electric-starter-app codebase iirc

main.js is written by shadow, not us. electric hot code reloading is implemented as a shadow hook but the actual compilation is performed by shadow

I've had a talk with Thomas this morning: https://clojurians.slack.com/archives/C6N245JGG/p1747811285609769

like thomas, i think you have a system issue, i think you should try on another computer. you can also clone inside a docker container if you don't have access to another computer. many people (100s) have successfully ran the starter app

the purpose of the shadow hook is to reload the clj namespaces when shadow reloads cljs namespaces, otherwise your electric app will desync

Okay, I don't know what I could have done to my system that is out of order. But I'll try the container version, thanks!

as thomas and I both said, macos has insane and obtuse security mechanisms on the file system

they have also changed over the years, is your mac mini up to date?

Yeah, I believe you. Tried a bunch of suggestions wrt mac file protections and they all don't work out or show what's wrong. If even LLM's are stumped I'm sure it's obtuse.

Yeah it's 15.5

Found the culprit, BitDefender.

Works in docker though, but that's not the dev version. Trying to get that to work by adjusting the dockerfile.

by try in docker, i meant use a generic image, apt-get install git, apt-get install clojure, git clone

Do you have any additional info about the BitDefender issue? I want to make a note of it

I tried the three things you mentioned, but this was always the result:

ETL-ing Item
{:item-id "CARROTSALAD00000",
 :org-id "164J0XYOR0OP335J",
 :neat-item
 {:id "CARROTSALAD00000",
  :price 239,
  :description "Bombay Carrot Salad",
  :kind :item.kind/product,
  :heading-id "SALADS0000000000"}}
{:new 4,
 :_diff
 #object[com.rpl.rama.diffs.NewValueDiff 0x764edee0 "NewValueDiff[[{:id \"CARROTSALAD00000\", :price 239, :description \"Bombay Carrot Salad\", :heading-id \"SALADS0000000000\", :kind :item.kind/product} {:id \"FALAFELSALAD0000\", :price 420, :description \"Falafel Salad\", :heading-id \"SALADS0000000000\", :kind :item.kind/product} {:id \"GARDENSALAD00000\", :price 235, :description \"Acme Garden Salad\", :heading-id \"SALADS0000000000\", :kind :item.kind/product} {:id \"SALADS0000000000\", :price nil, :description \"Salads\", :heading-id nil, :kind :item.kind/heading}]]"],
 :_old 4}

This is what happens at the log "ETL-ing Item":

(r/local-transform> [(path/keypath *org-id *item-id)
                         #_(path/submap (keys *neat-item))
                         (m->multi-path *neat-item)
                         #_(path/termval *neat-item)] $$org->items)

This is the proxy path:

[(path/keypath id) (path/subselect path/MAP-VALS (path/submap [:id :price :description :heading-id :kind]))]

I have this working in my project but I am away from a computer atm. I'll post a snippet when I'm back in an hour or so

Ok, this is what I am currently using. It's pretty similar to the old one, but I don't give it an initial value of nil.

(defn proxy-flow
  "Usage: (e/join (e/flow->incseq (proxy-flow (keypath key) pstate)))"
  [path pstate]
  #?(:clj
     (-> (m/observe
           (fn [!]
             (let [cp-ps (r/foreign-proxy-async path pstate
                           {:callback-fn
                            (fn [new _diff _old]
                              (! new))})]
               (fn cleanup [arg]
                 (r/close! @cp-ps)))))
         (m/relieve))))

(e/defn ReactiveProxy [path pstate-name]
  (e/server
    (e/join (e/flow->incseq (proxy-flow path (get deps/pstates pstate-name))))))

@cjmurphy i think your error is caused by the call site

The unserializable is almost certainly the middle argument of the callback function, diff, which is an object that won’t serialize without additional work.

If you emit new, as in Jed’s example, the error should disappear. In both cases, I would recommend closing the proxy with a .thenComposeAsyncor .whenCompleteAsync or similar to avoid locking up any threads in case it doesn’t deref in a timely manner.

I had a few things to sort out (getting away from files being .clj to .cljc, where I'm comfortable that Electric and Missionary are comfortable, then using Rama's subselect on end of the path to be returning a vector rather than one element). I believe not using the e/join was my main problem, and the cause of the serialisation error message. I'm now using @noonian’s way exactly, so can confirm that works for me. Thanks also Dustin and @henrik - I'll get to trying out closing the proxy a bit later...

The domain is products per organisation (say menu of a restaurant). The problem I ended up with is that every time a product is added all products are broadcast! Claude suggested this was the problem: ResyncDiff happens when query paths are too complex for incremental diffing.

Anyway that's a Rama issue...

That’s probably Claude hallucinating. I don’t know how complex your paths are, but I’ve never run into that problem. I have run into the problem when there’s been other issues however: the connection to Rama is interrupted (eg. because of network issues). Since diffs are pushed straight through from write, it’s not inconceivable that it could occur if you’re doing coarse grained transforms on the pstate. Also, I don’t think Electric will care if you’re only returning new either way. By the way, it’s probably a good idea to guard the emit with (when-not (instance? DestroyedDiff diff) …).

I've made the path this simple: [(path/keypath id) (path/subselect path/MAP-VALS)] . It used to be this [(path/keypath id) (path/subselect path/MAP-VALS (path/view transform-item))] but I'm doing the transform on the browser client now. I agree on the hallucinating - the docs seem confident any path will work. I'm getting more than ResyncDiff. Now seeing NewValueDiff. But the problem with it is it contains that whole menu, when all I'm doing from the UI is changing the price of one dish. And I can see one depot event and I'm not sure what could be wrong with the ETL.

When you write the map, are you updating the entire map in place? NewValueDiff suggests that it’s been written wholesale, rather than manipulated with eg. submap.

I'm not using submap no. This is the ETL update:

(r/local-transform> [(path/keypath *org-id *item-id) (path/termval *neat-item)] $$org->items)

And this is $$org->items:

(r/declare-pstate s $$org->items
      (r/map-schema String
        (r/map-schema String
          (r/fixed-keys-schema
            {:id String
             :price Long
             :description String
             :kind clojure.lang.Keyword
             :heading-id String})
          {:subindex? true})))

Well, if you’re termvaling an entire item, rather than writing eg. the price only with say (keypath *org-id *item-id :price), that would explain why you’re seeing NewValueDiffs at the leaf nodes, but not for the entire thing, since you are walking down via specific keys.

Rama does no diffing; the diffs emitted are entirely based on the semantics of the transform that updated them.

Apart from looking at the granularity of your updates, try being more specific in your proxy path, eg.: [(path/keypath id) (path/subselect path/MAP-VALS (submap [:id :price …])] and see what happens. I don’t know for sure that this makes a difference, but it might.

I'll do a more detailed intro-item that looks at every key and only changes the minimum then, and see the results. Thanks for your input here.

You can allow updating arbitrary parts of the map with some transform like (submap (keys *neat-item)) (termval *neat-item). This would allow *neat-item to contain only a subset of fields in the map and leave the rest of the fields alone. If that’s still too coarse, you can unfold it into a multi-path:

(defn m->multi-path
  [m]
  (apply multi-path (mapv (fn [[k v]] [(keypath k) (termval v)]) m)))