I don't know if you've seen https://github.com/clojure/clojure-clr/wiki/ByRef-and-params#byref-parameters, but it might help. I ran into something similar and couldn't get it working but I was probably doing something wrong.

Thanks. I’ve seen that, but I’m not sure how it applies here, or how I’m supposed to use it in this context

It would be helpful if the error message said what the value was 😅

I ran into that 'boxed ByRef-lib values' problem recently working on NREPL, but I don't recall why or how I solved it. I'll take a look.

Thanks, David!

The issue I ran into was:

% clojure.main
Clojure 1.11.0
user=> (let [bytes (.GetBytes System.Text.Encoding/UTF8 "hi")]
          (new |System.ReadOnlySpan`1[System.Byte]| bytes))
Execution error (InvalidProgramException) at System.Diagnostics.StackFrame/ (NO_FILE:0).
Cannot create boxed ByRef-like values.

Yes, it was along those lines. Span causes some issues. And there is an overload for Write that takes a ReadOnlySpan<Byte> that mucks things up. Now, how did I solve it. Looking ...

Interesting. Not sure why that’s triggered only when Close is called. If I remove that the Write call seems successful.

.Flush also triggers it

Ah, that makes sense, thanks

It might be the case that the Write to the underlying memory stream does not happen when you write to the StreamWriter. When you Close or explicitly Flush, the Write occurs at that point. Then you get into trouble.

The overload is here: https://learn.microsoft.com/en-us/dotnet/api/system.io.memorystream.write?view=net-7.0

If you do (def sw (doto (StreamWriter. mstr) (.set_AutoFlush true) (.Write "testing"))) you get the boxed error. Without the AutoFlush, the error happens when you flush explicitly or Close. It is the underlying Write on MemoryStream, I think. That and the imprecision of proxy.

This may be a deficiency in the underlying interop calling mechanism. Spans and such as stack allocated. And as is clear from the error message, we have trying to box it. I'm not surprised it is not happy about this. I'll have to think about how to work around this.

at user.proxy$MemoryStream$FF19274A$_INTERP.Write(ReadOnlySpan`1 )
   at System.IO.StreamWriter.Flush(Boolean flushStream, Boolean flushEncoder)
   at System.Dynamic.UpdateDelegates.UpdateAndExecute1[T0,TRet](CallSite site, T0 arg0)
   at user$eval24453__24458.__interop_Flush24460(Object )

👍

Did you manage to find a workaround?

Just to follow up on this — while it would be good to have this fixed, the close override turned out to be unneeded, as one can call .ToArray on the stream even after it’s been closed.

nope. This is going to take some digging. I'll open an issue on it so I don't forget.

Thanks David! I did open one in the Ask Clojure forum, so maybe you could link it to that. or whatever is the standard procedure.

Missed it over there. I already created the issue: https://clojure.atlassian.net/jira/software/c/projects/CLJCLR/issues/CLJCLR-132 I'll link to the Ask thread.

It probably matters that my call to Clojure is in a dynamically loaded plugin to another program I do not control. I see the Clojure assembly in the VS object browser in my plugin project, but maybe I need to explicitly load or initialize Clojure in some way? I don't have this issue with other dependencies though.

https://github.com/lucasteles/clojure-flappy-bird This is an example that works for me, using clj from C#. https://github.com/lucasteles/clojure-flappy-bird/blob/master/MonogameClj.csproj Perhaps the csproj needs to be configured to include the clj sources?

@UBN9SNVB4 this is a good example to have too since it shows using typical dotnet tools for dependency management of ClojureCLR projects

Yep, it's a good starting point, though I have to use .NETFramework, so the tools don't quit line up. I'm thinking I may be in uncharted waters. CLJCLR-88 (still open) indicates that calling Clojure from a DLL has an issue (with a workaround), but it's not the same issue I have. I'm doing that, but my dll is dynamically loaded and I don't have any control over that. Since it seems it may be a fundamental problem I'm going to start building test projects to either repro or fix this, or maybe along the way I'll find what I'm doing wrong, if that's the case. Reflection is failing to find things that are clearly present, so I am pretty sure it's fixable, but it's a bit deeper into reflection than I've gone before, and that was a while back. So, that's today's agenda. 🙂 I pretty much have this week to get the basic mechanism working in order to justify going with cljr or I'll have to go back to translating stuff to C#. (I have to get this part to work, and then call C# from the clojure code. That's where I was expecting trouble, so this one caught me by surprise.)

The worst part is it fails before it's even tried to load any of my code, so it's a setup thing. I'm sure there'll be issues with my code, but it hasn't even gotten to that yet. ClojureCLR can't initialize. The namespaces it can't find are StringBuilder and 2 from clojure.lang (BigInteger and BigDecimal). So it's something REALLY basic.

@U4LCNMMGS I'm in some back-to-back meetings most of today but if @U45FQSBF1 doesn't jump in and solve it for you by the time I'm free, I'll circle back to this later and see if I can help. I too am using .NET Framework (cuz I want to compile!) and most of my experience is with that for now (though I anticipate supporting .NET6+ soon).

Thanks! I'm doing a plugin, so the environment is dictated to me. I don't know how the program loads my assembly, but it has worked for my C# code for years. I'm going to try to repro it with test projects.

I'm off doing some code spelunking for a gig. I'll try to take a look this afternoon. It's odd to me if that failure is happening in CreateDefaultImportDictionary. That's just running through the loaded types in the System namespace in order to create the map that allows you to refer to classes such as String and Object without saying System.String and System.Object. It runs through all assemblies loaded at startup and grabs their types. It isn't doing any type lookup on its own. But I just saw your note on the namespaces it can't find. Those are encoded differently because they are not in System. Here is the code:

// ADDED THESE TO SUPPORT THE BOOTSTRAPPING IN THE JAVA CORE.CLJ
            d.Add(Symbol.intern("StringBuilder"), typeof(StringBuilder));
            d.Add(Symbol.intern("BigInteger"), typeof(clojure.lang.BigInteger));
            d.Add(Symbol.intern("BigDecimal"), typeof(clojure.lang.BigDecimal));

yeah, that's the code I was looking at. It's flagging the cctor, which isn't a thing at the C# source level, so that usually means static class initialization, and that's all of that that is visible. If it's not those statements, it's invisible initialization mechanics that don't even have source code (which is possible, I guess).

Looking at the IL, the typeof construct compiles to

IL_0034: call class clojure.lang.Symbol clojure.lang.Symbol::intern(string)
	IL_0039: ldtoken [mscorlib]System.Text.StringBuilder
	IL_003e: call class [mscorlib]System.Type [mscorlib]System.Type::GetTypeFromHandle(valuetype [mscorlib]System.RuntimeTypeHandle)

Well, I can get different error. Running under 4.8. I created a Console project, did the get-a-var-and-invoke-it. No problem. Restructured the code. created a DLL with the get-a-var-and-and-invoke-it in a static method. References that library project from the console project. Now it fails. System.TypeInitializationException. location is in the RT.doInit method, trying to load an assembly from file. the only assemblies doInit tries to load are the spec.alpha assemblies. Why can it see them when called from the executable, but not from the library? I restructured the code this way because it sounded more like your situation. The kicker here is: if I add the Clojure package reference back into the console project, it works again. Why can it find the spec dlls only when the reference is coming from the consolle project and not the library project? This is a .net mystery.

CLJCLR-88 is reporting that problem exactly, I think. Found that this morning

I jumped into the deep end, never done any of this in the easy case, so (when this meeting is over) I'm going to do a simple project that works, then complicate it moving towards my actual setup step by step to either solve or at least reliably repro this. I haven't even done the "get a var and invoke it" bit successfully yet. (I'm close lol)

Although it sounds like you just did what my next step was going to be and got stuck there. Only difference between what you did and my situation is that my DLL is dynamically loaded.

Even not dynamically loaded, there is a problem here. I'm looking around for situations where Assembly loads fail. Here is one suggestion: https://stackoverflow.com/questions/22242440/c-sharp-dll-assembly-load-fails-due-to-using-another-dll I'll keep digging. After lunch.

interesting

In my specific case, I can actually try that first option, but obviously not a general solution. And there are other places I might want to do this same thing where I couldn't do that. The last one seems like a general solution, but probably not an easy one. But yeah, it does kind of sound like this is a general .net problem not even specific to ClojureCLR

I think this one is spot on. In CLJCLR-88 he said the workaround was to move one of the clojure dll's to the calling program's directory (option 1 from that SO solution). Probably the only reason I got a different error from you was what specific thing it failed to find first. sigh However, it seems this is at least a well known issue, so it's a place to start. I'll report back if I make any useful progress. (I don't like option 1, but I'll fall back to it if it saves me writing C# code lol.)

(just to add to the fun, my Clojure code will also be in a dll and has to call c# code from other dlls)

Identical setup under .Net 6 runs just fine. So it looks a difference between how Framework deals with things. This is going to be to fix.

😞

https://github.com/dotnet/runtime/issues/18527 utility https://github.com/stazz/NuGetUtils/tree/develop/Source/Code/NuGetUtils.Lib.AssemblyResolving 1500+ lines! It's going to be a long ... night ... several days ... week ... . Any time it's a problem in Framework but not in later versions ... I heave a deep sigh.

yikes. I wanted to try to help with this in some way, but I can't manage to load my own clojure code at all.

dashes and underscores. arg. btw, when I mess that up it throws an index out of range error trying to report the exception out of RT.load. Not obvious why by looking at it, but it's they only place string.Format is found, so I finally realized it was a file not found error.

That index out of range is a bug I accidentally introduced in alpha2. I'll get alpha3 out with a fix soon. (I had fixed it before alpha2 went out but apparently wiped out the change somewhere along the way. Another sigh.)

it's a really confusing error, but I got it to run my code once I got all the dashes and underscores right. From local code, now I'm trying it with a dll. One step at a time.

@U45FQSBF1 we all appreciate your hard work to keep ClojureCLR going

Thanks.

Yeah, I'm sorry to be a pest, but I just want so much to be able to use it. 🙂

The fail in the RT cctor is, in my environment, trying to get types for assemblies from the host program, which is why you weren't seeing it. Swallowing exceptions from the GetTypes call fixes that.

Could you explain a little more? Also "fixes that" means that you now see the error I see, or something else?

yeah, it just fixes the fail in CreateDefaultImportDictionary. Now I'm getting "Could not locate clojure/core ...." which I assume is what you're getting.

I thought I found a promising solution for that, but those don't even hit ResolveAssembly()

Oh but with that one fix I could try the ugly brute force thing again (copy everything to the host program exe folder)...

Yep, if I copy everything to the host program folder, I can successfully invoke my clojure code. It's ugly, but it's a working stopgap. If I can then call back to C# code I'm good. Tomorrow's project. 🙂

What I did was add this to GetAllTypesInNamespace:

Func<Assembly, IEnumerable<Type>> getTypes = (Assembly a) => {
                try { return a.GetTypes(); } catch (Exception) { return new Type[0]; }
            };

where were the files you moved originally located?

my C# project's bin\Debug directory

So they included all the clojure dll's that got used. Also had to put my clojure source there. Haven't managed to get a compiled dll to work yet, though I suspect it's the same issue.

just trying to figure out why your main can find Clojure.dll (which it must it can blow up in the RT code) and the RT code cannot find the spec dlls. Trying to load assemblies the wrong way probably, perhaps LoadFile/LoadFrom issue. I'll play with it. I can also add the fix you put in for the type loading exception.

I'm finding the dlls because I moved them to the same directory the top level program exe lives in, which is always on the .net assembly search path

And it'll load my clojure code from compiled .dll files if I copy those as well, now that I figured out how to properly reference them. 🙂

Understood. I'd like to see if there is a way to find them where they were originally. Also odd that the problem I created is on Framework and not for .NET 6.

Oh I definitely hope for a better solution. But having a workaround lets me use clojure instead of translating to C#. 🙂 What is the problem you created?

my problem = the sample problem. Console uses Library. Library uses Clojure. Fails under Framework, works under .Net 6.

Are you sure anyone's ever tried it before? It's kind of specific. It sounds like they fixed some stuff between Framework and .Net 6. After all that SO thing you found indicated this was pretty common for Framework (didn't say anything about .Net x).

I found one that showed how to fix it with an assembly resolver, but those aren't even being called in this case.

(which is pretty weird)

i'll take a link to that fix. I do have an assembly resolver floating around in my code.

I'll try to remember how to do a PR tomorrow. 🙂 Yeah, I know, I found it, and I messed with it and saw when it was called and it's not called for the assemblies that show up as missing. I was disappointed, I thought I'd found a fix. 🙂 Here's what I was looking at (the question goes in a different direction, but the answers are relevant): https://stackoverflow.com/questions/1373100/how-to-add-folder-to-assembly-search-path-at-runtime-in-net

thanks

Happy to help

i'm only now able to try to get caught up on this thread. Can someone summarize for me the core issue - I gather that it's a difference in assembly resolution/loading between Framework and Core. I'd like to reproduce it as I'm working on cljr.exe and puzzling over how to handle various source paths etc.

I gather this is highly relevant to what I'm working on so I'd like to see if I can solve some of the issues in the way I handle cljr.exe requirements.

Core issue is that if you try to access clojure from an assembly loaded by your main assembly, it can't find the other assemblies it depends on. If you look at the stack overflow @U45FQSBF1 posted, you'll see that .net has issues loading assemblies from loaded assemblies. He created a simple program accessing Clojure from C#, which worked fine, and then moved that to a separate dll, and it did not.

(repeatable in framework but not .net 6.0)

In my case there was also a side issue obscuring this, that occurs when the assembly calling C# is itself called from another program (I'm building a plugin). I'll be PRing a fix for that today.

yes, specifically, I wrote some sample code. Fails on Framework okay on .net6. Code is: Console application, calls a static method in class that is in another project. The other project is a library project. uses Clojure. Calls the api to grab a var. That is is.

I hope to dig into later today, but have to take care of some chores first.

There is a workaround that lets me proceed, but it isn't generally applicable or relevant to your tools project, but it does provide a clue of sorts: If you copy all the assemblies to the folder the .exe originally loaded from, it can find them. So the problem is it's not looking for them in the right place.

Well that explains why I haven't seen it. 🙂

Yeah, it's a specific configuration that you're only going to hit in larger projects, and especially something like a plugin.

It's interesting that Clojure.dll can be found, given that it blows up in code in there. However, the static init code in RT tries Assembly.LoadFile for the two spec.alpha dlls. That's what blows up.

Clojure.dll is directly referenced by its calling project, so they know where it is. The others are dynamically loaded.

Btw looking into this, I saw several places saying that LoadFile does not load dependencies but Load and LoadFrom do.

Apparently, just because they know where to find Clojure.dll doesn't mean they will look in that location for anything else. That's why I thought having ResolveAssembly look in the same place would fix it, but it never got called. Maybe that is because it used LoadFile?

I will do a local build and package it (once I remember how to source nuget packages locally) with LoadFile replaced by Load or LoadFrom and see if that solves the problem.

You may need to also add a bit to ResolveAssembly based on the StackOverflow I linked yesterday. Here's the code I was trying in the case that the existing code returned null, but it wasn't being called for those assemblies.

string folderPath = Path.GetDirectoryName(Assembly.GetExecutingAssembly().Location);
string assemblyPath = Path.Combine(folderPath, new AssemblyName(args.Name).Name + ".dll");
var ex = File.Exists(assemblyPath);
if (!ex) return null;
Assembly assembly = Assembly.LoadFrom(assemblyPath);
return assembly;

I'm thinking with LoadFile you told it where to look, so it didn't fire that event, but if you use one of the others, it may fire the event in order for you to tell it where to look. This apparently solved the issue for lots of people, based on comments on that SO post.

The problem with using Assembly.Load is that it requires a full assembly name, with version info and other goodies. I can do that, it just means I have to edit that information whenever we change versions. (If someone knows how to grab the version info from the metadata on the packages at compile time, let me know.) I tried the suggestion above about using Assembly.GetExecutingAssembly().Location. in my test case, that is the same as as AppDomain.CurrentDomain.BaseDirectory. Off to read through some of the other links.

There are likely several different problems we need to address. I'm working with project references. I think @U4LCNMMGS is working with her DLL (that uses Clojure) being dynamically loaded. (Yes?) First, dealing with the error I have for project references. The problem is that the spec dlls that are referenced in the Clojure nuget package are not being copied to the bin directory of the console app. I think this may be because they are netstandard2.0/2.1 and not net462. There are hints of this problem here: https://stackoverflow.com/questions/51155274/copy-all-dependencies-from-net-standard-libraries-to-net-framework-console-app My first guess is this was a very old document (more than 5 years!) and would not apply. However, I followed the advice in the referenced article (https://www.hanselman.com/blog/referencing-net-standard-assemblies-from-both-net-core-and-net-framework) and it worked! All the files got copied. Now this only works because I could recompile the main program. That won't solve the problem for @U4LCNMMGS, if I'm reading her situation properly. The main program is not hers. Her program is being loaded dynamically by the main program. So my question now is: the main program finds her DLL, and from there it must be finding Clojure.dll. How does it manage to find that? What it is the magic path we must look for? Can we find it from some information available from her DLL or Clojure.dll as it is running? I don't know what that is. I'm afraid the only real solution here may be to copy all the dlls for clojure. But let's start with: How does the main program find her DLL (it is doing dynamic loading AND find Clojure.DLL?

LoadFrom seems to only need a path name

Agree that LoadFrom only needs the path name. But I don't know the path.

But yes my stuff is dynamically loaded. It find Clojure.dll from the nuget package. I assume that is because my dll directly referenced it. The other Clojure dlls are in the same build directory but are not found. I don't know why. I have been assuming it is because they are dynamically loaded by Clojure.dll, but that is a guess. If your fix was too get the dlls in the right place, that doesn't solve it in my case unless the right place is where the to level exe lives

It sounds like for your sample program, that was the case? Or are they still in the referenced project?

When I build my plugin project, I have a separate directory with my stuff and all my references, including clojure. The host program dynamically loads my plugin. For C# dependencies in that folder that are directly referenced by my plugin, it all works. For clojure, Clojure.dll works, but the others IT depends on are not found.

If I copy all of it (my entire plugin directory contents, I've not yet tried to winnow that down to see which files are necessary) to the host program folder, where the .exe lives, then it can find them and it all works.

(Well, trying to call back to C# from clojure is having an issue, but that's probably my fault... my dev environment is down due to a license issue and I wasn't able to keep going on that part. sigh)

I wish I knew exactly how it loads assemblies directly referenced from nuget packages, because that is clearly working. Possibly it just is able to see the full assembly names and paths and all.

I think the use of Assembly.GetExecutingAssembly().Location might actually work. When I dynamically load a dll and inspect the assembly object for it, the Location is right. If I use that to locate the spec dlls in Clojure.dll, it might work. Having a problem at the moment getting it to bring in my modified local nuget package. Sigh.

Success. Console app. no references to anything. Library app that uses Clojure. Copy everything from the bin of the library into another directory that I can find from where Console.exe is. Code in Console.exe:

Assembly assy = Assembly.LoadFrom(Path.Combine("goodies", "ClassLibrary1.dll"));
            Type t = assy.GetType("ClassLibrary1.Class1");
            MethodInfo mi = t.GetMethod("GoForIt",BindingFlags.Static|BindingFlags.Public);
            var result = mi.Invoke(null,Array.Empty<object>());

Here we are in DoInit!!!!
AppDomain.CurrentDomain.BaseDirectory: C:\Users\dmill\source\repos\ConsoleApp3\bin\Debug\
Assembly.GetCallingAssembly().Location dir: C:\Users\dmill\source\repos\ConsoleApp3\bin\Debug\goodies
(+ 1 2 3) => 6

FANTASTIC! Can't wait to try it out. BTW, just checked, and I'm on the contributors list, so you can get that PR in there as well for the type scanning bug.

Will do that first.

Having finally had the chance to fully test this, I just wanted to confirm alpha3 does indeed fix all the problems mentioned in this thread. Thank you @U45FQSBF1!

🎉