https://news.ycombinator.com/item?id=8072730 <-- is fascinating; looking at old projects, I tend to run into the wall at 1-2 k, and have never exceeded 10k in any project

the project ive been working on for the past 4 years is roughly 40k lines

but going through it with xargs now im happy to report that only one file is >1k

what’s helped is rigidly sticking to the pipes and filters design pattern for it. While it creates an interesting problem for the meta of the system (setting up and configuration, etc) it ensures that components for the most part only add linear complexity

I worked at DreamWorks throughout that period including the development of the successor to that Lighting program. [clarification: I was in management when all this happened. I didn’t write any of that code.] Perhaps of interest to this community is that the PDI/DreamWorks tools rested on an ascii scene description format and a proprietary scripting language for scene evaluation. A sort of DSL for worlds. I don’t think any of that was published.

The successor toolchain was rewritten in C++, architected where it made sense to be highly parallel dependency graph evaluations, and optimized on vector (AVX2, AVX512) architectures. The new lighting tool able to distribute interactive evaluations of complex datasets across multiple machines. The new animation tool received a Oscar at the “SciTech” awards session held separately from the red carpet one.

fascinating! did you work with / know the person who wrote that blog post? 🙂

Yes, but not very well. Lawrence, Dan, Ted, and Drew all worked at the SF studio and I was/am in Los Angeles. The combined R&D and systems staff at the time was ~300 engineers.

https://www.youtube.com/watch?v=8S1wqaB_l0M

@souenzzo looks like normal memory usage over time - what’s the issue?

Oh, sorry. I forgot to give the description @noisesmith The application hangs and start that sawtooth I think that the problem is on a recursive path from #specter. (6000 x recursive path on a structure, in this case). I changed to 1 recursive path, then another part of my app hangs. But now I know where is the problem 😅