what does the bsm in https://github.com/jgpc42/insn/blob/048806480ec11e2e2482025ac13b48bbb2dfe4a8/test/insn/core_test.clj#L289 mean ?

nothing interesting, just a method name like any other

bootstrap method

from what I'm reading, static = static members, special = initializer / constructors; interface = invoking on interface, virtual = calling on class method invokedynamic = some type of optimization, where first call = does type checking, after that, some type of magic happens ? what is going on with invokedynamic ?

right but it's not a particular method name required by indy is what I meant

you could call it foo

invokespecial can be used for invoking super methods and private methods too FYI, not just constructors

Think of invokedynamic as an atom that contains a method pointer. The atom's initial contents are provided by calling a bootstrap method.

and don't think about it as an optimization

it's more of a late-binding method resolution mechanism

@ghadi that's a good explanation. I've been trying to develop better intuition for invokedynamic myself for quite some time...seems very few people really understand how it works.

Has anyone tried using Arcadia (for Unity3D) to teach programming? My daughter is bored to death by (+ 2 2), but she was excited when she used Scratch to create movies, so I'm thinking of exposing her to a real language via 3D engine.

Is the Clojurians Slack Log dead? Last entries seem to be from last year https://clojurians-log.clojureverse.org/clojure/index.html

@ikitommi I think I remember somebody saying that the logging part is still logging, but the front end for viewing the log is broken and someone is working on fixing it. I don’t remember any more detail than that tho.

thanks. was looking for a old comment and got worried.

Hey, has anyone set up onyx? the only thing I can find is https://github.com/onyx-platform/onyx-benchmark/blob/master/ansible/roles/onyx_peers/tasks/onyx.yml which seems a bit dated

On two (only two) of my production server with ring I keep getting these logs about illeagal charactes

2018-02-17 12:22:03.486:WARN:oejh.HttpParser:qtp2141841267-16: Illegal character 0x16 in state=START for buffer HeapByteBuffer@29262d30[p=1,l=295,c=8192,r=294]={\x16<<<\x03\x01\x01"\x01\x00\x01\x1e\x03\x03\xFb\xA7\x19\xE4\xAe\xD1\xC7...\x03\x02\x03\x03\x02\x01\x02\x02\x02\x03\x00\x0f\x00\x01\x01>>>tf-8;q=0.7,*;q=0....\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00}
2018-02-17 12:22:03.486:WARN:oejh.HttpParser:qtp2141841267-16: badMessage: 400 Illegal character

that looks like it's on the http server level, that's nothing ring implements directly

I bet if you search for your http server plus "Illegal character" you'll find something

just googling the class and error, looks like somebody is sending something to jetty that isn't http https://bugs.eclipse.org/bugs/show_bug.cgi?id=471081 - are you in fact using jetty?

Yes, started the server with lein ring server-headless gave it handler and port

as an aside, lein in production is best avoided, if you must use it at least override the -server flag to the jvm - lein is designed in many ways to explicitly be a dev tool

basically that along with

# localhost
iptables -t nat -I OUTPUT -p tcp -d 127.0.0.1 --dport 80 -j REDIRECT --to-ports 8440
# external
iptables -t nat -I PREROUTING -p tcp --dport 80 -j REDIRECT --to-ports 8440

also jetty isn't the best ring-compatible server for most uses

oops had some connection problems...

hmm ok

tomcat?

aleph is good, or you can use a container with an uberwar like tomcat

(but usually if you are using tomcat it's a side-effect of using eg. elasticbeanstalk on aws)

aleph, have a look into that, I took it thus far for granted that ring and jetty were inseperable.

and run it in a jar, no excuse not to compile it, not much traffic so I thought, meh.

anyway, regarding the actual error you saw, it looks like jetty gives that error when it gets data that isn't valid for http, one cause of this can be mix ups with https vs. http traffic iirc

If I'm reading that post on http://bugs.eclipse.org properly, it looks like it will get that error if https traffic is directed to its http port

ok, this only happens in production, I had other webservice once where these charset errors were more annoying, complaining especially much about swedish characters, whatever the log was meaning with that. But Ill try aleph and see if it behaves better, but if its https redirecting to http then, its totally harmless.

I’m running out of memory when loading in around 1 million nested maps. The maps share a lot of the same keys. Is this kind of duplication optimized away in Clojure? I saw that only a recent version of the JVM 8 u20 deduplicates strings.

@hlolli Just as a data point, we saw a number of strange thread-related problems with Jetty/Ring so we switched to http-kit/Ring and we've been very happy with that combination in production. The downside is that New Relic doesn't work as well with http-kit because it can't hook into the HTTP request/response points (but they're looking into that). So we have "Non-web transactions" from http-kit instead of "Web transactions". Annoying but livable. /cc @noisesmith

@grav What are you JVM heap settings? Also, are the keys keywords or strings? How big are each of the maps? On first glance, I'd expect having 1M hash maps with non-trivial amounts of data all in memory to eat up a lot of heap... You also probably need to ensure you're not leaking memory by using a profiler.

IIRC keywords used to be interned, and they changed that at some point

if you build maps by calling conj or assoc on existing maps you'll get structural sharing that will help keep heap size down, but this wont' happen if you are reading the maps (or generating them) one by one

if some other clojure process is producing some of the maps, and then a new one consumes them, using cognitect/transit for serialization ensures that you can save memory by sharing structure

as long as the two maps sharing structure are sent in the same message

I am reading them one by one. I might not have to have everything in memory at the same time. Just trying to calculate a rough estimate if it is at all possible, before I start moving away from the current naive approach

@seancorfield

$ java -XX:+PrintFlagsFinal -version | grep HeapSize
    uintx ErgoHeapSizeLimit                         = 0                                   {product}
    uintx HeapSizePerGCThread                       = 87241520                            {product}
    uintx InitialHeapSize                          := 2147483648                          {product}
    uintx LargePageHeapSizeThreshold                = 134217728                           {product}
    uintx MaxHeapSize                              := 32210157568                         {product}

Keys are keywords. I haven’t got a stat on the map size. I can process 100k entities, so I’ll see if I can generate a stat from that

@noisesmith The data is in ndjson format, that is, one json object per line in a file. So there’s not structural sharing in the serialized format.

Am I reading those numbers right that you're starting with a 2GB heap and the max is 32GB and you're still getting Out of Memory exceptions? How much physical memory does the machine have that you're running this on?

Also, is genuinely an Out of Memory exception, or is it Unable to create native thread (which also claims to be an Out of Memory condition)?

@grav OK - I was just saying you could use less memory if you used a different message format, there's no automatic structural sharing

`(= (type (float-array [0])) (type (float-array [1])))` <-- is currently true; is this always guarnateed to be true ?

keywords are still interned @noisesmith

@ghadi I think I misremembered and the thing I was thinking of was the weak references (preventing the bug where user turns every random string into a keyword and runs out of heap)

I think maybe symbols were also in the past... Maybe that's what changed

@grav post your reading code if possible

could be a simple list head retention issue

@seancorfield I’m pretty sure it has 256 gigs. I haven’t got access to it right now. I guess it should be possible to boost the heap then. I believe the error was something in the lines of GC Out of memory.

If you don't already, I'd recommend adding the options to print verbose GC stats to a separate log file, just to get a bit more insight into how heap usage is.

@ghadi Not at the moment, but I’m keeping everything in memory because I’m looking for duplicates of various sorts

We run most of our processes with verbose GC logging

base_jvm_opts="${base_jvm_opts} -XX:+UseG1GC -XX:MaxGCPauseMillis=100 -XX:+PrintGCDetails -XX:+PrintGCTimeStamps "
                base_jvm_opts="${base_jvm_opts} -verbose:gc -Xloggc:${base}/ws-${program}-gc.log"

@seancorfield good tip, haven’t done any real profiling yet except for the processing time

(we rarely need to look at it, but it can be very useful when you do need it! 🙂 )