off-topic 2022-05-07 | Slack Archive

adi11:05:43

I don't know who needs to hear this, but I got bitten by grep's stdio buffering behaviour today. Frankly I'm astonished it did not bite me all this time! Here is https://www.perkin.org.uk/posts/how-to-fix-stdio-buffering.html. TL;DR, copied from the post: This will apparently do nothing (really, it's just buffering)...

$ tail -f /var/log/foo | grep -E -v 'some|stuff' | sed | awk

Using stdbuf -o0 fixes it (presumably with performance penalty, because of force-flushing every line to stdout).

$ tail -f /var/log/foo | stdbuf -o0 grep -E -v 'some|stuff' | sed | awk

apiology15:05:49

If MODE is 'L' the corresponding stream will be line buffered.  This
       option is invalid with standard input.

I’ve never used this before, but based on the man page, I think -o0 means flush with every byte. I wonder if -oL would be a bit faster and still do what you intend.

Cora (she/her)18:05:27

there's also unbuffer, a cli tool that is part of TCL expect

👀 1

Cora (she/her)18:05:50

https://linux.die.net/man/1/unbuffer

adi04:05:49

@U060KDHA6 thanks for the reference! Guess who read the blog post but not the manpage?

😂 1

apiology13:05:35

I’d guess it doesn’t make too much of a difference in practice, as the producer is probably producing things in larger chunks in most cases anyway.

👍 1

adi21:05:49

#Also sent to the channel

This minor snafu was on the way to getting a form of hot reload to work in the little static site generator I'm ~~yak-shaving~~ making for myself... And the reload is heating up! https://twitter.com/AdityaAthalye/status/1523776563241955328

adi21:05:49

replied to a thread:I don't know who needs to hear this, but I got bitten by `grep`'s stdio buffering behaviour today. Frankly I'm astonished it did not bite me all this time! Here is https://www.perkin.org.uk/posts/how-to-fix-stdio-buffering.html. TL;DR, copied from the post: This will apparently do nothing (really, it's just buffering)... $ tail -f /var/log/foo | grep -E -v 'some|stuff' | sed | awk Using `stdbuf -o0` fixes it (presumably with performance penalty, because of force-flushing every line to stdout). $ tail -f /var/log/foo | stdbuf -o0 grep -E -v 'some|stuff' | sed | awk

2022-05-07

Channels