Clojurians Log v2

Scraping with Firefox or Chrome I run into occasional cryptic errors

Execution error (ExceptionInfo) at etaoin.api/running? (api.clj:2230).
WebDriver process exited unexpectedly with a value: 69

Execution error (ExceptionInfo) at etaoin.api/running? (api.clj:2230).
WebDriver process exited unexpectedly with a value: 1

The code: https://github.com/kuhumcst/regionh/blob/master/src/dk/cst/regionh.clj#L14-L29

I cant’ even make it past 200 pages scraped before I get this error and the whole things fails…

What's the recommended recovery strategy? Wrapping the e/with-headless-driver in a try-catch and retrying every time?

I don't know @simongray! But... have you tried with a single thread. Does that work? Also, might want to check for https://cljdoc.org/d/etaoin/etaoin/1.0.40/doc/developer-guide#_webdriver_processes. If you clone Etaoin, from project root, you can use bb drivers and bb drivers kill. Also, have not really looked at your code and use case, if you are doing simple scraping, do you need Etaoin? Would simple direct HTTP GETs work?

No, that is why is am using Etaoin. It is a JS heavy site.

Gotcha, had a vague memory and dug up: https://github.com/clj-commons/etaoin/issues/379, wild shot but maybe related.

I'm not sure how many Etaoin users launch concurrent webdriver sessions, so my current guess is that your issue might be related to that.

My recommendation would be to start with a single webdriver session, work out all kinks with that then, maybe try concurrent webdriver sessions.

Ok, will try that tomorrow

Cool, please keep us posted on your progress! We'll do our best to lend a hand if you remain stuck.