Original link to the post: https://thewebscraping.club/p/ikea-scraping-kallax

Good stuff, at least there's a sort of solution inside and not only content marketing

Thanks for sharing!

Thank you for this article, it confirms my direct experience. Never run a benchmarking test but I can see this every day.

Companies spend thousands of dollars on these anti-bot solutions and then they are so misconfigured that using a specific user agent or faking browsing via mobile, bypasses them. Real life stories.

Often this is because you are hamstrung by old mobile apps or TV apps that can't be updated forcibly and so you break users. So your making a trade-off of user pain and bot deflection. So many times this is actually known and on purpose. Botters hitting that loophole helps prioritize closing that loophole in an agile customer experience and makes it easier for engineering and product to prioritize. Real life stories

Just shared some of these stories here https://twitter.com/pigivinciguerra/status/15715437943480893...

Totally agree.

Thank you for sharing my post, appreciated

I’ve got several years of experience of webscraping, mainly in python. Scrapy is the first choice for “basic websites” while playwright is used then things get difficult. I’m collecting my experience in using these tools in this “web scraping open knowledge project” on github (https://github.com/reanalytics-databoutique/webscraping-open...) and on my substack (http://thewebscraping.club/) for longer free content

Some high level principles for scaling a web data company

Yes, it's another post about web scraping