obbeel@lemmy.eco.brtoPiracy: ꜱᴀɪʟ ᴛʜᴇ ʜɪɢʜ ꜱᴇᴀꜱ@lemmy.dbzer0.com•How do I use Open Source scrapers? (Selenium, Scrapy, etc.)English
9·
5 days agoIt needs a driver and the web-browser to be executed in headless mode. For Chrome that’s chrome-driver. You can get it here.
To make a script for it, I recommend talking to a LLM. I have asked it to build scrapers before, so it does the job.
If you want a practical use of Selenium being demonstrated, you can see it in LucidWebSearch plugin for Oobabooga.
Gemini, Perplexity, Poe. Creating a Selenium script isn’t that hard for them. You can try running your own, but it’s more less likely that it will produce good results. Best coder LLM I’ve seen out there for hosting is Yi Coder 9B.