Ask HN: Can web scraping be the basis of a viable business model?
by fforflo on Hacker News.
I’m a data engineer at heart, and I never did or enjoyed front-end work. Having said that I always was happy to code and evolve crawlers and web scrapers. Now I’ve taken some time off from work and gigs and I’m working on a side-project I’ve been hacking for some time. Without getting into the details yet: it aims to make web data collection a little bit easier for non-devs. I’ll soon have an MVP and will start pitching to investors: aiming for an open-source business model (after a few months of stealth development) and eventually a typical SaaS offering for extra functionality. At this point I’m trying to consolidate and counter the steel-man counter-arguments I should expect from investors. The most obvious one: as one can imagine, the product it’s not magic and, after a certain point it does require some manual work from the customer, hence this is an aspect I should prepare for. I have done some preliminary analysis of the space of potential competitors (think import.io, Apify, Zyte/ScarpingHub, etc.) and described opportunities for differentiation. What I’m afraid of is getting sidetracked in a discussion of “um, this is web scraping and it’s hard to make a business on top of it”. I understand that there’s not much context now and one could easily say “well yeah, anything could be possible with a good team, product…”, but I’m reaching out to the HN community to gather some considerations, mental models and pointers, I may not think of myself at this point.