If we were really going to pull off a totally cloud-based crawler and keep it affordable, we had to think outside the box.
Size matters
Simply put, the internet is big. You'll get a different figure depending on who you ask, but many sources estimate the total amount of data on the internet is in the territory of multiple zettabytes. That is trillions of gigabytes. Of course, you're probably not going to be crawling all of that in one go, but it should put into perspective how a heavy crawl of a large ecommerce website can easily top 100 GB of data.
So how exactly can we afford to store all of that in the cloud without either wiping out a rainforest to build our own data centres or passing on eye-watering database costs to our users? We took things back to the drawing board and built an entire system which bundles our crawls into a custom, compressed, and queryable archive format, which only costs us a fraction of the raw data.
The numbers are the fun part: we achieved roughly 17x compression on a heavy crawl, into frames that can be decompressed in mere milliseconds. This makes it significantly cheaper to store and read crawls without having to pass on a silly bill to you. And it all happens automatically behind the scenes while you work on your crawls uninterrupted.
We're very excited to share more details with you about some of the other ways we've pushed the boundaries of cloud crawling, and we hope you will come along for the ride!
David Mortiboy
Co-Founder, Scuba