Engineering

How we optimised crawls for the cloud

Written by David Mortiboy

If we were really going to pull off a totally cloud-based crawler and keep it affordable, we had to think outside the box.

Size matters

Simply put, the internet is big. You'll get a different figure depending on who you ask, but many sources estimate the total amount of data on the internet is in the territory of multiple zettabytes. That is trillions of gigabytes. Of course, you're probably not going to be crawling all of that in one go, but it should put into perspective how a heavy crawl of a large ecommerce website can easily top 100 GB of data.

So how exactly can we afford to store all of that in the cloud without either wiping out a rainforest to build our own data centres or passing on eye-watering database costs to our users? We took things back to the drawing board and built an entire system which bundles our crawls into a custom, compressed, and queryable archive format, which only costs us a fraction of the raw data.

A hand-drawn diagram: big thing goes in, do some science, small thing comes out

The numbers are the fun part: we achieved roughly 17x compression on a heavy crawl, into frames that can be decompressed in mere milliseconds. This makes it significantly cheaper to store and read crawls without having to pass on a silly bill to you. And it all happens automatically behind the scenes while you work on your crawls uninterrupted.

We're very excited to share more details with you about some of the other ways we've pushed the boundaries of cloud crawling, and we hope you will come along for the ride!

David Mortiboy

Co-Founder, Scuba

GET STARTED

Crawl the lot,
Melt nothing

Cloud crawls on sites of any size, from your first 500 URLs to your millionth. Nothing to install, no fans screaming, no waiting around.

Launching 9 December 2026

Launching 9 December 2026