Almost certainly not. Google’s own guidance says crawl budget is an advanced topic for very large or fast-changing sites, and that if your new pages tend to be crawled the same day you publish them, you don’t need to read it. For a typical small business site with tens or a few hundred pages, “crawl budget” is rarely the reason a page isn’t showing in Google. The real cause is usually something more fixable: a blocked page, a noindex tag, weak internal links, an unreliable server, or a page Google doesn’t think is worth indexing yet.
Here’s what crawl budget actually is, the few cases where a smaller site should care, and what to check instead.
What “crawl budget” means
Google can’t crawl every URL on the web as often as it would like, so it limits how much time it spends on each site. Google’s crawl budget guide (last updated 22 July 2026, checked 10 November 2026) explains it as two things working together:
- Crawl capacity: how much Google can crawl without overloading your server. If your site responds quickly and reliably, this can rise. If it slows down or returns server errors (5xx) or “too many requests” (429), Google crawls less.
- Crawl demand: how much Google wants to crawl. That depends on things like how many URLs it knows about on your site, how popular they are, and how often they change.
Put together, crawl budget is the set of URLs Google can and wants to crawl.
Who Google says the guide is for
Google is unusually direct here. Its guide says it’s aimed mainly at:
- large sites with 1 million+ unique pages that change about weekly
- medium or larger sites with 10,000+ unique pages that change daily
- sites where a large share of URLs sit under “Discovered – currently not indexed” in Search Console.
Google calls those numbers rough estimates, not exact thresholds. Its Crawl Stats report help page (checked 10 November 2026) is blunter still: if your site has fewer than a thousand pages, you shouldn’t need that report or to worry about crawling at that level of detail. And the crawl budget guide itself says that, for Google Search, keeping your sitemap up to date and checking the Page indexing report regularly is enough for sites outside its target group.
Problems that look like crawl budget, but aren’t
If a page isn’t appearing in Google, it’s tempting to blame crawl budget because it sounds technical and out of your hands. These are the usual real culprits on small sites:
| What you see | More likely cause | Where to check |
|---|---|---|
| A new page isn’t in Google after a few weeks | No internal links point to it, or it’s missing from your sitemap | Your menus and related pages; the Sitemaps report |
| “Excluded by ‘noindex’ tag” | A plugin or theme setting is hiding the page | URL Inspection in Search Console |
| “Blocked by robots.txt” | A robots.txt rule left over from development | robots.txt report in Search Console |
| “Discovered – currently not indexed” on a handful of pages | Google found the URL but hasn’t prioritised it; often thin, duplicate or poorly linked pages | Page indexing report; the page itself |
| “Crawled – currently not indexed” | Google crawled it and chose not to index it, usually a quality or duplication question rather than a crawling one | The page’s content and its near-duplicates |
| Many pages dropped at once | Server errors, a broken robots.txt, or a site change | Crawl Stats “Host status”; your hosting status |
Our guide to checking whether a page is indexed walks through the URL Inspection steps for each of these.
When a smaller site should pay attention
Even without thousands of pages, a few situations can create far more URLs than you meant to have:
- Filters and sorting on a shop or listing page. Colour, size, price and sort order can combine into thousands of near-identical URLs. Google’s guide lists differently sorted versions of the same page as a classic example of crawling waste.
- Calendar or booking pages that generate a new URL for every date, stretching endlessly into the future.
- Site search results that are linked to and crawlable.
- Tracking or session parameters added to internal links.
- Long redirect chains after several redesigns. Google says these have a negative effect on crawling.
If you have any of these, it’s worth a look, not because your “budget” is running out, but because Google may spend its visits on junk URLs instead of your service pages. If your blog or gallery uses “load more” buttons, our guide to pagination and load more covers the related problem of pages Google can’t reach at all.
What Google recommends (the parts that apply to small sites)
From Google’s crawl budget guide, these are the habits worth keeping regardless of size:
- Keep one version of each page. Consolidate duplicates rather than letting several URLs show the same content.
- Return a real 404 or 410 for pages that are gone. Google says a 404 is a strong signal not to crawl that URL again. Avoid “soft 404s”, where a missing page shows a normal-looking page with a 200 status.
- Keep your sitemap current, listing only the pages you want in Google, with honest last-modified dates.
- Avoid redirect chains. Point old URLs straight to their final destination.
- Keep the site fast and reliable. Slow responses and server errors make Google back off.
Two things Google specifically says not to do: don’t use noindex to save crawling (Google still has to fetch the page to see the tag), and don’t use robots.txt to temporarily shift crawling to other pages. Google won’t reassign that crawling unless your site is already hitting its capacity limit.
The one thing that can stop crawling entirely: robots.txt errors
For most small sites, this is more worth knowing about than crawl budget. Your site doesn’t need a robots.txt file, and a “not found” (404) response is fine. But if the robots.txt address returns a server error (5xx) or “too many requests” (429), Google’s Crawl Stats help says it will stop crawling your site for the first 12 hours and keep retrying, then rely on the last good copy for up to 30 days.
So if your host has an outage, or a firewall or rate limit starts returning errors for that one file, crawling can pause without anything looking wrong to visitors. If pages suddenly stop updating in Google, check that yoursite.com/robots.txt loads.
A hypothetical example
This is an invented example. A bakery’s 40-page website adds an online order page with a date picker. Each date creates its own URL, linked from the page, running two years ahead. Search Console soon shows hundreds of “Discovered – currently not indexed” URLs, and the owner reads that this means a crawl budget problem.
The fix isn’t to chase more crawl budget. It’s to stop the date picker creating crawlable links (or keep those URLs out of the sitemap and consolidate them), make sure the 40 real pages are linked from the menu and listed in the sitemap, and then let Google settle. The real pages were never short of crawling; they were sharing the site with noise.
A five-minute check for your site
- In Search Console, open Pages (Page indexing). Are the pages you care about indexed?
- Open Sitemaps. Is your sitemap submitted and read successfully?
- Open Settings > Crawl stats and look at Host status. Green means no significant availability issues in the last 90 days.
- Count your URLs roughly: does Google know about far more URLs than you have real pages? If so, look for filters, calendars, site search or parameters.
- Load
/robots.txtin a browser to make sure it returns either a file or a plain “not found”.
If all five look fine, crawl budget isn’t your problem. Spend the time on the pages themselves.
What’s in your control, and what isn’t
In your control: keeping duplicate and junk URLs to a minimum, linking clearly to your important pages, maintaining an accurate sitemap, fixing server errors and redirect chains, and keeping robots.txt reachable.
Not in your control: how often Google chooses to crawl your site, and whether it indexes or ranks any page. You can’t ask Google to raise your crawl rate, and crawling a page doesn’t guarantee it will be indexed or shown.