Small Web Index Bot Small Web Index BotSmall Web Index Bot is polite and won't bother your server with repeated requests.
It may try to read once all html pages allowed by your robots.txt
It will also read robots.txt once per week to make sure already crawled data doesn't violate the new version.
If your website falls into the "Small Web" category, the information will be used only to add it to our index and make it discoverable for small web users.
Small Web Index will never use your website data to derive and present it as our own. The index will provide URL of your website with the keywords and important quotes and never full content.
Your website URL was discovered by our bot and first it tried to identify if it satisfies the "Small Web" criteria.
If true, it will slowly read more pages from your website following robots.txt.
It's understandable and totally fine. It may not be aligned with your vision.
Start by adding this to your website robots.txt
User-agent: Smallwebindexbot
Disallow: /
The bot will scan robots.txt within a week and will remove your website from this index.