The primary way you communicate with all robots is via the robots.txt file. This is a simple text file placed in your website’s root directory (e.g., https://yourdomain.com/robots.txt ). It acts like a "guest policy" for bots, telling them which areas of your site they are allowed or disallowed from scanning.
Never use Disallow: / unless you want to block your entire site from search engines. seorobots