The Control Crawling option of Search Engine Visibility lets you control what pages of your Website can be crawled, and generates the required file or tags.
From the Optimize tab, select Control Crawling to get started.
Each tab lets you perform the indicated function and generates the necessary file. On each tab you can do the following:
Once you have determined how you want to control crawling for your Website, click Get File. Click Create robots.txt or Create Meta Tag to generate the appropriate file. For more information on robots.txt and Meta Tags, see What's the difference between robots.txt and Meta Tags?