Home Guides Glossary

Selecting Website Pages

The Control Crawling option of Search Engine Visibility lets you control what pages of your Website can be crawled, and generates the required file or tags.

From the Optimize tab, select Control Crawling to get started.

To Control Crawling for your website

Each tab lets you perform the indicated function and generates the necessary file. On each tab you can do the following:

  • Allow All — Lets crawlers access all of your Website's pages.
  • Block specific web pages and search engines — Under this tab, click Add new or modify existing rule to choose specific pages of your Website to block by selecting from a list of search engines that cannot crawl the pages.
  • Block All — Blocks all of your Website's pages from crawlers. NOTE: This is not recommended.

Once you have determined how you want to control crawling for your Website, click Get File. Click Create robots.txt or Create Meta Tag to generate the appropriate file. For more information on robots.txt and Meta Tags, see What's the difference between robots.txt and Meta Tags?

Domain Registration

Pay less for website domain names. Register your own .com, .net or .org for as low as $10.18 per year. We have everything you need to get online with your new domain.

Website Builder

For as little as $3.89 per month you can build your Website online with Website Builder using our easy to use professional templates. Play Video - Demo

Quick Shopping Cart

Build and run your own successful online store in minutes. You're just five easy steps away! Shopping Cart works with Google® and eBay® Play Video

Website Hosting

Everything needed to give your website the high-performance home it deserves.  Protect transactions and secure your customer's data with a SSL Certificate

Copyright © 2005 - 2017. All rights reserved. Privacy Policy