An important aspect of Drupal SEO is the robots.txt file. Drupal 5 was the first version of Drupal that came with a robots.txt file, but it still needs some modifications.

Hiyas Just looking for opinions on robots.txt. Should I disallow items like newthread.php, newreply.php, profile.php etc., or should I just have no robots.txt and let the bots

Optimizing your WordPress robots.txt file will help prevent Google penalizing you for duplicate content and can also improve your search engine rankings.

User-agent: * Disallow: /ac.php. Disallow: /ae.php. Disallow: /album.php. Disallow: /ap.php. Disallow: /feeds/ Disallow: /p.php. Disallow: /photo_comments.php

User-agent: * Disallow: /search/. Allow: /

User-agent: * Disallow: /printer_friendly_story. Disallow: /projects/livestream # Sitemap: http://www.foxnews.com/google_news_news.xml. Sitemap: http://www.foxnews.com/google_news ...

The robots.txt file must be placed at the root of your domain (www.yourdomain.com/robots.txt). If you cannot put a robots.txt file up, read our exclusion policy.

User-agent: * Disallow: /search/ Disallow: /about/adops/ Disallow: /about/adops/hp/ Sitemap: http://www.sun-sentinel.com/sitemap.xml

Cam somebody who knows how to use a robot.txt file give me an example of what should be excluded to robots on your site. To help index the site better. I'm currently using this ...

User-agent: Googlebot Disallow: /* Disallow: /sitemap Disallow: /navigation*/ Disallow: /products/ User-Agent: * Disallow: /sitemap Disallow: /products/