solo pedirles un favorsote, pueden echarle un ojo a mi robots.txt
para saber si esta bien, si debo agregar/quitar/cambiar algo
por favor
gracias
robots.txt
Código:
User-agent: * Allow: / Disallow: Disallow: /access/ Disallow: /administracion/ Disallow: /ckeditor/ Disallow: /conexion_mysql/ Disallow: /css/ Disallow: /js/ Disallow: /source/ Sitemap: http://www.buzonxalapa.com/sitemap.xml User-agent: Googlebot # disallow all files ending with these extensions Disallow: /access/ Disallow: /administracion/ Disallow: /ckeditor/ Disallow: /conexion_mysql/ Disallow: /css/ Disallow: /js/ Disallow: /source/ User-agent: Googlebot-Image Disallow: Allow: /* # allow adsense bot on entire site User-agent: Mediapartners-Google* Allow: / Disallow: Disallow: /access/ Disallow: /administracion/ Disallow: /ckeditor/ Disallow: /conexion_mysql/ Disallow: /css/ Disallow: /js/ Disallow: /source/ User-agent: Slurp Allow: / Disallow: Disallow: /access/ Disallow: /administracion/ Disallow: /ckeditor/ Disallow: /conexion_mysql/ Disallow: /css/ Disallow: /js/ Disallow: /source/ User-agent: Scooter Allow: / Disallow: Disallow: /access/ Disallow: /administracion/ Disallow: /ckeditor/ Disallow: /conexion_mysql/ Disallow: /css/ Disallow: /js/ Disallow: /source/ User-agent: duggmirror Disallow: / User-agent: wget Disallow: / User-agent: MSIECrawler Disallow: / User-agent: WebCopier Disallow: / User-agent: HTTrack Disallow: / User-agent: Microsoft.URL.Control Disallow: / User-agent: libwww Disallow: / User-agent: noxtrumbot Crawl-delay: 30 User-agent: msnbot Crawl-delay: 30 User-agent: Slurp Crawl-delay: 30
