User-agent: SISTRIX Crawler Disallow: / User-agent: SISTRIX Disallow: / user-agent: AhrefsBot Disallow: / User-agent: Baiduspider Disallow: / User-agent: Yandex Disallow: / User-agent: netEstate NE Crawler Disallow: / User-agent: spbot Disallow: / User-agent: msnbot Crawl-delay: 1 User-agent: serpstatbot Disallow: / User-agent: PetalBot Disallow: / User-agent: Seekport Crawler Disallow: / User-agent: facebookexternalhit crawl-delay: 20 User-agent: facebookexternalhit/1.0 (+http://www.facebook.com/externalhit_uatext.php) crawl-delay: 20 User-agent: facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php) crawl-delay: 20 User-agent: facebookexternalhit/1.1 crawl-delay: 20