Converge Digital runs two small crawlers that visit websites on behalf of our clients. If you have seen either of the user agents below in your server logs, this page explains what it is and what it does.
ConvergeDigitalSEO-InternalLinks/1.0
Checks the internal links on websites we manage for Converge clients: it follows links between pages on the client’s own site to find broken links, redirect chains and pages that nothing links to, so we can fix them. It reads pages; it does not submit forms or log in.
User agent string: ConvergeDigitalSEO-InternalLinks/1.0
ConvergeContentBot/1.0
Runs content checks for Converge clients: it reads published pages and posts to confirm they are live, correctly titled and up to date, and to review them for our content work. It reads pages; it does not submit forms or log in.
User agent string: ConvergeContentBot/1.0
They respect robots.txt
Both crawlers read and obey robots.txt. To keep them off your site, or off part of it, add a rule for the user agent, for example:
User-agent: ConvergeDigitalSEO-InternalLinks
Disallow: /
User-agent: ConvergeContentBot
Disallow: /They also crawl at a modest rate so they do not put load on the sites they visit.
Questions or problems
If either crawler is causing a problem on your site, or you have a question about it, email dale@convergedigitalmarketing.com and we will look into it right away.