For site owners
About the Site Checkup crawler
If your access logs contain entries with SiteKenshin/1.0, this page is for you. These requests come from users of the Site Checkup smartphone app checking a website for outdated software or misconfigured settings. This page explains what it fetches, what it does not do, and how to stop it.
- The user’s phone sends the same requests (GET and HEAD) as viewing public pages. Requests to all servers under the same domain are spaced at least 1 second apart.
- It does not submit forms or send anything that counts as an attack.
- You can stop it by specifying
User-agent: SiteKenshinin robots.txt.
User-Agent
Both the iPhone and Android apps send the following string.
Mozilla/5.0 (iPhone; CPU iPhone OS 18_0 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Mobile/15E148 SiteKenshin/1.0 (+https://teractor.co.jp/site-diagnosis/bot)
Requests come directly from the user’s phone. They are not sent from our servers, so the source IP address belongs to the user’s network.
Purpose
Site Checkup is an app that identifies the CMS, plugins, JavaScript libraries and server software a site uses, along with their versions, and tells site administrators about known vulnerabilities, end-of-life software and misconfigured settings. Before starting a check, users confirm that they are the administrator of the site or have been asked by the administrator to check it.
What it fetches
- Pages on the same registered domain, starting from the entered page and following links and sitemaps. This includes subdomains such as www and shop, up to 100 pages in total and 3 levels deep
- Files those pages load, such as JavaScript and CSS (including files from external sources)
/robots.txton each host visited, plus the sitemap and/.well-known/security.txton the starting host- A check of whether the starting host redirects from http to https (once)
- A nonexistent URL on the starting host,
/sitekenshin-check-<random>, to check the 404 page (once)
The app also uses the operating system on the user’s device to look up DNS records for the domain (TXT, MX, CAA and others).
How requests are sent
| Request types | GET and HEAD only |
|---|---|
| Interval | One at a time. Requests to hosts on the same registered domain (www, shop and so on) are sent at a fixed interval of one per second in total |
| Volume | Up to 100 pages per check, subdomains included. Downloads stop at 5MB per file and 100MB in total |
| Stopping | The crawl stops after 5 errors in a row |
What it does not do
- Submit forms
- Modify URL parameters or send attack strings
- Send POST, PUT, DELETE or similar requests, or scan ports
- Open multiple connections at once to add load
How to stop it
Add the following to robots.txt, and Site Checkup will not crawl your site.
User-agent: SiteKenshin Disallow: /
To exclude only part of the site, specify a path such as Disallow: /private/. If there is no User-agent: SiteKenshin group, the User-agent: * rules apply. robots.txt is read for each host. To stop crawling of a subdomain, add the rules to that subdomain’s robots.txt.
- robots.txt, the sitemap, security.txt, the http redirect check and the 404 check are fetched regardless of these settings.
The crawling is done directly from the user’s phone and does not pass through our servers. For that reason, we cannot stop a particular user’s crawling from our side. If you want to stop the crawling, please use robots.txt.
Contact
Send questions or comments about the crawler to the address below. Including the date and time, the source IP address and the URLs accessed will help us look into it.
Operated by TERACTOR LLC