How our verification bots work, and how to control their access to your site
Kyxstart runs regulatory identity verification Know Your Business and Know Your Customer for financial institutions and regulated businesses. When one of our customers needs to verify a company, our software checks that company against public records: its own website, company registries, regulatory filings, and news.
Our bots exist only for that. Every request one of them makes is attributable to a specific verification of a specific named business, requested by a specific customer. We do not crawl the web broadly, we do not build a search index, and we do not collect training data.
Nothing our bots fetch is ever used to train,fine-tune, or otherwise improve a machine learning model. Pages we retrieve are attached as evidence to the one verification case that caused the request, along with a crypto graphic hash of the content so the record is auditable, and they are retained under that customer’s data-retention terms.
This is the main way we differ from the AI crawlers you may already be blocking. If you block us to keep your content out of a model, you are blocking something that was never going to a model.
Two bots, split by purpose. Each has its own robots.txt token and each can be controlled independently allowing one does not allow the other.
KyxstartBot is bounded by design andhard-capped in code: one host, at most 30 pages, at most 3 links deep, and roughly 60 seconds per crawl. It also waits at least one second between requests to thesame host longer if your Crawl-delay says so. It is not a whole-site archiver, andit cannot be pointed at your site indefinitely.
Neither bot fills in or submits forms, runs searches on your site, attempts to reach content behind a login or paywall, or tries to work around access controls.
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; KyxstartAgent/1.0; +https://kyxstart.com/bots
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; KyxstartBot/1.0; +https://kyxstart.com/bots
Match on the product token (KyxstartAgent, KyxstartBot), not the whole string. Tokens never change. We bump the version number when behavior changes; we do not rename a bot.
Both bots honor robots.txt. We have no “a human asked for it, so robots.txt does not apply” exception if you disallow us, we do not fetch, and the verification records the gap instead.
Block multi-page crawling but still allow a single evidence fetch:
User-agent: KyxstartBot
Disallow: /
Block both bots entirely:
User-agent: KyxstartAgent
Disallow: /
User-agent: KyxstartBot
Disallow: /
Slow a bot down instead of blocking it (seconds between requests):
User-agent: KyxstartBot
Crawl-delay: 1
Keep a bot out of one section only:
User-agent: KyxstartBot
Disallow: /private/
• Each token is independent. A rule for KyxstartBot does not apply to KyxstartAgent.
• Put robots.txt in the top-level directory of each subdomain you want covered robots.txt on example.com does not govern shop.example.com.
• We cache a site’s robots.txt for up to 1 hour, so a change takes effect within that window.
• Please do not block us by IP address alone. Blocking our addresses can also stop us reading your robots.txt, whichis what tells us your actual preference.
Anyone can copy a User-Agent string, so we let you verify our requests cryptographically. Requests from our bots can carry an RFC 9421 HTTP Message Signature using the Web Bot Auth profile:
Signature-Agent: "https://manage.<api-host>/.well-known/http-message-signatures-directory"
Signature-Input: sig1=("@authority" "@path" "signature-agent");created=...;expires=...;keyid="...";alg="ed25519";tag="web-bot-auth"
Signature: sig1=:<base64>:
To check one:
1. Read the Signature-Agent header it names our public key directory.
2. GET that URL. It returns a JSON Web Key Set of Ed25519 public keys.
3. Find the key whose kid equals the key id in Signature-Input (an RFC 7638 JWKthumbprint).
4. Rebuild the signature base from the covered components and verify the Ed25519 signature.
Signatures are short-lived (about 60 seconds) and are re-signed for each URL, including after a redirect so a signature cannot be replayed against a different host or path.
Cloudflare-protected sites can verify this at the edge without any work on your part.
For tooling that matches on addresses rather than signatures, we publish our crawler egress ranges as a JSON file:
https://kyxstart.com/bots.json
Same format Google, OpenAI and Apple use for their crawler lists, so existing tooling can consume it unchanged:
{
"creationTime": "2026-09-30T00:00:00Z",
"prefixes": [
{ "ipv4Prefix": "100.28.82.249/32" },
{ "ipv4Prefix": "44.198.48.62/32" }
]
}
One file covers both bots KyxstartAgent and KyxstartBot egress from the same infrastructure, so their address ranges are identical. Use the User-Agent token, not the address, to tell the two apart.
• Re-fetch periodically; creationTime changes when the ranges do. Do not hard-code the addresses.
• A request from one of these ranges is genuinely ours. A request claiming our User-Agent from outside them is not.
• Signature verification is the stronger check and needs no list-fetching at all an address range proves where a request came from, while the signature proves who sent it. Prefer the signature if your stack can verify one; use the IP list if it cannot.
• Please don’t block us by address alone (see above) it can also stop us reading your robots.txt, which is what tells us your actual preference.
Questions, or think one of our bots is misbehaving? Email bots@kyxstart.com.
Please write from an address on the domain you are asking about otherwise we cannot verify thatthe report concerns a site you control. Include the User-Agent you saw, a few example URLs, and timestamps, and we will trace it.
