Public endpoints on search.ai.cloudflare.com
The generated hostname is the starting point, not the end state. It gives you a working public endpoint without any extra setup, which is convenient until the URL starts escaping into places it should not. Logs, browser history, client code, and half-finished integrations have a habit of keeping things alive long after they should have been replaced.
When the endpoint stays public, treat it as exposed by design. That means the query surface needs the same discipline as any other unauthenticated API. If it is meant for open use, keep the access pattern simple and accept that the URL itself is part of the attack surface.
Move the endpoint onto a zone you own
Moving the endpoint onto a custom domain gives you control over the hostname and the surrounding DNS. A public endpoint can live on a zone you own, such as search.example.com, while keeping the same basic query behaviour. The endpoint does not become private just because the hostname looks more respectable, but it does become easier to place behind your own controls.
That matters when the endpoint is wired into browser code, server-side jobs, or agent flows that you expect to keep for a while. A custom domain gives you a stable place to hang controls, logs, and policy. It also keeps the Cloudflare-generated hostname out of places where it does not need to be seen.
Keep the old hostname in mind for anything already wired to it
The old hostname does not disappear just because a better one exists. Anything already pointed at the default endpoint will keep using it until you move it. That sounds obvious, which is usually how the mess starts.
If you switch hostnames, plan for stale config, cached URLs, and older clients that still call the old path. For a while, both endpoints may need to exist. If one of them is public and the other is gated, be clear which one is meant to survive.
Putting Cloudflare Access in front of AI Search
Cloudflare Access fits in front of a custom-domain endpoint and gives you a real gate instead of a polite request. For machine callers, that gate can be a service token. For people in a browser, the same endpoint can use identity provider sign-in.
That split is useful because the access pattern is usually different. Agents need something they can send without a login page. Humans need a browser flow that does not turn every visit into a token-passing exercise.
Service tokens for agents and browser sign-in for people
Service tokens are the practical choice for non-interactive callers. They are awkward enough to discourage casual misuse and simple enough to fit into automation. Browser users can authenticate through an identity provider without needing a separate public endpoint just for humans.
The useful boundary is not complicated: agents get tokens, people get sign-in. If both groups hit the same endpoint, the policy needs to make that distinction cleanly. A single unauthenticated route for everything is how a convenience layer turns into a liability.
Decide which endpoints stay public and which ones do not
Not every endpoint deserves the same treatment. Some routes can stay public if they are meant for open access and the contents are not sensitive. Others should be gated from the start, especially if they expose private content, internal documentation, or query paths that can be abused at scale.
The split often lands between discovery and consumption. Public search can be useful. Public access to a broader chat or MCP surface may not be. If the endpoint can be called directly, it should have a reason to exist in that state.
Namespace endpoints and discover crawling change the shape of the index
Namespace endpoints shift the model from a single source to a set of selected instances. The endpoint can expose /search, /chat/completions, and /mcp paths, and the request can select which instances to query through ai_search_options.instance_ids. That gives you one public entry point with a narrower index scope behind it.
It is a cleaner fit when related content lives in separate instances, such as docs and support. The query path stays simple while the routing happens in the request body. In practice, that is easier to manage than wiring a separate hostname for every slice of content.
Fan out queries across selected instances with aisearchoptions
ai_search_options.instance_ids lets a request fan out to selected instances instead of hitting everything. If a query only needs docs and support, there is no good reason to search the rest of the estate. Narrowing the query set is not just tidier, it cuts down on irrelevant results and avoids mixing unrelated content into the answer set.
That routing choice matters when content quality differs between instances. A broad query across a messy index can return a confident-looking pile of junk. Selecting the right instances keeps the response closer to the source material you actually want.
Use discover parse type when the sitemap is not the whole site
A sitemap is useful until it is incomplete, stale, or both. The discover parse type starts from the source URL and follows links it finds, instead of trusting the sitemap to be the whole truth. For sites with partial or awkward sitemaps, that is often the difference between a thin index and one that actually covers the site.
The crawler can combine sitemap pages with discovered links, which is the part that matters operationally. If the site has hidden corners, nested docs, or pages that only show up through navigation, discover crawling gives the index a better chance of finding them. The crawl settings can go deeper and wider too, with limits such as a depth of 3 and a link limit of 5000, but more crawling is not the same as better content. It just finds more of what is already there.
