AI Crawler Policies of 8 Korean Metropolitan Governments — From Fully Open to Path Whitelists

Why Metropolitan Government Homepages

In an era when citizens ask AI "where do I file a move-in report?", a government homepage is the source of official information. Whether AI can read that source depends on robots.txt and how the first page responds. We checked all 8 special and metropolitan city governments (Seoul, Busan, Daegu, Incheon, Gwangju, Daejeon, Ulsan, Sejong). The 8 are not a sample; they are the whole of that administrative category. This article does not rank anyone. It covers only the structural picture visible in public responses and what to improve. Measured on October 20, 2026; settings may have changed since.

AI Crawler Policy Fell Into Five Types

We classified policy against 17 AI crawlers (training, search, user-request, and control tokens), which we covered in our earlier post.

The first finding is that policy has not converged on one approach. That does not say which is right; it says intent and inertia have accumulated differently at each institution. A full block is the strongest form of policy here, but public responses alone cannot tell whether it was a deliberate decision or a leftover security setting. So this article does not judge whether any policy is right or wrong.

A Whitelist Is Not a Full Block — Where We Updated Our Tool for a New Pattern

Two sites looked like this.

User-agent: * / Disallow: / / Allow: /main / Allow: /news / Allow: /citizen ...

The standard (RFC 9309) prefers the longest matching rule, so this opens only the allowed paths rather than blocking everything. The earlier logic of our diagnostic tool (AEGIS Insight) judged this as "all 17 AI crawlers fully blocked." As the kinds of AI crawlers and the ways sites use robots.txt change quickly, this was a gap we missed when we first met the public-sector whitelist pattern at scale. We found it while checking government sites, rebuilt the logic around longest-match, and made it distinguish fully open, partially open, and fully blocked. Keeping measurement from falling behind the times, and disclosing what we missed rather than hiding it, is a principle of this series.

With a whitelist, the key check is whether paths for areas citizens often ask AI about — civil services, welfare, transport, tourism — are on the list.

When robots.txt Cannot Be Read

Two sites could not be judged at all. One returned 401 (authentication required) for the robots.txt request, and one had no file (404). A missing file means no restrictions under the standard, but the policy is undocumented, so intent is unknown. A 401 leaves crawlers unable to read the rules and needs its own check.

A Second Update — The Host After a Redirect

The site classified as a full block was first recorded as "robots.txt missing (404)." Its root address redirects to an authentication (SSO) server, and our tool looked for robots.txt on the host it was redirected to. The site's own policy was the opposite. We noticed the mismatch while writing, fixed the tool to check the host of the address the user entered first, and re-measured that institution. When a diagnostic's judgment cannot follow the variety of real-world structures, it can lead to wrong statements about a public body, so we fixed and re-measured as soon as we found it.

First Pages That Are Empty

Counting body words on the root URLs, two sites stood out.

This is the same structure as the "blank page for crawlers" problem in our prerendering post. People see a normal page, but crawlers that do not run JavaScript may never reach the body. Search grounding can compensate, so scores are less affected, but it is still the entrance to the information AI reads directly.

Five Improvements

The Measurement Tool Keeps Updating

The AI search environment changes quickly. Crawler types multiply and sites configure themselves differently. If a measurement tool cannot keep up, it ends up scoring against outdated standards. We keep updating the judgment logic of the AEGIS platform for every case we meet in the field so it does not fall behind the times.

This list will keep growing. Updating the logic every time we meet a new pattern, and publishing the history, is how we believe a measurement tool avoids going out of date.

Limits of This Check

This is one set of public responses from a Korean environment on October 20, 2026. robots.txt can change at any time, and some sites respond differently depending on User-Agent or IP. The 8 are the whole of their category but we do not generalize statistically. We did not judge whether any policy is right.

Next

The next post covers the structure scores (SEO, AEO, GEO) for the same 8, schema and sitemap status, and what to watch for when reading scores. You can diagnose your own institution with AEGIS Insight.