Which Malaysian websites block AI crawlers? I checked 100

Key takeaways

• On 20 October 2026, 13 of 100 major Malaysian websites blocked at least one of 17 AI crawlers in their robots.txt. None blocked all 17.
• News sites block the most: 5 of 15, among them the New Straits Times, Harian Metro and Bernama. None of the banks, telcos, travel sites or universities whose files I could read blocks any AI crawler.
• OpenAI's GPTBot is the most blocked crawler (9 sites), then CCBot and Bytespider (8 each) and Anthropic's ClaudeBot (7). Google-Extended, the token that keeps pages out of Google's AI models, is blocked by only 4.
• Most blocks are against training, not search: 10 of the 13 sites block a training crawler while leaving OpenAI's and Perplexity's search crawlers in.
• Two of the 13 blocking files are the same file, byte for byte: a block of rules marked "Cloudflare Managed content", on KWSP and EdgeProp.

13 of 100 block an AI crawler, and none blocks them all

Of the 100 major Malaysian websites I checked on 20 October 2026, 13 tell at least one AI crawler to stay out, and most of the rest say nothing about AI at all.

"Block" here means the file tells that crawler not to fetch the home page, "/", which keeps it out of the whole site unless other paths are opened for it. I counted 17 AI crawlers, from the list of 20 my free checker uses. The other three, Googlebot, Bingbot and Applebot, are search engine crawlers, and no site in the sample blocked Googlebot.

Most big Malaysian sites have not written down a choice about AI. Only 18 of the 78 files I could read name an AI crawler at all. The other 60 do not mention AI at all.

A larger survey points the same way. Zyte, a web data company, reports in its State of Web Access 2026 that 13.4% of the 1,106 Malaysian websites it looked at block AI crawlers, and 78.2% have a robots.txt. My sample is smaller, but every site in it is named.

News sites block the most

News is the only sector where blocking AI crawlers is common: 5 of the 15 news sites I checked block at least one.

By language, 4 of the 8 English news sites block an AI crawler: the New Straits Times, Free Malaysia Today, The Edge Malaysia and Bernama. 1 of the 4 Malay sites does, Harian Metro; Berita Harian refused my request, so I cannot say. None of the 3 Chinese papers, Sin Chew Daily, China Press and Oriental Daily, blocks any. The Star blocks one bot outside my list of 17: Diffbot, which extracts data from web pages ("user-agent: Diffbot", then "Disallow: /").

Outside news, blocking is rare and scattered: KWSP and Bank Negara Malaysia in government, Zalora and Lotus's in shopping, Maukerja and Ricebowl in jobs, EdgeProp in property, and foodpanda. Banks are the least readable sector: only 5 of 10 served a robots.txt my script could read.

Which AI crawlers Malaysian websites block

Training crawlers are blocked far more often than the bots that fetch pages for AI answers. GPTBot, which collects pages to train OpenAI's models, is blocked by 9 of the 100 sites.

10 of the 13 blocking sites keep out at least one training crawler, GPTBot, ClaudeBot or CCBot, while leaving OAI-SearchBot and PerplexityBot free to come in. Only 4 sites block any bot that fetches a page for AI search or for a person's question.

Google's AI is blocked less often than OpenAI's. Google-Extended, the robots.txt token that says whether Google may use pages for its AI models, is blocked by 4 sites, under half the 9 that block GPTBot. The New Straits Times and Harian Metro block GPTBot and ClaudeBot but write "Allow: /" for Google-Extended.

What the 13 files say

Each line below is a fact from the site's own robots.txt on 20 October 2026, with rules quoted as written.

The New Straits Times and Harian Metro are the strictest, blocking 12 of the 17 AI crawlers. Their two files are the same except for one folder name. The search and assistant bots of OpenAI, Anthropic and Perplexity are blocked from the home page but allowed into one section: "Allow: /plus/" on the NST, "Allow: /hmextra/" on Harian Metro. Both files carry the line "Content-Signal: search=yes,ai-input=no,ai-train=no", and the NST's opens with a heading dated 16 October 2026.

Bernama blocks five: GPTBot, ChatGPT-User, ClaudeBot, Bytespider and CCBot. Its file gives the reason in Malay, in a comment: "Halang Bot AI / Scraper Agresif (Ringankan Beban Server)", block AI bots and aggressive scrapers to lighten the load on the server.

The Edge Malaysia has the shortest blocking file in the sample: four lines, 54 bytes, which shut out GPTBot alone and allow everything else.

Free Malaysia Today names 15 AI crawlers and lets most of them in by name, GPTBot, ClaudeBot, PerplexityBot and Google-Extended included. It blocks three: CCBot, Bytespider and Amazonbot.

KWSP and EdgeProp serve the same file, byte for byte. It is a block of rules marked "# BEGIN Cloudflare Managed content" that keeps out eight crawlers and tokens tied to AI training, GPTBot, ClaudeBot and Google-Extended among them, and says "Content-Signal: search=yes,ai-train=no,use=reference". Two of the 13 blocking sites, then, block AI with a file their CDN provides.

Bank Negara Malaysia blocks one AI crawler, Common Crawl's CCBot ("User-agent: ccbot"), at the end of a long list of SEO tools and other search engines' bots it keeps out.

Lotus's blocks GPTBot, ChatGPT-User, Google-Extended and CCBot. It also names "anthropic-ai" and "Claude-Web", older names for Anthropic's bots, but not ClaudeBot, the name Anthropic's crawler uses now. So ClaudeBot falls under the file's general rules, which let it in.

foodpanda blocks GPTBot and gives OAI-SearchBot and ChatGPT-User "Allow: /". That is the split OpenAI's own documentation describes: out of training, still in ChatGPT search.

Zalora blocks only Meta's training crawler: "User-agent: meta-externalagent", then "Disallow: *".

Maukerja and Ricebowl, two job sites with near-identical general rules, block different sets. Maukerja blocks GPTBot, ClaudeBot, Google-Extended, Amazonbot and Bytespider; Ricebowl blocks ClaudeBot, Meta-ExternalAgent and Bytespider.

One more is worth a line, though I count it as allowing. JobStreet Malaysia gives GPTBot, CCBot, Google-Extended, Bytespider and three other bots "Disallow: /companies" and "Disallow: */job/". AI training crawlers are kept off company pages and job ads, and the home page stays open.

The sites that say yes by name

Some Malaysian sites name AI crawlers only to let them in, and these files are as deliberate as the blocks.

• Hong Leong Bank has a section headed "# AI CRAWLERS" that gives "Allow: /" to GPTBot, ChatGPT-User, ClaudeBot, PerplexityBot and Meta's crawler, among others.
• paultan.org names 16 AI crawlers in one group and adds "Content-Signal: ai-train=yes, search=yes, ai-input=yes".
• Lazada Malaysia names OpenAI's three bots, GPTBot, OAI-SearchBot and ChatGPT-User, each with "Allow: /".

The 13 sites I could not read, and the 9 with no file

13 of the 100 sites did not let my script read their robots.txt, which is a finding in itself.

8 answered "403 Forbidden": TM, Berita Harian, the Ministry of Health, PG Mall, Watsons, PropertyGuru, Batik Air and McDonald's Malaysia. Most of these answers came from a service in front of the site: the response named Cloudflare on four, and Akamai, Microsoft Azure's gateway and Envoy on one each. A 403 tells me the site screens unknown bots before they reach any file. It does not tell me what its robots.txt says, or how it treats GPTBot itself.

4 did not answer. Maybank2u closed the connection, Public Bank did not reply within 10 seconds, MyKasih reset the connection, and for JPA my script could not verify the site's certificate chain, a gap a browser can often fill on its own. Indeed Malaysia sent my request through three redirects to a sign-in page.

9 sites have no robots.txt at all. AmBank, Bank Islam, FPX, MetMalaysia, Universiti Malaya, UKM, USM and Air Selangor answered 404, and Firefly's robots.txt address leads to its home page. Under the robots.txt standard, no file means every crawler may fetch everything, so I count these 9 as letting every AI crawler in.

How I checked

On 20 October 2026, from 16:31 to 16:34 Malaysia time, a small script made one request to each site: GET /robots.txt over HTTPS, following at most three redirects, waiting at most 10 seconds, one site a second. Its user agent named this study and linked to my checker. It fetched nothing else.

• The sample. 100 major Malaysian websites in ten sectors, chosen by me. 39 come from Similarweb's public Top Websites rankings for Malaysia for September 2026; the rest are names most Malaysians know in their sector. For regional brands I used the Malaysian site, such as shopee.com.my. It is not a list of the 100 most visited sites.
• The reading. The same parser my free AI crawler checker uses, which follows the robots.txt standard, RFC 9309: a crawler named in its own group follows that group, and the longest matching rule wins.
• The address. A first run that afternoon showed 16 sites whose bare domain does not serve the site; jpj.gov.my, for one, does not resolve, and the site is www.jpj.gov.my. I moved those 16 to the address each site lives on before the run I counted.
• The check. A second script, sharing no code with the first, read all 78 files again and recomputed every number in this post. It found 0 differences in 1,560 verdicts.

What this cannot see:

• robots.txt is a request, not a lock. A crawler can ignore it.
• A site can block AI crawlers at its firewall or CDN without a line in robots.txt. Those blocks are invisible here.
• I checked only the home page path. A site that keeps AI crawlers out of some sections, like JobStreet, counts as allowing.
• My script is not GPTBot. A site may answer the real crawler differently.
• It is one afternoon. Files change; the NST's carries a date four days before my check.

The result for each of the 100 sites is in this CSV file, free to reuse with a link to this page.

What a Malaysian business should do with its robots.txt

Most big Malaysian sites have not written down a choice about AI crawlers, and yours may not have either. Saying nothing lets every crawler in, which is a choice too. Five steps.

1. Decide two things separately. May AI companies train their models on your pages? May AI search engines read your pages to quote and link them? The answers can differ, and robots.txt can say both. On this site I let them all in, because an answer cannot quote a page its crawler never read; a month of AI crawler visits showed me what that brings and what it does not.
2. If you want to be found in AI answers, leave the search and assistant bots in. OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User and PerplexityBot fetch pages for answers that can link back to you. Blocking GPTBot alone keeps you out of OpenAI's training, not out of ChatGPT search; foodpanda's file does exactly that.
3. Use the names in use today. A block on "anthropic-ai" or "Claude-Web" does not stop ClaudeBot. To keep pages out of Google's AI models without leaving Google Search, the token is Google-Extended, not Googlebot.
4. Check what sits in front of your file. If your CDN or firewall answers bots with a 403, your robots.txt may never be read, and the crawlers you meant to welcome may be turned away. To see what a crawler really gets back, request a page with its name, as I did in my own check.
5. Read your file the way a crawler does. robots.txt is the file AI crawlers actually ask for: in the 30 days to 13 October they fetched mine 605 times and my llms.txt not once. My free AI crawler checker reads your robots.txt for 20 crawlers at once and shows which it allows and which it blocks.

FAQ

Do Malaysian websites block AI crawlers?

A few do. On 20 October 2026, 13 of 100 major Malaysian websites I checked blocked at least one AI crawler in robots.txt, and none blocked all of them. 74 let every AI crawler in, and 13 did not let my script read their file.

Which Malaysian news sites block GPTBot?

On 20 October 2026, the New Straits Times, Harian Metro, Bernama and The Edge Malaysia blocked GPTBot in robots.txt. Free Malaysia Today named GPTBot and allowed it, while blocking CCBot, Bytespider and Amazonbot. The Star, Malaysiakini, Malay Mail, Astro Awani, Sinar Harian, Utusan, Sin Chew Daily, China Press and Oriental Daily blocked none of the 17 AI crawlers in this study; The Star blocks Diffbot, a data extraction bot outside that list.

Does blocking GPTBot keep my website out of ChatGPT?

Not entirely. GPTBot collects pages to train OpenAI's models, while ChatGPT's search uses OAI-SearchBot and a page someone asks about is fetched by ChatGPT-User. A site that blocks only GPTBot can still appear in ChatGPT answers, which is how foodpanda Malaysia's robots.txt is set up.

How do I check which AI crawlers my website blocks?

Open yourdomain.com/robots.txt and look for user-agent lines that name AI bots, such as GPTBot, ClaudeBot, PerplexityBot or Google-Extended, followed by "Disallow: /". A free AI crawler checker reads the file for 20 crawlers at once. Then ask whoever runs your CDN or firewall whether it blocks AI bots, because that never shows in robots.txt.

Is robots.txt enough to stop AI crawlers?

No. robots.txt is a request that well-behaved crawlers follow, not a lock, and a crawler that ignores it can only be stopped at your firewall or CDN. Use robots.txt to state your choice, and the firewall for bots that do not listen.