Which websites block AI crawlers? Data from 1647 top sites
We checked the robots.txt of 1647 of the most visited websites for 18 AI crawlers. Here is who blocks ChatGPT, Claude, Perplexity and Gemini, and who lets them in.
1647major websites checked
35%block at least one AI crawler
25%block GPTBot
14%block GPTBot but allow ChatGPT search
21%publish an llms.txt
Share of sites blocking each AI crawler
Percentage of the 1647 sites whose robots.txt fully disallows the bot for the whole site.
| User agent | Company | Purpose | Blocked by |
|---|---|---|---|
CCBot | Common Crawl | Open dataset used by many AI models | 28% |
Bytespider | ByteDance | Model training | 27% |
GPTBot | OpenAI | Model training | 25% |
ClaudeBot | Anthropic | Model training | 25% |
Applebot-Extended | Apple | Apple Intelligence training | 22% |
Meta-ExternalAgent | Meta | Meta AI training | 22% |
Google-Extended | Gemini training & grounding | 21% | |
Amazonbot | Amazon | Alexa and Amazon AI | 20% |
PerplexityBot | Perplexity | Perplexity search index | 19% |
ChatGPT-User | OpenAI | Live browsing for ChatGPT users | 14% |
DuckAssistBot | DuckDuckGo | DuckAssist answers | 12% |
Claude-SearchBot | Anthropic | Claude search index | 11% |
Claude-User | Anthropic | Live browsing for Claude users | 11% |
Perplexity-User | Perplexity | Live browsing for Perplexity users | 11% |
MistralAI-User | Mistral | Live browsing for Le Chat users | 11% |
OAI-SearchBot | OpenAI | ChatGPT search index | 10% |
Bingbot | Microsoft | Bing search and Microsoft Copilot | 1% |
Googlebot | Search and AI Overviews | 0% |
By industry
| Category | Sites | Block GPTBot | Block PerplexityBot |
|---|---|---|---|
| E-commerce & retail | 203 | 10% | 5% |
| Entertainment & gaming | 96 | 43% | 22% |
| Finance | 151 | 3% | 1% |
| Health, food & lifestyle | 147 | 31% | 33% |
| Jobs, property & classifieds | 53 | 13% | 8% |
| News & media | 381 | 59% | 51% |
| Public sector & education | 81 | 7% | 1% |
| Reference & knowledge | 92 | 27% | 10% |
| SaaS & software | 335 | 3% | 1% |
| Social & community | 44 | 45% | 30% |
| Travel & mobility | 64 | 2% | 3% |
Key findings
- Training crawlers are blocked far more often than AI search and live-browsing agents. Many sites want to be cited by AI assistants without feeding model training.
- News publishers block AI crawlers much more often than e-commerce and SaaS sites, which mostly want AI assistants to recommend their products.
- llms.txt is still rare: only 21% of the sites publish one.
Are AI assistants sending visitors to your website? Citelytic tracks traffic from ChatGPT, Perplexity, Gemini, Claude and Copilot and shows which pages they cite.
Start free – no credit cardMethodology
On 2026-11-05 we fetched the public robots.txt of 1647 well-known websites (394 further sites returned no usable robots.txt and are excluded) and evaluated the rules for 18 AI user agents at the site root. “Blocked” means the whole site is disallowed for that bot. Sites can additionally block bots at the CDN or firewall level, which robots.txt does not show. Data is refreshed regularly. You may cite this report with a link to this page.
All websites
E-commerce & retail
| Website | AI bots blocked | GPTBot |
|---|---|---|
| aa.com | 0/18 | Allowed |
| aboutyou.de | 0/18 | Allowed |
| agoda.com | 0/18 | Allowed |
| airbnb.com | 3/18 | Blocked |
| aldi-nord.de | 0/18 | Allowed |
| aldi-sued.de | 0/18 | Allowed |
| alibaba.com | 2/18 | Blocked |
| aliexpress.com | 6/18 | Blocked |
| allegro.pl | 0/18 | Allowed |
| alternate.de | 0/18 | Allowed |
| amazon.co.jp | 14/18 | Blocked |
| amazon.co.uk | 14/18 | Blocked |
| amazon.com | 14/18 | Blocked |
| amazon.de | 14/18 | Blocked |
| americanas.com.br | 3/18 | Blocked |
| ao.com | 0/18 | Allowed |
| apo-discounter.de | 0/18 | Allowed |
| apple.com | 0/18 | Allowed |
| asda.com | 8/18 | Blocked |
| autoscout24.de | 5/18 | Blocked |
| bahn.de | 0/18 | Allowed |
| baur.de | 0/18 | Allowed |
| bergfreunde.de | 0/18 | Allowed |
| bestbuy.com | 0/18 | Allowed |
| bhphotovideo.com | 0/18 | Allowed |
| bigcommerce.com | 0/18 | Allowed |
| bike-discount.de | 1/18 | Allowed |
| bilka.dk | 0/18 | Allowed |
| billiger.de | 0/18 | Allowed |
| blibli.com | 0/18 | Allowed |
| blokker.nl | 0/18 | Allowed |
| bol.com | 0/18 | Allowed |
| bonprix.de | 0/18 | Allowed |
| boohoo.com | 0/18 | Allowed |
| booking.com | 0/18 | Allowed |
| boots.com | 0/18 | Allowed |
| boozt.com | 0/18 | Allowed |
| breuninger.com | 0/18 | Allowed |
| c-and-a.com | 0/18 | Allowed |
| canadiantire.ca | 0/18 | Allowed |
| carrefour.es | 0/18 | Allowed |
| carrefour.fr | 0/18 | Allowed |
| cdiscount.com | 0/18 | Allowed |
| ceneo.pl | 3/18 | Allowed |
| check24.de | 0/18 | Allowed |
| chefkoch.de | 8/18 | Blocked |
| chewy.com | 1/18 | Allowed |
| clasohlson.com | 0/18 | Allowed |
| costco.com | 0/18 | Allowed |
| czc.cz | 0/18 | Allowed |
| decathlon.de | 0/18 | Allowed |
| dell.com | 0/18 | Allowed |
| delta.com | 0/18 | Allowed |
| dm.de | 0/18 | Allowed |
| docmorris.de | 0/18 | Allowed |
| ebay.com | 8/18 | Blocked |
| ebay.de | 8/18 | Blocked |
| empik.com | 0/18 | Allowed |
| engelhorn.de | 0/18 | Allowed |
| enterprise.com | 0/18 | Allowed |
| esselunga.it | 0/18 | Allowed |
| etsy.com | 0/18 | Allowed |
| europcar.com | 0/18 | Allowed |
| expedia.com | 0/18 | Allowed |
| fahrrad-xxl.de | 0/18 | Allowed |
| farfetch.com | 0/18 | Allowed |
| flipkart.com | 0/18 | Allowed |
| flixbus.de | 0/18 | Allowed |
| fressnapf.de | 0/18 | Allowed |
| gap.com | 0/18 | Allowed |
| geizhals.de | 1/18 | Allowed |
| globus-baumarkt.de | 0/18 | Allowed |
| gmarket.co.kr | 15/18 | Allowed |
| gousto.co.uk | 0/18 | Allowed |
| groupon.com | 8/18 | Blocked |
| guenstiger.de | 0/18 | Allowed |
| harveynorman.com.au | 0/18 | Allowed |
| hellofresh.com | 0/18 | Allowed |
| hema.nl | 0/18 | Allowed |
| hilton.com | 0/18 | Allowed |
| hofer.at | 0/18 | Allowed |
| hornbach.de | 0/18 | Allowed |
| hostelworld.com | 0/18 | Allowed |
| hp.com | 0/18 | Allowed |
| hugendubel.de | 0/18 | Allowed |
| ikea.com | 1/18 | Allowed |
| instacart.com | 14/18 | Blocked |
| interdiscount.ch | 0/18 | Allowed |
| jacob.de | 0/18 | Allowed |
| jbhifi.com.au | 0/18 | Allowed |
| jd.com | 0/18 | Allowed |
| johnlewis.com | 0/18 | Allowed |
| jysk.dk | 0/18 | Allowed |
| kayak.com | 0/18 | Allowed |
| klingel.de | 1/18 | Allowed |
| kmart.com.au | 0/18 | Allowed |
| kohls.com | 0/18 | Allowed |
| lampenwelt.de | 1/18 | Allowed |
| lazada.com.ph | 0/18 | Allowed |
| lazada.sg | 0/18 | Allowed |
| leroymerlin.fr | 0/18 | Allowed |
| lidl.de | 0/18 | Allowed |
| llbean.com | 0/18 | Allowed |
| lpp.com | 0/18 | Allowed |
| lufthansa.com | 0/18 | Allowed |
| lyft.com | 0/18 | Allowed |
| manor.ch | 0/18 | Allowed |
| marksandspencer.com | 0/18 | Allowed |
| marriott.com | 0/18 | Allowed |
| matas.dk | 0/18 | Allowed |
| mediaexpert.pl | 0/18 | Allowed |
| mediamarkt.at | 2/18 | Allowed |
| mediamarkt.de | 3/18 | Allowed |
| medpex.de | 0/18 | Allowed |
| mercari.com | 0/18 | Allowed |
| microspot.ch | 0/18 | Allowed |
| migros.ch | 0/18 | Allowed |
| mobile.de | 0/18 | Allowed |
| mydealz.de | 11/18 | Blocked |
| mytheresa.com | 0/18 | Allowed |
| newegg.com | 0/18 | Allowed |
| nike.com | 0/18 | Allowed |
| nordstrom.com | 0/18 | Allowed |
| nykaa.com | 0/18 | Allowed |
| obi.de | 0/18 | Allowed |
| oldnavy.com | 0/18 | Allowed |
| omio.com | 1/18 | Allowed |
| otto.at | 0/18 | Allowed |
| otto.de | 0/18 | Allowed |
| overstock.com | 0/18 | Allowed |
| peek-cloppenburg.de | 0/18 | Allowed |
| petsmart.com | 0/18 | Allowed |
| poco.de | 0/18 | Allowed |
| pollin.de | 0/18 | Allowed |
| porta.de | 6/18 | Allowed |
| power.no | 0/18 | Allowed |
| prettylittlething.com | 0/18 | Allowed |
| rakuten.com | 0/18 | Allowed |
| rakuten.de | 0/18 | Allowed |
| reichelt.de | 0/18 | Allowed |
| rentalcars.com | 0/18 | Allowed |
| rewe.de | 0/18 | Allowed |
| roller.de | 0/18 | Allowed |
| rosebikes.de | 1/18 | Allowed |
| rossmann.de | 1/18 | Allowed |
| ryanair.com | 0/18 | Allowed |
| samsclub.com | 0/18 | Allowed |
| samsung.com | 0/18 | Allowed |
| sanicare.de | 0/18 | Allowed |
| saturn.de | 3/18 | Allowed |
| screwfix.com | 0/18 | Allowed |
| shein.com | 0/18 | Allowed |
| shop-apotheke.com | 0/18 | Allowed |
| shopee.com | 0/18 | Allowed |
| shopee.com.br | 0/18 | Allowed |
| shopee.sg | 0/18 | Allowed |
| shopify.com | 0/18 | Allowed |
| shpock.com | 2/18 | Allowed |
| sixt.com | 0/18 | Allowed |
| skyscanner.net | 1/18 | Allowed |
| sncf-connect.com | 0/18 | Allowed |
| spar.at | 0/18 | Allowed |
| sportscheck.com | 0/18 | Allowed |
| ssense.com | 0/18 | Allowed |
| stockx.com | 0/18 | Allowed |
| taobao.com | 0/18 | Allowed |
| target.com | 0/18 | Allowed |
| tchibo.de | 0/18 | Allowed |
| thomann.de | 1/18 | Blocked |
| tiki.vn | 0/18 | Allowed |
| tokopedia.com | 0/18 | Allowed |
| tomtailor.de | 0/18 | Allowed |
| toom.de | 0/18 | Allowed |
| trainline.com | 0/18 | Allowed |
| trendyol.com | 0/18 | Allowed |
| tripadvisor.com | 7/18 | Blocked |
| trivago.com | 0/18 | Allowed |
| uber.com | 0/18 | Allowed |
| ulta.com | 0/18 | Allowed |
| unieuro.it | 0/18 | Allowed |
| universal.at | 0/18 | Allowed |
| veepee.fr | 0/18 | Allowed |
| verivox.de | 0/18 | Allowed |
| verkkokauppa.com | 0/18 | Allowed |
| voelkner.de | 0/18 | Allowed |
| vrbo.com | 0/18 | Allowed |
| walgreens.com | 0/18 | Allowed |
| walmart.com | 0/18 | Allowed |
| wayfair.com | 0/18 | Allowed |
| wayfair.de | 0/18 | Allowed |
| wehkamp.nl | 1/18 | Allowed |
| westwing.de | 0/18 | Allowed |
| willhaben.at | 0/18 | Allowed |
| windeln.de | 0/18 | Allowed |
| wish.com | 0/18 | Allowed |
| witt-weiden.de | 0/18 | Allowed |
| woocommerce.com | 0/18 | Allowed |
| zalando.co.uk | 8/18 | Blocked |
| zalando.de | 8/18 | Blocked |
| zalando.se | 8/18 | Blocked |
| zalora.com.ph | 1/18 | Allowed |
| zappos.com | 0/18 | Allowed |
| zooplus.de | 0/18 | Allowed |
Entertainment & gaming
Finance
Health, food & lifestyle
Jobs, property & classifieds
| Website | AI bots blocked | GPTBot |
|---|---|---|
| bayut.com | 0/18 | Allowed |
| bazos.cz | 0/18 | Allowed |
| blocket.se | 0/18 | Allowed |
| boligsiden.dk | 0/18 | Allowed |
| craigslist.org | 0/18 | Allowed |
| dba.dk | 0/18 | Allowed |
| dice.com | 0/18 | Allowed |
| dubizzle.com | 0/18 | Allowed |
| emlakjet.com | 0/18 | Allowed |
| finn.no | 0/18 | Allowed |
| fotocasa.es | 0/18 | Allowed |
| gumtree.com | 0/18 | Allowed |
| hemnet.se | 0/18 | Allowed |
| homes.co.jp | 0/18 | Allowed |
| idealista.com | 0/18 | Allowed |
| immobiliare.it | 0/18 | Allowed |
| infojobs.net | 13/18 | Blocked |
| ingatlan.com | 0/18 | Allowed |
| inmuebles24.com | 0/18 | Allowed |
| jobkorea.co.kr | 18/18 | Blocked |
| jobs.ch | 0/18 | Allowed |
| jobware.de | 7/18 | Blocked |
| jofogas.hu | 7/18 | Blocked |
| kariyer.net | 0/18 | Allowed |
| karriere.at | 0/18 | Allowed |
| kijiji.ca | 0/18 | Allowed |
| leboncoin.fr | 1/18 | Allowed |
| magicbricks.com | 0/18 | Allowed |
| markt.de | 0/18 | Allowed |
| marktplaats.nl | 0/18 | Allowed |
| meinestadt.de | 0/18 | Allowed |
| oikotie.fi | 0/18 | Allowed |
| olx.pl | 0/18 | Allowed |
| olx.ro | 0/18 | Allowed |
| onthemarket.com | 0/18 | Allowed |
| otodom.pl | 0/18 | Allowed |
| otomoto.pl | 0/18 | Allowed |
| pap.fr | 0/18 | Allowed |
| propertyfinder.ae | 0/18 | Allowed |
| quoka.de | 1/18 | Allowed |
| realestate.com.au | 11/18 | Blocked |
| reed.co.uk | 0/18 | Allowed |
| rumah123.com | 0/18 | Allowed |
| saramin.co.kr | 2/18 | Blocked |
| seloger.com | 0/18 | Allowed |
| sreality.cz | 16/18 | Blocked |
| stellenanzeigen.de | 0/18 | Allowed |
| storia.ro | 0/18 | Allowed |
| suumo.jp | 0/18 | Allowed |
| tori.fi | 0/18 | Allowed |
| totaljobs.com | 1/18 | Allowed |
| wallapop.com | 1/18 | Allowed |
| zonaprop.com.ar | 0/18 | Allowed |
News & media
Public sector & education
| Website | AI bots blocked | GPTBot |
|---|---|---|
| admin.ch | 0/18 | Allowed |
| argentina.gob.ar | 0/18 | Allowed |
| belastingdienst.nl | 0/18 | Allowed |
| berlin.de | 1/18 | Allowed |
| bmas.de | 0/18 | Allowed |
| bmf.de | 0/18 | Allowed |
| boe.es | 0/18 | Allowed |
| borger.dk | 0/18 | Allowed |
| boun.edu.tr | 0/18 | Allowed |
| bsi.bund.de | 0/18 | Allowed |
| bund.de | 0/18 | Allowed |
| bundesgesundheitsministerium.de | 0/18 | Allowed |
| bundesregierung.de | 0/18 | Allowed |
| bundestag.de | 0/18 | Allowed |
| bzst.de | 0/18 | Allowed |
| canada.ca | 0/18 | Allowed |
| cornell.edu | 0/18 | Allowed |
| cuni.cz | 0/18 | Allowed |
| ed.ac.uk | 0/18 | Allowed |
| elte.hu | 2/18 | Blocked |
| frankfurt.de | 8/18 | Blocked |
| gob.mx | 0/18 | Allowed |
| gov.br | 0/18 | Allowed |
| gov.co | 0/18 | Allowed |
| gov.cz | 0/18 | Allowed |
| gov.pl | 0/18 | Allowed |
| gov.sg | 0/18 | Allowed |
| gov.uk | 0/18 | Allowed |
| gov.vn | 0/18 | Allowed |
| governo.it | 0/18 | Allowed |
| hamburg.de | 0/18 | Allowed |
| handwerkskammer.de | 0/18 | Allowed |
| helsinki.fi | 0/18 | Allowed |
| ihk.de | 0/18 | Allowed |
| imperial.ac.uk | 0/18 | Allowed |
| irs.gov | 0/18 | Allowed |
| kaist.ac.kr | 0/18 | Allowed |
| kfw.de | 0/18 | Allowed |
| ki.se | 0/18 | Allowed |
| koeln.de | 0/18 | Allowed |
| kormany.hu | 0/18 | Allowed |
| ku.dk | 0/18 | Allowed |
| kyoto-u.ac.jp | 0/18 | Allowed |
| metu.edu.tr | 0/18 | Allowed |
| muenchen.de | 0/18 | Allowed |
| nasa.gov | 0/18 | Allowed |
| ntu.edu.sg | 0/18 | Allowed |
| oesterreich.gv.at | 0/18 | Allowed |
| polimi.it | 0/18 | Allowed |
| princeton.edu | 0/18 | Allowed |
| regeringen.se | 0/18 | Allowed |
| rijksoverheid.nl | 0/18 | Allowed |
| service-public.fr | 0/18 | Allowed |
| sorbonne-universite.fr | 0/18 | Allowed |
| stiftung-warentest.de | 6/18 | Blocked |
| stuttgart.de | 8/18 | Blocked |
| suomi.fi | 0/18 | Allowed |
| test.de | 6/18 | Blocked |
| tudelft.nl | 0/18 | Allowed |
| turkiye.gov.tr | 0/18 | Allowed |
| u-tokyo.ac.jp | 0/18 | Allowed |
| uba.ar | 0/18 | Allowed |
| uchile.cl | 0/18 | Allowed |
| ucl.ac.uk | 0/18 | Allowed |
| ui.ac.id | 0/18 | Allowed |
| uio.no | 0/18 | Allowed |
| uj.edu.pl | 16/18 | Blocked |
| umich.edu | 0/18 | Allowed |
| unibo.it | 0/18 | Allowed |
| unibuc.ro | 0/18 | Allowed |
| unicamp.br | 0/18 | Allowed |
| up.edu.ph | 0/18 | Allowed |
| usa.gov | 0/18 | Allowed |
| utoronto.ca | 0/18 | Allowed |
| uva.nl | 0/18 | Allowed |
| uw.edu.pl | 0/18 | Allowed |
| verbraucherzentrale.de | 0/18 | Allowed |
| whitehouse.gov | 0/18 | Allowed |
| wien.gv.at | 0/18 | Allowed |
| yale.edu | 0/18 | Allowed |
| zoll.de | 0/18 | Allowed |
Reference & knowledge
SaaS & software
Social & community
| Website | AI bots blocked | GPTBot |
|---|---|---|
| 500px.com | 0/18 | Allowed |
| 9gag.com | 0/18 | Allowed |
| behance.net | 7/18 | Blocked |
| bsky.app | 0/18 | Allowed |
| change.org | 0/18 | Allowed |
| dev.to | 0/18 | Allowed |
| deviantart.com | 0/18 | Allowed |
| discord.com | 0/18 | Allowed |
| dribbble.com | 0/18 | Allowed |
| facebook.com | 16/18 | Blocked |
| flickr.com | 11/18 | Blocked |
| giphy.com | 0/18 | Allowed |
| gofundme.com | 0/18 | Allowed |
| hashnode.com | 0/18 | Allowed |
| imgur.com | 0/18 | Allowed |
| indiehackers.com | 12/18 | Blocked |
| instagram.com | 16/18 | Blocked |
| lemmy.world | 0/18 | Allowed |
| linkedin.com | 14/18 | Blocked |
| lobste.rs | 16/18 | Blocked |
| mastodon.social | 7/18 | Blocked |
| meetup.com | 1/18 | Blocked |
| news.ycombinator.com | 0/18 | Allowed |
| nextdoor.com | 16/18 | Blocked |
| onlyfans.com | 0/18 | Allowed |
| patreon.com | 1/18 | Allowed |
| pexels.com | 5/18 | Blocked |
| pinterest.com | 15/18 | Blocked |
| pixabay.com | 6/18 | Blocked |
| producthunt.com | 0/18 | Allowed |
| reddit.com | 18/18 | Blocked |
| slashdot.org | 0/18 | Allowed |
| snapchat.com | 8/18 | Blocked |
| threads.net | 16/18 | Blocked |
| tiktok.com | 15/18 | Blocked |
| tildes.net | 1/18 | Blocked |
| tumblr.com | 6/18 | Allowed |
| twitch.tv | 0/18 | Allowed |
| unsplash.com | 3/18 | Allowed |
| vimeo.com | 2/18 | Allowed |
| vk.com | 0/18 | Allowed |
| whatsapp.com | 6/18 | Blocked |
| x.com | 16/18 | Blocked |
| youtube.com | 0/18 | Allowed |
Travel & mobility
| Website | AI bots blocked | GPTBot |
|---|---|---|
| ab-in-den-urlaub.de | 0/18 | Allowed |
| adac.de | 0/18 | Allowed |
| airasia.com | 0/18 | Allowed |
| alaskaair.com | 0/18 | Allowed |
| amtrak.com | 0/18 | Allowed |
| austrian.com | 0/18 | Allowed |
| auto-motor-und-sport.de | 14/18 | Blocked |
| autobild.de | 8/18 | Allowed |
| avis.com | 0/18 | Allowed |
| bvg.de | 0/18 | Allowed |
| cd.cz | 0/18 | Allowed |
| condor.com | 0/18 | Allowed |
| decolar.com | 0/18 | Allowed |
| despegar.com | 0/18 | Allowed |
| dsb.dk | 0/18 | Allowed |
| europcar.de | 0/18 | Allowed |
| eurowings.com | 0/18 | Allowed |
| expedia.de | 0/18 | Allowed |
| fewo-direkt.de | 0/18 | Allowed |
| finnair.com | 0/18 | Allowed |
| france.fr | 0/18 | Allowed |
| greyhound.com | 0/18 | Allowed |
| holidaycheck.de | 1/18 | Allowed |
| hopper.com | 0/18 | Allowed |
| hrs.de | 3/18 | Allowed |
| ita-airways.com | 0/18 | Allowed |
| italia.it | 0/18 | Allowed |
| italotreno.it | 0/18 | Allowed |
| jalan.net | 0/18 | Allowed |
| jetblue.com | 0/18 | Allowed |
| miles-mobility.com | 0/18 | Allowed |
| mvg.de | 0/18 | Allowed |
| nationalrail.co.uk | 0/18 | Allowed |
| norwegian.com | 0/18 | Allowed |
| ns.nl | 0/18 | Allowed |
| obilet.com | 0/18 | Allowed |
| opodo.de | 0/18 | Allowed |
| orbitz.com | 0/18 | Allowed |
| priceline.com | 0/18 | Allowed |
| regiojet.com | 0/18 | Allowed |
| renfe.com | 0/18 | Allowed |
| sas.se | 0/18 | Allowed |
| saudia.com | 0/18 | Allowed |
| sixt.de | 0/18 | Allowed |
| sj.se | 0/18 | Allowed |
| southwest.com | 0/18 | Allowed |
| spain.info | 0/18 | Allowed |
| swiss.com | 0/18 | Allowed |
| tarom.ro | 0/18 | Allowed |
| transavia.com | 0/18 | Allowed |
| traveloka.com | 0/18 | Allowed |
| trenitalia.com | 0/18 | Allowed |
| trivago.de | 0/18 | Allowed |
| tui.co.uk | 0/18 | Allowed |
| tui.com | 0/18 | Allowed |
| tuifly.com | 0/18 | Allowed |
| turo.com | 0/18 | Allowed |
| vietnamairlines.com | 0/18 | Allowed |
| virginatlantic.com | 0/18 | Allowed |
| visitbritain.com | 0/18 | Allowed |
| volaris.com | 0/18 | Allowed |
| vr.fi | 0/18 | Allowed |
| westjet.com | 0/18 | Allowed |
| wizzair.com | 0/18 | Allowed |
FAQ
What percentage of websites block GPTBot?
In our 2026-11-05 check of 1647 major websites, 25% fully block GPTBot in their robots.txt.
Which AI crawler is blocked most often?
CCBot is blocked most often, by 28% of the sites we checked.
Should I block AI crawlers on my website?
It depends on your goal. Blocking training crawlers keeps your content out of future models, but blocking AI search and browsing agents also stops ChatGPT, Perplexity and Claude from citing and linking to you. Most businesses that want customers from AI assistants keep search agents allowed.
Are AI assistants sending visitors to your website? Citelytic tracks traffic from ChatGPT, Perplexity, Gemini, Claude and Copilot and shows which pages they cite.
Start free – no credit card