{"id":2699,"date":"2026-08-20T12:40:51","date_gmt":"2026-08-20T10:40:51","guid":{"rendered":"https:\/\/mairateam.com\/?post_type=glossary&p=2699"},"modified":"2026-08-20T15:48:45","modified_gmt":"2026-08-20T13:48:45","slug":"ai-crawler","status":"publish","type":"glossary","link":"https:\/\/mairateam.com\/cs\/slovnik\/ai-crawler\/","title":{"rendered":"AI crawler (GPTBot, ClaudeBot)"},"content":{"rendered":"

AI crawlery<\/strong> jsou roboti jazykov\u00fdch model\u016f, kte\u0159\u00ed sb\u00edraj\u00ed obsah web\u016f: GPTBot pro OpenAI, ClaudeBot pro Anthropic, PerplexityBot, Amazonbot a dal\u0161\u00ed; u Googlu jde o token Google-Extended v robots.txt. Kdy\u017e se na v\u00e1\u0161 web nedostanou, va\u0161e zna\u010dka pro danou AI p\u0159est\u00e1v\u00e1 existovat: model o v\u00e1s nem\u00e1 co v\u011bd\u011bt a v odpov\u011bd\u00edch v\u00e1s nahrad\u00ed konkurence, kterou crawler p\u0159e\u010d\u00edst mohl.<\/p>\n

V kostce<\/h2>\n\n\n\n\n\n\n
Co to je<\/strong><\/td>\nRoboti sb\u00edraj\u00edc\u00ed obsah pro tr\u00e9nink a odpov\u011bdi jazykov\u00fdch model\u016f<\/td>\n<\/tr>\n
Dva druhy<\/strong><\/td>\nIndexa\u010dn\u00ed crawler pln\u00ed znalostn\u00ed b\u00e1zi. Crawler pro odpov\u011bdi na\u010d\u00edt\u00e1 str\u00e1nku a\u017e p\u0159i konkr\u00e9tn\u00ed odpov\u011bdi<\/td>\n<\/tr>\n
Kde se blokuj\u00ed<\/strong><\/td>\nV robots.txt v\u011bdom\u011b, \u010dasto ale nev\u011bdom\u011b na firewallu nebo WAF hostingu<\/td>\n<\/tr>\n
Jak to ov\u011b\u0159it<\/strong><\/td>\nP\u0159\u00edm\u00fdm dotazem na server s identifikac\u00ed bota. Kontrola robots.txt nesta\u010d\u00ed<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

Jak to funguje<\/h2>\n

T\u0159i role, t\u0159i r\u016fzn\u00ed roboti<\/h3>\n

Ka\u017ed\u00fd velk\u00fd provozovatel model\u016f provozuje n\u011bkolik robot\u016f a ka\u017ed\u00fd z nich m\u00e1 jinou \u00falohu. Rozli\u0161en\u00ed podle jm\u00e9na (user agenta) je z\u00e1klad, proto\u017ee pravidla v robots.txt i ve firewallu se p\u00ed\u0161\u00ed pr\u00e1v\u011b na jm\u00e9no.<\/p>\n\n\n\n\n\n\n
Tr\u00e9nink modelu<\/strong><\/td>\nGPTBot, ClaudeBot, Amazonbot, Bytespider, CCBot. Sb\u00edraj\u00ed obsah do tr\u00e9ninkov\u00fdch dat, v\u00fdsledek se projev\u00ed a\u017e v p\u0159\u00ed\u0161t\u00ed generaci modelu.<\/td>\n<\/tr>\n
Index pro vyhled\u00e1v\u00e1n\u00ed asistenta<\/strong><\/td>\nOAI-SearchBot, Claude-SearchBot, PerplexityBot. Pln\u00ed index, ze kter\u00e9ho asistent vyb\u00edr\u00e1 zdroje pro odpov\u011bdi s citacemi.<\/td>\n<\/tr>\n
Na\u010dten\u00ed na pokyn u\u017eivatele<\/strong><\/td>\nChatGPT-User, Claude-User, Perplexity-User. St\u00e1hnou konkr\u00e9tn\u00ed str\u00e1nku ve chv\u00edli, kdy ji u\u017eivatel vlo\u017e\u00ed nebo kdy ji asistent pot\u0159ebuje pro pr\u00e1v\u011b psanou odpov\u011b\u010f.<\/td>\n<\/tr>\n
\u0158\u00eddic\u00ed token bez vlastn\u00edho robota<\/strong><\/td>\nGoogle-Extended, Applebot-Extended. Nemaj\u00ed vlastn\u00ed user agent. Jen \u0159\u00edkaj\u00ed Googlebotu a Applebotu, zda sm\u00ed u\u017e sta\u017een\u00fd obsah pou\u017e\u00edt: Google-Extended pro tr\u00e9nink a grounding Gemini, Applebot-Extended pro tr\u00e9nink model\u016f Apple.<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

Praktick\u00fd d\u016fsledek: z\u00e1kaz Google-Extended nezm\u011bn\u00ed nic na vyhled\u00e1v\u00e1n\u00ed Google ani na AI Overviews, ty b\u011b\u017e\u00ed na Googlebotu. A povolen\u00fd ChatGPT-User bez OAI-SearchBota znamen\u00e1, \u017ee v\u00e1s asistent p\u0159e\u010dte jen tehdy, kdy\u017e na va\u0161i URL naraz\u00ed jinde.<\/p>\n

Cesta po\u017eadavku a kde se p\u0159eru\u0161\u00ed<\/h3>\n

Robot, kter\u00fd dodr\u017euje pravidla, si nejd\u0159\u00edv st\u00e1hne robots.txt, najde skupinu se sv\u00fdm jm\u00e9nem a \u0159\u00edd\u00ed se \u0159\u00e1dky Allow a Disallow. Pak pos\u00edl\u00e1 b\u011b\u017en\u00e9 HTTP po\u017eadavky na jednotliv\u00e9 URL. Ka\u017ed\u00fd z nich proch\u00e1z\u00ed stejnou cestou jako n\u00e1v\u0161t\u011bvn\u00edk: CDN a firewall (WAF, bot management), pravidla na serveru (.htaccess, bezpe\u010dnostn\u00ed plugin) a teprve potom aplikace, kter\u00e1 vr\u00e1t\u00ed HTML. Blokace na kter\u00e9koli vrstv\u011b p\u0159ed aplikac\u00ed vrac\u00ed 403, v\u00fdzvu pro prohl\u00ed\u017ee\u010d nebo pr\u00e1zdnou str\u00e1nku a robots.txt v tu chv\u00edli nehraje roli. Cloudflare se nav\u00edc od \u010dervence 2025 u nov\u011b p\u0159idan\u00fdch dom\u00e9n pt\u00e1, zda AI crawlery pustit, a v\u00fdchoz\u00ed volba je blokace. K z\u00e1kazu tak \u010dasto sta\u010d\u00ed zalo\u017eit \u00fa\u010det u CDN. Druh\u00e9 omezen\u00ed: AI crawlery zpravidla nespou\u0161t\u011bj\u00ed JavaScript. Ceny, popisy a recenze, kter\u00e9 se vykresluj\u00ed a\u017e v prohl\u00ed\u017ee\u010di, pro n\u011b nemus\u00ed existovat.<\/p>\n

Modelov\u00fd p\u0159\u00edklad:<\/strong> e-shop s 5 000 URL nastav\u00ed pro AI roboty limit 1 po\u017eadavek za sekundu. Cel\u00fd web robot na\u010dte za 5 000 sekund, tedy zhruba za 83 minut, a z\u00e1t\u011b\u017e serveru je zanedbateln\u00e1. Limit je proto rozumn\u011bj\u0161\u00ed ne\u017e blok.<\/p>\n

Ov\u011b\u0159en\u00ed: dv\u011b ot\u00e1zky, dva postupy<\/h3>\n

Prvn\u00ed ot\u00e1zka zn\u00ed, zda se robot na web dostane. Odpov\u011b\u010f d\u00e1 jen po\u017eadavek zven\u010d\u00ed s user agentem konkr\u00e9tn\u00edho bota na n\u011bkolik typ\u016f str\u00e1nek (homepage, kategorie, produkt, \u010dl\u00e1nek), proto\u017ee pravidla WAF b\u00fdvaj\u00ed odstup\u0148ovan\u00e1 podle cesty. Vedle n\u00e1vratov\u00e9ho k\u00f3du se kontroluje i t\u011blo odpov\u011bdi: k\u00f3d 200 s v\u00fdzvou k ov\u011b\u0159en\u00ed prohl\u00ed\u017ee\u010de je pro robota stejn\u00e1 p\u0159ek\u00e1\u017eka jako 403. Druh\u00e1 ot\u00e1zka zn\u00ed, zda n\u00e1v\u0161t\u011bvn\u00edk v logu, kter\u00fd se hl\u00e1s\u00ed jako GPTBot, je opravdu GPTBot. \u0158et\u011bzec user agenta si m\u016f\u017ee nastavit kdokoli. Provozovatel\u00e9 proto zve\u0159ej\u0148uj\u00ed rozsahy IP adres sv\u00fdch robot\u016f (OpenAI nap\u0159\u00edklad jako ve\u0159ejn\u00fd soubor JSON) a shoda IP s t\u00edmto seznamem je jedin\u00fd spolehliv\u00fd d\u016fkaz. Bez n\u00ed ned\u00e1v\u00e1 smysl po\u010d\u00edtat n\u00e1v\u0161t\u011bvy AI robot\u016f ani na n\u011b stav\u011bt pravidla.<\/p>\n

Dva crawlery, dv\u011b r\u016fzn\u00e9 \u0161kody<\/h2>\n

Poskytovatel\u00e9 model\u016f pou\u017e\u00edvaj\u00ed odd\u011blen\u00e9 roboty. Indexa\u010dn\u00ed crawler, nap\u0159\u00edklad ClaudeBot nebo GPTBot, pln\u00ed znalostn\u00ed b\u00e1zi modelu: bez n\u011bj se o v\u00e1s model nedozv\u00ed s\u00e1m od sebe. Crawler pro odpov\u011bdi, nap\u0159\u00edklad Claude-User nebo ChatGPT-User, na\u010d\u00edt\u00e1 str\u00e1nku a\u017e ve chv\u00edli, kdy na ni v konkr\u00e9tn\u00ed odpov\u011bdi p\u0159ijde \u0159ada. Blokace ka\u017ed\u00e9ho z nich m\u00e1 jin\u00fd d\u016fsledek a m\u016f\u017ee nastat nez\u00e1visle.<\/p>\n

Z na\u0161\u00ed praxe: blokovan\u00ed roboti, o kter\u00fdch nikdo nev\u00ed<\/h2>\n

P\u0159i auditech testujeme crawlery jazykov\u00fdch model\u016f p\u0159\u00edm\u00fdm dotazem. \u010c\u00e1st z nich b\u011b\u017en\u011b dost\u00e1v\u00e1 HTTP 403, p\u0159esto\u017ee je robots.txt v\u00fdslovn\u011b povoluje. Blokuje je firewall na \u00farovni hostingu nebo CDN, vrstva, do kter\u00e9 se b\u011b\u017en\u011b nikdo ned\u00edv\u00e1. Typick\u00fd detail: crawler pro odpov\u011bdi t\u00e9ho\u017e poskytovatele projde, blokovan\u00fd je jen indexa\u010dn\u00ed.<\/p>\n

Blokovat AI crawlery v\u011bdom\u011b je konkuren\u010dn\u00ed nev\u00fdhoda, ne opatrnost. A kdo je blokuje, obvykle o tom ani nev\u00ed: blokace vznik\u00e1 na firewallu nebo v CDN, ne rozhodnut\u00edm.<\/p>\n

\u010cast\u00e9 chyby<\/h2>\n