³ë¹«Çö ´ëÅë·É ¹è³Ê
  ±è¼ºÅÂÀÇ Tech Tips(Linux, PHP, Apache, DBMS, Mobile)
  http://www.supersky.pe.kr  
¾È³çÇϽʴϱî? ±è¼ºÅÂÀÔ´Ï´Ù.
Linux, Apache, PHP, Mysql, Mobile °ü·Ã Tech Tips Á¤º¸¸¦ Á¦°øÇÕ´Ï´Ù.
 
<<   2006 Dec   >>
S M T W T F S
262728293012
3456789
10111213141516
17181920212223
24252627282930
31123456
1750076 256
  
  + Article List  :  2006³â 12¿ù (2)
2006/12/12     robots.txtÆÄÀÏÀ» ÀÌ¿ëÇÑ `ÀÎÅÍ³Ý °Ë»ö¿£Áø ¹èÁ¦Ç¥ÁØ`À» µû¸£´Â ¹æ¹ý  (12)
2006/12/06     ¼Ö¶ó¸®½º ¹öÀü È®ÀÎÇÏ±â  (12)

DNS Powered by DNSEver.com
  ++ robots.txtÆÄÀÏÀ» ÀÌ¿ëÇÑ `ÀÎÅÍ³Ý °Ë»ö¿£Áø ¹èÁ¦Ç¥ÁØ`À» µû¸£´Â ¹æ¹ý  -  2006/12/12 17:21
`ÀÎÅÍ³Ý °Ë»ö¿£Áø ¹èÁ¦Ç¥ÁØ`À» µû¸£´Â ¹æ¹ý

`ÀÎÅÍ³Ý °Ë»ö¿£Áø ¹èÁ¦Ç¥ÁØ`À» µû¸£´Â ¹æ¹ý

 

 °Ë»ö¿£Áø¿¡¼­ ³»È¨ÆäÀÌÁö°¡ ¼­Ä¡´çÇÏÁö ¾Ê±âÀ§ÇØ ¾²´Â ¹æ¹ý

ÃÖ±Ù À¥ °Ë»ö¿£ÁøÀÇ ¼º´ÉÀÌ ¿ùµîÈ÷ Çâ»óµÇ¸é¼­ HTML·Î ÀÛ¼ºµÈ À¥ÆäÀÌÁöÀÇ ³»¿ëÀº ¹°·Ð À¥»çÀÌÆ®¿¡ ¿Ã·Á³õÀº PDF³ª DOC°°Àº ¹®¼­ÆÄÀÏ ³»¿ë±îÁö °Ë»öÀÌ °¡´ÉÇÕ´Ï´Ù.

ÀÌ·± Á¤º¸À¯Ãâ ¹æÁö ´ëÃ¥ÀÇ ÀÏȯÀ¸·Î 'ÀÎÅÍ³Ý °Ë»ö¿£Áø ¹èÁ¦Ç¥ÁØ(Robots Exclusion Protocol)'À» Àû¿ëÇϽñ⠹ٶø´Ï´Ù.

À̸¦ ÀÌ¿ëÇÏ¿© »çÀÌÆ®ÀÇ ¸ðµç ÆäÀÌÁö¿¡ ´ëÇÏ¿© ³ëÃâÀº Â÷´ÜÇÒ ¼ö ÀÖ°í, Â÷´ÜÀ» ¿øÇÏ´Â ÆäÀÌÁö¿¡ ´ëÇؼ­¸¸ ³ëÃâÂ÷´Üµµ °¡´ÉÇÕ´Ï´Ù.

ÀÎÅÍ³Ý °Ë»ö¿£Áø ¹èÁ¦Ç¥ÁØÀ̶õ º¸¾ÈÀÌ ÇÊ¿äÇÑ ³»¿ëÀÌ °Ë»ö¿£Áø¿¡ À¯Ãâ µÇÁö ¸øÇϵµ·Ï À¥ÆäÀÌÁö¸¦ ÀÛ¼ºÇÏ´Â ¹æ¹ýÀ» ±â¼úÇÑ ±¹Á¦±â¼úÇ¥ÁØÀÔ´Ï´Ù.

 

robots.txt ÀÛ¼º¹æ¹ý

1. robots.txt¸¦ À¥ ¼­¹öÀÇ È¨ÆäÀÌÁö ÃÖ»óÀ§ µð·ºÅ丮¿¡ ÀúÀå

»çÀÌÆ®°¡ ƯÁ¤ È£½ºÆ®(host)¿Í Æ÷Æ®(port) ¹øÈ£¿¡¼­ HTTP ¼­¹ö·Î ¿î¿µµÇ´Â °ÍÀ¸·Î Á¤ÀǵǾúÀ¸¸é, ·Îº¿Àº ´Ü¼øÈ÷ »çÀÌÆ® URI¿¡¼­ "/robots.txt"¸¦ ãÀ» °ÍÀÔ´Ï´Ù. robots.txtÀÇ À§Ä¡ ¿¹Á¦ÀÔ´Ï´Ù.

»çÀÌÆ® URI                        robots.txtÀÇ URI
http://www.w3.org/           http://www.w3.org/robots.txt
http://www.w3.org:80/       http://www.w3.org:80/robots.txt
http://w3.org/                    http://w3.org/robots.txt

 

½ÎÀÌÆ®¿¡´Â ÇϳªÀÇ "/robots.txt" ¸¸À» °¡Áú ¼ö ÀÖ½À´Ï´Ù. ±¸Ã¼ÀûÀ¸·Î, "robots.txt" È­ÀϵéÀº ·Îº¿ÀÌ ±×°ÍÀ» »ç¿ëÀÚ µð·ºÅ丮¿¡¼­ ãÁö ¾Ê±â ¶§¹®¿¡, »ç¿ëÀÚ µð·ºÅ丮¿¡ À§Ä¡½ÃÄѼ­´Â ¾ÈµË´Ï´Ù. »ç¿ëÀÚ°¡ ÀÚ½ÅÀÇ "robots.txt"¸¦ ¸¸µé±â¸¦ ¿øÇϸé, ´ÜÀÏ "/robots.txt" ¾È¿¡ ¸ðµÎ ÅëÇÕ(merge)ÇÒ ÇÊ¿ä°¡ ÀÖ½À´Ï´Ù.

2. robots.txtÀÇ ³»¿ë

- ¸ðµç ·Îº¿¿¡ ´ëÇÑ ¸ðµç µð·ºÅ丮 °Ë»ö °ÅºÎ½Ã

User-agent:*

Disallow:/

- ¸ðµç ·Îº¿¿¡ ´ëÇÑ ¼­¹ö ÀϺκи¸ °Ë»ö°ÅºÎ½Ã

   (¿¹: /secure µð·ºÅ丮, /policy µð·ºÅ丮¸¸ °Ë»ö°ÅºÎ)

User-agent:*

Disallow:/secure/

Disallow:/policy/

 

URI´Â ´ë¼Ò¹®ÀÚ ±¸º°Çϸç, "/robots.txt" ¹®ÀÚ¿­Àº ¸ðµÎ ¼Ò¹®ÀÚÀ̾î¾ß Çϸç, °ø¹éÀº Çã¿ëµÇÁö ¾Ê½À´Ï´Ù.

¸¸ÀÏ ±× °ªÀÌ "*" À̸é, ±× ·¹ÄÚµå´Â ´Ù¸¥ ·¹ÄÚµåµé°ú Çϳªµµ ¸ÂÁö ¾Ê´Â ¾î¶² ·Îº¿¿¡¼­³ª °¡´ÉÇÑ µðÆúÆ® Á¢¼Ó(access) Á¤Ã¥(policy)À» ³ªÅ¸³À´Ï´Ù. "/robots.txt" ¾È¿¡¼­ ¿©·¯°³ÀÇ ±×·¯ÇÑ ·¹Äڵ带 °®´Â °ÍÀº Çã¿ëµÇÁö ¾Ê½À´Ï´Ù.

"Disallow"(Çã¿ë ¾ÈÇÔ) Çʵå´Â ¹æ¹® ÇÒ ¼ö ¾ø´Â ºÎºÐ URI¸¦ ÁöÁ¤ÇÕ´Ï´Ù. ÀÌ´Â ¿ÏÀü °æ·Î(full path) ¶Ç´Â ºÎºÐ °æ·Î°¡ µÉ ¼ö ÀÖ½À´Ï´Ù. ÀÌ °ªÀ¸·Î ½ÃÀ۵Ǵ URI´Â ÀÐÇôÁöÁö ¾ÊÀ» °ÍÀÔ´Ï´Ù. ¿¹¸¦ µé¾î


Disallow: /help

/help.html°ú /help/index.html µÑ ´Ù Çã¿ë ¾ÈÇÔ,


Disallow: /help/

/help/index.html´Â Çã¿ë ¾ÈÇϳª, /help.htmlÀº Çã¿ë µÊ.


"Disallow"¿¡¼­ ºó °ªÀº, ¸ðµç URIµéÀÌ ÀÐÇô Áú ¼ö ÀÖ½¿À» °¡¸®±é´Ï´Ù. robots.txt È­ÀÏ¿¡´Â ÃÖ¼ÒÇÑ ÇÑ°³ÀÇ "Disallow" Çʵå(field)°¡ ÀÖ¾î¾ß ÇÕ´Ï´Ù.

 

      robots.txt »ý¼º ¿¹½Ã

# http://www.xxx.com/¿¡ /robots.txt ÆÄÀÏÀ» »ý¼ºÇÏ´Â ¿¹½Ã

User-agent: webcrawler

Disallow:

óÀ½ µÎ¶óÀÎÀº '#'·Î ½ÃÀÛÇÏ´Â µ¥, À̰͵éÀº ÄÚ¸àÆ®À̹ǷΠ·Îº¿µéÀÌ ¹«½ÃÇÕ´Ï´Ù.

webcrawler¶ó´Â ·Îº¿¿¡ ´ëÇØ ¾Æ¹«°Íµµ ºÒÇãÇÏÁö ¾Ê´Â´Ù´Â °ÍÀ» ¾ê±âÇÕ´Ï´Ù. ´Ù½Ã¸»ÇØ ¾îµð¿¡³ª °¥ ¼ö ÀÖ´Ù°í Çã¶ôÇÏ´Â °ÍÀÔ´Ï´Ù.

 

User-agent: lycra

Disallow: /

lycra¶ó´Â ·Îº¿¿¡ ´ëÇØ, '/'¿¡ »ó´ëÀûÀ¸·Î ¾Æ·¡ ÀÖ´Â ¸ðµç URL¿¡ ´ëÇØ Á¢±ÙÀ» Á¦ÇÑÇÏ°Ú´Ù´Â ¶æÀÔ´Ï´Ù. ¸ðµç URLÀº '/'·ÎºÎÅÍ ½ÃÀ۵ǹǷΠÀÌ°ÍÀº ÀÌ ·Îº¿¿¡ ÀÌ »çÀÌÆ® Àüü¸¦ ºÒÇãÇÏ°Ú´Ù´Â ¶æÀÔ´Ï´Ù.

 

User-agent: *

Disallow: /tmp

Disallow: /logs

¸ðµç ·Îº¿µé¿¡ ´ëÇØ /tmp³ª /logs·Î ½ÃÀÛÇÏ´Â URLÀ» ÀоÁö ¸øÇϵµ·Ï ¸·´Â °ÍÀ» ¾ê±â ÇÕ´Ï´Ù. .


3. À¥¹®¼­¸¦ ¸¸µå´Â °³ÀÎÀÌ ÇÒ ¼ö ÀÖ´Â ¹èÁ¦ ¼ö´ÜÀº ¾Æ·¡¿Í °°Àº ¸ÞŸÅ±׸¦ HTML ¹®¼­ ÀÛ¼º½Ã <head> </head> ÅÂ±× »çÀÌ¿¡ ³ÖÀ¸¸é(ÀÔ·ÂÇϸé) µË´Ï´Ù.

<meta NAME="ROBOTS" CONTENT="NOINDEX,NOFOLLOW">

        ¸ÞŸÅÂ±× Àû¿ë ¿¹½Ã

<html>

<head>

<title>Çѱ¹Çؾç´ëÇб³</title>

<meta NAME="ROBOTS" CONTENT="NOINDEX,NOFOLLOW">

</head>

<body>

          test

</body>

</html>


* ÀÚ¼¼ÇÑ ·Îº¿¹èÁ¦¿¡ ´ëÇÑ Ç¥ÁØÀº ÀÎÅͳÝ(www.robotstxt.org)À» ÀÌ¿ëÇϽñ⠹ٶø´Ï´Ù.

Âü°í»çÀÌÆ® :

http://tool.motoricerca.info/robots-checker.phtml

http://www.robotstxt.org







      << prev     1  2     next >>