Writing /var/www/bithack.se/projects/methabot/data/cache/9/938b3cee3e8e5f0f141dae1e719ce810.i failed
Unable to save cache file. Hint: disk full; file permissions; safe_mode setting.
Writing /var/www/bithack.se/projects/methabot/data/cache/9/938b3cee3e8e5f0f141dae1e719ce810.xhtml failed
Differences
This shows you the differences between the selected revision and the current version of the page.
| docs:robotstxt 2009/02/20 12:41 | docs:robotstxt 2009/02/20 12:42 current | ||
|---|---|---|---|
| Line 15: | Line 15: | ||
| ====Disallow==== | ====Disallow==== | ||
| This directive is used to disallow the web crawler from accessing a part of the website matching the given pattern. Example: | This directive is used to disallow the web crawler from accessing a part of the website matching the given pattern. Example: | ||
| - | <code> | + | <code robots> |
| Disallow: /private/ | Disallow: /private/ | ||
| </code> | </code> | ||
| ====Allow==== | ====Allow==== | ||
| Used in combination with disallow to allow access to a document of path inside the disallow-pattern. Example: | Used in combination with disallow to allow access to a document of path inside the disallow-pattern. Example: | ||
| - | <code> | + | <code robots> |
| Disallow: /private/ | Disallow: /private/ | ||
| Allow: /private/public.html | Allow: /private/public.html | ||