geeklabor_legacy/static/robots.txt

User-agent: *
Disallow: /feed.atom.xml
Disallow: /categories/*
Disallow: /authors/*
Disallow: /author/*
Disallow: /tags/*
Disallow: /page/*
Sitemap: https://geeklabor.de/sitemap.xml
add a minimal robots.txt 2018-12-20 23:21:04 +00:00			`User-agent: *`
			`Disallow: /feed.atom.xml`
disallow all taxonomy subdirs for web crawler 2019-02-05 14:28:00 +00:00			`Disallow: /categories/*`
			`Disallow: /authors/*`
			`Disallow: /author/*`
			`Disallow: /tags/*`
disallow paging sites for web crawler 2019-02-05 15:05:33 +00:00			`Disallow: /page/*`
add a minimal robots.txt 2018-12-20 23:21:04 +00:00			`Sitemap: https://geeklabor.de/sitemap.xml`