Opened 6 years ago

Closed 6 years ago

#1765 closed unknown (fixed)

http://alpha.bewelcome.org/robots.txt needs to Disallow everything

Reported by: guaka Owned by: planetcruiser
Priority: major Milestone: 1.1
Component: ServerSetup Keywords: seo, robots.txt
Cc:

Change History (3)

comment:1 Changed 6 years ago by jsfan

This file is not specific to alpha. The robots.txt is in git and thus shared between alpha and www.

comment:2 Changed 6 years ago by guaka

Then there should be a mechanism to have a different robots.txt.

Otherwise there's tons of duplicate content, and people might actually arrive at alpha when searching for stuff.

Last edited 6 years ago by guaka (previous) (diff)

comment:3 Changed 6 years ago by jsfan

  • Milestone changed from unassigned to 1.1
  • Resolution set to fixed
  • Status changed from new to closed

I have now removed robots.txt from the repo and created one for alpha on puma just as the one you suggested. With the next release www will get a static robots.txt as well.

Note: See TracTickets for help on using tickets.