Case-Based Critical Thinking Question
Tyler does not want parts of his website indexed, so he is creating a robots.txt file.
-Tyler uses the code Disallow: /to instruct bots ____.
A) not to index the entire website
B) to index the entire website
C) to index only the main directory
D) to index the website at their own discretion

Question

Quizplus · Accepted Answer

The Disallow command in a robots.txt file instructs bots not to crawl or index certain parts of a website. In this case, Tyler is using Disallow: /, which tells bots not to index anything on the website, making A the best choice.