Situation-Based Question
Imagine you’re managing a blog with a private section meant only for internal use. You don’t want these pages to show up in search engine results, but you also want other public pages to rank. What is a meta robots tag, and how would you use it to handle this scenario?
Exact Answer
A meta robots tag is an HTML tag used to instruct search engine crawlers on how to index or follow links on a webpage. It works by specifying directives such as “noindex” to prevent a page from appearing in search results or “nofollow” to stop link equity from being passed to external links.
Explanation
The meta robots tag is a key tool for controlling how search engines interact with your site. Placed in the <head> section of your HTML, this tag communicates specific instructions to crawlers about indexing and crawling behavior.
Here’s how it works:
- Directives
index: Allows search engines to index the page (default behavior).noindex: Prevents the page from being indexed and appearing in search results.follow: Allows search engines to follow and pass link equity to the links on the page (default behavior).nofollow: Prevents search engines from following links or passing link equity.- Combinations: For example,
noindex, followhides the page but lets crawlers follow its links.
- Use Cases
- Private Content: Use
noindexfor pages like internal documents or admin panels. - Duplicate Content: Prevent duplicates from appearing in search results by using
noindex. - Link Management: Use
nofollowon affiliate or untrusted links to prevent link equity from passing.
- Private Content: Use
- Syntax
Add the meta robots tag in the<head>section of your webpage:
<meta name=”robots” content=”noindex, follow”> - Impact
- Helps maintain a clean search index by excluding irrelevant or sensitive pages.
- Manages link equity to maximize SEO potential.
- Provides more granular control over SEO behavior than robots.txt.
Example
Let’s revisit your blog scenario. You want your public articles to rank, but internal team pages (like meeting notes) must stay private.
Solution: Add the following meta robots tag to your internal pages:
<meta name=”robots” content=”noindex, nofollow”>
This ensures:
- These pages won’t appear in search engine results.
- Search engines won’t crawl or pass link equity from links on these pages.
For public pages, you keep the default index, follow directive so they rank and contribute to your SEO.