How to Make a robots.txt file

Sometimes you want parts of your website to be excluded from search engine listings. You don't want the search engine to see these pages in your website. There could be a lot of reasons for this like the pages are incomplete or not ready for search engine listing. Or maybe you want to keep the information private.

If you use google adsense and you use their search engine function on your website you also have to exclude the search results page from the google index bot. This is a requirement of google. So as a minimum you should have this noted in your robots.txt file.

Putting a robots.txt file is easy and I am going to show you exactly how to do it.

Let's Make a robots.txt file

What is a robots.txt file? It is a simple text file that you create in a simple word processor program like notepad. It has commands in it that tell spiders specific instructions like don't crawl certain pages in the site.

A simple robots.txt file

Create a document in notepad and save it as robots.txt

What does SEO include?

SEO can be broken down into two major task groups:
1. On-page optimization :-

2. Off-page optimization :-

1. On-page optimization :-

Includes all the work associated directly with your web site and may include some or all of the following:

- review of site architecture
- keyword research (very important)
- writing page titles
- recommendations on additional content
- tweaking of existing content
- adding appropriate image “alt” attributes
- review or modification of the “robots” meta tag
- internal linking
- review of server configuration

The above steps often include sub-tasks and may be very complex. The goal of SEO is to get the right visitors to the web site so they can decide if they will take further action (such as buy a product). SEO is only part of the equation.

Once a visitor arrives at a web page the overall user experience will determine what actions they take. This is why many use the phrase “Content is King” in web site development. Even so, professional marketers understand how the economy and competition can affect what happens in the marketplace no matter what you include on your web site. This is why it is so important to make your web site an integral part of your marketing strategy; building in the necessary ingredients to secure positive results.

2. Off-page optimization :-


Includes all the efforts made away from your web site. It usually takes place after the on-page optimization is completed and may include some or all of the following, depending on your budget and marketing goals:

- Link building
- Social bookmarking
- Press releases
- Articles
- Blog post
- Web based ads (especially with link text)
- Appropriate forum posts (no spamming)
- Appropriate blog comment posts (no spamming)

Often, off-page SEO should include traditional forms of advertising such as printed ads in newspapers or magazines. Even radio and TV may be appropriate. All of these bring traffic to the site and may result in a link “recommendation” by a visitor.

What is SEO?

“Search engine optimization is about putting your site’s best foot forward when it comes to visibility in search engines.”

Similar to the way a librarian uses the Dewey Decimal System to categorize a book based on it’s subject matter, Google and other search engines read each page of your web site and categorize (index) them based on the information they find.

Which of the following statements is true?

Get highly ranked in Google and become famous!
Get famous then you will be highly ranked in Google!

Good SEO is good marketing:

• Identify your market
• Add keyword rich content to your site that will draw your market
• Get your site noticed and recommended using all practical means

SEO includes numerous minor adjustments to each major page of your web site to ensure it is properly indexed by Google and recognized as a quality source for something specific. Adjustments may include such things as writing unique and useful page titles, adding alt attributes to images, organizing the content of the page so it is easy for humans and robots to follow to name but a few.

Google sends out robots (also called spiders) to crawl (visit and read) each page of a web site. These spiders take note of many characteristics of each page. Google takes the data collected from your web site and using an algorithm decides what the page is about and how it should be categorized (that is: under what keywords will this page be found?).

In addition, Google takes into account how many other sites “recommend” your web site. A recommendation is a link from outside your domain to a page in your web site. The more quality backlinks you have the more important your web site becomes. How such links are structured and where they are placed has much to do with how well your web site does in the rankings.