Problems and solutions in optimizing a database-driven website for the search engines
There are problems optimizing a website for the search engines, where the pages of that site are created from pieces pulled from a database. If you use cgi, Cold Fusion, Microsoft ASP, or other various proprietary shopping carts like xCart -- these make the kind of web pages where the URL (as you see it in your browser's address bar) contains question marks ("?") or other symbols. The links within this kind of website don't go to existing html pages. The links are set up so that when you click on them, the pages are created instantly for you from information and HTML code stored in a database on the server. It is all put together for you instantly "on the fly" when you click on the link.

Basically, search engine spiders aren't smart enough to figure out how to interact with a database to create those pages, so sometimes they never make it past the first page of the site. While indexing your site and trying to follow the links from your main page, if a search engine spider finds a question mark in the URL that you are linking to, the spider will disregard that link and move on. If your whole site is built this way, the busy spider leaves your site without being able to index anything.

None of the search engines can interface directly to your database and read what is in it -- they just aren't set up to do that.

If you have a database-driven site then special actions need to be taken handle the behavior of the search engine spiders. We do not cover them in detail on this page, this is just an outline of what you need to do:

The main (home) page of the site should link to several static pages (which will need to be created if they don't already exist) which then interact with the database. It helps to have about a dozen of these static (non-changing) pages: these are .html pages which are NOT created by the database "on the fly". Typically they would describe the main categories of what you are selling. The links FROM these "category" pages can create other pages based on information pulled from the database, but these static category pages themselves should not be created newly every time someone clicks a link from the main page of the site.

Each of these static category HTML pages should then be optimized using the appropriate key word phrases for that category. These static HTML pages will then be picked up and indexed by the search engines after going through the submission process. You stand a much better chance of getting good search engine rankings for your website by creating and optimizing static pages like this.

There are other more technical ways to handle it, some of which involve changes to the configuration of the server, and there are various software solutions which will make your site look more friendly to a search engine -- but this method of making some static "category" pages is usually the simplest to implement and meets with the least resistance from your webmaster. The other methods one can use are described in this article at the Web Developer's Journal and depend upon whether one is using ColdFusion, CGI scripts, ASP, or some other method of displaying the information from your database on your web pages.

For another tutorial (with solutions for your specific problem) on this subject, go here: Search Engine Ethics.

For another view on this, see Paul Bruemmer's article about optimizing dynamic sites.

If you have a database-driven site, contact us for an inspection and suggestions about what else can and should be done. We've seen some beautifully designed websites--very pretty, interactive and doing a good job of selling products--which were invisible to the search engines. The search engine spiders are simply not designed to crawl through the site and index pages created from a database in response to clicks from visitors.

The workaround outlined above using category pages work well to get your site listed and doing better in the search engines. Unfortunately, it involves some re-design of the way the site works. Your webmaster may have some resistance to this because it means a lot of work -- but it's the only way we know of to get good placement in the search engines for a website built around a database of products and information about them.

Source: http://www.wordsinarow.com/database.html

Module mod_rewrite:
Rewriting URLs With Query Strings
------------------------------------------------------
by
Dirk Brockhausen
------------------------------------------------------
The Apache server's module mod_rewrite is typically
used to rewrite one URL to turn it into another one.

Example:
--------
RewriteRule ^index\.html$ homepage.html

"^index\.html$" is a regular expression.
"^" represents the beginning and "$" the end of a
string.
The dot "." in a regular expression is a meta symbol
(wildcard) and signifies any random character.
If you want to use an actual, real period/dot "."
character, you will have to mask it with a preceding
backslash.

The next example is slightly more complicated:

RewriteRule ^(.*)/(.*)/(.*)/(.*)$  shop.cgi?$1=$2&$3=$4

This rule rewrites URL:
< http://www.yourdomain.com/cat/cars/product/bmw >

into:
< http://www.yourdomain.com/shop...roduct=bmw >

This way, you can submit static URLs (the first one
in our example above) to the search engines, with
visitors still being directed to a cgi script with
dynamic parameters.

The two examples above are part and parcel of
mod_rewrite's standard features.

Matters tend to get more sophisticated and involved,
however, if you want to redirect URLs with different
tags to various different web pages. This procedure
will now be explained in detail.

For a practical example, let's consider three PPC
campaigns with Overture. In Overture's admin center
your will assign your domain's URL, followed by unique
tags to track the traffic generated, e.g.:

http://www.yourdomain.com/?ov1
http://www.yourdomain.com/?ov2
http://www.yourdomain.com/?ov3

Overture actually recommends tracking URLs. However, we
don't want to restrict ourselves to mere traffic
analysis, we also want to guide visitors to different
pages.

To this effect, the following ".htaccess" file entries
are generated:

RewriteEngine on
Options +FollowSymlinks
RewriteBase /
RewriteCond %{QUERY_STRING}  ^ov1$
RewriteRule ^$ /product1.html [L]
RewriteCond %{QUERY_STRING}  ^ov2$
RewriteRule ^$ /product2.html [L]
RewriteCond %{QUERY_STRING}  ^ov3$
RewriteRule ^$ /product3.html [L]

This code example makes use of rewrite conditions by
analyzing the query string, i.e, everything following
the question mark "?" character.

Our line:

RewriteCond %{QUERY_STRING} ^ov1$

is made up of the following three parts:

  Directive: "RewriteCond"
 TestString: "%{QUERY_STRING}"
CondPattern: "^ov1$"

TestString is a server variable. CondPattern is a
regular expression. "^" represents the beginning and
"$" the end of a string.

It follows that the query string must exactly be "ov1"
and nothing else. For example, "ov11", would not
work: in this case, the condition would not be met.

The next line:

RewriteRule ^$ /product1.html [L]

will rewrite the URL "^$" into the new URL
"/product1.html".

The regular expression "^$" matches, if there is no
file name included in the URL. This is exactly the case
for these URLs:

http://www.yourdomain.com/?ov1
http://www.yourdomain.com/?ov2
http://www.yourdomain.com/?ov3

Here, we only have the domain name, followed by
"/?ov1", etc. Thus, there is no file name included.

A URL like the following:

http://www.yourdomain.com/page1.html?ov1

will not match the condition.

The [L] flag stops the rewriting process
if a condition matches.

You can achieve an even shorter form by making use of
backreferences, e.g.:

RewriteEngine on
Options +FollowSymlinks
RewriteBase /
RewriteCond %{QUERY_STRING}  ^ov(1)$ [or]
RewriteCond %{QUERY_STRING}  ^ov(2)$ [or]
RewriteCond %{QUERY_STRING}  ^ov(3)$
RewriteRule ^$ /product%1.html? [L]

By adding the brackets in "ov(1)", etc. the value
included within brackets can be stored in a variable.
This variable can then be used for further processing.

In our second example further above:

RewriteRule ^(.*)/(.*)/(.*)/(.*)$  shop.cgi?$1=$2&$3=$4

we worked with backreferences already. Here, the
variables were "$1", "$2", etc.

If you want to make use of variables from a RewriteCond
entry in the following RewriteRule, the variables have
to adhere to the syntax "%1", "%2", etc.

There's a further detail to be found in the expression
"/product%1.html?": the trailing question mark "?"
character will inhibit transfer of the the query string
from the old to the new URL. Without this trailing "?"
character, we would get the following results:

old URL - http://www.yourdomain.com/?ov1
new URL - http://www.yourdomain.com/product1.html?ov1

As you can see, the query string "ov1" would be added
to the new URL, which is not desirable.

The above goes to illustrate how you can make use of
very sparse mod_rewrite syntax to efficiently
manipulate URLs for your SEO campaigns.

http://www.fantomaster.com/faa...ngurls.txt

Dynamic sites require highly specialized search engine marketing strategies that differ from those used for static sites. It's still hard to get dynamic sites indexed unless they're properly optimized. While search engines say they now index dynamic sites, and they do; many times it doesn't happen without a little help. And certainly the positioning of pages is another issue.

There are a number of strategies that can be used to convert your dynamic URLs into search-engine friendly URLs. Before we get into that, let's look at how the dynamic databases used by e-commerce sites and other large sites are created and why they are hard to index.

What Keeps Dynamic Sites Hidden?

Dynamic pages are created on the fly with technology such as ASP, Cold Fusion, Perl and the like. These pages function well for users visiting the site, but they don't work well for search engine crawlers.

That's because dynamically generated pages don't actually exist until a user selects the variable(s) that generate them. A search engine spider can't select such variables, so the pages don't get generated or indexed.

The big problem is that crawlers such as Google can't read the entire dynamic database of URLs, which either contain a query string (?) or other database characters (#&*!%) known to be spider traps. Because search crawlers have problems reading deep into a dynamic database, these crawlers have been programmed to detect and ignore many dynamic URLs.

We recently increased a client's search engine potential from six (6) to six-hundred fifty-nine (659) pages. Considering that Google saw only a half-dozen pages originally, we think hundreds of optimized pages will significantly increase our client's search engine visibility.

Making Dynamic Sites Visible

There are a few dynamic-page optimization techniques that can be used to facilitate the indexing of dynamic sites. The first that comes to mind is to make use of static pages. There are also many fixes to convert dynamic URLs to search-engine friendly URLs. Another good way to get visibility is to use paid inclusion and trusted feed programs that guarantee the indexing of dynamic sites or a number of click-throughs.

Static Pages. Place links to your dynamic pages on your static pages, submitting your static pages to the search engines manually according to each search engine's guidelines. This is easily done with a Table of Contents displaying links to dynamic pages. While the crawlers can't index the entire dynamic page, they will index most of the content.

Active Server Pages (ASP). XQASP from Exception Digital Enterprise Solutions is an excellent tool for converting dynamic ASP pages into search-engine compatible formats.

For example, the following URL contains both "?" and "&," making it non-indexable:

   http://www.planet-source-code....CodeId=769

Below, it has been made search engine-friendly (all "?" and "&" and "=" characters replaced with alternate characters):

   http://www.planet-source-code....owCode.htm

Once you've converted the URL, don't forget to use search engine optimization techniques to modify the HTML tags and content within the tags before submitting all pages in accordance with each search engine's submission guidelines.

ColdFusion. This might be an easy fix. Reconfigure ColdFusion on your server so that the "?" in a query string is replaced with a "/," passing the value to the URL. You will still have to deal with optimization of your pages and making your site respond quickly when a crawler does come by for a visit.

CGI/ Perl. Path_Info and Script_Name are environment variables in a dynamic application containing the complete URL address, including query string information. Your solution is to write a script that removes all the information before the query string, making the remaining information equal to a variable and using that variable in your URL address. Again, optimization is required to show up in the top editorial listings.

Apache Software. The Apache server has a rewrite module (mod_rewrite) available for Apache 1.2 and beyond that converts requested URLs on the fly. You can rewrite URLs containing query strings into URLs that can be indexed by search engines. The module doesn't come with Apache by default, so find out from your Web hosting firm whether it's available for your server.

Indexing Dynamic Sites With Paid Inclusion Programs

Most major search engines now offer paid inclusion and trusted feed programs based on refresh indexing or cost-per-click. Engines offering such programs include AltaVista, AskJeeves, FAST AllTheWeb, Inktomi, LookSmart, Lycos, and Teoma.

These programs alone are not good enough for search engine positioning. When indexing dynamic sites through XML feed, you must first ensure the site is properly optimized using professional search engine optimization (SEO) techniques.

Good SEO contractors have access to Web-based automated feeds with creation and management application to generate XML optimized feeds for multiple search engine inclusion programs. Such contractors can map any large size e-commerce site's entire catalog, generating an automated XML optimized feed.

The key to this XML procedure is keyword matching between the dynamic site content and various search engine databases. Using special filters and parameters, this process then generates thousands of keywords with page-specific meta information. The result is a distinctive representation of each product page on the target search engine, and a more accurate representation of your dynamic site's services, products, etc.

Internet users search with mind-boggling combinations of your strategic keywords. Users at various search levels must find you before they find your competition. That's why keyword analysis and research is so important to the success of your SEO campaign. Professional optimization techniques covering site architecture, copywriting, editorial linking, etc. are also important for top positioning on major search portals.

Source: http://www.searchengineguide.c...2_wi1.html

Invite Search Engine Spiders Into Your Dynamic Web Site
by Larisa Thomason
"Come into my parlor and index my content," said the dynamic Web site to the search engine spider.
February 28, 2001

But did the spider actually visit? Maybe not. Dynamic page content is often invisible to most search engine spiders, so it never gets indexed. Increase the traffic to your dynamic site by making your valuable content visible to search engine spiders.

Dynamic Pages Are Easy To Maintain
The content of static pages doesn't change unless you actually code the changes into your HTML file: open the file, edit the content, save the file, and upload it to the server. All search engine spiders can index static Web pages.

A dynamic Web page is a template that displays specific information in response to queries. Most of the page content comes from the database connected to the Web site. Visitors love them since they get quick access to the information they want. These sites are easy for webmasters to update: as product offerings or prices change, just edit your database instead of hundreds of individual Web pages.

Search engine spiders have a much tougher time with dynamic sites. Some get stuck because they can't supply the information the site needs to generate the page. Other spiders deliberately stay away from dynamic pages to avoid getting trapped in the site.

What Was The Question Again?
Visitors find information in a dynamic site by using a search query. That query can either be typed into a search form by the visitor or already be coded into a link on the home page - making the link a pre-defined search of the site's catalog. In that later case, the portion of the link containing the search parameters is called a 'query string.'

But a search engine spider doesn't know to use your search function - or what questions to ask. Dynamic scripts often need certain information before they can return the page content: cookie data, session id, or a query string are common requirements. Spiders usually stop indexing a dynamic site because they can't answer the question.

If the spider does accidentally wander deeper into your site, it could inadvertently get caught in a "spider trap": a badly written CGI script that requests information the spider can't supply. Then the spider and your server navigate a never-ending loop where a request for a page is met with a request for information.

Getting a spider trapped inside your server is not just bad for the spider. The repeated requests for pages can crash the server. If you share server space with other Web sites and have a problem with site downtime, ask your Web host to check for CGI script problems on other sites.

It's All In The Name
A page's actual URL address often poses a problem too because most dynamic page URLs contain query strings. Here's an example of the URL for a book search result page on Barnes and Noble's Web site:

http://shop.barnesandnoble.com...quiry.asp?
userid=2IMXLT5XN1&mscssid=QEUFGRFF5X2G9H2UCMJQLAKJ8
JV83FMD&isbn=0452269350

Look closely at the URL. See the question mark after /isbnInquiry.asp? Most search engine spiders get to the "?" in the query string and stop indexing because of the probability of getting caught in a spider trap.

Attract Spiders To Your Web
So, you've got all this invisible content - what do you do? Search engines know about the problem, but most have shown very little interest in addressing it. Infoseek and HotBot are the exception. Their search engine spiders can index dynamic page content, but don't do it automatically. You have to invite them in.

HotBot recommends that you submit your dynamic page with all the arguments added onto the URL (www.website.com/products/search/product_query.asp?prod_id=22929). You can also submit a static page that contains links to the dynamic URLs you need indexed.

Infoseek's spider, called Slurp, will index dynamic pages that you submit, but won't crawl through your dynamic Web site by default.

You do have options to get indexed by the other search engines, but no matter which you select, you'll have to spend some time and effort to make sure your dynamic content gets indexed.

Add Dynamic Links To Static Pages
Include links to important dynamic content on your static pages. The simplest way is a straightforward table of contents page that links to your most important dynamic pages. It gives spiders a way to index content without having to answer any questions. If you have a small site with few products, this is a quick way to get more of your content indexed.

However, the table of contents won't help you with search engine spiders that stop at query strings. Increase your chances by including good, descriptive links to your major product categories on a static products page. Search engines that stop at query strings will still index the content of the products page - including your link titles. Other search engines that can follow dynamic links can visit the actual dynamic page content without a query.

Remove Query Strings From Dynamic URLs
Amazon.com uses this method to get its product selections indexed by search engines. For instance, a search on Google for Rachael Carson's book, Silent Spring, returns a result that takes you directly to the appropriate dynamic page at Amazon: www.amazon.com/exec/obidos/ISB...womensha/. Because the URL doesn't contain any query strings, all search engines can index Amazon's product line.

This method works, but it's also the most technically demanding solution. If you decide to use this method, you can select from several different options, depending on the type of Web server you use and the software you're using to integrate your database with your Web site:

Cold Fusion: Reconfigure your Cold Fusion setup to replace the "?" in a query string with a '/' and pass the value to the URL. The browser interprets that as a static URL page.

Instead of http://www.mystore.com/products.cfm?prod_id=22343, you get a string like this: http://www.mystore.com/products.cfm/22343.

CGI Scripts: Path_Info (or Script_Name) is a variable in a dynamic application that contains the complete URL address (including the query string information). Write a script that strips out all the information before the query string and set the balance of the information equal to a variable. You can then use that variable in your URL address.

Apache: has a special rewrite module that allows you to translate URL's containing query strings into URL addresses that search engine spiders can follow. The module, mod_rewrite, isn't automatically installed with Apache software. Check with your Web host or administrator and see if it's available on your server.

Visit the Apache Web site for more information on the mod_rewrite module.

Active Server Pages: Most search engines will index .asp pages if the "?" is removed from the URL. XQASP offers a product that will automatically remove the query strings from your .asp pages and replace them with "/" marks.


A note of caution: these four methods make your dynamic page appear to have its own sub-directory, so the browser will look for images and links there. You can completely avoid broken links and pages by using all absolute URL addresses on your page, but that will make maintenance more difficult later. Alternatively, use URL addresses that are relative to the root directory of your site, not the document. Use /homepage.htm instead of ../homepage.htm and you'll be fine.

Remember The Rules
Don't get so caught up in modifying your page design or URL addresses that you forget the basic rules for search engine optimization. Your pages need to have good content, META tags, a high link popularity score, appropriate keywords, and more before you can climb to the top of the search engine ranking.


 
Source: http://www.webdevelopersjourna..._site.html

Optimizing a Frames-Built Website
for the Search Engines
SEO for Frame-based Websites
If you have a frames-built website, you probably won't like our advice: If your website was built using frames, re-design it so it no longer uses frames.

We know that is harsh advice, but rather than pull punches and do workarounds that don't work all that well, you might as well know that, in our experience, any website that has been fully optimized for the search engines has done away with using frames. To get the best rankings in as many search engines as possible, it is a necessity to re-design the site and get rid of the frames entirely.

Most search engines (and there are exceptions) will not read any further into your site than your index.html page if the site is built in frames. They don't look at the information which is called for and displayed by the frames themselves. The search engines only look at the info between the <noframes> and </noframes> tags on that page.

So, what CAN you do if you are determined to stick with frames?

Put in a <noframes> tag in the <HEAD> of your main frame page, and cram it with what amounts to a whole web page worth of text, headings, alt tags, links, and so on. Put the same things in the noframes tag that we advise you to put in a web page on our Search Engine Optimizing page.

Often we find that the only thing in a <noframes> tag is some smart-alec comment about how you should upgrade your browser to one that supports frames. That's not enough as far as the search engines are concerned.

For best Search Engine Optimizing results, build a <noframes> tag that contains duplicates of the text, headings, meta tags and alt tags (basically everything on our Search Engine Optimizing page) from the info displayed in the frames area of the page.

In other words, build a basic web page INSIDE the <noframes> tag that contains all the points we mention on our Search Engine Optimizing page.

If you are going to do that, however, be warned that it is more trouble than simply re-designing the website to do away with Frames entirely. Hence our advice: Redesign the website to do away with frames.

Other factors mitigating against frames:

Using frames can cause the browser's "back" button to fail to work, trapping people in a page of your website. This can be very frustrating for the person visiting the site.


Unless you are very careful, your links can cause duplicate frames to come up within your frames.


Clicking on a link from a framed site frequently causes the linked site to show up inside your frame. Users don't like that.


It is nearly impossible to bookmark a specific page in a frames-built website. Instead, all you can bookmark is the index.html page displaying the frames. The content of the various frames pages themselves cannot be specifically bookmarked by the average visitor.
This subject of "problems with frames sites" comes up a lot - here is an article about it at PromotionData.com from January 2003.

All in all -- if you really want the extra traffic that full search engine optimizing can give your website, then you're better off re-designing it to get rid of the frames. If you need help redesigning your website so it no longer uses frames, feel free to contact us. We've done it many times and are experts at the easy way to take websites out of frames in order to fully optimize them for the search engines.

Source: http://www.wordsinarow.com/frames.html