Get Free Quote

Dynamic Sites and Mod Rewrites

by: Susan Esparza and Aaron Landerkin, May 2005

The problem for many dynamic sites is that the long URLs and multiple parameters often will cause a spider to give up on indexing a page or session ids will cause a page to get indexed multiple times causing a duplicate content penalty. The result is underexposure in the first case and a penalty in the second. Either way, the site will suffer.

For pages with many query string parameters or with long query string parameter values, most search engines will try to retrieve a page without the query strings attached and then add the query parameters until it gets the page with the simplest URL possible (or until the spider gives up on retrieving a valid page).

In addition, note that the same page can exist twice in the Google index, with the only difference being the order that the query parameters appear in the URL. While the page functions properly no matter the order of the URL parameters, each entire URL is distinct enough for Google to consider it unique. The result is that the same page has been indexed twice in Google, resulting in a potential "duplicate content" problem.

The cleanest way to resolve this issue is to have the URLs appear to be static to the search engine spider. By static, we mean that the URL should have no query strings parameters within it.

In those cases, one possible solution is to mod rewrite the url so that it won't be as confusing for a spider. Mod rewrites are a function available to those running on an Apache server. The function of a mod rewrite is to write the URL of a page on the fly as it is called from the database. Rather than displaying a long URL full of parameters which spiders don't like--Google says they stop trying if there are three or more parameters in a URL--the mod rewrite will use regular expression parsing to create a new, more readable URL.

The method of implementing a mod rewrite relies on the Apache Web Server, which uses. The Apache Web Server allows for a configuration file, called .htaccess, which resides inside the root folder of the web site. In this file, you may use code to issue redirects, change default pages, and also use mod rewrite. To implement mod rewrite, the .htaccess file must contain:

RewriteEngine On RewriteBase /

Then, you may enter the mod rewrite code to perform specific re-mappings.

RewriteRule ^Category1/Category2/?$ /Example/category3.jsp?Ne=89&N=98&categoryId=98&pCategoryId=89&gpCategoryId=0 [L]

Obviously, you would not want to map every individual category and product, as the script could become enormous and slow the performance of the web server to an unacceptable level. Mod rewrite gets around this problem by using Regular Expressions, which enable you to match patterns instead of individual pages. Using regular expressions, you can use a static URL like

Although this is not as descriptive as /Category1/Category2/, the presentation of a static page is most important.

More information about mod rewrite and regular expressions can be found at: