Google Crawling Past HTML Forms

Regular readers of our blog will probably attest to the idea that we are big proponents of the Google search engine. Let's face it, we abide by Google's suggested rules of engagements and our efforts result in quality listings for our clients.

It is rare that we speak negatively of the Google "system" outside of rising Pay Per Click costs and a general lack of competition.

We have a new gripe.

Google Is Crawling Past HTML Forms

A recent post to the Google Webmaster Central Blog tells all about how Google is programming their spider to explore content behind HTML forms. The main reason for this work is to "discover new web pages and URLs that we otherwise couldn't find and index..."

They go on to speak about adhering to good Internet citizenry practices such as following noindex, nofollow, and robots.txt directives. They obviously won't get past a login form but they will quickly bypass a GET form pointing to database results.

This is an SEO nightmare.

Optimized Your Post-Form Pages Much?

If you've done your homework, your database site has a navigational flow to the deep content that you want indexed. Your .php and .asp form results are undoubtedly pure user functionality with little care to metatags and headers.

The forms are just a way for your visitors to get right to the goods, right?

Well, now your non-optimized, form-result pages with non-specific metatags (if they exist) are available to the Googlebot.

Don't build pages for Google!

How many times have you heard that statement? Don't build content for Google. Don't optimize content for Google. Yeah, right.

By definition their search engine looks for, and appropriately ranks, relevant content. The only way that content gets relevant is if someone makes it relevant. They're about to climb through some forms and find a wealth of unoptimized, irrelevant trash.

In their own words, "This experiment is part of Google's broader effort to increase its coverage of the web."

Unfortunately this effort will uncover some real garbage. I have a feeling most of that garbage is titled "Search Results".

SEO Isn't Just Metatags

What do you think of when you hear or read the acronym SEO? Is it good metatags? Rich content and links? Or maybe Pay Per Clicks (PPCs)? Well if you put all those together in your website you would have what I might have said was good SEO.

That was until I joined the First Scribe Team. I knew when I accepted my position with them I would have the opportunity to learn and grow as a web developer but never thought I would see so many new ideas and techniques in the realm of SEO.

While some of it is a "Why didn't I think of that?" type of scenario, there a lot of technical, behind the scenes, ways to improve websites that I didn't know about.

Savvy Visitors

As the Internet matures and users become more sophisticated, sites that used to perform well just don't rank as high as they used to. It is far more than looking pretty and giving users enough information to buy something from you. It doesn't mean anything unless the user can find your site.

Let's be honest here for a moment -- When was the last time you searched for something and you went past the first page of results from your favorite search engine? Once or twice, maybe. If you didn't find what you were looking for on the first page, you probably went back to your search terms and modified them.

SEO Works

I have looked at traffic reports from some sites that this team has taken over and improved from an SEO standpoint. The results are remarkable. I can say I have never seen such a drastic improvement in traffic in such a short period of time. All due to the SEO techniques used here at First Scribe.

Keyword Research and Design lead to Increased Conversion

In early June Stoney Wolf Productions, which sells videos for sportsmen contacted us for help. They were slipping in the rankings for their keyword terms and showing decreased traffic and a decreased conversion rates across the board.

Our first step in the process was to make the site have more of a structured feel. The past design had quite a bit of text on the home page and the content area had around a dozen different directions available from the home page. The content in the shopping cart area was organized into a few broad categories with too many choices.

Our Approach

First, we re-organized the shopping cart area into more specific categories to make things more intuitive to find. Next we cleaned out the content area of the home page, trimmed the text down and added some nice photographic images to depict a few of the more popular categories. In the end, only 4 categories were show-cased on the home page.

After design came the all important keyword research. The first thing we looked at was the old keyword terms in use. We saw a lot of fishing video, hunting video, etc. but became evident that the last time this site was optimized there was not YouTube and internet videos were not the norm.

In today's searchable content videos are all over the place. We concluded people were coming to the site to see hunting videos, and when they found out it was DVD's they quickly left the site which was hurting our conversions. We researched new terms around DVD's and VHS to get qualified traffic to the site and raise conversion rates. The results surprised even us...

June 2007:

July 2007:
We made our changes the last week of July.

August 2007:
The first full month after the changes have gone live. A 7.1% conversion rate blows the previous conversion rates out of the water.

September 1-20 2007:
About 2/3 of the month as an example. Conversion rate is over 8%

Quality analytics helped us track the conversions on the site allowing us to show the client the immediate results of our work on their site.

Differing Opinions of SEO

I received a comment on a past blog post today and it really struck me as great feedback. Essentially the comment stated that they like our blog because their opinion(s) of SEO often differ from my opinion(s) here at First Scribe.

Perfect!

SEO is an evolving skill set where you must continue to learn or the competing market will pass you by. There will always be more competition, more finicky visitors and refinements to search algorithms. We don't work for the search engines so we need to look for subtle SEO symptoms and then track the symptom to the source.

I try to dedicate 20% of my time to competitive research but it usually amounts to less than 10%. I will start by saying this - If you find an SEO symptom/trend, you must test it yourself or there's no point in wasting your time. NOBODY is going to tell completely spell out their philosophy. You must read between the lines and try it for yourself.

Here are the sources of my research, in order of preference:

1. SEO forums - The forums tend to be chock full of new SEO staffers who freak out and ask questions every time their rankings slide. These cries for help have turned me on to algorithm changes earlier than any other source.
2. Competitors - I generally don't look at my competitor's websites because SEO firms rarely dedicate their full efforts to their own website. Go to their latest portfolio entries and look for strategy.
3. Blogs - Find an active blog written by someone you feel is credible. You don't need to agree with them, in fact it's better if you disagree but you need to trust that they aren't intentionally disseminating false information.
4. My own websites - The last place I look (for research purposes) is at my own websites. If I'm looking at them for research purposes, I specifically look at the sites with volatile SERP results.

Changes in Google caching?

We are noticing a significant difference in the amount of time it takes Google to cache and subsequently rank a new page.

Two months ago, 3 to 6 weeks was a fair estimate to cache status with a PageRank of 0/10 in Google. A quality page rank would arrive somewhere on order of 3 months later.

Now we are noticing a cache time of 6 to 10 weeks with a grey PageRank bar and quality rank on order of 4 months later. This is calculated off of existing pages with an exiting Page Rank of 5/10, and a quality text link pointing to relevent content.

Could it be that Google is sending people to the Sandbox for a longer period?

The only good advice is to consistantly add quality content to your website. MSN seems to pick it up first and Google will appreciate it later.

Local Search Is Real

Rarely will an Internet Marketing Firm give away their secrets but here is a great one:

People use Local search in large numbers - and in many cases it's FREE.

What is Local Search?

The more things change, the more they stay the same. A local search is not unlike opening up an old fashioned Yellow Pages™ on the Internet.

Demographically speaking, there is a vast number of people on the Internet who do not know how to make the search engines work for them. They get search and receive too much irrelevent information. The get confused and they search for something familiar - a Yellow Pages.

Prominent Local Search directories

There are many local search directories but here are a few of the prominant locations:

• www.google.com/local
• local.yahoo.com/
• www.dexonline.com

Local With Maps

Yahoo and Google have the added bonus of seeing your search over a map which is nifty but the text-based directories seem to drive more traffic (for various reasons). Cover your bases and submit to each of them.

Listings are Free

Free listings are available for the city listed in your mailing address. Most of the directories offer a broader area for an additional fee but beware - the fee structure is steep.

Ken Kralick
SEO Specialist
http://www.firstscribe.com/