Back around the 1st of February I started a new website. It had a
couple of inbound links from other sites of mine and I did a manual
submission to Google, MSN, and Yahoo. Since Teoma doesn't take manual
submission, I have to wait for them to find me which they finally did
over the weekend.

Now, after five weeks, the search engine results are rather
interesting. Here is the list of indexed page counts for queries of
the form "site:worldbeachlist.com" on the four engines:

Google 1
MSN 32
Yahoo 11
Temoa 0

Teoma is a lost cause, but Google is rather disapointing right now.
What is more interesting is that Google has been deep crawling my site
for nearly two weeks now and has looked at over 15,000 pages. MSN and
Yahoo have only touched a few pages, but those they did made it into
the index in a few days.

Google seems to be really slow and dealing with new sites. I know they
are better about adding new content for existing sites being that I see
USENET posts, which are mirrored on several websites, showing up in
just a few days.

What experiences have others had with their first deep crawl for a
site? Does Google usually not index until it has finished a deep
crawl, or is it just slow at adding pages for new sites even when it
starts the crawl?

Thanks.

Beach Comber

Beach Comber wrote:

> Back around the 1st of February I started a new website. It had a
> couple of inbound links from other sites of mine and I did a manual
> submission to Google, MSN, and Yahoo. Since Teoma doesn't take manual
> submission, I have to wait for them to find me which they finally did
> over the weekend.
>
> Now, after five weeks, the search engine results are rather
> interesting. Here is the list of indexed page counts for queries of
> the form "site:worldbeachlist.com" on the four engines:
>
> Google 1
> MSN 32
> Yahoo 11
> Temoa 0

Strange indeed /if/ your site has been up for months. Google should
typically be around 90% overall.

> Teoma is a lost cause, but Google is rather disappointing right now.
> What is more interesting is that Google has been deep crawling my site
> for nearly two weeks now and has looked at over 15,000 pages. MSN and
> Yahoo have only touched a few pages, but those they did made it into
> the index in a few days.

From my experience, Google takes at least a month to show something
sensible. It's organising its data in long cycles.

> Google seems to be really slow and dealing with new sites. I know they
> are better about adding new content for existing sites being that I see
> USENET posts, which are mirrored on several websites, showing up in
> just a few days.

These post most probably reside on domains which have existed for quite some
time and have gained good ranking.

> What experiences have others had with their first deep crawl for a
> site? Does Google usually not index until it has finished a deep
> crawl, or is it just slow at adding pages for new sites even when it
> starts the crawl?

How long has the site been up? How long has Google been crawling it? How
many inbound links can you identify?

Roy

--
Roy Schestowitz
http://schestowitz.com

> How long has the site been up? How long has Google been crawling it?
How
> many inbound links can you identify?

The site has been up for five weeks.

Google started a deep crawl about two weeks ago. Since then it has
craweled 15,000 pages. There are a total of about 70,000 pages on the
site of which about 65,000 are tagged as being valid for indexing.
5,000 pages are labeled as not being valid for indexing (using robots
meta tags) to avoid duplicate content.

Using "link:worldbeachlist{dot}com", only one link comes up currently.
Doing a '"worldbeachlist{dot}com" -site:worldbeachlist{dot}com', 37
matches are found.

On 8 Mar 2005 07:37:58 -0800, "Beach Comber"
<BeachNews@worldbeachlist.com> wrote:

>The site has been up for five weeks.
>
>Google started a deep crawl about two weeks ago. Since then it has
>craweled 15,000 pages. There are a total of about 70,000 pages on the
>site of which about 65,000 are tagged as being valid for indexing.
>5,000 pages are labeled as not being valid for indexing (using robots
>meta tags) to avoid duplicate content.

Assuming you have a fair amount of links to the site I'd expect to see
a reasonable number of those pages added to the database within weeks
and many more as the weeks pass.

Might be worth double checking your robots meta tags for errors, just
in case (I didn't look at your site).

Any page you want indexing fully don't use the robots meta tag. Look
at the bottom of
http://www.seo-gold.com/tutorial/met...imization.html and
compare your robots meta tags to the examples at the page above.

David
--
Free Search Engine Optimization Tutorial
http://www.seo-gold.com/tutorial/

> Might be worth double checking your robots meta tags for errors, just

> in case (I didn't look at your site).

I'm only using the robots meta tag to exclude pages. This only applies
to dynamic pages where I can't use the robots.txt file to exclude an
entire path.

I wish there was a way to tell the robots "follow these links first".
Then I could flag the high content pages for earlier indexing and let
the robots worry about the lower interest areas later.

On 8 Mar 2005 08:10:36 -0800, "Beach Comber"
<BeachNews@worldbeachlist.com> wrote:

>I wish there was a way to tell the robots "follow these links first".
>Then I could flag the high content pages for earlier indexing and let
>the robots worry about the lower interest areas later.

Pages you want indexed first link to them from the home page and any
other pages you have indexed. Same is true for new pages you want
indexed ASAP. You can remove the home page links after indexing if
they don't work long term for the site structure etc...

David
--
Free Search Engine Optimization Tutorial
http://www.seo-gold.com/tutorial/

> Pages you want indexed first link to them from the home page and any
> other pages you have indexed. Same is true for new pages you want
> indexed ASAP. You can remove the home page links after indexing if
> they don't work long term for the site structure etc...

That's good to know. Even better knowing that I'm already doing it.
The home page always shows links to the last 10 beaches added and the
last 10 reviews added. The problem I'm seeing right now is that while
there is about 160 beaches there are over 60,000 towns and Google
appears to be lost in the towns. At this rate it won't finish for
another month or two. Once everything is in the robots shouldn't have
trouble keeping up with the new content going in.

Dave, really liked the tutorial pages. Nice to know that following
most of the guidelines. This isn't the first site I've done, but it is
the most ambitious. The biggest headache will be long term going from
me researching and adding new locations to getting the users to
contribute some of the research along with their own reviews.

The few pages indexed into MSN and Yahoo are already generating a
couple of hits per day. Hopefully, once more of the core content is
indexed, this will improve.

Again, thanks for the help, insight, and suggestions.

On 8 Mar 2005 07:37:58 -0800, "Beach Comber"
<BeachNews@worldbeachlist.com> wrote:

>> How long has the site been up? How long has Google been crawling it?
>How
>> many inbound links can you identify?
>
>The site has been up for five weeks.
>
>Google started a deep crawl about two weeks ago. Since then it has
>craweled 15,000 pages. There are a total of about 70,000 pages on the
>site of which about 65,000 are tagged as being valid for indexing.
>5,000 pages are labeled as not being valid for indexing (using robots
>meta tags) to avoid duplicate content.

You want to exclude them with robots.txt. Rather more reliable than
the robots meta tags.

>Using "link:worldbeachlist{dot}com", only one link comes up currently.
>Doing a '"worldbeachlist{dot}com" -site:worldbeachlist{dot}com', 37
>matches are found.

Google has been telling porkies with the link: command for some time.
It doesn't actually mean the pages aren't indexed.

BB
--
www.kruse.co.uk/ SEO@kruse.demon.co.uk
Affordable SEO!
--

On 8 Mar 2005 08:10:36 -0800, "Beach Comber"
<BeachNews@worldbeachlist.com> wrote:

>> Might be worth double checking your robots meta tags for errors, just
>
>> in case (I didn't look at your site).
>
>I'm only using the robots meta tag to exclude pages. This only applies
>to dynamic pages where I can't use the robots.txt file to exclude an
>entire path.

Ah. I'm beginning to wonder if there isn't a clue to your predicament
in there somewhere. I dunno what, though. You sure there's not some
complication with your "unhide" facility? I never saw something like
that before.

>I wish there was a way to tell the robots "follow these links first".
>Then I could flag the high content pages for earlier indexing and let
>the robots worry about the lower interest areas later.

You could do that with structure but I imagine it's way too late to do
that now.

BB
--
www.kruse.co.uk/ SEO@kruse.demon.co.uk
Affordable SEO!
--

>> I'm only using the robots meta tag to exclude pages. This only
applies
>> to dynamic pages where I can't use the robots.txt file to exclude an

>> entire path.

> Ah. I'm beginning to wonder if there isn't a clue to your predicament

> in there somewhere. I dunno what, though. You sure there's not some
> complication with your "unhide" facility? I never saw something like
> that before.

I hope not, but anything is possible. I actually posted a question a
few weeks ago regarding using robots tag on dynamic pages. Nobody had
an answer. I'm only using it on the browse page and not the town page
where Google has been concentrating for the last week. Being that the
tag is never used on the town or beach pages I don't think it will foul
up those results, but who knows what goes on in the heart of Google.

>> I wish there was a way to tell the robots "follow these links
first".
>> Then I could flag the high content pages for earlier indexing and
let
>> the robots worry about the lower interest areas later.

> You could do that with structure but I imagine it's way too late to
do
> that now.

Somewhat covered in another post (latest 10 always on the home page),
but in general I care more about making the site friendly to people
than robots. That's why all the filtering is in place to begin with so
that the users don't have to wade through a mountain of links with no
serious content behind them. The links expose as content is added.
I'll see what Google does over the next few weeks. In the mean time
I'll keep working on new content for the site so that some day when (or
if) the users come marching in they'll have more to look at.