Basically all search engine spiders function on the same principle – they crawl the Web and index pages, which are stored in a database and later use various algorithms to determine page ranking, relevancy, etc of the collected pages. While the algorithms of calculating ranking and relevancy widely differ among search engines, the way they index sites is more or less uniform and it is very important that you know what spiders are interested in and what they neglect.
Search engine spiders are robots and they do not read your pages the way a human does. Instead, they tend to see only particular stuff and are blind for many extras (Flash, JavaScript) that are intended for humans. Since spiders determine if humans will find your site, it is worth to consider what spiders like and what don't.

Very good post about spiderablity and crawl behavior , basically we think more about seo prospective this should be happen or this should be not but we should care for the user behavior for web pages.

Good info Sam, thanks. Just as an add: google spiders like sitemaps, so having one on your website is a good idea.

A Web crawler is a computer program that browses the World wide web in a methodical, automated manner. Other terms for Web crawlers are ants, automatic indexers, bots, and wormsor Web spider, Web robot.

This process is called Web crawling or spidering. Many sites, in particular Search engines use spidering as a means of providing up-to-date data. Web crawlers are mainly used to create a copy of all the visited pages for later processing by a search engine that will index the downloaded pages to provide fast searches. Crawlers can also be used for automating maintenance tasks on a Web site, such as checking links or validating HTML code. Also, crawlers can be used to gather specific types of information from Web pages, such as harvesting e-mail addresses (usually for SPAM).

__________________
Outsource Web design company,outsourcing web services

Well what I like to know is if Yahoo has spiders or not? If yes do, are they termed as? Are Yahoo spiders important?

Thanks in advance,

__________________
Dr. Berger: First Reversal Surgeon

Google insinuate to first spider the home page, then a couple of weeks later it will get the pages mentioned on the home page. Then a couple of weeks behind it will get the pages that were mentioned in the second set. And so on. Eventually, when google has transmitted, a lot of your pages, it'll start visiting to amend the old pages, and you'll have a regular steady stream of Google visits.

__________________
micro sd card

Now search engines are capable to read flash & designing file. SE are more intelligent and efficient to serve specific data as per user query.

__________________
SEO Company India | SEO India

Crawler is an automated software program which collects the information about the webpages and stores the data for future use.

__________________
website developer