Web crawling, Google Spiders, indexing websites–our guess is that you’ve heard these terms somewhere before on the news, online, or your nerdy friend that loves the Internet and search engines (the entire Stellar Blue Crew). Maybe you kind of knew what the terms meant, or at least enough to get the jest of things, which is good, but we want to clear it up.
It’s important to clarify what these terms mean because let’s face it, we Google things daily. Search engines are now a part of our lives and you should know some search engine anatomy terms without getting too techy. Just as you know the heart is an organ that pumps blood, you should know a web spider is a program that automatically finds web pages. It’s not a big deal if you don’t know the heart “is enclosed in a double-walled sac called the pericardium.” We didn’t either until we Googled it. But you at least know a key reason of your existence is the heart, which pumps blood in your body. A key reason to why we could search “heart” on Google and find about 657,000,000 results is a web spider.
Make Sense? Let’s Learn.
Crawling & Spiders: Web crawling is Google’s servers visiting pages on the Internet and reading their contents. Web Spiders are the “vehicle” that preforms the action of crawling/finding documents and content on the web. Google’s spiders are constantly crawling and filtering content (gets rid of spam) before placing it in it’s web index (brought up later). Web crawling is basically doing the dirty work and putting together the good chunk (millions+) website together before you make the search.
Keywords & Queries: These are what you type in a search box. A keyword is a single word, like “heart.” A query is putting keywords together in the act of searching, like “What is the main functions of a heart?”
Web Index & Indexing (and Algorithms): A web index is where the search engine stores all the sites found by the web spiders during crawling. Indexing means analyzing the sites in the web index and determining which page matches the searched keyword or queries the best. Indexing follows rules, or algorithms to be more efficient and precise.
Now, we just touched the surface. If you want more information about how often spiders crawl, the best keywords to use or how big Google’s index is contact Stellar Blue! We know exactly how to rank your page higher and would be delighted to tech talk more with you.