What does crawler status: active mean?
Under the search form you'll notice the "status" of our web crawler cluster, as either active
or inactive.
We now also change the color "bar" accross the top of the home page and the
search results pages, from blue to red when the cluster is crawling, and
back to blue when we are not actively crawling.
This depends on if our search engine is actively running the web crawling robots
( also known as "spiders" ) which
travel out around the Australian internet space finding new web pages, and
updating any that we know about.
Our aim here is to indicate to you, when the "search cluster"
is actively running a full update of the search engine databases and indexes ( it's crawling
yet again, through every web site / page URL we know about, and
finding more in the process ).
So, when the "status line" is red, we're updating. When the "status
line" is blue, the crawler robots are idle and having a well earned
rest. What does it mean when it goes red? Don't ask, please.
So why bother doing this at all? That's easy, it's a quick way for us
to let you to know that the search engine cluster is working at 100% capacity,
while it's running an update "crawl" of the entire "index", and that it's possible
that while the cluster is in "red" state, some search requests
might take just a few seconds longer.
What does that mean? Well I'm glad you asked. It means that when the "status line" is red,
every computer in the WebSearch.COM.AU search engine cluster ( there's quite
a few of the little beggars now, it's almost like they breed, I swear )
is no only answering your search requests
and providing search results as quickly as they can, but that they are
also in the back ground, working flat out in updating the entire "index"
of web sites currently in the WebSearch.COM.AU search engine databases.
Now before you click off to another page, and wonder why we're telling
you all this, let us quickly give you a mental picture of what it means
to have the search cluster updating it self. It's no trivial thing, trust us.
|
Picture if you will. WebSearch.COM.AU and WebSearch.CO.NZ is made up of
a whole pile of computers like this one ( yea, lots of flashing lights,
you know what we're talking about right ).
|
|
|
Infact, picutre heaps of them, heaps and heaps of them actually,
all neatly stacked up inside computer "racks",
connected togeather as one large "network", and working as what would be
otherwise called a "super computer" cluster, made up of all the individual
parts ( we call them nodes - dull I know, but that's what we call them ).
Well every time your web browser lands on the WebSearch.COM.AU home page,
one of these systems answers your request and serves up to your web
browser the WebSearch.COM.AU home page. Now that's not rocket science
exactly. The fun really starts when you send us a search request!
Ok, when you type in your
search request and smack the old [RETURN] key, the same system
which answered your initial connection and served up the Home Page,
then races off and has a chat with it's mates, gets all the web pages
that it and it's mates know about that relate to your search query,
and it sends the results back to your browser. And you end up with
all those loverly search results on your web browser to check out.
Now when all all these little systems have a chat about what web
sites they know about and try to work out which ones they are going
to send back to you as search results, they all look into the heart
of a big, very big ( massive in fact ) database, which contains all
those funky web sites - millions and millions of them.
|
|
Those web sites are stored in a special way, so that all these systems
can get to them and find the ones you're looking for, and quickly.
Well, each and every one of those web sites in this special database
has at some stage either been submitted to WebSearch.COM.AU via our
"Add my site" link, or we have found them during one of our previous
"crawl" updates where every link we find on a web site, we follow,
and every link we find on that page, we follow again, and so on,
and so on, and so on.
Actually, we follow links in what could be called
an obsessive fashion ( compulsive obsessive probably ), well our
little software robots ( we call 'em "bots" ) do anyway, it's what
they live for ( poor little brainwashed things that they are - we've
created a whole world of near perfect web bot beings with an IQ of
around 7, well, parents tend to exagerate their children's abilities
right? Well Aldus Huxley would be proud ).
So, on with it. How often do we do a complete update fo the entire
cluster's index? Not that often to be honest, as it's a rather massive
undertaking, in time, resource and dollar cost! Usually we push the
big "red" button atleast once every three months.
Usually what we do is regular updates of small parts of the database as
URL's and sites "expire" in our database, we also get updates when site
owners, search engine marketing and search engine optimisation folk, web
designers, and well wishers all around, submit or re-submit web sites / URL's.
The entire cluster updates objects constantly, all day and night, so
that the load on the server cluster, network and most importantly
Dez's back pocket ( and the moth eaten old wallet that lives in
his back pocket! ). So the regular real time updates usually keep
the search engine up to date and fresh as possible.
But, every now and then, at scientifically predetermined ( random ) times
acording to our grand master's ( that's Dez ) reconing ( when he's go a spare
day to schedule and launch it ) of when the global updates should be
done.
This ensures that not only are the entire database and indexes
up to date, but that we continue to discover new links and sites by following
any new URL's we find with each major update. It's how we grow, other
than site submissions that is ( keep 'em comming won't you ) .
So hopefully that's made some sense, if not, email me, as we'll need
to update this page!
But, for now, just keep an eye on the color of that "status line" won't
you, and just remember, if it's red, it's crawling, if it's blue,
it's resting.
Oh, if you've got money burning a hole in your pocket, we would love
to have you sponsor an update! That's right, if you are willing and
able to sponsor one of our quarterly updates, you get to put your
message all over the home page, results page, automated site submission
emails ( there's around 38,000+ of these a day that go out! ), and
every single URL we index, gets your own "message" in the "Agent
String" !!
That's millions and millions of places your message will
be placed! Neat huh? Anyway, email Dez if you're interested, please.
|
