You probably already know that search engines send out their spiders (aka robots) out to websites in order to index their content into their search databases so you should already appreciate the importance of having content that is easily accessible to spiders, but what do you do if you’re not sure how visible your content actually is?
Perhaps you rely on JavaScript, frames or multiple URL query strings to present content. How do you know whether your content can be easily read by search engine spiders?
If search engine spiders can’t get access to your website content then you can’t expect them to be able to return that content in search results. If organic search engine traffic is important to you then you must ensure your content can be spidered fully.
Here are some ways in which you can get an idea of what the spiders see:
- Use a search engine’s cache in order to view your website pages as they’ve recorded them. If all of the content that should be on your webpages is visible in the cached pages then you know it’s visible to the spiders.
- Use a text-based web browser such as Lynx to view your website. Click on the thumbnail above to see what this website looks like through Lynx.
- Use an online text browser simulator to view your website.
- Use a search engine spider emulator.
- Search for content you want to check. If you can find it through a search engine then you know it’s visible!
Apart from actually using the search engines themselves, how accurate these methods are at representing what spiders see is questionable. Search engines are known for their veil of secrecy and search technology is ever-evolving. What search engine spiders can’t see today they might see tomorrow. Sounds like the strap line for a horror movie…