Searchine uses self made spider technology to crawl your site and analyse the documents structure and text.
We will take notice of hyperlinks to other pages / documents and follow those as well.
The crawler will take notice of noindex and nofollow requests as it will process robots / XML sitemaps and behave just like any other crawler.
Questions about our crawler and indexing technology
How does Searchine index my site ?
How can I improve the index results ?
Can I run Searchine on my own computer (on-premise) ?
Can I prevent some pages from being indexed ?
How can I remove a page from the index ?
How often do you crawl my website ?
Do you support robots and sitemap.xml ?
Do you support canonical URL’s ?
Which file types are supported ?
Can you crawl AJAX websites ?