Statement on Web Crawling and Deep Indexing

Elsevier supports and encourages our institutional and corporate customers who want programmatic (i.e. automated) access to the content on ScienceDirect and other Elsevier Internet databases.  We have developed complimentary services for our customers that are specifically designed to support machine-to-machine access, including a comprehensive suite of APIs and content syndication services.  In order to ensure stability and maintain the performance of our user-facing platforms, we recommend that our customers use those Elsevier services to access content.   

Elsevier has also put in place policies and technical measures to prevent crawling by robots or other mechanized devices.  Web crawling is an inefficient method of harvesting large quantities of content from sites similar to our sites and imposes a high volume burden on our systems which interferes with access for human users in the research community.  http://public.lanl.gov/herbertv/papers/f140-nelson.pdf).

To aid discovery for researchers, we also partner with major search engines such as Google and Google Scholar to enable them to index our content.

Customers who would liketo use Elsevier’s services to deep-index full text and bibliographic data on ScienceDirect, or other Elsevier Internet databases (for example to build indexes for the purpose of recreating navigational schemes to Elsevier content on a customer's or company's local servers), should contact their Account Manager for an overview of the commercial solutions we provide for this purpose.

We also recognise that some of our customers want to be able perform text or data mining within this content, and we encourage them to do so. To enable that functionality (and ensure that is not falsely identified as crawling), customersshould discuss content mining needs with their Account Manager so that they can utilize these services. Elsevier also has sophisticated services for delivering Journal metadata including bibliographic information and abstracts for the inclusion in 3^rd party A&I databases, agents’ consolidated services, and others.   

If you are a researcher, developer, or other end-user and you would like to use Elsevier data to build applications, perform text or data mining, or for other uses, please contact your institution’s library staff for assistance.