As part of the OpenWebSearch.eu project, the Chair of Data Science crawled parts of the WWW.
For this purpose, some crawler experiments are carried out under the agent string: OSAlphaXCrawl or hgfAlphaXCrawl/1.0.
In addition to the content, some statistical data will also be collected, such as the average size of the web pages, the size of the net text content of the pages and the connection structure between web pages (e.g. number of outgoing links per page).
Further details about the OpenWebSearch.eu project and the crawling activities can be found at http://www.openwebsearch.eu
