Graduate Thesis Or Dissertation

An automated web crawl methodology to analyze the online privacy landscape

Public Deposited

Downloadable Content

Download PDF


Attribute NameValues
  • Protecting end-users privacy and building trust are the two most important factors needed to support the growth of ecommerce. The increased dependence on the Internet for a wide variety of daily transactions causes a corresponding loss in privacy for many users, as virtually all websites collect data from users directly or indirectly while performing business with them. In this thesis I have used a web crawler named “iWatch” which serves as an instrument to collect basic statistics on the state of privacy, security, and data-collection practices on the web. I have looked at several interesting practices, and ways of examining the data. This thesis is also meant to serve as a point for reflection and discussion about which practices to observe, and how the raw data from such a system can and should be evolved and made available to a wider audience. The purpose of this thesis is to show web-crawling is a valid approach to mass data collection over the internet with the aim of predicting privacy practices and analyzing how they have evolved in the last three years in terms of geography, legislation, risks, biases and flows. Finally I demonstrate methods to show how to control bias while collecting data, and I propose a probabilistic mathematical model to limit the depth of search to achieve wider breadth for web crawling techniques in the future.
Resource Type
Date Available
Date Issued
Degree Level
Degree Name
Degree Field
Degree Grantor
Commencement Year
Committee Member
Academic Affiliation
Non-Academic Affiliation
Rights Statement
File Format
File Extent
  • 709283 bytes



This work has no parents.

In Collection: