Web Scraping

Shook‐Sa, B. E., Hudgens, M. G., Kavee, A. L., & Rosen, D. L. (2022). Estimating the number of persons with HIV in jails via web scraping and record linkage. Journal of the Royal Statistical Society. Series A, Statistics in Society, 185(Suppl 2), S270–S287. https://doi.org/10.1111/rssa.12909

This article proposes web scraping as a way to gather generalized information on populations when the group is rare or available data is incomplete.

Speckmann, F. (2021). Web Scraping: A Useful Tool to Broaden and Extend Psychological Research. Zeitschrift Für Psychologie, 229(4), 241–244. https://doi.org/10.1027/2151-2604/a000470

This paper surveys the utility of web scraping as a tool for developing psychological research in a more naturalistic environment than a lab study. It discusses possible shortcomings, as well as the potential use as a method for developing testable hypotheses.

Ulbricht, L. (2020). Scraping the demos. Digitalization, web scraping and the democratic project. Democratization, 27(3), 426–442. https://doi.org/10.1080/13510347.2020.1714595

Increasingly politicians rely on web scraping to assess the inscrutable will of their constituents to inform their policy and campaign decisions. In this paper, Ulbricht examines the shortcomings of this approach, discussing the biases it creates and the technocratic connotations.