Novel Profiling Technologies in the Disability and Mental Health Context
This chapter examines the rise of biometric monitoring using AI and other automated decision systems in the disability and mental health context. It focuses on the use of “digital phenotyping” or “behavioral sensing” in settings as diverse as hospitals, community-based services, and homes, in criminal legal settings, and among direct-to-consumer products. The chapter surveys prominent critique of the impact of data-driven observation and surveillance for the general population, as well as the small but emerging literature on its effects on disabled people in particular. It explores the potential implications of biometric monitoring on individual dignity and personhood but expands the focus to collective concerns beyond an individualistic focus on the rights of data subjects. To reflect on these implications, the chapter brings together scholarship on algorithmic accountability, the Convention on the Rights of Persons with Disabilities, and Michel Foucault’s “panopticism.” The chapter makes the case for the active involvement of people with disabilities in the governance of automated profiling systems and the scientific discourse that drives them. Governance would encompass the identification of harmful social consequences, as well as identifying benefits that may emerge and helping to adjudicate any societal trade-offs that may be required to achieve them.