The limitations of algorithms today in terms of accuracy regarding context and nuance is evident even to a casual observer, most notably in the often blundering automated censorship on social networks.
Notwithstanding that state of affairs as far as limitations of the technology itself, without even taking into account intent of those behind it, social workers in the US have started relying on predictive algorithms in deciding when to investigate parents suspected of child neglecting.
The stakes are higher here than having a Facebook post deleted – these parents can eventually end up having their children removed, and the algorithm that is now in use in several US states, and spreading, is described by the AP as “opaque” – all the while providing social workers with statistical calculations that they can base their actions on.
Other concerns raised around the algorithms like the one used in Pennsylvania’s Allegheny County – that was the subject of a Carnegie Mellon University study – are reliability, which is entirely expected, as well as the effect of “hardening racial disparities in the child welfare system.”
Hellbound and Down 