<
https://c4osl.org/false-positives-real-harm-when-child-safety-systems-get-it-wrong/>
"When Jonas (not his real name) posted a photo of himself in sports clothes to
his own Instagram account, the last thing that he expected was for his account
to be suspended for suspected child exploitation. Although Jonas is in his 20s,
and his sports clothes do not sexualize him at all, evidently an AI image
classifier used in automated content moderation had falsely flagged his upload
as possible child sexual abuse material (CSAM). Jonas initially reacted with
disbelief when this happened – but this was soon followed by a mounting sense
of fear about what being under investigation for child exploitation might mean
for him.
Adriana (not her real name) was also shocked when she was banned from a popular
adult-only platform after its AI moderation system incorrectly flagged her use
of common terms within BDSM communities. Despite the platform claiming to use
human review, no one assessed her case before the ban was enforced, revealing a
significant operational failure in how moderation systems are applied. She
writes:
As a survivor of CSA, being banned for safe, sane, and consensual kink
practices was deeply triggering. It feels hypocritical to punish adults for
sexual expression while simultaneously failing to build stronger safeguards
against child exploitation. Human review should be mandatory whenever
content is flagged – both to protect children and prevent false positives."
Cheers,
*** Xanni ***
--
mailto:xanni@xanadu.net Andrew Pam
http://xanadu.com.au/ Chief Scientist, Xanadu
https://glasswings.com.au/ Partner, Glass Wings
https://sericyb.com.au/ Manager, Serious Cybernetics
