<
https://the-riotact.com/government-test-shows-humans-far-outperforming-ai-because-they-understand-the-task-better/805612>
"A government trial of artificial intelligence has shown it was quite poor at
summarising public submissions when compared to how humans do the work.
As the Federal Government continues to stress that its workforce must learn how
to embrace the responsible use of AI, a targeted trial at the Australian
Securities and Investments Commission (ASIC) has delivered results that are, to
say the least, very interesting.
Submitted summaries were marked by reviewers who were unaware AI played any
role in the summaries at all, with the outcome being that human work was graded
roughly twice as highly as the AI papers.
Across all criteria, humans far outperformed AI.
A report of the test concluded that the use of AI could potentially create
unnecessary workloads due to the greater need for fact-checking.
Amazon Web Services (AWS) conducted a test for ASIC to assess the capability of
generative AI to summarise a sample of public submissions made to an external
parliamentary joint committee inquiry.
Meta’s Llama2-70B model was prompted to focus on ASIC references and explain
what they meant while summarising the submissions.
ASIC staff were given the same task, using identical directions and prompts.
The reviewers found wrong information, lack of nuance, misplaced context and
overlooked emphases in some of the summaries they were marking – which turned
out to be AI-generated work."
Cheers,
*** Xanni ***
--
mailto:xanni@xanadu.net Andrew Pam
http://xanadu.com.au/ Chief Scientist, Xanadu
https://glasswings.com.au/ Partner, Glass Wings
https://sericyb.com.au/ Manager, Serious Cybernetics
