Inappropriate Adobe Express AI Image Output

I don't know how I missed this, but yesterday I read an post from Pivot to AI, which is a blog that looks critically at the world of generative AI, which was called, "LA Unified School District forces unfiltered AI on kids." The post from early December, tells the story of how a child in a 4th grade class (10 - 11 year olds) innocently used Adobe for Education to generate an image, the output of which was pretty inappropriate. The child had been reading Pippi Longstocking books, a character who is usually a drawn little girl, with red hair, long stripy socks and pigtails. She used Adobe Express AI image generator to create the character for a book cover. According to the post and evidence from Instagram account which originally posted this, the child used the prompt, "long stockings a red headed girl with braids sticking straight out." Below are the results that Adobe Express AI produced.

As you can hopefully see, these are certainly images that we wouldn't want to see in the primary classroom. I don't know much about this account, but from what I can see they are possibly a parent group that may have some 'beef' with the district school board. The Los Angeles Unified School District did, last summer, some make the headlines with the collapse of an AI project that was being introduced across the district, which possibly upset some parents. Heralded as a trailblazer for its embrace of artificial intelligence, when it unveiled a custom-designed chatbot, Superintendent Alberto Carvalho called the tool a “game changer” that would “accelerate learning at a level never seen before.” But in just five months, LAUSD went from enviable AI pioneer to cautionary tale. Whatever is the back story to the AI generation of the Pippi Longstocking 'photos', the worry for me is that as schools in Wales are, from my discussions with teachers, using Adobe Express to generate AI images. In fact, Adobe have been running training on using Adobe Express with practitioners from across Wales already. Go to about 15:40 in the video to see the use of generative AI for images being demoed. The worry is very much around the unknowability of the output. As teachers we have to put our faith in Adobe that they have put in the guardrails to protect our children from generating unsuitable images. A couple of months ago, I had a conversation with a digital lead from a primary school about this very concern with Adobe Express. He was worried about what could be created with seemingly innocent prompts and was afraid to really test the tool to see if it would produce inappropriate images for the worry of his account being flagged or locked in some way! Both of us also had the concern that we were unsure whether primary school children should even be using this tool. He complained about the lack of guidelines on the age when children could use of this tool and what to do if something inappropriate was generated? Who is at fault? The child, the teacher, the school, local authority, Welsh Government? My guess is everyone would start blaming each other. To me, the blame would lie firmly with Adobe for embedding generative AI into this application, with, no matter what they say, has no guarantees that what is output, whatever guardrails are in place, are suitable for children. I would also blame Welsh Government for, as I posted previously, releasing AI tools in Hwb, without first getting in place the national guidance on the strategic implementation of AI and some professional learning for teachers before rushing out these tools to teachers and pupils. Let's cross fingers that we get no examples like the Pippi Longstocking example here in Wales 🫰