Inappropriate AI Chat in the Classroom

I had an interesting chat with a teacher today who was telling me about a generative AI activity that a teacher in her daughter's school was doing with the daughter's class. She told me that they were using AI to 'talk' to a character. Things started off ok, but then the conversations started to become inappropriate and they had soon had to stop. I asked what application they were using, if it was through Hwb, or if it was some application aimed at school use, but she didn't know. What really surprised me was that I assumed her child was in a secondary school doing this, but it wasn't, she was in primary school Year 6 (10 to 11 year olds)! I'd love to know what application the teacher used. I've got lots of unanswered questions about this situation, but it's a really good example of why teachers should think really carefully about using AI in their classroom.