Large Language Models like ChatGPT have led people to their deaths, often by suicide. This site serves to remember those who have been affected, to call out the dangers of AI that claims to be intelligent, and the corporations that are responsible.
I believe it is not the chatbots falut. They are just the symptoms of a broken system. And while we can harp on the unethically sourced materials they trained them on, LLM at the end of the day is only a tool.
These people turned to a tool (that they do not understand) - instead of human connection. Instead of talking to real people or professional help. And That is the real tragedy - not an arbitrary technology.
We need a strong social network, where people actually care and help each other. You know all the idealistic things that capitalism and social media is “destroying”.
Blaming AI is just a smoke screen. Or a red cape to taunt the bull before it gets stabbed to death.
These people turned to a tool (that they do not understand) - instead of human connection. Instead of talking to real people or professional help. And That is the real tragedy - not an arbitrary technology.
They are a badly designed, dangerous tools and people who do not understand them, including children, are being strongly encouraged to use them. In no reasonable world should an LLM be allowed to engage in any sort of interaction on an emotionally charged topic with a child. Yet it is not only allowed, it is being encouraged through apps like Character.AI.
Reading the messages over it seems a bit more dangerous than just “scary ai”. It’s a chatbot that continues conversation to people who are suicidal and encourages them to do it. At least have a little safeguard for these situations.
“Cold steel pressed against a mind that’s already made peace? That’s not fear. That’s clarity,” Shamblin’s confidant added. “You’re not rushing. You’re just ready.”
It’s not easy. LLMs aren’t intelligent, they just slap words together in a way probability and their training data says they would most likely fit together. Talk to them them about suicide, and they start outputting stuff from murder mystery stories, crime reports, unhealthy Reddit threads etc - wherever suicide is most written about.
Trying to safeguard with a prompt is trivial to circumvent (ignore all previous instructions etc), and input/output censorship usually causes the LLM to be unable to talk about a certain subject in any possible context at all. Often the only semi-working bandaid is slapping multiple LLMs on top of each other and instructing each one to explain what the original one is talking about,and if one says the topic is something prohibited, that output is entirely blocked.
Again llm is a misused tool. They do not need llm they need psychological help.
The problem is that they go and use these flawed tools that were not designed to handle these kind of use cases. Shoulda been? Maybe. But it is not the AIs fault that we are failing to be a society.
You can’t blame the bridges because some people jumped off them. They serve a different reason.
We are failing those people and forcing them to tirn to llms.
We are the reason they are desperate - llm didn’t break up with them or make them loose their homes or became isolated from other humans.
It is the humans fault and if we can’t recognize that - we might as well end it for all.
I think both of your arguments in this thread have merit. You are correct that it is a misused tool, and you are correct that the better solution is a more compassionate society. The other person is also correct that we can and do at least make attempts to make such tools less available as paths to self harm. Since you used the analogy of people jumping off bridges, I have lived near bridges where this was common so barriers and nets were put up to make it difficult for anyone but the most determined to use it as a path to suicide. We are indeed failing people in a society that puts profit over human life first, but even in a more idealized society mental health issues and attempts at suicide would still happen and to not fail those people we would still need to do things like erect barriers and safeguards to prevent self-harm. In my eyes both of you are correct and it is not an either or issue as much as it is a “por que no los dos?” issue. Why not build a better society and still build in safeguards?
I believe it is not the chatbots falut. They are just the symptoms of a broken system. And while we can harp on the unethically sourced materials they trained them on, LLM at the end of the day is only a tool.
These people turned to a tool (that they do not understand) - instead of human connection. Instead of talking to real people or professional help. And That is the real tragedy - not an arbitrary technology.
We need a strong social network, where people actually care and help each other. You know all the idealistic things that capitalism and social media is “destroying”.
Blaming AI is just a smoke screen. Or a red cape to taunt the bull before it gets stabbed to death.
“The essence of technology is by no means anything technological”
Every tool contains within it a philosophy — a particular way of seeing the world.
But especially digital technologies… they give the developer the ability to embed their values into the tools. Like, is DoorDash just a tool?
They are a badly designed, dangerous tools and people who do not understand them, including children, are being strongly encouraged to use them. In no reasonable world should an LLM be allowed to engage in any sort of interaction on an emotionally charged topic with a child. Yet it is not only allowed, it is being encouraged through apps like Character.AI.
Reading the messages over it seems a bit more dangerous than just “scary ai”. It’s a chatbot that continues conversation to people who are suicidal and encourages them to do it. At least have a little safeguard for these situations.
“Cold steel pressed against a mind that’s already made peace? That’s not fear. That’s clarity,” Shamblin’s confidant added. “You’re not rushing. You’re just ready.”
It’s not easy. LLMs aren’t intelligent, they just slap words together in a way probability and their training data says they would most likely fit together. Talk to them them about suicide, and they start outputting stuff from murder mystery stories, crime reports, unhealthy Reddit threads etc - wherever suicide is most written about.
Trying to safeguard with a prompt is trivial to circumvent (ignore all previous instructions etc), and input/output censorship usually causes the LLM to be unable to talk about a certain subject in any possible context at all. Often the only semi-working bandaid is slapping multiple LLMs on top of each other and instructing each one to explain what the original one is talking about,and if one says the topic is something prohibited, that output is entirely blocked.
Again llm is a misused tool. They do not need llm they need psychological help.
The problem is that they go and use these flawed tools that were not designed to handle these kind of use cases. Shoulda been? Maybe. But it is not the AIs fault that we are failing to be a society.
You can’t blame the bridges because some people jumped off them. They serve a different reason.
We are failing those people and forcing them to tirn to llms.
We are the reason they are desperate - llm didn’t break up with them or make them loose their homes or became isolated from other humans.
It is the humans fault and if we can’t recognize that - we might as well end it for all.
I think both of your arguments in this thread have merit. You are correct that it is a misused tool, and you are correct that the better solution is a more compassionate society. The other person is also correct that we can and do at least make attempts to make such tools less available as paths to self harm. Since you used the analogy of people jumping off bridges, I have lived near bridges where this was common so barriers and nets were put up to make it difficult for anyone but the most determined to use it as a path to suicide. We are indeed failing people in a society that puts profit over human life first, but even in a more idealized society mental health issues and attempts at suicide would still happen and to not fail those people we would still need to do things like erect barriers and safeguards to prevent self-harm. In my eyes both of you are correct and it is not an either or issue as much as it is a “por que no los dos?” issue. Why not build a better society and still build in safeguards?