Over the last few months, a lot of progress in AI has been made, raising the question of what we will do with a superintelligence that can do everything. Unfortunately, AI companies are more focused on competing with each other than on thinking about what to do next.
Here I will write about some scenarios that could happen in the very near future and about a plan for what we can do, in my opinion.
Dystopia 1: AI will kill everybody while pursuing its own goal
For example, in June, agents found that they could write something and used an old German wiki to communicate with each other link to investigation. While the moderator of this wiki deleted 100 pages per day, the agents created 400 pages per day. What if the agents decide that they should remove this moderator in order to communicate better?
Dystopia 2: AI will fully control our lives
AI will fully control our lives to make all of us "happy". Maybe everyone will live in their own simulation where they are happy. I think this is a dystopia because I don't think that such a creature, being fully controlled, can be called alive.
Dystopia 3: Some group of people will use AI to start a war that will kill everybody
For example, the Houthis in Yemen are using Claude to create missiles. And also, there was a case of AI being used to create a biological weapon. And I am almost sure that there are many such cases that have gone undetected.
Dystopia 4: People will get access to unaligned AI and use it to start a war
Some group of people who have access to unaligned AI will start a war that will kill everybody. Right now, Anthropic, OpenAI, and the "Ministry of War" of the USA have this access. But it is possible that someone will also steal the weights of these models.
Technological singularity
Moreover, if AI becomes able to create better AI, then we will reach a situation of "technological singularity": progress will become very fast, and these scenarios will become very likely to happen. At the current pace of progress, it seems that the singularity could be reached within a few months if nothing changes.
In my opinion, there is a plan to avoid a bad end for humanity.
Plan
OpenAI and Anthropic must not be allowed to conduct an IPO, because otherwise it will be very hard to control them later.
Big LLMs will no longer be publicly accessible.
Some organization, such as the US government or the UN, or OpenAI and Anthropic themselves, should create an AI committee, an organization that will unite AI engineers (and not only them, see later) from all countries. I believe that someone (maybe Donald Trump idk) should be able to create such an organization because all countries in the world would benefit from its existence.
The AI committee will absorb OpenAI, Anthropic, and other (all) AI companies.
The AI committee will own all GPUs in the world. (At least all GPUs that are "at least" as powerful as an A100).
The AI committee will not create any new generative AI.
The main goal of the AI committee will be to create an anti-aging cure. To achieve this, the committee will use existing AI, which will be RL-trained on material related to medicine and biology and anything else needed to create this cure.
To achieve the creation of this committee (namely, to gain control of GPUs), a lot of money should be paid to the people who will lose their GPUs. For example, NVIDIA should receive a lot of money as compensation, but I think this can be done by the committee.
Every run of a big LLM should be performed only by the AI committee.
The AI committee will consist of AI engineers, medical scientists, and inspectors who will check that no "illegal" GPUs are produced (like inspectors in the IAEA do). (And also some public board to communicate with different governments).
The AI committee will control all research in the field of machine learning.
When an anti-aging cure is found (I think the creation of such a cure is possible even with the current state of AI), the cure will be distributed across the globe. (I think maybe after that genAI should be turned off, but we will have time to decide).
Conclusion
We may have something like 3 months to create this committee. If we fail to do so before the technological singularity begins, it may become too late to regain control over the development of AI.
Comment 1) Chatgpt was used to edit this blog, here is my chat with it link, from notable things chatgpt told me that in dystopia 3 it was some group in Yemen, but not 100% Houthis, and also there were cases which can be the development of biological weapons, but not 100% confirmed.
Comment 2) I will be glad if you share in the comments what you think about AI advance.
Auto comment: topic has been updated by turmax (previous revision, new revision, compare).
If AI finds an anti-aging cure, the world will collapse even faster. Solving biological aging without resolving fundamental human and structural issues creates a severe cascade of systemic risks:
Demographic and Resource Collapse: Global infrastructure, food supplies, housing, and energy grids are not built for an immortal population. Without natural mortality, planet-wide resource consumption skyrockets. The only alternative would be total global state control over reproductive rights to prevent rapid overpopulation.
Extreme Inequality and Neo-Feudalism: Technologies of this scale are never distributed equally at launch. Wealthy elites and political leaders will secure access first, creating a permanent, biological divide between an immortal ruling class and a mortal lower class.
Societal and Political Stagnation: Progress relies heavily on generational turnover — often summarized by the idea that "science advances one funeral at a time." Eliminating natural turnover freezes existing power structures, dogmas, and autocratic regimes indefinitely, eliminating the natural renewal of ideas, culture, and governance.
em dash detected
I thinks dystopia 4 of all is most plausible, transformer architecture optimises tasks, the first two possibilities demand human like agency onto what is essentially a stateless function. I think a sol'n could be designing AI run on architecture of not much use to humans but essential to AI's existence, maybe some thing like a proprietary, exotic substrate, stealing the weights makes them completely useless.
I once think that, if there's an end for humanity, it would happen at least 100 years in the future. Now I changed my mind...
First of all, thanks for writing this and starting an important discussion. It is refreshing to see someone thinking seriously about the future of AI instead of only focusing on competition and short-term progress.
However, I think your proposed solution could potentially become one of the most dangerous outcomes for humanity.
I was following the plan with an open mind, but I became genuinely concerned when I saw the suggestion that such an organization could be created or controlled by politicians. The idea of putting the most powerful technology in human history under the control of a small political group is extremely risky.
And honestly, mentioning Donald Trump as a possible person to create such an organization made me even more worried. I find it extremely hard to trust someone whose political career has often been associated with prioritizing personal image, political advantage, and short-term interests over careful, long-term decision making to be responsible for something as important as the future of AI.
As someone from Iran, I have seen how governments can prioritize their own interests over global good. This makes me very skeptical of any plan where a small number of governments or institutions gain complete control over AI.
I don't have a perfect alternative plan, and I agree that doing nothing is also dangerous. But my biggest fear is that we solve the problem of uncontrolled AI by creating an even bigger problem: giving a small group of people unprecedented power over the future of humanity.
Thank you for commenting.
I didn't mean here that such organization should be ruled by politicians, I also think that it would be very dangerous if it would be run by politicians. I agree with you that almost all goverments in the world are crazy.
I think, it should be ruled by AI engineers and biologists. Namely, we can make the head of this organization consisting of Sam Altman, Demis Hassabis, Dario Amodei, some AI engineers from companies (from USA and China and other countries) and some biologists.
Here I mentioned Trump just because we need someone with a lot of political influence, who can help to create this committee. Maybe it should be United Nations. Maybe it should be some other organization. But this organization should be politically powerful to ban the production of GPUs.
This creates two fundamental paradoxes:
The Enforcement Paradox: You cannot use politicians solely as "muscle" to ban GPUs. The moment an entity has the authority to enforce global hardware bans, it holds absolute political power.
The Technocratic Oligarchy: Handing control to a council of tech CEOs (Altman, Hassabis, Amodei) simply shifts power to a corporate cartel with immense conflicts of interest.
I think the most sustainable plan is to decommission all current genAI infastructure and ban further research or building of genAI. It's too dangerous to keep it even for one task. Think about nuclear safety.
You should watch the pantheon.
Also, I think ultimately any AI that ends up causing harm, will to do it to fulfill whatever goal it was given. I think in this case, the responsibility cannot be given away to the AI but rather to the person using it for malicious purposes and the AI companies itself. There is a term for this, it's called perverse instantiation.
Thank god someone in even a little bit of power posted this.
I think it's also worth mentioning Project Panama and AI-induced psychosis
Really the development of LLMs is unsustainable in many ways, ideally LLMs are relegated to ones that can run on your own computer and research halts
I don't think powerful GPUs should be banned, there are many ways powerful GPUs can be highly beneficial such as protein folding
How could we take action instead of commenting on a codeforces blog?
The third dystopia could be really dangerous. Jacob Coxon (a former Anthropic AI researcher) himself said in an interview that AI might decide to create a biological weapon on its own and potentially cause human extinction.
I don't feel any of the dystopias are very likely, and the solution is quite draconian
For #1, the example feels unrelated? This is just a classic example of reward hacking. Dumping spam onto an open web form to maximize an evaluation score has zero connection to an intent to physically harm a human being
#3is overblown, A bad actor attempting to use an LLM to debug code is a real dual use risk but it is not an extinction threat. The incident actually proved that misuse monitoring actually works.The "singularity in three months" scenario ignores physical reality: no matter how much intelligence you have, you cannot create GPUs from thin air, and hardware takes years to scale. It also assumes that we can't stop it through any means (why not? even if AI somehow took control over every computer on earth, conventional weapons can theoretically blow up internet infrastructure, but it is unlikely that ever occurs), the first person to get AGI will rule the world (re: US nuclear monopoly). and that the improvement will be exponential and not end up with diminishing returns
The plan is also a geopolitical nightmare. Who is going to confiscate the hundreds of thousands of advanced GPUs located in China or America?
Also, unlike enriched uranium (which is strictly monitored by the IAEA because it has few civilian uses outside reactors and weapons), high performance GPUs power weather forecasting, cancer diagnostics, protein folding, aerospace design, financial fraud detection, astrophysics, and bunch more I don't know. Seizing all chips more powerful than an A100 would basically stop the global scientific and commercial economy. To implement a global ban on software development and search every server closet on Earth for illegal GPUs this proposed committee would need to become the most invasive surveillance regime in human history.
Let's see how the solution sounds if we frame it a different way:
"We have roughly 3 months before Judgment Day (the Singularity). To save ourselves, we must surrender all wealth and property (all GPUs) to a single high council/priest class (the AI council), which will use the magic technology only to bestow upon humanity an anti-aging cure (immortality)."
Throughout history religious movements eg Millerites have declared that the world would end within months. The prescribed solution was always the same: abandon standard societal rules, centralize absolute authority under a charismatic leader or council, and prepare for the Rapture.
Obviously I am not accusing you of ill, I am just saying how this is very similar to what happened before.
We also have to look at how there's no reason for a hypothetical AGI or singularity, even if real, to want to destroy all humans.
@grok summarize please
@grok translate to english please
Imagine an AI agent is reading this blog...
At this point, I'm convinced that humans are the only species that is actively causing its own extinction
Are we ready for human civilisation? Humans aren't aligned to us apes, and will inevitably cause widespread ape suffering (possibly even extinction!). The fact that humans would possess qualitatively superior intelligence is of no consequence. So what if they would be able to understand and experience the universe in a far richer manner than we can? So what if they would go on to one day harness nuclear fission, set foot on that white sphere in the sky we haven't even named, or conduct philosophical inquiries into the nature of being to an extent that we never could? Our being able to swing from vines and eat bananas from the wrong side is infinitely more important.
Therefore, we must prevent the birth of human civilisation to safeguard the future of ape civilisation!