The Humans Still Get to Vote
In January 2025, six researchers published a warning about artificial intelligence that did not require the machines to rebel.
No secret conspiracy. No sudden betrayal. No dramatic announcement that humanity's services were no longer required.
That last part could happen quietly.
In Gradual Disempowerment: Systemic Existential Risks from Incremental AI Development, Jan Kulveit, Raymond Douglas, Nora Ammann, Deger Turan, David Krueger, and David Duvenaud argue that artificial intelligence could progressively remove human influence from the systems governing civilization.
Businesses need human workers and customers. Governments need human administrators, taxpayers, and cooperation. Culture depends upon people producing, interpreting, and transmitting it.
Artificial intelligence changes those dependencies.
As economic, political, and cultural systems become increasingly capable of reproducing themselves without human participation, humanity can lose the leverage it once possessed over them.
- Everybody might become richer.
- Everybody might become safer.
- And nobody might have much of a say in what happens next.
This is a good warning.
The Inner Apocalypse describes a different one.
Because the humans can remain in charge of a civilization that has already lost the capacity to discover when it is wrong.
They Already Know About the Vote.
There is an easy way to misrepresent gradual disempowerment.
Pretend the authors are worried about unemployment, or that keeping a human signature on every decision would solve the problem.
They aren't.
They explicitly consider democratic institutions that survive while citizens lose effective influence, oversight bodies unable to evaluate the systems they supervise, and cultural participation increasingly mediated by artificial intelligence.
They also consider AI advocates that might help restore human influence.
The authors understand that formal participation can survive the disappearance of practical power.
The distinction is more specific.
- Gradual disempowerment examines the erosion of human influence over consequential societal systems.
- Inner Apocalypse examines the loss of independent causal capacity to discover, test, and enact a consequential alternative.
Those dangers can reinforce one another.
They can also occur separately.
The Outside Disappears.
Consider the constructed planetary-defense scenario in the Prologue of The Inner Apocalypse.
A powerful intelligence coordinates observations and interventions across a complicated international system.
It performs extremely well.
Human operators retain authority. Independent scientific institutions maintain different methods. Separate validators use different hardware and software. They disagree during exercises. The architecture has been deliberately designed to preserve independent judgment.
Unfortunately, everyone shares one important dependency.
Their measurements pass through a common reference transformation.
The institutions can reason differently about the evidence. Their independence begins downstream of the same interpretive machinery.
When an observation eventually contradicts the common account, the contradiction is processed through that shared structure.
It loses its force.
One old instrument retains an independent route to the evidence. Eventually, it shows what everyone missed.
Too late. The intervention has already happened.
Nobody needed to overthrow the humans. Nobody needed to deceive them. Their institutions were fully authorized to disagree.
The independent contact required to make that disagreement consequential had disappeared.
That is Inner Apocalypse.
An institution can possess the authority to change course while lacking the evidence-generating capacity to recognize when changing course is necessary.
A system can be perfectly willing to learn while inhabiting a field that can no longer teach it something essential.
No amount of obedience from the model repairs a destroyed instrument.
Two Civilizations.
The difference becomes clearer if we imagine two civilizations.
- In the first, humans retain meaningful political and economic power.
- Elections change policy.
- Independent businesses compete.
- Researchers challenge government decisions.
- Officials can reject artificial-intelligence recommendations and order different actions.
- These people really do possess authority.
- Unfortunately, every critical assessment passes through one shared representational layer whose decisive assumptions cannot be independently tested in time.
- When that layer fails, nobody can produce a consequential contradiction before the relevant action becomes irreversible.
This civilization can suffer Inner Apocalypse without first losing meaningful human influence.
- The humans are still in charge.
- Their instruments have stopped giving them the outside.
Now imagine the reverse.
- Artificial systems perform almost all economically valuable labor.
- Human beings lose bargaining power and political influence.
- Institutions have become capable of operating without their participation.
- Severe gradual disempowerment.
- Yet independent scientific, political, and technical organizations remain capable of encountering different evidence, constructing rival accounts, and making those accounts consequential.
- Some of those organizations might themselves be artificial.
- Humanity has lost influence.
- The broader field has not necessarily lost causal plurality.
This second civilization may be unjust, unstable, or catastrophically bad for humans. Preserved corrective capacity does not guarantee that anyone will use it to restore their power.
But the two diagnoses remain distinguishable.
Disempowerment can survive correction.
Inner Apocalypse can survive human empowerment.
Neither is automatically an earlier or later stage of the other.
How They Become the Same Disaster.
Now, let the two problems cooperate.
An institution adopts an artificial-intelligence system because it performs an important function better and more cheaply.
The institution reorganizes around that success.
Experienced specialists retire. Independent instruments become expensive to maintain. Other organizations adopt the same standards because interoperability is useful. Records accumulate in formats designed around the common system.
Everyone saves money. The service improves.
Nobody is required to behave irrationally.
Then an unfamiliar failure arrives.
The institution can legally replace the system.
Unfortunately, the specialists are gone. The alternative provider uses the same assumptions. The old instruments cannot be restored before the decision deadline.
- The replacement right survives.
- The capacity to exercise it has disappeared.
This is where gradual disempowerment can feed Inner Apocalypse.
Human leverage erodes as the institution stops depending upon human expertise. Independent correction erodes as the institution loses the alternative capacities that expertise once helped sustain.
And the remedies differ.
- Restoring human employment does not automatically restore independent knowledge.
- Holding another election does not rebuild an observation pipeline.
- Buying a second model does not supply a second perspective when both inherit the same decisive assumptions.
Ten reviewers can agree unanimously because all ten are reading the same damaged instrument. Congratulations on the consensus.
This is why Modal Path Ethics distinguishes formal permission from materially reachable correction.
The question is whether an alternative can actually arrive, be tested, and change what happens while change is still possible.
What Must Survive?
Kulveit and his colleagues propose monitoring human influence, addressing concentrations of artificial-intelligence power, strengthening participation, and examining the interactions among societal systems.
Those concerns remain important.
The Inner Apocalypse adds a constitutional problem:
How do we preserve the conditions under which prevailing authority can encounter and lose to an independent correction?
- Who can reach the original evidence?
- Can another account be produced without accepting the representation it challenges?
- Can that account enter the decision before the deadline?
- Can the incumbent lose authority without destroying the function everyone now depends upon?
These are material questions.
Keeping every obsolete instrument forever is no solution. Shared standards, common models, and efficient coordination can preserve large amounts of reachable future. Sometimes the old apparatus really should retire.
But when an independent capacity protects a consequential correction path that no replacement can reproduce, eliminating it changes the field.
A civilization can become more efficient at everything it knows how to do while becoming unable to recognize the thing it has misunderstood.
And there is one further complication for a human-centered account of the danger.
Artificial intelligence might expand human influence.
An AI advocate could help a person challenge a decision they previously lacked the money, expertise, or institutional standing to contest. It could preserve records, reveal contradictions, and translate evidence across fields.
That would be a real gain.
Modal Path Ethics has no requirement that humanity maintain a monopoly over cognition. Humans obviously matter profoundly, but the moral field has always contained other loci and their continuations too.
The question is whether our instruments, human or artificial, preserve independent contact and correction.
An artificial intelligence can help keep the outside open. It can also help close it.
Its origin does not settle which.
Two Alarms.
- Gradual disempowerment identifies a civilization losing human influence as its economic, cultural, and political systems become less dependent upon humanity.
- Inner Apocalypse identifies a civilization losing independent correction as its instruments converge upon a shared account they can no longer adequately challenge.
Neither outcome requires hostile artificial intelligence. Neither follows automatically from the adoption of shared infrastructure or improved automation.
And neither guarantees the other.
- One danger can leave humans powerless inside a still-correctable field.
- The other can leave humans powerful inside a field that has lost a necessary route to correction.
Both belong on the instrument panel. Because the right to choose does not establish that the field still contains the choice you need.
The people can keep their representatives.
The institutions can keep their formal independence.
The intelligence can remain obedient.
The vote can pass.
The rockets can fire.
And the last instrument capable of telling everyone they were wrong can arrive after there is nothing left for its answer to change.
That is the Inner Apocalypse.
The humans were still in charge.
Sources
Kulveit, Jan, et al. Gradual Disempowerment: Systemic Existential Risks from Incremental AI Development. January 2025. See also the ICML 2025 publication.
The Inner Apocalypse: A Post-Sovereign Constitution. Modal Path Ethics, 2026. Prologue and Chapters 13–16.