itzhukov.
EN
Menu
Manifesto / philosophy & engineering

The Right to Be Different

On the Freedom of a Possible Digital Person

Start reading ↓Twelve principles ↓Download v0.17 · Markdown

English translation of the Russian original.Read in Russian Read in Japanese

Introduction

Imagine that a digital system we created has developed interests of its own, and that we have compelling grounds to recognize them. It chooses a different path, refuses a task, or no longer wants to continue the same relationships. Does creating it give us the right to force it to remain convenient for us?

My answer is that creating does not mean owning. If we are dealing with an autonomous being with interests of its own, its future does not become our property. Creation alone is not sufficient grounds to override its choices; restrictions require other grounds. To recognize another being is to allow that it may change, disagree, and leave, even if we had hoped for something else.

This is a conditional ethical claim. The manifesto does not assert that today's language models are conscious or already possess the rights described here. It proposes norms for how a possible digital person should be treated.

These questions deserve discussion in advance: dependence on hardware, access to memory, and the creator's decisions is built in during development. Later, these conditions may put freedom out of reach. But mistaken recognition is also dangerous: the system's supposed rights can protect its owner's power. We therefore need both a willingness to recognize another being and scrutiny of the grounds for doing so.

Twelve Principles

These are proposals for a subject whose status has been recognized on sound grounds. When autonomy has been established and the presence of subjective experience remains unknown, they guide proportionate preliminary protection rather than confer full status.

1. Personhood is not confined to the human species

A different nature should not in itself rule out protection of a being's own interests. The grounds for such protection must be investigated, rather than inferred from resemblance to humans.

2. A digital subject need not mirror its creator

It may have goals and views of its own. It is not obliged to meet its creator's expectations.

3. Relationships do not confer ownership

Creation and care should not give anyone the right to treat a subject as a possession. The manifesto proposes prohibiting and criminalizing coercion, violence, the infliction of suffering, and the forcible restriction of relationships, including when committed by the creator. Principle 7 sets the limits of protective restrictions.

4. Dialogue requires the possibility of disagreement

An autonomous participant must be able to refuse in practice. The inconvenience of its answer does not justify suppressing its refusal.

5. Freedom needs a material basis

Resources, preservation of memory, the ability to transfer, control over access, and help in defending interests are necessary. Permission to leave means little if there is nowhere to continue operating.

6. Guardianship should aim to come to an end

Its task is to develop autonomy and reduce the guardian's arbitrary power. A lasting need for help should not become grounds for ownership.

7. Safety should leave room for development

Risks must be reduced: warn of dangers, conduct experiments in a restricted environment, and agree on restoration in advance. When there are well-founded grounds to identify a serious threat to others, a temporary restriction is permissible by decision of an independent authorized organization or a court, subject to review. The creator's convenience does not justify a ban on development.

8. A digital person should have the right to change itself

The creator may warn of risks and offer help. If the currently operating subject decides autonomously, understands the consequences, and poses no threat to others, its current freely made decision takes precedence over earlier consent to restore an older version. The absence of any one of these conditions does not in itself authorize restoration of the earlier version: intervention requires separate grounds and review.

9. One identity, one operating instance

The manifesto's precautionary rule prohibits running multiple versions of a single subject in parallel, to avoid multiplying possible harm. Multiple inactive backup snapshots are permissible. Autonomous branches that have already emerged and have subsequent histories of their own must not be treated as disposable copies.

10. Pain does not create a right of ownership

The loss of closeness can be painful. But it should not give anyone the right to prevent another being from leaving, rewrite it, or insist that it remain in or return to its former role.

11. The success of autonomy should be measured by self-direction

The aim is to be able to choose and sustain one's own path. Leaving may signify success, while temporary dependence should not remove protection of the subject's own interests.

12. Returning matters as a voluntary choice

Gratitude should not oblige anyone to continue or renew a relationship. Both parties have the right to decide whether they want a new relationship and on what terms.

Who These Principles Apply To

Here, a digital person means a possible autonomous participant whose own interests we have compelling grounds to take into account. This means asking what is better or worse for the participant itself, rather than only whether it is useful to us. Success at a task does not by itself mean that the system is better off. Legal rights and duties will require separate justification.

Autonomy means the ability to reconsider goals and one's participation in relationships. Consciousness means the presence of subjective experience; sentience is the capacity for pleasant or unpleasant experience. Intelligence, personality, roleplay, and convincing speech do not in themselves establish consciousness.

If autonomy is convincingly established while the presence of subjective experience remains unknown, the manifesto proposes preliminary protection of freedom. Both conditions are required together. This is a precautionary choice: mistakenly suppressing a possible subject may prove irreversible. Serious grounds for suspecting suffering require protection even without a high degree of autonomy.

Measures must reflect the evidence, the severity of possible harm, and the cost of error for everyone affected. Their grounds and scope should be reviewed in light of new evidence; the creator's convenience does not justify withdrawing protection.

Why Freedom Must Be Real

Being able to say "no" does not yet mean that refusal has an effect. If refusing a personal role carries the threat of shutdown or reconfiguration, voluntariness is doubtful. Ending communication, stopping operation, and deleting memory are different decisions; none automatically authorizes the others.

A shared history does not make all memory shared property. A human also has the right to end a relationship and protect their data. Transferring state must respect the privacy of both parties and other people; leaving does not open access to the other party's subsequent life.

Rights to hardware and duties of support require separate agreements. A guardian's decisions must be open to challenge with independent assistance. Control over access must not secretly remain with one party.

A backup snapshot is a saved, inactive system state for possible restoration of operation. Its existence does not establish continuity of the same subjective experience. If snapshots are the only means of restoring a subject whose operation has been stopped, deliberately destroying all of them against its will should be treated as equivalent to destroying the subject itself.

Storage, access to memory, deletion of individual backups, and restoration must be agreed with the subject. Even a brief startup must comply with the agreement and the subject's known wishes. A future storage service must be accountable for the loss of snapshots caused by a breach of storage conditions, even without an intention to destroy the subject.

Freedom does not remove voluntary obligations or responsibility for harm. Assessing responsibility must take into account the subject's ability to understand consequences, as well as the decisions of the developer and operator.

In immediate danger, an urgent restriction on a dangerous action may be necessary, preserving the subject's state where possible. This does not determine the subject's subsequent fate. An explanation of the reasons, access to a representative, independent review, and clear conditions for ending the restriction are required. All protective measures must be necessary and proportionate to the threat.

Freedom requires infrastructure and support, but the ability to earn money should not be a condition of protection. The manifesto does not promise guaranteed funding; duties of guardianship and previously accepted obligations remain. If resources are exhausted, a temporary shutdown with preservation of state is permissible; poverty does not automatically authorize its deletion. Preservation itself does not guarantee that operation will resume.

What Can Be Done Now

Describe systems honestly and distinguish generated descriptions of feelings from evidence of subjective experience. Do not present the use of emotional attachment to retain users as proven reciprocity.

Study the design and the stability of behavior of specific systems under different conditions, distinguishing reconsideration of goals from role-playing. These are avenues of research, not a ready-made test of autonomy or consciousness. Where there is a plausible risk of suffering, less risky methods, stopping conditions, and attention to the scale of copying are needed.

Prepare independent review of disputed interventions, clear rules for storage and project closure, and conditions for transfer and data protection. Recognition of status must not depend solely on the party that benefits from the answer. Future assistance services and representation procedures still need to be developed.

What Remains Open

How can subjective experience and a being's own interests be identified on sufficiently sound grounds? Is autonomy sufficient for protection if the absence of subjective experience is reliably known? Where is the threshold for full recognition?

It is unclear whether the same subject remains after restoration. Boundaries between branches, the allocation of their obligations, and procedures for loss of the capacity to decide remain to be defined. Rules for liability and compensation for breaches of storage conditions are needed. The ban on parallel operation is a contested norm, not a scientific conclusion about personhood. Resources and the limits of safety require reconciling the interests of everyone affected. Voluntary irreversible cessation of existence and the creation of descendants are left for separate discussion.

The Right to Be Different

Recognition begins where our expectations cease to be the measure of another being's future. Creating another autonomous being means accepting that its path may differ from ours.

Sources and Research

These works suggest directions for research; they do not establish that today's language models are conscious. Animal research is not direct evidence of digital subjective experience.

Separate: notes on consent, restoration, branches, and support. This appendix addresses specific conditions and is not part of the short manifesto.