Penumbra Improvement Proposal (UIP) process

Read UIP-1 for information on the UIP process.

Meetings

Penumbra Improvement Proposals (UIPs)

Contributing

Files in this repo must conform to markdownlint. Install markdownlint and then run:

markdownlint --config .markdownlint.yaml '**/*.md'

Running the site locally

Prerequisites:

1. Install Rust
2. Install mdbook

mdbook serve -o

Table of Contents

• What is a UIP?
• UIP Rationale
• UIP Types
• UIP Work Flow
  • Shepherding a UIP
  • Core UIPs
  • UIP Process
• What belongs in a successful UIP?
• UIP Formats and Templates
• UIP Header Preamble
  • author header
  • discussions-to header
  • type header
  • category header
  • created header
  • requires header
• Linking to External Resources
  • Data Availability Specifications
  • Consensus Layer Specifications
  • Networking Specifications
  • Digital Object Identifier System
• Linking to other UIPs
• Auxiliary Files
• Transferring UIP Ownership
• UIP Editors
• UIP Editor Responsibilities
• Style Guide
  • Titles
  • Descriptions
  • UIP numbers
  • RFC 2119 and RFC 8174
• History
• Copyright

What is a UIP

UIP stands for Penumbra (UM) Improvement Proposal. A UIP is a design document providing information to the Penumbra community, or describing a new feature for Penumbra or its processes or environment. The UIP should provide a concise technical specification of the feature and a rationale for the feature. The UIP author is responsible for building consensus within the community and documenting dissenting opinions.

UIP Rationale

We intend UIPs to be the primary mechanisms for proposing new features, for collecting community technical input on an issue, and for documenting the design decisions that have gone into Penumbra. Because the UIPs are maintained as text files in a versioned repository, their revision history is the historical record of the feature proposal.

For Penumbra software clients and core devs, UIPs are a convenient way to track the progress of their implementation. Ideally, each implementation maintainer would list the UIPs that they have implemented. This will give end users a convenient way to know the current status of a given implementation or library.

UIP Types

There are three types of UIP:

• Standards Track UIP describes any change that affects most or all Penumbra implementations, such as a change to the network protocol, a change in block or transaction validity rules, proposed standards/conventions, or any change or addition that affects the interoperability of software using Penumbra. Standards Track UIPs consist of three parts: a design document, an implementation, and (if warranted) an update to the formal specification. Standards Track UIPs are marked as either being "Consensus" or "Non-Consensus", depending on whether they affect the consensus-critical state transition function. Consensus UIPs SHOULD be approved by an on-chain signaling proposal to signal acceptance by the community.
• Meta UIP describes a process surrounding Penumbra or proposes a change to (or an event in) a process. Meta UIPs are like Standards Track UIPs but apply to areas other than the Penumbra protocol itself. They may propose an implementation, but not to Penumbra’s codebase; they often require community consensus; unlike Informational UIPs, they are more than recommendations, and users are typically not free to ignore them. Examples include procedures, guidelines, changes to the decision-making process, and changes to the tools or environment used in Penumbra development.
• Informational UIP describes a Penumbra design issue, or provides general guidelines or information to the Penumbra community, but does not propose a new feature. Informational UIPs do not necessarily represent Penumbra community consensus or a recommendation, so users and implementers are free to ignore Informational UIPs or follow their advice.

It is highly recommended that a single UIP contain a single key proposal or new idea. The more focused the UIP, the more successful it tends to be. A change to one client doesn’t require a UIP; a change that affects multiple clients, or defines a standard for multiple apps to use, does.

A UIP must meet certain minimum criteria. It must be a clear and complete description of the proposed enhancement. The enhancement must represent a net improvement. The proposed implementation, if applicable, must be solid and must not complicate the protocol unduly.

Penumbra Improvement Proposal (UIP) Workflow

Shepherding a UIP

Parties involved in the process are you, the champion or UIP author, the UIP editors, and the Penumbra Core Developers.

Before diving into writing a formal UIP, make sure your idea stands out. Consult the Penumbra community to ensure your idea is original, saving precious time by avoiding duplication. We highly recommend opening a discussion thread on the Penumbra forum for this purpose.

Once your idea passes the vetting process, your next responsibility is to present the idea via a UIP to reviewers and all interested parties. Invite editors, developers, and the community to give their valuable feedback through the relevant channels. Assess whether the interest in your UIP matches the work involved in implementing it and the number of parties required to adopt it. For instance, implementing a Core UIP demands considerably more effort than a CRC, necessitating adequate interest from Penumbra client teams. Be aware that negative community feedback may hinder your UIP's progression beyond the Draft stage.

Consensus UIPs

For Consensus UIPs, you'll need to either provide a client implementation or persuade clients to implement your UIP, given that client implementations are mandatory for Consensus UIPs to reach the Final stage (see "UIP Process" below).

To effectively present your UIP to client implementers, request a Penumbra CoreDevsCall (CDC) call by posting a comment linking your UIP on a CoreDevsCall agenda GitHub Issue.

The CoreDevsCall allows client implementers to:

• Discuss the technical merits of UIPs
• Gauge which UIPs other clients will be implementing
• Coordinate UIP implementation for network upgrades

These calls generally lead to a "rough consensus" on which UIPs should be implemented. Rough Consensus is informed based on the IETF's RFC 7282 which is a helpful document to understand how decisions are made in Celestia CoreDevCalls. This consensus assumes that UIPs are not contentious enough to cause a network split and are technically sound. One important excerpt from the document that highlights based on Dave Clark's 1992 presentation is the following:

We reject: kings, presidents and voting. We believe in: rough consensus and running code.

On-chain voting is one way to signal community sentiment, but it is only one aspect of rough consensus.

:warning: The burden falls on client implementers to estimate community sentiment, obstructing the technical coordination function of UIPs and AllCoreDevs calls. As a UIP shepherd, you can facilitate building community consensus by ensuring the Penumbra forum thread for your UIP encompasses as much of the community discussion as possible and represents various stakeholders.

In a nutshell, your role as a champion involves writing the UIP using the style and format described below, guiding discussions in appropriate forums, and fostering community consensus around the idea.

UIP Process

The standardization process for all UIPs in all tracks follows the below status:

• Idea: A pre-draft idea not tracked within the UIP Repository.
• Draft: The first formally tracked stage of a UIP in development. A UIP is merged by a UIP Editor into the UIP repository when properly formatted.
  • ➡️ Draft: If agreeable, UIP editor will assign the UIP a number (generally the next available number) and merge your pull request. The UIP editor will not unreasonably deny a UIP.
  • ❌ Draft: Reasons for denying Draft status include being too unfocused, too broad, duplication of effort, being technically unsound, not providing proper motivation or addressing backwards compatibility, or not in keeping with the Penumbra values and code of conduct.
• Review: A UIP Author marks a UIP as ready for and requesting Peer Review.
• Last Call: The final review window for a UIP before moving to Final. A UIP editor assigns Last Call status and sets a review end date (last-call-deadline), typically 14 days later.
  • ❌ Review: A Last Call which results in material changes or substantial unaddressed technical complaints will cause the UIP to revert to Review.
  • ✅ Final: A successful Last Call without material changes or unaddressed technical complaints will become Final.
• Final: This UIP represents the final standard. A Final UIP exists in a state of finality and should only be updated to correct errata and add non-normative clarifications. A PR moving a UIP from Last Call to Final should contain no changes other than the status update. Any content or editorial proposed change should be separate from this status-updating PR and committed prior to it.

Other Statuses

• Stagnant: Any UIP in Draft, Review, or Last Call that remains inactive for 6 months or more is moved to Stagnant. Authors or UIP Editors can resurrect a proposal from this state by moving it back to Draft or its earlier status. If not resurrected, a proposal may stay forever in this status.
• Withdrawn: The UIP Author(s) have withdrawn the proposed UIP. This state has finality and can no longer be resurrected using this UIP number. If the idea is pursued at a later date, it is considered a new proposal.
• Living: A special status for UIPs designed to be continually updated and not reach a state of finality. This status caters to dynamic UIPs that require ongoing updates.

As you embark on this exciting journey of shaping Penumbra's future with your valuable ideas, remember that your contributions matter. Your technical knowledge, creativity, and ability to bring people together will ensure that the UIP process remains engaging, efficient, and successful in fostering a thriving ecosystem for Penumbra.

What belongs in a successful UIP

A successful Penumbra Improvement Proposal (UIP) should consist of the following parts:

• Preamble: RFC 822 style headers containing metadata about the UIP, including the UIP number, a short descriptive title (limited to a maximum of 44 words), a description (limited to a maximum of 140 words), and the author details. Regardless of the category, the title and description should not include the UIP number. See below for details.
• Abstract: A multi-sentence (short paragraph) technical summary that provides a terse and human-readable version of the specification section. By reading the abstract alone, someone should be able to grasp the essence of what the proposal entails.
• Motivation (optional): A motivation section is crucial for UIPs that seek to change the Penumbra protocol. It should clearly explain why the existing protocol specification is insufficient for addressing the problem the UIP solves. If the motivation is evident, this section can be omitted.
• Specification: The technical specification should describe the syntax and semantics of any new feature. The specification should be detailed enough to enable competing, interoperable implementations for any of the current Penumbra clients.
• Parameters: Summary of any parameters introduced by or changed by the UIP.
• Rationale: The rationale elaborates on the specification by explaining the reasoning behind the design and the choices made during the design process. It should discuss alternative designs that were considered and any related work. The rationale should address important objections or concerns raised during discussions around the UIP.
• Backwards Compatibility (optional): For UIPs introducing backwards incompatibilities, this section must describe these incompatibilities and their consequences. The UIP must explain how the author proposes to handle these incompatibilities. If the proposal does not introduce any backwards incompatibilities, this section can be omitted.
• Test Cases (optional): Test cases are mandatory for UIPs affecting consensus changes. They should either be inlined in the UIP as data (such as input/expected output pairs) or included in ../assets/uip-###/<filename>. This section can be omitted for non-Consensus proposals.
• Reference Implementation (optional): This optional section contains a reference/example implementation that people can use to better understand or implement the specification. This section can be omitted for all UIPs ( mandatory for Consensus UIPs to reach the Final stage).
• Security Considerations: All UIPs must include a section discussing relevant security implications and considerations. This section should provide information critical for security discussions, expose risks, and be used throughout the proposal's life-cycle. Examples include security-relevant design decisions, concerns, significant discussions, implementation-specific guidance, pitfalls, an outline of threats and risks, and how they are addressed. UIP submissions lacking a "Security Considerations" section will be rejected. A UIP cannot reach "Final" status without a Security Considerations discussion deemed sufficient by the reviewers.
• Privacy Considerations: All UIPs must include a section discussing relevant privacy implications and considerations. This section should provide information critical for privacy discussions, expose risks, and be used throughout the proposal's life-cycle. Examples include privacy-relevant design decisions, concerns, significant discussions, implementation-specific guidance, pitfalls, an outline of threats and risks, and how they are addressed. UIP submissions lacking a "Privacy Considerations" section will be rejected. A UIP cannot reach "Final" status without a Pecurity Considerations discussion deemed sufficient by the reviewers.
• Copyright Waiver: All UIPs must be in the public domain. The copyright waiver MUST link to the license file and use the following wording: Copyright and related rights waived via CC0.

UIP Formats and Templates

UIPs should be written in markdown format. There is a UIP template to follow.

UIP Header Preamble

Each UIP must begin with an RFC 822 style header preamble in a markdown table. In order to display on the UIP site, the frontmatter must be formatted in a markdown table. The headers must appear in the following order:

• uip: UIP number (this is determined by the UIP editor)
• title: The UIP title is a few words, not a complete sentence
• description: Description is one full (short) sentence
• author: The list of the author’s or authors’ name(s) and/or username(s), or name(s) and email(s). Details are below.
• discussions-to: The url pointing to the official discussion thread
• status: Draft, Review, Last Call, Final, Stagnant, Withdrawn, Living
• last-call-deadline: The date last call period ends on (Optional field, only needed when status is Last Call)
• type: One of Standards Track, Meta, or Informational
• consensus: Yes or No (Always No for Meta or Informational)
• created: Date the UIP was created on
• requires: UIP number(s) (Optional field)
• withdrawal-reason: A sentence explaining why the UIP was withdrawn. (Optional field, only needed when status is Withdrawn)

Headers that permit lists must separate elements with commas.

Headers requiring dates will always do so in the format of ISO 8601 (yyyy-mm-dd).

author header

The author header lists the names, email addresses or usernames of the authors/owners of the UIP. Those who prefer anonymity may use a username only, or a first name and a username. The format of the author header value must be:

Random J. User <address@dom.ain>

or

Random J. User (@username)

or

Random J. User (@username <address@dom.ain>

if the email address and/or GitHub username is included, and

Random J. User

if neither the email address nor the GitHub username are given.

At least one author must use a GitHub username, in order to get notified on change requests and have the capability to approve or reject them.

discussions-to header

While an UIP is a draft, a discussions-to header will indicate the URL where the UIP is being discussed.

The preferred discussion URL is a topic on Penumbra Forums. The URL cannot point to Github pull requests, any URL which is ephemeral, and any URL which can get locked over time (i.e. Reddit topics).

type header

The type header specifies the type of UIP: Standards Track, Meta, or Informational.

consensus header

The consensus header specifies whether the UIP is consensus-critical.

created header

The created header records the date that the UIP was assigned a number. Both headers should be in yyyy-mm-dd format, e.g. 2001-08-14.

requires header

UIPs may have a requires header, indicating the UIP numbers that this UIP depends on. If such a dependency exists, this field is required.

A requires dependency is created when the current UIP cannot be understood or implemented without a concept or technical element from another UIP. Merely mentioning another UIP does not necessarily create such a dependency.

Linking to External Resources

Other than the specific exceptions listed below, links to external resources SHOULD NOT be included. External resources may disappear, move, or change unexpectedly.

The process governing permitted external resources is described in UIP-3.

Linking to other UIPs

References to other UIPs should follow the format UIP-N where N is the UIP number you are referring to. Each UIP that is referenced in an UIP MUST be accompanied by a relative markdown link the first time it is referenced, and MAY be accompanied by a link on subsequent references. The link MUST always be done via relative paths so that the links work in this GitHub repository, forks of this repository, the main UIPs site, mirrors of the main UIP site, etc. For example, you would link to this UIP as ./uip-1.md.

Auxiliary Files

Images, diagrams and auxiliary files should be included in a subdirectory of the assets folder for that UIP as follows: assets/uip-N (where N is to be replaced with the UIP number). When linking to an image in the UIP, use relative links such as ../assets/uip-1/image.png.

Transferring UIP Ownership

It occasionally becomes necessary to transfer ownership of UIPs to a new champion. In general, we'd like to retain the original author as a co-author of the transferred UIP, but that's really up to the original author. A good reason to transfer ownership is because the original author no longer has the time or interest in updating it or following through with the UIP process, or has fallen off the face of the 'net (i.e. is unreachable or isn't responding to email). A bad reason to transfer ownership is because you don't agree with the direction of the UIP. We try to build consensus around an UIP, but if that's not possible, you can always submit a competing UIP.

If you are interested in assuming ownership of an UIP, send a message asking to take over, addressed to both the original author and the UIP editor. If the original author doesn't respond to the email in a timely manner, the UIP editor will make a unilateral decision (it's not like such decisions can't be reversed :)).

UIP Editors

The current UIP editors are

• Henry de Valence (@hdevalence)
• Finch (@plaidfinch)

If you would like to become a UIP editor, please check UIP-2.

UIP Editor Responsibilities

For each new UIP that comes in, an editor does the following:

• Read the UIP to check if it is ready: sound and complete. The ideas must make technical sense, even if they don't seem likely to get to final status.
• The title should accurately describe the content.
• Check the UIP for language (spelling, grammar, sentence structure, etc.), markup (GitHub flavored Markdown), code style

If the UIP isn't ready, the editor will send it back to the author for revision, with specific instructions.

Once the UIP is ready for the repository, the UIP editor will:

• Assign an UIP number (generally the next unused UIP number, but the decision is with the editors)
• Merge the corresponding pull request
• Send a message back to the UIP author with the next step.

Many UIPs are written and maintained by developers with write access to the Penumbra codebase. The UIP editors monitor UIP changes, and correct any structure, grammar, spelling, or markup mistakes we see.

The editors don't pass judgment on UIPs. We merely do the administrative & editorial part.

Style Guide

Titles

The title field in the preamble:

• Should not include the word "standard" or any variation thereof; and
• Should not include the UIP's number.

Descriptions

The description field in the preamble:

• Should not include the word "standard" or any variation thereof; and
• Should not include the UIP's number.

UIP numbers

When referring to UIPs, it must be written in the hyphenated form UIP-X where X is that UIP's assigned number.

RFC 2119 and RFC 8174

UIPs are encouraged to follow RFC 2119 and RFC 8174 for terminology and to insert the following at the beginning of the Specification section:

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119 and RFC 8174.

History

This document was adapted fairly directly from the Celestia CIP process.

That process was in turn derived heavily from Ethereum's EIP Process written by Hudson Jameson which is derived from Bitcoin's BIP-0001 written by Amir Taaki which in turn was derived from Python's PEP-0001. In many places text was simply copied and modified. Although the PEP-0001 text was written by Barry Warsaw, Jeremy Hylton, and David Goodger, they are not responsible for its use in the Celestia Improvement Process, and should not be bothered with technical questions specific to Penumbra, Celestia or the UIP. Please direct all comments to the UIP editors.

Copyright

Copyright and related rights waived via CC0.

Abstract

UIP stands for Penumbra (UM) Improvement Proposal. A UIP is a design document providing information to the Penumbra community, or describing a new feature for Penumbra or its processes or environment. The UIP should provide a concise technical specification of the feature and a rationale for the feature. The UIP author is responsible for building consensus within the community and documenting dissenting opinions.

This UIP describes the recommended process for becoming an UIP editor.

Specification

Application and Onboarding Process

Anyone having a good understanding of the UIP standardization and network upgrade process, intermediate level experience on the core side of the Celestia blockchain, and willingness to contribute to the process management may apply to become a UIP editor. Potential UIP editors should have the following skills:

• Good communication skills
• Ability to handle contentious discourse
• 1-5 spare hours per week
• Ability to understand "rough consensus"

The best available resource to understand the UIP process is UIP-1. Anyone desirous of becoming an UIP editor MUST understand this document. Afterwards, participating in the UIP process by commenting on and suggesting improvements to PRs and issues will familliarize the procedure, and is recommended. The contributions of newer editors should be monitored by other UIP editors.

Anyone meeting the above requirements may make a pull request adding themselves as an UIP editor and adding themselves to the editor list in UIP-1. If every existing UIP editor approves, the author becomes a full UIP editor. This should notify the editor of relevant new proposals submitted in the UIPs repository, and they should review and merge those pull requests.

Copyright

Copyright and related rights waived via CC0.

Abstract

Penumbra (UM) Improvement Proposals (UIPs) occasionally link to resources external to this repository. This document sets out the requirements for origins that may be linked to, and the process for approving a new origin.

Specification

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119.

Definitions

• Link: Any method of referring to a resource, including: markdown links, anchor tags (<a>), images, citations of books/journals, and any other method of referencing content not in the current resource.
• Resource: A web page, document, article, file, book, or other media that contains content.
• Origin: A publisher/chronicler of resources, like a standards body (eg. w3c) or a system of referring to documents (eg. Digital Object Identifier System).

Requirements for Origins

Permissible origins MUST provide a method of uniquely identifying a particular revision of a resource. Examples of such methods may include git commit hashes, version numbers, or publication dates.

Permissible origins MUST have a proven history of availability. A origin existing for at least ten years and reliably serving resources would be sufficient—but not necessary—to satisfy this requirement.

Permissible origins MUST NOT charge a fee for accessing resources.

Origin Removal

Any approved origin that ceases to satisfy the above requirements MUST be removed from UIP-1. If a removed origin later satisfies the requirements again, it MAY be re-approved by following the process described in Origin Approval.

Finalized UIPs (eg. those in the Final or Withdrawn statuses) SHOULD NOT be updated to remove links to these origins.

Non-Finalized UIPs MUST remove links to these origins before changing statuses.

Origin Approval

Should the editors determine that an origin meets the requirements above, UIP-1 MUST be updated to include:

• The name of the allowed origin;
• The permitted markup and formatting required when referring to resources from the origin; and
• A fully rendered example of what a link should look like.

Rationale

Unique Identifiers

If it is impossible to uniquely identify a version of a resource, it becomes impractical to track changes, which makes it difficult to ensure immutability.

Availability

If it is possible to implement a standard without a linked resource, then the linked resource is unnecessary. If it is impossible to implement a standard without a linked resource, then that resource must be available for implementers.

Free Access

The Penumbra ecosystem is built on openness and free access, and the UIP process should follow those principles.

Copyright

Copyright and related rights waived via CC0.

UIP: Spend Backreferences

Abstract

This specification introduces a method to improve Penumbra sync speeds by adding additional data that can be used by DAGSync clients. Spend actions will contain a new encrypted_backref field, allowing clients to traverse their transaction graph backwards and quickly recover their entire transaction history.

Motivation

DAGSync is a graph-aware fast syncing algorithm. A client, upon detecting a single transaction involving them, can check that outputs visible to them are spent or not. If the output is unspent, then they have identified a live note they can potentially spend in the future, else if the output is unspent, they can continue the process forwards in the transaction graph, until they reach unspent notes.

The design of Penumbra currently does not allow traversal backwards through the transaction graph, only forwards. A Spend intentionally does not reveal the note being spent, only the nullifier that is revealed. By including on the Spend an encrypted reference back to the note commitment being spent, such that only the note owner can view it, we enable DAGSync clients to efficienctly reconstruct the transaction history both backwards and forwards.

Specification

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119 and RFC 8174.

Modification to SpendBody

The Spend action will be augmented with an additional field encrypted_backref on the SpendBody:

message SpendBody {
    // A commitment to the value of the input note.
    penumbra.core.asset.v1.BalanceCommitment balance_commitment = 1;
    // The nullifier of the input note.
    penumbra.core.component.sct.v1.Nullifier nullifier = 6;
    // The randomized validating key for the spend authorization signature.
    penumbra.crypto.decaf377_rdsa.v1.SpendVerificationKey rk = 4;
    // NEW: An encryption of the commitment of the input note to the sender's OVK.
    bytes encrypted_backref = 7;
}

Clients MAY populate the encrypted_backref field with the encrypted note commitment corresponding to the note they are spending.

Transaction parsing rules MUST ensure the length of the encrypted_backref bytes field on a Spend has either 48 or zero bytes in length.

This allows for a phased adoption period such that clients have time to implement support for Spend backreferences. See the Backwards Compatibility section for further discussion.

Backreference Key

We derive a new symmetric key, the Backreference Key $brk$, from the OutgoingViewingKey $ovk$ using the BLAKE2b-256 hash function and personalization string "Penumbra_Backref":

#![allow(unused)]
fn main() {
brk = BLAKE2b_256("Penumbra_Backref", ovk)
}

One advantage of using a new key is that it has a single purpose with a new capability: it can be disclosed to show the transaction graph only and provides no other information.

Another advantage of using a single key is that we can scan all spends without having to do key derivation each time.

For incoming scanning, for each note, we currently do Diffie-Hellman (DH) key exchange between the Incoming Viewing Key and the ephemeral public key associated with the note. This allows us to derive the key that may have been used to encrypt the note.

For outgoing scanning, for each note, we first attempt to decrypt the OvkWrappedKey using a key derived from the OutgoingViewingKey and the other public fields (value commitment, note commitment, and ephemeral public key). This approach allows us to identify if the action belongs to us prior to doing DH key exchange. The same benefit of avoiding a DH key exchange is also true of scanning with the Backreference Key.

Encryption of Spend Backreference

The encrypted_backref should be encrypted using the Backreference key $brk$ and ChaCha20-Poly1305. RFC-8349 specifies that (key, nonce) pairs MUST NOT be reused.

The first 12 bytes of the nullifier nf on the spend is used as the nonce $n$:

#![allow(unused)]
fn main() {
n = nf[:12]
}

There is a single nullifier per spend/note, thus this nonce will not repeat, satisfying the requirement that no (key, nonce) pair be reused for encrypting different plaintexts.

Encryption of the 32-byte note commitment $cm$ is performed using ChaCha20-Poly1305 with the $(brk, n)$ tuple and outputs the 32-byte ciphertext $c$ and a 16-byte MAC:

#![allow(unused)]
fn main() {
(c, MAC) = ChaCha20_Poly1305(brk, n, cm)
}

The transmitted data in the encrypted_backref field consists of a concatenation of the ciphertext $c$ and MAC. The encrypted_backref is thus 48 bytes (32 byte ciphertext + 16 byte MAC).

EffectHash

Currently the EffectHash for the Spend action is computed as:

effect_hash = BLAKE2b-256(len(type_url) || type_url || proto_encode(proto))

where type_url is the bytes of a variable-length Type URL defining the proto message, len(type_url) is the length of the Type URL encoded as 8 bytes in little-endian order, and proto represents the proto used to represent the effecting data, and proto_encode represents encoding the proto message as a vector of bytes.

EffectHash Backwards Compatibility

The EffectHash computation is unchanged if the new encrypted_backref field is not populated. The EffectHash computation is a domain-separated hash of the Protobuf encoding of the Spend message. Protobuf encoding rules skip encoding default values. The new encrypted_backref field is a bytes field with a default value of an empty array, thus if it is not populated, it will be skipped, ensuring backwards compatibility.

For spends that populate a 48-byte encrypted_backref field, the field will be included in the EffectHash per the existing proto_encode method as described above.

Transaction Perspectives and Views

The TransactionPerspective and TransactionView will be unchanged. The backreference is treated as an internal sync optimization detail.

Rationale

ZCash has considered a similar approach wherein backwards syncing is enabled using references encoded into the memo fields. Wallets can periodically construct transactions that stuff references to previous transaction hashes into the memo field of the dummy transaction. The advantage of the memo-stuffing approach is that DAGSync-aware clients can populate these fields without a change to the consensus rules. The disadvantage, however, is that the user's transaction history is polluted with dummy transactions, and a client must scan forward to find one of these dummy transactions before it can go backwards.

Non-Unique Note Commitments

Note commitments correspond to the contents of a note, not to individual note instances. If two note instances have the same exact contents, they will share the same note commitment. This requires two notes to be generated with the same Rseed: for honest users the Rseed is generated randomly, but an honest user may nevertheless receive two notes constructed with the same Rseed. However, the Penumbra protocol allows this possibility of duplicate note commitments, so during syncing clients should allow the possibility of selecting a note commitment that appears in multiple transaction IDs. In the rare case that the encrypted_backref field refers to a note commitment that is a duplicate note commitment, the client should continue syncing using each transaction ID.

Backwards Compatibility

There should be no compatibility issues since the EffectHash for a Spend will be unchanged if the encrypted_backref field is absent. Once all clients have added encrypted_backref support, a future UIP could make the field mandatory.

Security Considerations

This specification considered several security considerations:

1. Encryption: The symmetric encryption scheme used for encrypted_backref uses a symmetric key derived from the OVK. Using a nonce derived from the nullifier field that is guaranteed to be unique for double-spend protection, we ensure that no duplicate (key, nonce) pairs can appear.
2. Malleability prevention: Including encrypted_backref in the EffectHash transaction signing mechanism ensures that the field cannot be replaced by an adversary. If the field is malleable and the adversary knows the client is using DAGSync, an adversary may attempt to force clients to forget or lose funds.

Privacy Considerations

Adding the encrypted_backref field introduces a potential distinguisher for client software based on the presence or absence of the field. The privacy leak is that the field signals whether a user has updated to a specific client version or higher, i.e. one that supports encrypted_backref. No other information is revealed. The privacy impact can be mitigated entirely by requiring encrypted_backref for all spend actions in a future protocol upgrade once there is broad client support.

The design decision to include encrypted_backref reflects the fact that the information leakage is minor, and is justified to improve sync performance, reducing user friction and improving protocol adoption and thus the anonymity set of the network.

Copyright

Copyright and related rights waived via CC0.

Abstract

This proposal describes a simple, backwards-compatible mechanism to safeguard node operators against running or migrating with the wrong version of PD. It works by saving the current app version in non-consensus storage, allowing the node to detect if a migration is running against the wrong version, or the node is being started against the wrong version.

Motivation

Starting PD with pd start or migrating during upgrade with pd migrate require using the correct version of PD, otherwise the resulting node will be operating with the wrong app hash, preventing it from syncing with the rest of the network. This is problematic during an upgrade, which depends on sufficient nodes (by voting power) reaching consensus on the new state of the network; errors here can delay upgrades significantly.

For example, during the chain upgrade on mainnet to v0.80.0, at height 501975, there was confusion about apphash mismatches when the network resumed, due to operator error: one validator operator mistakenly ran the pd migrate command using the old version of pd, i.e. 0.79.x, when they should have used v0.80.0 instead. This resulted in a different app hash in that validator’s state, preventing the network from reaching consensus on the first post-upgrade block. Fortunately, the problem was quickly diagnosed, and the validator was able to rerun the migration from backed up state, resolving the problem and allowing the chain to resume.

This kind of error can be prevented at the software level, preventing this as a potential operator error.

Specification

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119 and RFC 8174.

We add a new non-consensus state key: app_version_safeguard (UTF-8), which can be used to store a u64 version value, as 8 little endian bytes.

Starting

When starting, PD SHOULD check that the app version safeguard is either:

• not present,
• or equal to the APP_VERSION constant in the app crate.

Then, PD SHOULD write the APP_VERSION constant into the app_version_safeguard slot.

Migrating

When migrating, PD SHOULD, in the context of an atomic migration transaction,

• check that app version safeguard is absent, or equal to the APP_VERSION constant of the pre-migration version of the app crate
• write the APP_VERSION constant of the post-migration version of the app crate into the app_version_safeguard slot.

Storing the post-migration version after the migrations are performed will ensure that on the next start, the version will match that of PD.

Backwards Compatability

This proposal is backwards compatible, because we never assume that the safeguard value is present in the state.

Rationale

We want to make sure that mechanism is backwards compatible, so that node operators are not forced to upgrade to the point release, and only gain benefits by doing so.

We also want a point release to be possible for this change, so that node operators can benefit from the safeguard ahead of a future upgrade, rather than only after it.

Security Considerations

There are no security considerations for this proposal.

Privacy Considerations

There are no privacy considerations for this proposal.

Copyright

Copyright and related rights waived via CC0.

Note: READ UIP-1 BEFORE USING THIS TEMPLATE! This is the suggested template for new UIPs. After you have filled in the requisite fields, please delete these comments. Note that an UIP number will be assigned by an editor. When opening a pull request to submit your UIP, please use an abbreviated title in the filename, uip-draft_title_abbrev.md. The title should be 44 characters or less. It should not repeat the UIP number in title, irrespective of the category.

TODO: Remove the note before submitting

Abstract

The Abstract is a multi-sentence (short paragraph) technical summary. This should be a very terse and human-readable version of the specification section. Someone should be able to read only the abstract to get the gist of what this specification does.

TODO: Remove the previous comments before submitting

Motivation

This section is optional.

The motivation section should include a description of any nontrivial problems the UIP solves. It should not describe how the UIP solves those problems, unless it is not immediately obvious. It should not describe why the UIP should be made into a standard, unless it is not immediately obvious.

With a few exceptions, external links are not allowed. If you feel that a particular resource would demonstrate a compelling case for your UIP, then save it as a printer-friendly PDF, put it in the assets folder, and link to that copy.

TODO: Remove the previous comments before submitting

Specification

The Specification section should describe the syntax and semantics of any new feature. The specification should be detailed enough to allow competing, interoperable implementations for any relevant Penumbra software.

It is recommended to follow RFC 2119 and RFC 8170. Do not remove the key word definitions if RFC 2119 and RFC 8170 are followed.

TODO: Remove the previous comments before submitting

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119 and RFC 8174.

Rationale

The rationale fleshes out the specification by describing what motivated the design and why particular design decisions were made. It should describe alternate designs that were considered and related work, e.g. how the feature is supported in other languages.

The current placeholder is acceptable for a draft.

TODO: Remove the previous comments before submitting

Backwards Compatibility

This section is optional.

All UIPs that introduce backwards incompatibilities must include a section describing these incompatibilities and their severity. The UIP must explain how the author proposes to deal with these incompatibilities. UIP submissions without a sufficient backwards compatibility treatise may be rejected outright.

The current placeholder is acceptable for a draft: "No backward compatibility issues found."

TODO: Remove the previous comments before submitting

Test Cases

This section is optional.

The Test Cases section should include expected input/output pairs, but may include a succinct set of executable tests. It should not include project build files. No new requirements may be be introduced here (meaning an implementation following only the Specification section should pass all tests here.)

If the test suite is too large to reasonably be included inline, then consider adding it as one or more files in ../assets/uip-####/. External links will not be allowed

TODO: Remove the previous comments before submitting

Reference Implementation

This section is optional.

The Reference Implementation section should include a minimal implementation that assists in understanding or implementing this specification. It should not include project build files. The reference implementation is not a replacement for the Specification section, and the proposal should still be understandable without it.

If the reference implementation is too large to reasonably be included inline, then consider adding it as one or more files in ../assets/uip-####/. External links will not be allowed.

TODO: Remove the previous comments before submitting

Security Considerations

All UIPs must contain a section that discusses the security implications/considerations relevant to the proposed change. Include information that might be important for security discussions, surfaces risks and can be used throughout the life cycle of the proposal. For example, include security-relevant design decisions, concerns, important discussions, implementation-specific guidance and pitfalls, an outline of threats and risks and how they are being addressed. UIP submissions missing the "Security Considerations" section will be rejected. A UIP cannot proceed to status "Final" without a Security Considerations discussion deemed sufficient by the reviewers.

The current placeholder is acceptable for a draft.

TODO: Remove the previous comments before submitting

Privacy Considerations

All UIPs must contain a section that discusses the privacy implications/considerations relevant to the proposed change. Include information that might be important for privacy discussions, surfaces risks and can be used throughout the life cycle of the proposal. For example, include privacy-relevant design decisions, concerns, important discussions, implementation-specific guidance and pitfalls, an outline of threats and risks and how they are being addressed. UIP submissions missing the "Privacy Considerations" section will be rejected. A UIP cannot proceed to status "Final" without a Privacy Considerations discussion deemed sufficient by the reviewers.

The current placeholder is acceptable for a draft.

TODO: Remove the previous comments before submitting

Copyright

Copyright and related rights waived via CC0.