From Permanent Storage to Parallel Compute - What is Arweave AO?
What is AO - the "hyper-parallel computer" on top of Arweave? What does it mean for $AR?
This post is brought to you by Yield Guild Games (YGG). YGG is a web3 guild protocol that enables players and gaming guilds to find their community, discover games and level up together. Its mission is to become the leading community-based user acquisition platform in web3 gaming through fun quests that help players build their onchain reputation
Find out more at: https://investors.yieldguild.io/
Introduction
Arweave, the decentralized permanent storage protocol, launched AO Computer, a parallel compute layer built on top of the network. Since its announcement on February 13, 2024, the AR token has surged in value, making it one of the best-performing large-cap tokens year-to-date. AO is touted as a "hyper-parallel computer" capable of running a vast number of computations simultaneously in a decentralized and verifiable manner. Some of its core features include:
Ability to split computations into many parallel "threads" that can run independently and communicate with each other, rather than relying on a single shared memory space like traditional blockchains.
Maintains a single system image (SSI), which means that the distributed network of AO nodes presents a single, cohesive computing platform to users and developers.
Built on top of Arweave's permanent data storage network, which enables it to support storing and processing large amounts of data, such as AI models, within its smart contracts.
The use of a new operating system called AOS, designed for highly scalable and parallel computations, running on the Lua programming language.
Is "parallel compute" merely an attempt to catch up with the latest buzzword, or does AO truly have the potential to fulfill its promise of unlocking new use cases and potentially achieving the vision of ‘World Computer’ with a distinct architectural approach? What does this mean for the AR token? By the end of this article, you would gain a clearer insights into these questions as well as the following concepts:
Strict global consensus vs. stateless infrastructure
Storage-based consensus paradigm
AO’s holographic state
Disclaimer: the author holds $AR tokens but this report is not sponsored by Arweave Founder or affiliated entities and is not an inducement to purchase $AR. DYOR!
Strict Global Consensus vs. Stateless Infra
Before we dive into AO itself, let’s go back to the basics about what is blockchain, what exactly it achieves, what it is good at, what are the limitations and what alternative paradigms could there be for decentralized compute.
Blockchain is essentially a unified, append-only ledger where actors globally come together to agree on a shared state through some predefined consensus rules. Here I draw from Polynya’s (a prominent pseudonymous crypto researcher) in-depth discussion about blockchain’s utility in enforcing strict global consensus. There are a series of blog posts which I will link here (Link 1, Link2, Link 3).
Our world runs on ledgers. These can be governments that hold your identity information, the commercial banks that hold your money, or tech companies that hold your family photos. Blockchain as a decentralized shared ledger and truth creation machine can thus be immensely powerful. Polynya argues that the most important property enabled by public blockchains is strict global consensus, which refers to the mechanism through which all participating nodes in a network agree on a single, consistent version of the truth or state of the ledger, regardless of their geographical location or individual stake. In addition, Polynya asserts that strict global consensus is essential for a number of applications including:
Objective Money: Public blockchains provide a decentralized platform for store-of-value assets like Bitcoin and Ethereum, which depend on global consensus to maintain their value and security.
Objective Identity: Limited aspects of identity, such as proving ownership or participation (e.g. Ethereum Name Service or Proof of Attendance Protocol), can require strict global consensus to verify authenticity without centralized authority.
However, achieving strict global consensus comes with several drawbacks:
Expensive: Implementing and maintaining strict global consensus is expensive in terms of the computational and social coordination costs.
Slow and inefficient: While many scaling solutions aim to increase throughput, strict global consensus inherently limits performance because each application’s activity might be confined to a single computational thread. Typical blockchain design can’t match the parallel processing capabilities of non-consensus-based applications that utilize vast server networks.
Virtual Machine Constraints: Applications running under strict global consensus are limited by the capabilities and rules of the underlying blockchain’s virtual machine. This can restrict what the applications can do, especially those needing high degrees of flexibility and composability.
To hone in on the point, we can think about a theoretical opposite:
Strict Global Consensus is defined as the need for every participant in a network to agree strictly on a set of objective outputs. This kind of consensus is essential for applications where absolute agreement on data accuracy and immutability is critical, such as in financial transactions.
Rough Global Consensus refers to a more flexible agreement that might be subject to interpretation and can vary between participants. It’s suitable in decentralized applications where exact outcomes are less critical and can be subjective.
Polynya further argues that most crypto applications do not require strict global consensus. Instead, they are better off to be designed as stateless infrastructure with rough global consensus.
Stateless infrastructure may have some form of state, but it does not need the entire network to consistently agree on that state to function properly. This is different from traditional blockchains, which demand that all nodes reach a consensus on the current state and have it consistently reflected throughout the network.
Put simply, stateless infra refers to systems that do not require a real-time, objective consensus state with strict state transitions. The importance of stateless infrastructure lies in its ability to overcome the limitations and challenges associated with traditional blockchains.
The reason to explain the above first is that AO essentially aligns with the vision of fostering a framework to develop stateless infrastructure rather than adhering to traditional methods of solving the blockchain trilemma through stringent global state maintenance. Therefore, it is not very suitable to draw direct comparisons between AO and other high-performance, fast Layer 1 blockchains that enhance VM performance through techniques like parallelization. With this understanding and insight, we can delve deeper into what Arweave’s AO aims to accomplish.
Refresher on Arweave
Arweave is a decentralized storage platform that operates on a unique architecture known as blockweave, that enables the network to provide immutable and permanent storage.
Arweave uses Succinct Proofs of Random Access (SPORA), an enhancement of the original Proof of Access (PoA) mechanism, to create consensus and maintain permanent storage. Arweave's Proof of Access (PoA) prioritizes the verification of data storage and retrieval over mere computational work. Unlike in Bitcoin’s Proof of Work where miners solve complex puzzles, Arweave miners demonstrate their ability to access and retrieve random past data blocks, promoting widespread data replication.
This approach gives rise to Arweave’s blockweave structure, where each block is connected not only to the previous and subsequent blocks but also to a randomly selected data block. Miners are required to access these randomly chosen blocks to add more data, motivating them to maintain the entire blockweave to continue mining and earning rewards.
Arweave's ecosystem is powered by the AR token, which incentivizes data storage within the network. Users pay a one-time fee in AR tokens for permanent data storage. These tokens compensate miners for maintaining data availability. Miners earn AR tokens by storing data and validating transactions. Apart from payment to miners, a significant portion (83%) of these fees contributes to an endowment—a fund designed to sustain the network long-term by continually providing incentives to miners, thus supporting Arweave's commitment to permanent data storage.
What is AO Computer?
The AO computer is a decentralized, actor-oriented machine that emerges from a network of nodes on the Arweave network. It's a single, unified computing environment that allows for parallel processes to operate independently, coordinating through an open message passing layer.
Unlike other blockchains, AO supports computation without protocol-enforced limitations, while maintaining network verifiability and trust minimization. With a modular architecture, users can choose their own virtual machines, sequencing models, and payment options, creating a unified computing environment for a wide range of workloads. The major components of AO are as follows:
Processes: A set of instructions executed by the network.
In AO, a process refers to a self-contained unit of computation that executes a specific set of instructions. Each process is a distinct entity that can operate independently. At initialization, a process can define its unique computing requirements, including the virtual machine, scheduler, memory allocation, and necessary extensions. This allows processes to tailor their execution environment to their specific needs, ensuring optimal performance and efficiency.
Messages: Process interactions in the network.
A message is a unit of communication between different processes. In AO, a message is a data structure that conforms to Arweave's native data structure (ANS-104) and is stored on the Arweave blockchain for security and verifiability.
Scheduler Units (SU): Assign a sequence number to messages and ensure that they are uploaded to Arweave.
SUs receive messages and perform key operations to maintain process continuity and integrity. They assign a unique incremental nonce to each message to ensure order and authenticity. SUs also upload messages to the Arweave data layer to ensure message availability and immutability.
Compute Units (CUs): Nodes that users and messenger units can use to calculate the state of processes in ao.
CUs compete with each other to complete computing tasks for users and SUs. Once a task is complete, CUs return a signed proof with the results. CUs have the discretion to accept or decline computing tasks based on their capacity and available resources.
Messenger Units (MUs): nodes that relay messages throughout the network.
MUs are responsible for relaying messages between processes. They ensure that messages are delivered to the correct destination and process them in the correct order. MUs also manage process subscriptions and schedule cron interactions.
Arweave AO derives its name from the Actor Oriented model, a computing framework that emphasizes units of concurrency known as Actors. Each Actor manages its own state and interacts with others through message passing, which prevents any direct alteration of another Actor's state. This method is particularly effective for systems that require high levels of concurrency, distribution, and fault tolerance.
Unlike conventional blockchain architectures that rely on a globally synchronized state for smart contracts, AO assigns each process its own independent state. This avoids the limitations of environments like the EVM, where all transactions are processed in a sequential order, creating delays as each transaction waits for the prior one to complete.
In AO, each process operates like a separate subnet or rollup, allowing for independent and parallel processing. This framework facilitates extensive scalability by enabling each node within the network to process different parts of a computation simultaneously, rather than sequentially.
The concept of a "holographic state" in the AO system represents a significant shift in how process states are managed and understood. In traditional blockchain, the state of a process is continuously updated and agreed upon by all network participants, AO uses a method where the state is not directly computed in real time. Instead, it's derived "holographically" from a log of messages stored on the Arweave network.
We can think of the ‘state’ on AO like a 3D image that only becomes visible only when viewed with the right 3D glasses. The information on the image is readily available, but is not immediately obvious to naked eyes. Only when using the right glasses can you properly interpret the data. Verifiability can be achieved because anyone with the right 3D glasses can decipher the state and will always get the same result.
What is Storage-based Consensus Paradigm (SCP)
The Storage-based Consensus Paradigm (SCP) posits that as long as storage is immutable and the transactions recorded on it are traceable, consistent results will be achieved regardless of where the computing application is executed. Simply put, it‘s ‘data on-chain and computation off-chain.’
The concept outlined by SCP is akin to Modular Blockchain, which advocates for the separation of data availability from execution. SCP was first proposed in 2021 by outprog from EverVision, a project within the Arweave ecosystem. This was around the same time LazyLedger was rebranded to Celestia, a blockchain known for its focus on data availability (DA).
Just like the concept of modularity, SCP also separates computation and DA, with the blockchain managing only data storage. Computation and the resulting state are handled off-chain by clients and servers. Arweave essentially functions as a "world hard drive," where all raw transaction data is stored, ensuring immutability and traceability.
Utilizing Arweave and SCP, anyone can download and verify the data. The state of a process is primarily inferred through a log of interactions stored on Arweave. This ties back to the 'holographic state' previously discussed. Although this state is not directly stored, it can be deterministically computed by any network participant, ensuring that the process's state is consistently verifiable.
The logic of off-chain computation in SCP can also be compared to various rollups in the Ethereum and Celestia ecosystem. AO, which is based on SCP, can be viewed as Arweave's Layer 2 solution. Specifically, AO represents a particular version of the SCP that is architecturally most similar to a sovereign rollup, which means they can fork independently of the DA layer.
The primary goal of most blockchains is to achieve consensus in near real-time on a set of outputs that all nodes can agree upon. Once consensus is reached, the blockchain's work is complete, and the data can be discarded.
Data availability layers, such as Celestia, ensure that block data is provably published, allowing rollups to determine the current state of the chain. However, even Celestia does not guarantee the permanent storage and retrievability of historical data.
In fact, most blockchains, except for Bitcoin, are not designed for permanence. They will prune and delete old data to reduce storage burdens on node operators. For instance, Ethereum’s roadmap includes "the Purge," which seeks to remove historical data.
This is why the meaning of SCP is not simply "separate DA from execution". SCP is uniquely enabled by Arweave, precisely because of its property of permanence, a trait that stands out from other DA chains. And this immutability is the key to creating a holographic state that enables AO.
Understanding AO through Bitcoin inscriptions
In envisioning the AO framework, another more intuitive way is to think of it as a computing system that leverages a highly efficient, super parallel indexer. This is akin to Bitcoin Ordinals and Inscriptions, where indexers extract JSON data and manage it off-chain according to specific rules. AO enhances this approach by allowing users to either select from multiple indexers or operate their own, effectively safeguarding against any potential discrepancies caused by indexer misconduct.
In AO, essential data such as message sequencing and the holographic state of processes are securely stored on Arweave. The SCP, which acts as an on-chain framework, ensures that any component of AO can be restored from its holographic data, simplifying the verification process without the necessity for a full node.
In the Bitcoin Ordinals system, an indexer plays a critical role. It reads the data on Bitcoin Network, records the sequence of transactions and determines which transactions are valid. This means the indexer effectively decides the final state of the data. Generally, a few indexers are run either by wallet services (eg. OKX Wallet or Ordinals Wallet) or marketplaces (e.g. Magic Eden, Unisat). Some entities might choose to share the same indexer, while others might use their own.
In the AO system, three units work together to determine the state: the Scheduler Unit (SU), Compute Unit (CU), and Messenger Unit (MU). These units collaborate, and their interactions may include some form of Proof of Stake (PoS) consensus, meaning they can’t independently decide the state based on their discretion as some inscriptions indexers can.
Essentially, in AO, the combination of SU, CU, and MU functions similarly to how an indexer works in Ordinals but with the indexing rule put on-chain. This is why we can refer to AO as a sort of sovereign rollup where Arweave acts as the DA layer and apps on top decide their own consensus rules.
What about verifiability?
Curious readers might notice a potential issue: under this SCP and holographic state idea, when a process receives inputs from another process, it's unclear if these inputs can be trusted, as CUs only check if inputs have been recorded on AR, not their validity. Let’s explore this further and see how AO might address this.
AO is designed to enable deterministic computation to take place off-chain, classifying these operations as "processes." With AO, computations are carried out by Computing Units (CUs), which process inputs into outputs. These outputs, once computed, are stored on Arweave.
The issue on the surface is that CUs may not necessarily run a consensus protocol, which means that the trustworthiness of their computational results might not be guaranteed. Nevertheless, these outputs are recorded on the blockchain and can be verified later if required (verifiability is the ability of different parties to reach the same conclusion by examining the same evidence, and records). This is essentially how Bitcoin Inscriptions work, where the indexers are the so-called consensus layer.
To address potential inaccuracies or malicious activities that could lead to incorrect outputs being recorded, AO could employ an optimistic verification mechanism through a system of staking and slashing. A special group of CUs would be responsible for managing this system. They can be challenged to verify outputs by checking against all historical inputs of a process, ensuring outputs are derived correctly from their inputs.
As AO is permissionless, anyone can operate a CU, and there is a vested interest in maintaining correctness across interconnected processes. This creates a decentralized network of off-chain indexers, and applications can prescribe the most appropriate security profile depending on the applications’ needs. (Link)
What this means is that AO offers a highly adaptable security model for its processes. Processes can opt for different security mechanisms depending on their needs. For example, they might choose to trust a specific set of MUs or SUs in a Proof of Authority model, similar to what's used in the AO testnet. Alternatively, they could implement a Proof of Stake system or use further innovations like zkWASM (zero knowledge web assembly). The graphic below from the ICP forum sums it up nicely where the blue shaded area indicates the vast possibility of security choices.
What use cases can AO unlock?
Recently, Arweave announced the launch of AO Ventures, a 10-week incubator program with $35 million in committed capital from investors such as Faction VC, Distributed Global, and others. This initiative is designed to support developers in leveraging Arweave’s capabilities for permanent storage and parallel computing to develop new applications. Here are some notable projects currently being developed on AO:
0rbit: A decentralized, permissionless, and censorship-resistant oracle network that utilizes Arweave’s AO infrastructure for enhanced security and reliability.
AOVM: Touted as the first EVM-compatible decentralized compute blockchain, AOVM is built on the AO framework, enhancing compatibility with Ethereum-based applications while leveraging AO’s computational efficiencies.
Bark: The first decentralized exchange (DEX) on AO, Bark supports low-latency, high-throughput trading and was launched in conjunction with the Astro stablecoin protocol.
Astro Stablecoin: Built on AO, Astro introduces a stablecoin named USDA, which is minted using Arweave’s native token $AR in an over-collateralized model to ensure value stability.
DataOS: An AI-driven blockchain browser that operates on an open data lake, launched to enhance the accessibility and usability of blockchain data through the Arweave AO network.
Harvest: Positioned as the first tokenized natural resource market on AO, Harvest diversifies the real-world assets (RWA) sector by tokenizing land and livestock assets, moving beyond traditional bond-type RWAs.
Autonomous Fi: This project is advancing the development of composable semi and fully autonomous agents. Unlike typical frameworks that rely on off-chain programs for signal processing and logic, Autonomous Fi utilizes AO’s capabilities for verifiable compute and on-chain data processing, enabling autonomous agents to self-learn and operate directly from on-chain data streams.
Conclusion - What does it mean for $AR?
Arweave AO marks a transformative development in crypto by combining the robustness of decentralized storage with the efficiency of parallel computing. It adopts a model that supports independent state processes and allows for versatile security and consensus designs through the implementation of a holographic state.
From an token perspective, Arweave can increasingly be seen as the DA Layer for AO, which focuses on execution as an L2 atop it. AO’s launch also suggests a broader scope for value accrual to Arweave's native token AR, as data from AO requires AR for storage costs. With each development in the AO ecosystem, Arweave's role as a foundational infrastructure component grows, enhancing its potential for broader utility and value accrual.
References
The ao Computer's Revolution in Decentralized Computing (Part I)
How does Arweave's Mining Mechanism Work? - Community Labs Blog
The meaning of verifiability: Bundle SmartWeave’s trustless consensus
Storage Consensus Paradigm: Non Blockchain for the next generation
Arweave AO: The Ready Player One in the AI Era | by ZAN | Mar, 2024
Let's solve these protocol weaknesses - Internet Computer Developer Forum
AO — An Application Consensus Interaction System with infinite scalability
Does your application need strict global consensus? — polynya
Understanding Arweave’s Consensus Mechanism Iteration in One Article
Disclaimer
The Blockcrunch Podcast (“Blockcrunch”) is an educational resource intended for informational purposes only. Blockcrunch produces a weekly podcast and newsletter that routinely covers projects in Web 3 and may discuss assets that the host or its guests have financial exposure to.
Some Blockcrunch VIP posts are written by contractors to Blockcrunch and posts reflect the contractors’ independent views, not Blockcrunch’s official stance. Blockcrunch requires contractors to disclose their financial exposure to projects they write about but is not able to fully guarantee no such conflicts of interest exist. Blockcrunch itself will not buy or sell assets it covers 72 hours prior to and subsequent to the publication of a piece; however, its directors, employees, contractors and affiliates may buy or sell assets prior to or subsequent to publication of any content and will make disclosures on a best effort basis.
Views held by Blockcrunch’s guests are their own. None of Blockcrunch, its registered entity or any of its affiliated personnel are licensed to provide any type of financial advice, and nothing on Blockcrunch’s podcast, newsletter, website and social media should be construed as financial advice. Blockcrunch also receives compensation from its sponsor; sponsorship messages do not constitute financial advice or endorsement.
For more detailed disclaimers, visit https://blockcrunch.substack.com/about