Cuprate Architecture

WIP

Cuprate's architecture book.

Sections are notated with colors indicating how complete they are:

ColorMeaning
⚪️Empty
🔴Severely lacking information
🟠Lacking some information
🟡Almost ready
🟢OK

Continue to the next chapter by clicking the right > button, or by selecting it on the left side.

All chapters are viewable by clicking the top-left button.

The entire book can searched by clicking the top-left 🔍 button.

Last change: 2024-09-08, commit: 0162553

Foreword

Monero1 is a large software project, coming in at 329k lines of C++, C, headers, and make files.2 It is directly responsible for 2.6 billion dollars worth of value.3 It has had over 400 contributors, more if counting unnamed contributions.4 It has over 10,000 node operators and a large active userbase.5

The project wasn't always this big, but somewhere in the midst of contributors coming and going, various features being added, bugs being fixed, and celebrated cryptography being implemented - there was an aspect that was lost by the project that it could not easily gain again: maintainability.

Within large and complicated software projects, there is an important transfer of knowledge that must occur for long-term survival. Much like an organism that must eventually pass the torch onto the next generation, projects must do the same for future contributors.

However, newcomers often lack experience, past contributors might not be around, and current maintainers may be too busy. For whatever reason, this transfer of knowledge is not always smooth.

There is a solution to this problem: documentation.

The activity of writing the what, where, why, and how of the solutions to technical problems can be done in an author's lonesome.

The activity of reading these ideas can be done by future readers at any time without permission.

These readers may be new prospective contributors, it may be the current maintainers, it may be researchers, it may be users of various scale. Whoever it may be, documentation acts as the link between the past and present; a bottle of wisdom thrown into the river of time for future participants to open.

This book is the manifestation of this will, for Cuprate6, an alternative Monero node. It documents Cuprate's implementation from head-to-toe such that in the case of a contributor's untimely disappearance, the project can continue.

People come and go, documentation is forever.

— hinto-janai


2

git ls-files | grep "\.cpp$\|\.h$\|\.c$\|CMake" | xargs cat | wc -l on cc73fe7

3

2024-05-24: $143.55 USD * 18,151,608 XMR = $2,605,663,258

4

git log --all --pretty="%an" | sort -u | wc -l on cc73fe7

6

https://github.com/Cuprate/cuprate

Last change: 2024-09-08, commit: 0162553

Intro

Cuprate is an alternative Monero node implementation.

This book describes Cuprate's architecture, ranging from small things like database pruning to larger meta-components like the networking stack.

A brief overview of some aspects covered within this book:

  • Component designs
  • Implementation details
  • File location and purpose
  • Design decisions and tradeoffs
  • Things in relation to monerod
  • Dependency usage

Source code

The source files for this book can be found on at: https://github.com/Cuprate/architecture-book.

Last change: 2024-09-08, commit: 0162553

Who this book is for

Maintainers

As mentioned in Foreword, the group of people that benefit from this book's value the most by far are the current and future Cuprate maintainers.

Cuprate's system design is documented in this book such that if you were ever to build it again from scratch, you would have an excellent guide on how to do such, and also where improvements could be made.

Practically, what that means for maintainers is that it acts as the reference. During maintenance, it is quite valuable to have a book that contains condensed knowledge on the behavior of components, or how certain code works, or why it was built a certain way.

Contributors

Contributors also have access to the inner-workings of Cuprate via this book, which helps when making larger contributions.

Design decisions and implementation details notated in this book helps answer questions such as:

  • Why is it done this way?
  • Why can it not be done this way?
  • Were other methods attempted?

Cuprate's testing and benchmarking suites, unknown to new contributors, are also documented within this book.

Researchers

This book contains the why, where, and how of the implementation of formal research.

Although it is an informal specification, this book still acts as a more accessible overview of Cuprate compared to examining the codebase itself.

Operators & users

This book is not a practical guide for using Cuprate itself.

For configuration, data collection (also important for researchers), and other practical usage, see Cuprate's user book.

Observers

Anyone curious enough is free to learn the inner-workings of Cuprate via this book, and maybe even contribute someday.

Last change: 2024-09-08, commit: 0162553

Required knowledge

General

  • Rust
  • Monero
  • System design

Components

Storage

  • Embedded databases
  • LMDB
  • redb

RPC

  • axum
  • tower
  • async
  • JSON-RPC 2.0
  • Epee

Networking

  • tower
  • tokio
  • async
  • Levin

Instrumentation

  • tracing
Last change: 2024-09-08, commit: 0162553

How to use this book

Maintainers

Contributors

Researchers

Last change: 2024-09-08, commit: 0162553

⚪️ Bird's eye view

Last change: 2024-09-08, commit: 0162553

⚪️ Map

Last change: 2024-09-08, commit: 0162553

⚪️ Components

Last change: 2024-09-08, commit: 0162553

⚪️ Formats, protocols, types

Last change: 2024-09-08, commit: 0162553

⚪️ monero_serai

Last change: 2024-09-08, commit: 0162553

⚪️ cuprate_types

Last change: 2024-09-08, commit: 0162553

⚪️ cuprate_helper

Last change: 2024-09-08, commit: 0162553

⚪️ Epee

Last change: 2024-09-08, commit: 0162553

⚪️ Levin

Last change: 2024-09-08, commit: 0162553

⚪️ Storage

Last change: 2024-09-08, commit: 0162553

⚪️ Database abstraction

Last change: 2024-09-08, commit: 0162553

⚪️ Blockchain

Last change: 2024-09-08, commit: 0162553

⚪️ Transaction pool

Last change: 2024-09-08, commit: 0162553

⚪️ Pruning

Last change: 2024-09-08, commit: 0162553

RPC

monerod's daemon RPC has three kinds of RPC calls:

  1. JSON-RPC 2.0 methods, called at the /json_rpc endpoint
  2. JSON (but not JSON-RPC 2.0) methods called at their own endpoints, e.g. /get_height
  3. Binary (epee) RPC methods called at their own endpoints ending in .bin, e.g. /get_blocks.bin

Cuprate's RPC aims to mirror monerod's as much as it practically can.

This includes, but is not limited to:

  • Using the same endpoints
  • Receiving the same request data
  • Sending the same response data
  • Responding with the same HTTP status codes
  • Following internal behavior (e.g. /pop_blocks)

Not all monerod behavior can always be followed, however.

Some are not followed on purpose, some cannot be followed due to technical limitations, and some cannot be due to the behavior being monerod specific such as the /set_log_categories endpoint which uses monerod's logging categories.

Both subtle and large differences between Cuprate's RPC and monerod's RPC are documented in the Differences with monerod section.

Main RPC components

The main components that make up Cuprate's RPC are noted below, alongside the equivalent monerod code and other notes.

Cuprate cratemonerod (rough) equivalentPurposeNotes
cuprate-json-rpcjsonrpc_structs.h, http_server_handlers_map2.hJSON-RPC 2.0 implementationmonerod's JSON-RPC 2.0 handling is spread across a few files. The first defines some data structures, the second contains macros that (essentially) implement JSON-RPC 2.0.
cuprate-rpc-typescore_rpc_server_commands_defs.hRPC request/response type definitions and (de)serialization
cuprate-rpc-interfacecore_rpc_server.hRPC interface, routing, endpoints
cuprate-rpc-handlercore_rpc_server.cppRPC request/response handlingThese are the "inner handler" functions that turn requests into responses
Last change: 2024-09-08, commit: 0162553

JSON-RPC 2.0

Cuprate has a standalone crate that implements the JSON-RPC 2.0 specification, cuprate-json-rpc. The RPC methods at the /json_rpc endpoint use this crate's types, functions, and (de)serialization.

There is nothing too special about Cuprate's implementation. Any small notes and differences are noted in the crate documentation.

As such, there is not much to document here, instead, consider reading the very brief JSON-RPC 2.0 specification, and the cuprate-json-rpc crate documentation.

TODO: document method/params vs flattened base when figured out.

Last change: 2024-09-08, commit: 0162553

The types

Cuprate has a crate that defines all the types related to RPC: cuprate-rpc-types.

The main purpose of this crate is to port the types used in monerod's RPC and to re-implement (de)serialization for those types, whether that be JSON, epee, or a custom mix.

The bulk majority of these types are request & response types, i.e. the inputs Cuprate's RPC is expecting from users, and the output it will respond with.

Example

To showcase an example of the kinds of types defined in this crate, here is a request type:

#![allow(unused)]
fn main() {
#[serde(transparent)]
#[repr(transparent)]
struct OnGetBlockHashRequest {
	block_height: [u64; 1],
}
}

This is the input (params) expected in the on_get_block_hash method.

As seen above, the type itself encodes some properties, such as being (de)serialized transparently, and the input being an array with 1 length, rather than a single u64. This is to match the behavior of monerod.

An example JSON form of this type would be:

{
  "jsonrpc": "2.0",
  "id": "0",
  "method": "on_get_block_hash",
  "params": [912345] // <- This can (de)serialize as a `OnGetBlockHashRequest`
}
Last change: 2024-09-08, commit: 0162553

Misc types

Other than the main request/response types, this crate is also responsible for any miscellaneous types used within monerod's RPC.

For example, the status field within many RPC responses is defined within cuprate-rpc-types.

Types that aren't requests/responses but exist within request/response types are also defined in this crate, such as the Distribution structure returned from the get_output_distribution method.

Last change: 2024-09-08, commit: 0162553

Base RPC types

There exists a few "base" types that many types are built on-top of in monerod. These are also implemented in cuprate-rpc-types.

For example, many requests include these 2 fields:

{
  "status": "OK",
  "untrusted": false,
}

This is rpc_response_base in monerod, and ResponseBase in Cuprate.

These types are flattened into other types, i.e. the fields from these base types are injected into the given type. For example, get_block_count's response type is defined like such in Cuprate:

#![allow(unused)]
fn main() {
struct GetBlockCountResponse {
	// The fields of this `base` type are directly
	// injected into `GetBlockCountResponse` during
	// (de)serialization.
	//
	// I.e. it is as if this `base` field were actually these 2 fields:
	// status: Status,
	// untrusted: bool,
    base: ResponseBase,
	count: u64,
}
}

The JSON output of this type would look something like:

{
  "status": "OK",
  "untrusted": "false",
  "count": 993163
}

RPC payment

monerod also contains RPC base types for the RPC payment system. Although the RPC payment system is pseudo deprecated, monerod still generates these fields in responses, and thus, so does Cuprate.

Last change: 2024-09-08, commit: 0162553

The type generator macro

Request and response types make up the majority of cuprate-rpc-types.

  • Request types are the inputs expected from users
  • Response types are what will be outputted to users

Regardless of being meant for JSON-RPC, binary, or a standalone JSON endpoint, all request/response types are defined using the "type generator macro". This macro is important because it defines all request/response types.

This macro:

  • Defines a matching pair of request & response types
  • Implements many derive traits, e.g. Clone on those types
  • Implements both serde and epee on those types
  • Automates documentation, tests, etc.

See here for example usage of this macro.

Last change: 2024-09-08, commit: 0162553

Metadata

cuprate-rpc-types also provides some traits to access some metadata surrounding RPC data types.

For example, trait RpcCall allows accessing whether an RPC request is restricted or not.

monerod has a boolean permission system. RPC calls can be restricted or not. If an RPC call is restricted, it will only be allowed on un-restricted RPC servers (18081). If an RPC call is not restricted, it will be allowed on all RPC server types (18081 & 18089).

This metadata is used in crates that build upon cuprate-rpc-types, e.g. to know if an RPC call should be allowed through or not.

Last change: 2024-09-08, commit: 0162553

(De)serialization

A crucial responsibility of cuprate-rpc-types is to provide the correct (de)serialization of types.

The input/output of Cuprate's RPC should match monerod (as much as practically possible).

A simple example of this is that /get_height should respond with the exact same data for both monerod and Cuprate:

{
  "hash": "7e23a28cfa6df925d5b63940baf60b83c0cbb65da95f49b19e7cf0ce7dd709ce",
  "height": 2287217,
  "status": "OK",
  "untrusted": false
}

Behavior would be considered incompatible if any of the following were true:

  • Fields are missing
  • Extra fields exist
  • Field types are incorrect (string instead of number, etc)

JSON

(De)serialization for JSON is implemented using serde and serde_json.

cuprate-rpc-interface (the main crate responsible for the actual output) uses serde_json for JSON formatting. It is mostly the same formatting as monerod, although there are slight differences.

Technically, the formatting of the JSON output is not handled by cuprate-rpc-types, users are free to choose whatever formatting they desire.

Epee

(De)serialization for the epee binary format is handled by Cuprate's in-house cuprate-epee-encoding library.

Bitcasted structs

TODO: https://github.com/monero-project/monero/issues/9422

Compressed data

TODO: https://github.com/monero-project/monero/issues/9422

Last change: 2024-09-08, commit: 0162553

The interface

This section is short as cuprate-rpc-interface contains detailed documentation.

The RPC interface, which includes:

  • Endpoint routing (/json_rpc, /get_blocks.bin, etc)
  • Route function signatures (async fn json_rpc(...) -> Response)
  • Type (de)serialization
  • Any miscellaneous handling (denying restricted RPC calls)

is handled by the cuprate-rpc-interface crate.

Essentially, this crate provides the API for the RPC.

cuprate-rpc-interface is built on-top of axum and tower, which are the crates doing the bulk majority of the work.

Request -> Response

The functions that map requests to responses are not implemented by cuprate-rpc-interface itself, they must be provided by the user, i.e. it can be customized.

In Rust terms, this crate provides you with:

#![allow(unused)]
fn main() {
async fn json_rpc(
	state: State,
	request: Request,
) -> Response {
	/* your handler here */
}
}

and you provide the function body.

The main handler crate is cuprate-rpc-handler. This crate implements the standard RPC behavior, i.e. it mostly mirrors monerod.

Although, it's worth noting that other implementations are possible, such as an RPC handler that caches blocks, or an RPC handler that only accepts certain endpoints, or any combination.

Last change: 2024-09-08, commit: 0162553

The handler

TODO: fill after cuprate-rpc-handler is created.

Last change: 2024-09-08, commit: 0162553

🔴 The server

TODO: fill after cuprate-rpc-server or binary impl is created.

Last change: 2024-09-08, commit: 0162553

Differences with monerod

As noted in the introduction, monerod's RPC behavior cannot always be perfectly followed by Cuprate.

The reasoning for the differences can vary from:

  • Technical limitations
  • Behavior being monerod-specific
  • Purposeful decision to not support behavior

This section lays out the details of the differences between monerod's and Cuprate's RPC system.

Last change: 2024-09-08, commit: 0162553

JSON field ordering

When serializing JSON, monerod has the behavior to order key fields within a scope alphabetically.

For example:

{
  "id": "0",
  "jsonrpc": "2.0",
  "result": {
    "blockhashing_blob": "...",
    "blocktemplate_blob": "...",
    "difficulty": 283305047039,
    "difficulty_top64": 0,
    "expected_reward": 600000000000,
    "height": 3195018,
    "next_seed_hash": "",
    "prev_hash": "9d648e741d85ca0e7acb4501f051b27e9b107d3cd7a3f03aa7f776089117c81a",
    "reserved_offset": 131,
    "seed_hash": "e2aa0b7b55042cd48b02e395d78fa66a29815ccc1584e38db2d1f0e8485cd44f",
    "seed_height": 3194880,
    "status": "OK",
    "untrusted": false,
    "wide_difficulty": "0x41f64bf3ff"
  }
}

In the main {}, id comes before jsonrpc, which comes before result.

The same alphabetical ordering is applied to the fields within result.

Cuprate uses serde for JSON serialization, which serializes fields based on the definition order, i.e. whatever order the fields are defined in the code, is the order they will appear in JSON.

Some struct fields within Cuprate's RPC types happen to be alphabetical, but this is not a guarantee.

As these are JSON maps, the ordering of fields should not matter, although this is something to note as the output will technically differ.

Example incompatibility

An example of where this leads to incompatibility is if specific line numbers are depended on to contain specific fields.

For example, this will print the 10th line:

curl http://127.0.0.1:18081/json_rpc -d '{"jsonrpc":"2.0","id":"0","method":"get_block_template","params":{"wallet_address":"44GBHzv6ZyQdJkjqZje6KLZ3xSyN1hBSFAnLP6EAqJtCRVzMzZmeXTC2AHKDS9aEDTRKmo6a6o9r9j86pYfhCWDkKjbtcns","reserve_size":60}' -H 'Content-Type: application/json' | sed -n 10p

It will be "height": 3195018 in monerod's case, but may not necessarily be for Cuprate.

By all means, this should not be relied upon in the first place, although it is shown as an example.

Last change: 2024-09-08, commit: 0162553

JSON formatting

In general, Cuprate's JSON formatting is very similar to monerod, but there are some differences.

This is a list of those differences.

Pretty vs compact

TODO: decide when handlers are created if we should allow custom formatting.

Cuprate's RPC (really, serde_json) can be configured to use either:

monerod uses something similar to pretty formatting.

As an example, pretty formatting:

{
  "number": 1,
  "array": [
    0,
    1
  ],
  "string": "",
  "array_of_objects": [
    {
      "x": 1.0,
      "y": -1.0
    },
    {
      "x": 2.0,
      "y": -2.0
    }
  ]
}

compact formatting:

{"number":1,"array":[0,1],"string":"","array_of_objects":[{"x":1.0,"y":-1.0},{"x":2.0,"y":-2.0}]}

Array of objects

monerod will format an array of objects like such:

{
  "array_of_objects": [{
    "x": 0.0,
    "y": 0.0,
  },{
    "x": 0.0,
    "y": 0.0,
  },{
    "x": 0.0,
    "y": 0.0
  }]
}

Cuprate will format the above like such:

{
  "array_of_objects": [
    {
      "x": 0.0,
      "y": 0.0,
    },
    {
      "x": 0.0,
      "y": 0.0,
    },
    {
      "x": 0.0,
      "y": 0.0
    }
  ]
}

Array of maps containing named objects

An method that contains outputs like this is the peers field in the sync_info method:

curl \
    http://127.0.0.1:18081/json_rpc \
    -d '{"jsonrpc":"2.0","id":"0","method":"sync_info"}' \
    -H 'Content-Type: application/json'

monerod will format an array of maps that contains named objects like such:

{
  "array": [{
    "named_object": {
      "field": ""
    }
  },{
    "named_object": {
      "field": ""
    }
  }]
}

Cuprate will format the above like such:

{
  "array": [
    {
      "named_object": {
        "field": ""
      }
    },
    {
      "named_object": {
        "field": ""
      }
    }
  ]
}
Last change: 2024-09-08, commit: 0162553

JSON strictness

This is a list of behavior that monerod's JSON parser allows, that Cuprate's JSON parser (serde_json) does not.

In general, monerod's parser is quite lenient, allowing invalid JSON in many cases. Cuprate's (really, serde_json) JSON parser is quite strict, essentially sticking to the JSON specification.

Cuprate also makes some decisions that are different than monerod, but are not necessarily more or less strict.

Missing closing bracket

monerod will accept JSON missing a final closing }.

Example:

curl \
	http://127.0.0.1:18081/json_rpc \
	-d '{"jsonrpc":"2.0","id":"0","method":"get_block_count"' \
	-H 'Content-Type: application/json'

Trailing ending comma

monerod will accept JSON containing a final trailing ,.

Example:

curl \
	http://127.0.0.1:18081/json_rpc \
	-d '{"jsonrpc":"2.0","id":"0","method":"get_block_count",}' \
	-H 'Content-Type: application/json'

Allowing - in fields

monerod allows - as a valid value in certain fields, not a string "-", but the character -.

The fields where this is allowed seems to be any field monerod does not explicitly look for, examples include:

  • jsonrpc
  • id
  • params (where parameters are not expected)
  • Any ignored field

The JSON-RPC 2.0 specification does state that the response id should be null upon errors in detecting the request id, although in this case, this is invalid JSON and should not make it this far. The response will contain the default id: 0 in this case.

Example:

curl \
	http://127.0.0.1:18081/json_rpc \
	-d '{"jsonrpc":-,"id":-,"params":-,"IGNORED_FIELD":-,"method":"get_block_count"}' \
	-H 'Content-Type: application/json'
Last change: 2024-09-08, commit: 0162553

JSON-RPC strictness

This is a list of behavior that monerod's JSON-RPC implementation allows, that Cuprate's JSON-RPC implementation does not.

In general, monerod's JSON-RPC is quite lenient, going against the specification in many cases. Cuprate's JSON-RPC implementation is slightly more strict.

Cuprate also makes some decisions that are different than monerod, but are not necessarily more or less strict.

Allowing an incorrect jsonrpc field

The JSON-RPC 2.0 specification states that the jsonrpc field must be exactly "2.0".

monerod allows jsonrpc to:

  • Be any string
  • Be an empty array
  • Be null
  • Not exist at all

Examples:

curl \
	http://127.0.0.1:18081/json_rpc \
	-d '{"jsonrpc":"???","method":"get_block_count"}' \
	-H 'Content-Type: application/json'
curl \
	http://127.0.0.1:18081/json_rpc \
	-d '{"jsonrpc":[],"method":"get_block_count"}' \
	-H 'Content-Type: application/json'
curl \
	http://127.0.0.1:18081/json_rpc \
	-d '{"jsonrpc":null,"method":"get_block_count"}' \
	-H 'Content-Type: application/json'
curl \
	http://127.0.0.1:18081/json_rpc \
	-d '{"method":"get_block_count"}' \
	-H 'Content-Type: application/json'

Allowing id to be any type

JSON-RPC 2.0 responses must contain the same id as the original request.

However, the specification states:

An identifier established by the Client that MUST contain a String, Number, or NULL value if included

monerod does not check this and allows id to be any JSON type, for example, a map:

curl \
    http://127.0.0.1:18081/json_rpc \
	-d '{"jsonrpc":"2.0","id":{"THIS":{"IS":"ALLOWED"}},"method":"get_block_count"}' \
	-H 'Content-Type: application/json'

The response:

{
  "id": {
    "THIS": {
      "IS": "ALLOWED"
    }
  },
  "jsonrpc": "2.0",
  "result": {
    "count": 3210225,
    "status": "OK",
    "untrusted": false
  }
}

Responding with id:0 on error

The JSON-RPC specification states:

If there was an error in detecting the id in the Request object (e.g. Parse error/Invalid Request), it MUST be Null.

Although, monerod will respond with id:0 in these cases.

curl \
    http://127.0.0.1:18081/json_rpc \
	-d '{"jsonrpc":"2.0","id":asdf,"method":"get_block_count"}' \
	-H 'Content-Type: application/json'

Response:

{
  "error": {
    "code": -32700,
    "message": "Parse error"
  },
  "id": 0,
  "jsonrpc": "2.0"
}

Responding to notifications

TODO: decide on Cuprate behavior https://github.com/Cuprate/cuprate/pull/233#discussion_r1704611186

Requests that have no id field are "notifications".

The JSON-RPC 2.0 specification states that requests without an id field must not be responded to.

Example:

curl \
	http://127.0.0.1:18081/json_rpc \
	-d '{"jsonrpc":"2.0","method":"get_block_count"}' \
	-H 'Content-Type: application/json'

Upper/mixed case fields

monerod will accept upper/mixed case fields on:

  • jsonrpc
  • id

method however, is checked.

The JSON-RPC 2.0 specification does not outright state what case to support, although, Cuprate only supports lowercase as supporting upper/mixed case is more code to add as serde by default is case-sensitive on struct fields.

Example:

curl \
	http://127.0.0.1:18081/json_rpc \
	-d '{"jsONrPc":"2.0","iD":0,"method":"get_block_count"}' \
	-H 'Content-Type: application/json'
Last change: 2024-09-08, commit: 0162553

HTTP methods

monerod endpoints supports multiple HTTP methods that do not necessarily make sense.

For example:

curl \
	http://127.0.0.1:18081/get_limit \
	-H 'Content-Type: application/json' \
	--request DELETE

This is sending an HTTP DELETE request, which should be a GET.

monerod will respond to this the same as GET, POST, PUT, and TRACE.

Cuprate's behavior

TODO: decide allowed HTTP methods for Cuprate https://github.com/Cuprate/cuprate/pull/233#discussion_r1700934928.

Last change: 2024-09-08, commit: 0162553

RPC payment

The RPC payment system in monerod is a pseudo-deprecated system that allows node operators to be compensated for RPC usage.

Although this system is pseudo-deprecated, monerod still generates related fields in responses. Cuprate follows this behavior.

However, the associated endpoints and actual functionality are not supported by Cuprate. The associated endpoints will return an error upon invocation.

TODO: decide on behavior and document https://github.com/Cuprate/cuprate/pull/233#discussion_r1700870051.

Last change: 2024-09-08, commit: 0162553

Custom strings

Many JSON response fields contain strings with custom messages.

This may be error messages, status, etc.

Although the field + string type will be followed, Cuprate will not always have the exact same message, particularly when it comes to error messages.

Last change: 2024-09-08, commit: 0162553

Unsupported RPC calls

TODO: compile unsupported RPC calls after handlers are created.

Last change: 2024-09-08, commit: 0162553

RPC calls with different behavior

TODO: compile RPC calls with different behavior after handlers are created.

Last change: 2024-09-08, commit: 0162553

⚪️ ZMQ

Last change: 2024-09-08, commit: 0162553

TODO

Last change: 2024-09-08, commit: 0162553

⚪️ Consensus

Last change: 2024-09-08, commit: 0162553

⚪️ Verifier

Last change: 2024-09-08, commit: 0162553

⚪️ TODO

Last change: 2024-09-08, commit: 0162553

⚪️ Networking

Last change: 2024-09-08, commit: 0162553

⚪️ P2P

Last change: 2024-09-08, commit: 0162553

⚪️ Dandelion++

Last change: 2024-09-08, commit: 0162553

⚪️ Proxy

Last change: 2024-09-08, commit: 0162553

⚪️ Tor

Last change: 2024-09-08, commit: 0162553

⚪️ i2p

Last change: 2024-09-08, commit: 0162553

⚪️ IPv4/IPv6

Last change: 2024-09-08, commit: 0162553

Instrumentation

Cuprate is built with instrumentation in mind.

Last change: 2024-09-08, commit: 0162553

⚪️ Logging

Last change: 2024-09-08, commit: 0162553

⚪️ Data collection

Last change: 2024-09-08, commit: 0162553

⚪️ Binary

Last change: 2024-09-08, commit: 0162553

⚪️ CLI

Last change: 2024-09-08, commit: 0162553

⚪️ Config

Last change: 2024-09-08, commit: 0162553

⚪️ Logging

Last change: 2024-09-08, commit: 0162553

Resources

Last change: 2024-09-08, commit: 0162553

⚪️ File system

Last change: 2024-09-08, commit: 0162553

Index of PATHs

This is an index of all of the filesystem PATHs Cuprate actively uses.

The cuprate_helper::fs module defines the general locations used throughout Cuprate.

dirs is used internally, which follows the PATH standards/conventions on each OS Cuprate supports, i.e.:

Cache

Cuprate's cache directory.

OSPATH
WindowsC:\Users\Alice\AppData\Local\Cuprate\
macOS/Users/Alice/Library/Caches/Cuprate/
Linux/home/alice/.cache/cuprate/

Config

Cuprate's config directory.

OSPATH
WindowsC:\Users\Alice\AppData\Roaming\Cuprate\
macOS/Users/Alice/Library/Application Support/Cuprate/
Linux/home/alice/.config/cuprate/

Data

Cuprate's data directory.

OSPATH
WindowsC:\Users\Alice\AppData\Roaming\Cuprate\
macOS/Users/Alice/Library/Application Support/Cuprate/
Linux/home/alice/.local/share/cuprate/

Blockchain

Cuprate's blockchain directory.

OSPATH
WindowsC:\Users\Alice\AppData\Roaming\Cuprate\blockchain\
macOS/Users/Alice/Library/Application Support/Cuprate/blockchain/
Linux/home/alice/.local/share/cuprate/blockchain/

Transaction pool

Cuprate's transaction pool directory.

OSPATH
WindowsC:\Users\Alice\AppData\Roaming\Cuprate\txpool\
macOS/Users/Alice/Library/Application Support/Cuprate/txpool/
Linux/home/alice/.local/share/cuprate/txpool/

Database

Cuprate's database location/filenames depend on:

  • Which database it is
  • Which backend is being used

cuprate_blockchain files are in the above mentioned blockchain folder.

cuprate_txpool files are in the above mentioned txpool folder.


If the heed backend is being used, these files will be created:

FilenamePurpose
data.mdbMain data file
lock.mdbDatabase lock file

For example: /home/alice/.local/share/cuprate/blockchain/lock.mdb.

If the redb backend is being used, these files will be created:

FilenamePurpose
data.redbMain data file

For example: /home/alice/.local/share/cuprate/txpool/data.redb.

Last change: 2024-09-08, commit: 0162553

Sockets

Last change: 2024-09-08, commit: 0162553

Index of ports

This is an index of all of the network sockets Cuprate actively uses.

Last change: 2024-09-08, commit: 0162553

⚪️ Memory

Last change: 2024-09-08, commit: 0162553

Concurrency and parallelism

It is incumbent upon software like Cuprate to take advantage of today's highly parallel hardware as much as practically possible.

With that said, programs must setup guardrails when operating in a concurrent and parallel manner, for correctness and safety.

There are "synchronization primitives" that help with this, common ones being:

These tools are relatively easy to use in isolation, but trickier to do so when considering the entire system. It is not uncommon for the bottleneck to be the poor orchastration of these primitives.

Analogy

A common analogy for a parallel system is an intersection.

Like a parallel computer system, an intersection contains:

  1. Parallelism: multiple individual units that want to move around (cars, pedestrians, etc)
  2. Synchronization primitives: traffic lights, car lights, walk signals

In theory, the amount of "work" the units can do is only limited by the speed of the units themselves, but in practice, the slow cascading reaction speeds between all units, the frequent hiccups that can occur, and the synchronization primitives themselves become bottlenecks far before the maximum speed of any unit is reached.

A car that hogs the middle of the intersection on the wrong light is akin to a system thread holding onto a lock longer than it should be - it degrades total system output.

Unlike humans however, computer systems at least have the potential to move at lightning speeds, but only if the above synchronization primitives are used correctly.

Goal

To aid the long-term maintenance of highly concurrent and parallel code, this section documents:

  1. All system threads spawned and maintained
  2. All major sections where synchronization primitives are used
  3. The asynchronous behavior of some components

and how these compose together efficiently in Cuprate.

Last change: 2024-09-08, commit: 0162553

⚪️ Map

Last change: 2024-09-08, commit: 0162553

⚪️ The RPC server

Last change: 2024-09-08, commit: 0162553

⚪️ The database

Last change: 2024-09-08, commit: 0162553

⚪️ The block downloader

Last change: 2024-09-08, commit: 0162553

⚪️ The verifier

Last change: 2024-09-08, commit: 0162553

⚪️ Thread exit

Last change: 2024-09-08, commit: 0162553

Index of threads

This is an index of all of the system threads Cuprate actively uses.

Last change: 2024-09-08, commit: 0162553

⚪️ External Monero libraries

Last change: 2024-09-08, commit: 0162553

⚪️ Cryptonight

Last change: 2024-09-08, commit: 0162553

RandomX

https://github.com/tari-project/randomx-rs

Last change: 2024-09-08, commit: 0162553

monero_serai

https://github.com/serai-dex/serai/tree/develop/coins/monero

Last change: 2024-09-08, commit: 0162553

⚪️ Benchmarking

Last change: 2024-09-08, commit: 0162553

⚪️ Criterion

Last change: 2024-09-08, commit: 0162553

⚪️ Harness

Last change: 2024-09-08, commit: 0162553

⚪️ Testing

Last change: 2024-09-08, commit: 0162553

⚪️ Monero data

Last change: 2024-09-08, commit: 0162553

⚪️ RPC client

Last change: 2024-09-08, commit: 0162553

⚪️ Spawning monerod

Last change: 2024-09-08, commit: 0162553

⚪️ Known issues and tradeoffs

Last change: 2024-09-08, commit: 0162553

⚪️ Networking

Last change: 2024-09-08, commit: 0162553

⚪️ RPC

Last change: 2024-09-08, commit: 0162553

⚪️ Storage

Last change: 2024-09-08, commit: 0162553

Appendix

Last change: 2024-09-08, commit: 0162553

Crates

This is an index of all of Cuprate's in-house crates it uses and maintains.

They are categorized into groups.

Crate documentation for each crate can be found by clicking the crate name or by visiting https://doc.cuprate.org. Documentation can also be built manually by running this at the root of the cuprate repository:

cargo doc --package $CRATE

For example, this will generate and open cuprate-blockchain documentation:

cargo doc --open --package cuprate-blockchain

Consensus

Networking

CrateIn-tree pathPurpose
cuprate-epee-encodingnet/epee-encoding/Epee (de)serialization
cuprate-fixed-bytesnet/fixed-bytes/Fixed byte containers backed by byte::Byte
cuprate-levinnet/levin/Levin bucket protocol implementation
cuprate-wirenet/wire/TODO

P2P

CrateIn-tree pathPurpose
cuprate-address-bookp2p/address-book/TODO
cuprate-async-bufferp2p/async-buffer/A bounded SPSC, FIFO, asynchronous buffer that supports arbitrary weights for values
cuprate-dandelion-towerp2p/dandelion-tower/TODO
cuprate-p2pp2p/p2p/TODO
cuprate-p2p-corep2p/p2p-core/TODO

Storage

CrateIn-tree pathPurpose
cuprate-blockchainstorage/blockchain/Blockchain database built on-top of cuprate-database & cuprate-database-service
cuprate-databasestorage/database/Pure database abstraction
cuprate-database-servicestorage/database-service/tower::Service + thread-pool abstraction built on-top of cuprate-database
cuprate-txpoolstorage/txpool/Transaction pool database built on-top of cuprate-database & cuprate-database-service

RPC

CrateIn-tree pathPurpose
cuprate-json-rpcrpc/json-rpc/JSON-RPC 2.0 implementation
cuprate-rpc-typesrpc/types/Monero RPC types and traits
cuprate-rpc-interfacerpc/interface/RPC interface & routing
cuprate-rpc-handlerrpc/handler/RPC inner handlers

1-off crates

CrateIn-tree pathPurpose
cuprate-cryptonightcryptonight/CryptoNight hash functions
cuprate-pruningpruning/Monero pruning logic/types
cuprate-helperhelper/Kitchen-sink helper crate for Cuprate
cuprate-test-utilstest-utils/Testing utilities for Cuprate
cuprate-typestypes/Shared types across Cuprate
Last change: 2024-09-08, commit: 0162553

Contributing

https://github.com/Cuprate/cuprate/blob/main/CONTRIBUTING.md

Last change: 2024-09-08, commit: 0162553

Build targets

  • x86
  • ARM64
  • Windows
  • Linux
  • macOS
  • FreeBSD(?)
Last change: 2024-09-08, commit: 0162553

Protocol book

https://monero-book.cuprate.org

Last change: 2024-09-08, commit: 0162553

⚪️ User book

Last change: 2024-09-08, commit: 0162553