Understanding the mining process

Note

Proof-of-Work (PoW) mining is no longer used for achieving consensus on Ethereum. Newer virtual machines, beginning with the ParisVM, assume a Proof-of-Stake (PoS) consensus mechanism which lies beyond the scope of the execution layer. This guide is for educational purposes only.

From the Cookbook we can already learn how to use the Chain class to create a single blockchain as a combination of different virtual machines for different spans of blocks.

In this guide we want to build up on that knowledge and look into the actual mining process that was once important for achieving consensus on mainnet Ethereum.

Note

Mining is an overloaded term and in fact the names of the mentioned APIs are subject to change.

Mining

The term mining can refer to different things depending on our point of view. Most of the time when we read about mining, we talk about the process where several parties are competing to be the first to create a new valid block and pass it on to the network.

In this guide, when we talk about the mine_block() API, we are only referring to the part that creates, validates and sets a block as the new canonical head of the chain but not necessarily as part of the mentioned competition to be the first. In fact, the mine_block() API is internally also called when we import existing blocks that others created.

Mining an empty block

Usually when we think about creating blocks we naturally think about adding transactions to the block first because, after all, one primary use case for the Ethereum blockchain is to process transactions which are wrapped in blocks.

For the sake of simplicity though, we’ll mine an empty block as a first example (meaning the block will not contain any transactions)

As a refresher, here is how we create a chain as demonstrated in the Using the chain object recipe from the cookbook.

from eth.db.atomic import AtomicDB
from eth.chains.mainnet import MAINNET_GENESIS_HEADER

# increase the gas limit
genesis_header = MAINNET_GENESIS_HEADER.copy(gas_limit=3141592)

# initialize a fresh chain
chain = chain_class.from_genesis_header(AtomicDB(), genesis_header)

Since we decided to not add any transactions to our block let’s just call mine_block() and see what happens.

# initialize a fresh chain
chain = chain_class.from_genesis_header(AtomicDB(), genesis_header)

chain.mine_block()

Aw, snap! We’re running into an exception at check_pow(). Apparently we are trying to add a block to the chain that doesn’t qualify the Proof-of-Work (PoW) rules. The error tells us precisely that the mix_hash of our block does not match the expected value.

Traceback (most recent call last):
  File "scripts/benchmark/run.py", line 111, in <module>
    run()
  File "scripts/benchmark/run.py", line 52, in run
    block = chain.mine_block()  #**pow_args
  File "/py-evm/eth/chains/base.py", line 545, in mine_block
    self.validate_block(mined_block)
  File "/py-evm/eth/chains/base.py", line 585, in validate_block
    self.validate_seal(block.header)
  File "/py-evm/eth/chains/base.py", line 622, in validate_seal
    header.mix_hash, header.nonce, header.difficulty)
  File "/py-evm/eth/consensus/pow.py", line 70, in check_pow
    encode_hex(mining_output[b'mix digest']), encode_hex(mix_hash)))

eth.exceptions.ValidationError: mix hash mismatch;
0x7a76bbf0c8d0e683fafa2d7cab27f601e19f35e7ecad7e1abb064b6f8f08fe21 !=
0x0000000000000000000000000000000000000000000000000000000000000000

Let’s lookup how check_pow() is implemented.

def check_pow(
    block_number: int,
    mining_hash: Hash32,
    mix_hash: Hash32,
    nonce: bytes,
    difficulty: int,
) -> None:
    validate_length(mix_hash, 32, title="Mix Hash")
    validate_length(mining_hash, 32, title="Mining Hash")
    validate_length(nonce, 8, title="POW Nonce")
    cache = get_cache(block_number)
    mining_output = hashimoto_light(
        get_dataset_full_size(block_number),
        cache,
        mining_hash,
        nonce,
    )
    if mining_output["mix_digest"] != mix_hash:
        raise ValidationError(
            f"mix hash mismatch; expected: {encode_hex(mining_output['mix_digest'])} "
            f"!= actual: {encode_hex(mix_hash)}.\n    "
            f"Mix hash calculated from block #{block_number},\n    "
            f"mine hash: {encode_hex(mining_hash)},\n    "
            f"nonce: {encode_hex(nonce)},\n    "
            f"difficulty: {difficulty}"
        )
    result = big_endian_to_int(mining_output["result"])
    validate_lte(result, 2**256 // difficulty, title="POW Difficulty")

Just by looking at the signature of that function we can see that validating the PoW is based on the following parameters:

  • block_number - the number of the given block

  • difficulty - the difficulty of the PoW algorithm

  • mining_hash - hash of the mining header

  • mix_hash - together with the nonce forms the actual proof

  • nonce - together with the mix_hash forms the actual proof

The PoW algorithm checks that all these parameters match correctly, ensuring that only valid blocks can be added to the chain.

In order to produce a valid block, we have to set the correct mix_hash and nonce in the header. We can pass these as key-value pairs when we call mine_block() as seen below.

chain.mine_block(nonce=valid_nonce, mix_hash=valid_mix_hash)

This call will work just fine assuming we are passing the correct nonce and mix_hash that corresponds to the block getting mined.

Retrieving a valid nonce and mix hash

Now that we know we can call mine_block() with the correct parameters to successfully add a block to our chain, let’s briefly go over an example that demonstrates how we can retrieve a matching nonce and mix_hash.

Note

Py-EVM currently doesn’t offer a stable API for actual PoW mining. The following code is for demonstration purpose only.

Mining on the main ethereum chain is a competition done simultaneously by many miners, hence the mining difficulty is pretty high which means it will take a very long time to find the right nonce and mix_hash on commodity hardware. In order for us to have something that we can tinker with on a regular laptop, we’ll construct a test chain with the difficulty set to 1.

Let’s start off by defining the GENESIS_PARAMS.

from eth import constants

GENESIS_PARAMS = {
      'difficulty': 1,
      'gas_limit': 3141592,
      'timestamp': 1514764800,
  }

Next, we’ll create the chain itself using the defined GENESIS_PARAMS and the latest ByzantiumVM.

from eth import MiningChain
from eth.vm.forks.byzantium import ByzantiumVM
from eth.db.backends.memory import AtomicDB


klass = MiningChain.configure(
    __name__='TestChain',
    vm_configuration=(
        (constants.GENESIS_BLOCK_NUMBER, ByzantiumVM),
    ))
chain = klass.from_genesis(AtomicDB(), GENESIS_PARAMS)

Now that we have the building blocks available, let’s put it all together and mine a proper block!

from eth.consensus.pow import mine_pow_nonce


# We have to finalize the block first in order to be able read the
# attributes that are important for the PoW algorithm
block_result = chain.get_vm().finalize_block(chain.get_block())
block = block_result.block

# based on mining_hash, block number and difficulty we can perform
# the actual Proof of Work (PoW) mechanism to mine the correct
# nonce and mix_hash for this block
nonce, mix_hash = mine_pow_nonce(
    block.number,
    block.header.mining_hash,
    block.header.difficulty)

block = chain.mine_block(mix_hash=mix_hash, nonce=nonce)

>>> print(block)
Block #1

Let’s take a moment to fully understand what this code does.

1. We call finalize_block() on the underlying VM in order to retrieve the information that we need to calculate the nonce and the mix_hash.

2. We then call mine_pow_nonce() to retrieve the proper nonce and mix_hash that we need to mine the block and satisfy the validation.

  1. Finally we call mine_block() and pass along the nonce and the mix_hash

Note

The code above will essentially perform finalize_block twice. Keep in mind this code is for demonstration purpose only and that Py-EVM will provide a pluggable system in the future to allow PoW mining among other things.

Mining a block with transactions

Now that we’ve learned the basics of how the mining process works, let’s revisited our example and add a transaction before we mine another block. There are a couple of concepts we need to dive into in order to accomplish that goal.

Every transaction goes from a sender Address to a receiver Address. Each transaction takes some computational power to get processed that is measured in a unit called gas.

In practice, we have to pay the miners to put our transaction in a block. However, there is no technical reason why we have to pay for the computing power, but only an economical, i.e. in reality we’ll usually have trouble finding a miner who’s willing to include a transaction that doesn’t pay for its computational costs.

In this example, however, we are the miner which means we are free to include any transactions we like. In the spirit of this guide, let’s start simple and create a transaction that sends zero ether from one address to another address. Keep in mind that even if the value being transferred is zero, there’s still a computational cost for the processing but since we are the miner, we’ll mine it anyway even if no one is willing to pay for it!

Let’s first setup the sender and receiver.

from eth_keys import keys
from eth_utils import decode_hex
from eth_typing import Address

SENDER_PRIVATE_KEY = keys.PrivateKey(
  decode_hex('0x45a915e4d060149eb4365960e6a7a45f334393093061116b197e3240065ff2d8')
)

SENDER = Address(SENDER_PRIVATE_KEY.public_key.to_canonical_address())

RECEIVER = Address(b'\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\x02')

One thing that strikes out here is that we only need the plain address for the receiver whereas for the sender we are obtaining an address derived from the SENDER_PRIVATE_KEY. That’s because we obviously can not send transactions from an address that we don’t have the private key to sign it for.

With sender and receiver prepared, let’s create the actual transaction.

vm = chain.get_vm()
nonce = vm.state.get_nonce(SENDER)

tx = vm.create_unsigned_transaction(
    nonce=nonce,
    gas_price=0,
    gas=100000,
    to=RECEIVER,
    value=0,
    data=b'',
)

Every transaction needs a nonce not to be confused with the nonce that we previously mined as part of the PoW algorithm. The transaction nonce serves as a counter to ensure all transactions from one address are processed in order. We retrieve the current nonce by calling get_nonce(sender)().

Once we have the nonce we can call create_unsigned_transaction() and pass the nonce among the rest of the transaction attributes as key-value pairs.

  • nonce - Number of transactions sent by the sender

  • gas_price - Number of Wei to pay per unit of gas

  • gas - Maximum amount of gas the transaction is allowed to consume before it gets rejected

  • to - Address of transaction recipient

  • value - Number of Wei to be transferred to the recipient

The last step we need to do before we can add the transaction to a block is to sign it with the private key which is as simple as calling as_signed_transaction() with the SENDER_PRIVATE_KEY.

signed_tx = tx.as_signed_transaction(SENDER_PRIVATE_KEY)

Finally, we can call apply_transaction() and pass along the signed_tx.

chain.apply_transaction(signed_tx)

What follows is the complete script that demonstrates how to mine a single block with one simple zero value transfer transaction.

>>> from eth_keys import keys
>>> from eth_utils import decode_hex
>>> from eth_typing import Address
>>> from eth import constants
>>> from eth.chains.base import MiningChain
>>> from eth.consensus.pow import mine_pow_nonce
>>> from eth.vm.forks.byzantium import ByzantiumVM
>>> from eth.db.atomic import AtomicDB


>>> GENESIS_PARAMS = {
...     'difficulty': 1,
...     'gas_limit': 3141592,
...     # We set the timestamp, just to make this documented example reproducible.
...     # In common usage, we remove the field to let py-evm choose a reasonable default.
...     'timestamp': 1514764800,
... }

>>> SENDER_PRIVATE_KEY = keys.PrivateKey(
...     decode_hex('0x45a915e4d060149eb4365960e6a7a45f334393093061116b197e3240065ff2d8')
... )

>>> SENDER = Address(SENDER_PRIVATE_KEY.public_key.to_canonical_address())

>>> RECEIVER = Address(b'\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\x02')

>>> klass = MiningChain.configure(
...     __name__='TestChain',
...     vm_configuration=(
...         (constants.GENESIS_BLOCK_NUMBER, ByzantiumVM),
...     ))

>>> chain = klass.from_genesis(AtomicDB(), GENESIS_PARAMS)
>>> genesis = chain.get_canonical_block_header_by_number(0)
>>> vm = chain.get_vm()

>>> nonce = vm.state.get_nonce(SENDER)

>>> tx = vm.create_unsigned_transaction(
...     nonce=nonce,
...     gas_price=0,
...     gas=100000,
...     to=RECEIVER,
...     value=0,
...     data=b'',
... )

>>> signed_tx = tx.as_signed_transaction(SENDER_PRIVATE_KEY)

>>> chain.apply_transaction(signed_tx)
(<ByzantiumBlock(#Block #1...)

>>> # Normally, we can let the timestamp be chosen automatically, but
>>> # for the sake of reproducing exactly the same block every time,
>>> # we will set it manually here:
>>> chain.set_header_timestamp(genesis.timestamp + 1)

>>> # We have to finalize the block first in order to be able read the
>>> # attributes that are important for the PoW algorithm
>>> block_result = chain.get_vm().finalize_block(chain.get_block())
>>> block = block_result.block

>>> # based on mining_hash, block number and difficulty we can perform
>>> # the actual Proof of Work (PoW) mechanism to mine the correct
>>> # nonce and mix_hash for this block
>>> nonce, mix_hash = mine_pow_nonce(
...     block.number,
...     block.header.mining_hash,
...     block.header.difficulty,
... )  
... # (takes too long for doctest to process)

>>> chain.mine_block(mix_hash=mix_hash, nonce=nonce)  
<ByzantiumBlock(#Block #1-0xe372..385c)>