Benchmark individual proofs

In this section we will cover how to run the benchmarks for the individual proofs. The benchmarks are located in the light-client crate folder. Those benchmarks are associated with programs that are meant to reproduce a production environment settings. They are meant to measure performance for a complete end-to-end flow.

The numbers we've measured using our production configuration are further detailed in the following section.

Sync committee change

Benchmark that will run a proof generation for the sync committee change program. This program will execute a hash for the received LightClientStore::current_sync_committee to ensure that the signature is from the previous sync committee set, execute a LightClientStore::process_light_client_update and finally generate the hash for the new LightClientStore::current_sync_committee.

On our production configuration, we currently get the following results for SNARK generation for this benchmark:

For STARKS:

{
  // Time in milliseconds, 2 minutes 02s ~
  "proving_time": 122772,
  "verification_time": 4376
}

For SNARKS:

{
  // Time in milliseconds, 10 minute 54s ~
  "proving_time": 654767,
  "verification_time": 1
}

Storage inclusion

Benchmark that will run a proof generation for the storage inclusion program. This program will execute a hash for the received CompactStore::sync_committee to ensure that the signature is from the current known sync committee set, execute a CompactStore::validate_compact_update to confirm that the received block information is one signed by the committee, and finally run an EIP1186Proof::verify against the state root of the finalized execution block header.

On our production configuration, we currently get the following results for SNARK generation for this benchmark:

For STARKS:

{
  // Time in milliseconds, 1 minutes 02s ~
  "proving_time": 62165,
  "verification_time": 2886
}

For SNARKS:

{
  // Time in milliseconds, 7 minute 51s ~
  "proving_time": 531021,
  "verification_time": 3
}

Running the benchmarks

Using Makefile

To ease benchmark run we created a Makefile in the light-client crate folder. Just run:

make benchmark

You will then be asked for the name of the benchmark you want to run. Just fill in the one that is of interest to you:

$ make benchmark
Enter benchmark name: committee_change

  ...

Manual

Run the following command:

SHARD_BATCH_SIZE=0 cargo bench --bench execute -- <benchmark_name>

Interpreting the results

Before delving into the details, please take a look at the cycle tracking documentation from SP1 to get a rough sense of what the numbers mean.

The benchmark will output a lot of information. The most important parts are the following:

Total cycles for the program execution

This value can be found on the following line:

INFO summary: cycles=63736, e2e=2506, khz=25.43, proofSize=2.66 MiB

It contains the total number of cycles needed for the program, the end-to-end time in milliseconds, the frequency of the CPU in kHz, and the size of the proof generated.

Specific cycle count

In the output, you will find a section that looks like this:

DEBUG ┌╴read_inputs    
DEBUG └╴9,553 cycles    
DEBUG ┌╴verify_merkle_proof    
DEBUG └╴40,398 cycles

These specific cycles count are generated by us to track the cost of specific operations in the program.

Proving time

The proving time is an output at the end of a benchmark in the shape of the following data structure, with each time in milliseconds:

{
  ratchet_proving_time: 100000,
  merkle_proving_time: 100000
}

Alternative

Another solution to get some information about proving time is to run the tests located in the light-client crate. They will output the same logs as the benchmarks, only the time necessary to generate a proof will change shape:

Starting generation of Merkle inclusion proof with 18 siblings...
Proving locally
Proving took 5.358508094s
Starting verification of Merkle inclusion proof...
Verification took 805.530068ms

To run the test efficiently, first install nextest following its documentation. Ensure that you also have the previously described environment variables set, then run the following command:

SHARD_BATCH_SIZE=0 cargo nextest run --verbose --release --profile ci --package ethereum-lc --no-capture --all-features

Note

The --no-capture flag is necessary to see the logs generated by the tests.

Some tests are ignored by default due to heavier resource requirements. To run them, pass --run-ignored all to nextest.

A short list of useful tests: