Nxt Blockchain Tutorial

From Nxt Wiki
Jump to: navigation, search

1 Introduction

1.1 Objective

Use a shell script to scan the Nxt blockchain, with an emphasis on understanding blockchain technology rather than speed. Based on the scan() method found in the Nxt software version 1.3.5 source file nxt/src/java/nxt/BlockchainProcessorImpl.Java

1.2 Assumptions

Nxt Java software running on your system, and familiarity with bash shell scripts running in a console window on a linux system. You have the jshon, bc and sha256sum utilities installed.

2 Table of Contents

3 H2 Database Interfaces

The Nxt blockchain is stored in a Java H2 database (DB), located in your Nxt software installation directory (assumed to be nxt) on the path nxt/nxt_db. There are three tools available for interacting with the DB, two of which (DB Shell and H2 Console) use a browser interface and one of which (DB Shell) can operate concurrently with the Nxt Software without changing the nxt/conf/nxt.properties file.

3.1 Shell

To interact with the DB from a linux command line, use the Java Shell tool available in nxt/lib. For example

$ cd nxt; java -cp lib/h2*.jar org.h2.tools.Shell \
  -url 'jdbc:h2:./nxt_db/nxt;DB_CLOSE_ON_EXIT=FALSE;CACHE_SIZE=439808;AUTO_SERVER=TRUE' \
  -driver 'org.h2.Driver' -user sa -password sa \
  -sql 'select base_target from block where height=0;'

returns

BASE_TARGET
153722867
(1 row, 19 ms)

The Shell tool will only connect with the DB if the Nxt software is not running, because it locks the DB. To avoid this problem add

nxt.dbUrl=jdbc:h2:nxt_db/nxt;DB_CLOSE_ON_EXIT=FALSE;AUTO_SERVER=TRUE

to the nxt/conf/nxt.properties file and restart the Nxt software. The AUTO_SERVER clause enables DB access from multiple processes simultaneously.

The Shell tool has an interactive mode accessible by invoking it without the -sql option. The command line mode is slow because a DB connection is established every time.

3.2 DB Shell

Enter http://localhost:7876/dbshell into a browser. A page loads with a prompt for an SQL statement. dbshell is built into the Nxt software and so must be used while the latter is running. The SQL query

SELECT * FROM INFORMATION_SCHEMA.COLUMNS WHERE TABLE_NAME='BLOCK';

gives a list of the columns (fields) of the block table, along with information such as data type. This tool is adequate, but the H2 Console is preferred because of its graphical interface.

3.3 H2 Console

In your Nxt software installation directory there is a library subdirectory (lib) with a jar file for the H2 DB. If launched as follows

$ cd nxt; java -cp "lib/h2*:lib/*" org.h2.tools.Console

a page opens in your browser with a DB connection dialog. Specify:

 Saved Settings: Generic H2 (Embedded)
 Setting Name: Generic H2 (Embedded)
 Driver Class: org.h2.Driver
 JDBC URL: jdbc:h2:nxt_db/nxt;DB_CLOSE_ON_EXIT=FALSE;AUTO_SERVER=TRUE;CACHE_SIZE=439808
 User Name: sa
 Password: sa

Clicking on Connect will connect with the DB but only if the Nxt software is not running, because it locks the DB. To avoid this problem add

nxt.dbUrl=jdbc:h2:nxt_db/nxt;DB_CLOSE_ON_EXIT=FALSE;AUTO_SERVER=TRUE

to the nxt/conf/nxt.properties file and restart the Nxt software. The AUTO_SERVER clause enables DB access from multiple processes simultaneously.

Use ^C in the shell console to terminate the DB engine, or click on the tool tray icon, then exit.

4 Database Fields

4.1 The BLOCK Table

In the left hand column of the H2 Console all tables in the DB are listed. Expanding the BLOCK table by clicking on "+" shows the block table columns (fields). Clicking the "+" next to each column name shows the column data type:

Field Name Data Type NOT NULL
DB_ID BIGINT(19) yes
ID BIGINT(19) yes
VERSION INTEGER(10) yes
TIMESTAMP INTEGER(10) yes
PREVIOUS_BLOCK_ID BIGINT(19) no
TOTAL_AMOUNT BIGINT(19) yes
TOTAL_FEE BIGINT(19) yes
PAYLOAD_LENGTH INTEGER(10) yes
GENERATOR_PUBLIC_KEY VARBINARY(32) yes
PREVIOUS_BLOCK_HASH VARBINARY(32) no
CUMULATIVE_DIFFICULTY VARBINARY(2147483647) yes
BASE_TARGET BIGINT(19) yes
NEXT_BLOCK_ID BIGINT(19) no
HEIGHT INTEGER(10) yes
GENERATION_SIGNATURE VARBINARY(64) yes
BLOCK_SIGNATURE VARBINARY(64) yes
PAYLOAD_HASH VARBINARY(32) yes
GENERATOR_ID BIGINT(19) yes

These 18 fields define a block on the blockchain in the current version of the Nxt software. Note that the block table has evolved since the genesis block. A complete history of the DB structure can be found in the Nxt software source file nxt/src/java/nxt/NxtDbVersion.java

Note that most fields cannot be NULL. The exceptions are the PREVIOUS_... and NEXT_... fields which link the blocks into a chain both forward and backward. The genesis block has a NULL PREVIOUS_BLOCK_ID and the last (current) block has a NULL NEXT_BLOCK_ID.

Below the column list is a list of indexes. The indexes are all used for sorting various columns for fast retrieval, but the following columns are also restricted to have unique values: DB_ID, HEIGHT, ID, TIMESTAMP. They are all used to uniquely identify blocks. DB_ID is the autoincrement field of the table. It normally increases by one with each new block but gaps can occur in the sequence due to occasional deleted blocks. HEIGHT is zero for the genesis block and increases by one with each block in the blockchain. There are no gaps in this sequence. ID is a unique block ID derived from the hash of some of the block fields. TIMESTAMP is the block creation time measured in number of seconds elapsed since the genesis block.

Note that blocks stored in the BLOCK table are associated with transactions stored in the TRANSACTION table through the fields PAYLOAD_LENGTH and PAYLOAD_HASH, and TOTAL_AMOUNT and TOTAL_FEE. PAYLOAD_LENGTH is the total number of bytes of certain fields of all transactions associated with the block and PAYLOAD_HASH is the hash of all those fields. TOTAL_AMOUNT and TOTAL_FEE are the total amounts and fees of all transactions associated with the block. All four of these block fields are zero when there are no transactions associated with the block.

4.2 The TRANSACTION Table

In the left hand column of the H2 Console all tables in the DB are listed. Expanding the TRANSACTION table by clicking on "+" shows the transaction table columns (fields). Clicking the "+" next to each column name shows the column data type:

Field Name Data Type NOT NULL
DB_ID BIGINT(19) yes
ID BIGINT(19) yes
DEADLINE SMALLINT(5) yes
SENDER_PUBLIC_KEY VARBINARY(32) yes
RECIPIENT_ID BIGINT(19) no
AMOUNT BIGINT(19) yes
FEE BIGINT(19) yes
HEIGHT INTEGER(10) yes
BLOCK_ID BIGINT(19) yes
SIGNATURE VARBINARY(64) yes
TIMESTAMP INTEGER(10) yes
TYPE TINYINT(3) yes
SUBTYPE TINYINT(3) yes
SENDER_ID BIGINT(19) yes
BLOCK_TIMESTAMP INTEGER(10) yes
FULL_HASH VARBINARY(32) yes
REFERENCED_TRANSACTION_FULL_HASH VARBINARY(32) no
ATTACHMENT_BYTES VARBINARY(2147483647) no
VERSION TINYINT(3) yes
HAS_MESSAGE BOOLEAN(1) yes
HAS_ENCRYPTED_MESSAGE BOOLEAN(1) yes
HAS_PUBLIC_KEY_ANNOUNCEMENT BOOLEAN(1) yes
EC_BLOCK_HEIGHT INTEGER(10) no
EC_BLOCK_ID BIGINT(19) no
HAS_ENCRYPTTOSELF_MESSAGE BOOLEAN(1) yes

These 25 fields define a transaction the current version of the Nxt software. Note that the transaction table has evolved since the genesis block. A complete history of the DB structure can be found in the Nxt software source file nxt/src/java/nxt/NxtDbVersion.java

Note that most fields cannot be NULL. The exceptions are RECIPIENT_ID, REFERENCED_TRANSACTION_FULL_HASH, ATTACHMENT_BYTES and the EC_BLOCK... fields. A transaction is valid without any of these fields specified.

Below the column list is a list of indexes. The indexes are all used for sorting various columns for fast retrieval, but the following columns are also restricted to have unique values: DB_ID, ID, FULL_HASH.

Note that transactions stored in the TRANSACTION table are associated with blocks stored in the BLOCK table through the fields HEIGHT, BLOCK_ID and BLOCK_TIMESTAMP.

5 The Nxt API

We could interact directly with the DB as shown above, but for the purpose of this tutorial, our script will obtain block and transaction data using Nxt software API calls.

5.1 The getBlock Call

The getBlock API call can be made from the shell using curl while the Nxt software is running:

$ h=300000; curl -s "http://localhost:7876/nxt?requestType=getBlock&height=$h"

This returns a JSON object for block height 300000. There are no extra spaces or newlines in the object, and the object does not include fields with null values. For clarity the object can be formatted as:

{
 "previousBlockHash": "f5006dc3ae97f52aa27dc7c51888db5c6b0553aeeb51ad4bc74350edbef6e84e",
 "payloadLength": 370,
 "totalAmountNQT": "0",
 "generationSignature": "5d08857979a8a900da73d37dc7b8c9d15acbf2644755523dcc3dccae722185ae",
 "generator": "15893842156456397710",
 "generatorPublicKey": "85d34459911c4c5eff7da30966334388d900fefe1082fe7b4d2cbe18706df422",
 "baseTarget": "769068704",
 "payloadHash": "0fcff1f8c4b384ee8e8940328827dd99d5d2aa88f401328195e55a5d5afba226",
 "generatorRS": "NXT-ELWG-9FPV-KTP6-F7L6T",
 "nextBlock": "3635605354783461399",
 "requestProcessingTime": 1,
 "numberOfTransactions": 2,
 "blockSignature": "11a916927177d737ce8869ad892403b61c7b8008d63690ce9ff92250c982630a98d8d9f7b52717af585a97b7475d26d14cd108e6f0704cc5a6cbaef52d21b129",
 "transactions": [
  "5119943785891977024",
  "5340614876563792980"
 ],
 "version": 3,
 "totalFeeNQT": "200000000",
 "previousBlock": "3095547095745888501",
 "cumulativeDifficulty": "11255618847538250",
 "block": "16287921892019728450",
 "height": 300000,
 "timestamp": 32363409
}

5.2 The getTransaction Call

The getTransaction API call can be made from the shell using curl while the Nxt software is running:

$ tx=5119943785891977024; curl -s "http://localhost:7876/nxt?requestType=getTransaction&transaction=$tx"

This returns a JSON object for one of the two transactions at block height 300000. There are no extra spaces or newlines in the object, and the object does not include fields with null values. For clarity the object can be formatted as:

{
 "senderPublicKey": "2fc5b938a98829d3cbefa60472eac11fd2cc09e6173b370ad6cc0ede666c6475",
 "signature": "ae92f26b1ded2fd0f154e20cedf9b949ea0b1a1d637bf8e4ebaa52f6dcce7008c6373d2c85b84ea2f919fdd66a6d495005fa3e6b52c89330f7064fabf91ea870",
 "feeNQT": "100000000",
 "requestProcessingTime": 0,
 "type": 2,
 "confirmations": 8217,
 "fullHash": "40570f24bfb10d47692bdfa25f3e175768b713b3ccf4bc4cb9cd7cf2ca094bca",
 "version": 1,
 "ecBlockId": "9591128410195604705",
 "signatureHash": "4c9203be1378e5f8ca940794b180e343ec19ac9d38b7fe2f3c2ad9c792e6c305",
 "attachment": {
  "version.AskOrderCancellation": 1,
  "order": "2574190861734682725"
 },
 "senderRS": "NXT-ADH7-TCCK-R46P-AHZSR",
 "subtype": 4,
 "amountNQT": "0",
 "sender": "10080164044348861925",
 "ecBlockHeight": 299990,
 "block": "16287921892019728450",
 "blockTimestamp": 32363409,
 "deadline": 1440,
 "transaction": "5119943785891977024",
 "timestamp": 32363262,
 "height": 300000
}

5.3 The getTransactionBytes Call

The getTransactionBytes API call can be made from the shell using curl while the Nxt software is running:

$ tx=5119943785891977024; curl -s "http://localhost:7876/nxt?requestType=getTransactionBytes&transaction=$tx"

This returns the signed and unsigned byte arrays of the transaction fields above packed in a certain order according to a complex algorithm that has changed since the genesis block:

{
 "unsignedTransactionBytes": "0214fed2ed01a0052fc5b938a98829d3cbefa60472eac11fd2cc09e6173b...",
 "requestProcessingTime": 0,
 "confirmations": 12154,
 "transactionBytes": "0214fed2ed01a0052fc5b938a98829d3cbefa60472eac11fd2cc09e6173b370ad6cc..."
}

6 Comparison of Database and API fields

6.1 Block

Note that there is not an exact correspondence between the 18 BLOCK table fields and the 21 getBlock fields:

DB fields API fields
DB_ID
ID block
VERSION version
TIMESTAMP timestamp
PREVIOUS_BLOCK_ID previousBlock
TOTAL_AMOUNT totalAmountNQT
TOTAL_FEE totalFeeNQT
PAYLOAD_LENGTH payloadLength
GENERATOR_PUBLIC_KEY generatorPublicKey
PREVIOUS_BLOCK_HASH previousBlockHash
CUMULATIVE_DIFFICULTY cumulativeDifficulty
BASE_TARGET baseTarget
NEXT_BLOCK_ID nextBlock
HEIGHT height
GENERATION_SIGNATURE generationSignature
BLOCK_SIGNATURE blockSignature
PAYLOAD_HASH payloadHash
GENERATOR_ID generator
generatorRS
numberOfTransactions
transactions
requestProcessingTime

The missing DB field is DB_ID which is not needed to uniquely identify a block. The extra API fields are the Reed-Solomon version of the generator ID, from which it can be computed, and transaction number and IDs associated with the block, and finally the API request processing time.

6.2 Transaction

Note that there is not an exact correspondence between the 25 TRANSACTION table fields and the 25 getBlock fields:

DB fields API fields
DB_ID
ID transaction
DEADLINE deadline
SENDER_PUBLIC_KEY senderPublicKey
RECIPIENT_ID recipient
recipientRS
AMOUNT amountNQT
FEE feeNQT
HEIGHT height
BLOCK_ID block
SIGNATURE signature
signatureHash
TIMESTAMP timestamp
TYPE type
SUBTYPE subtype
SENDER_ID sender
senderRS
BLOCK_TIMESTAMP blockTimestamp
FULL_HASH fullHash
REFERENCED_TRANSACTION_FULL_HASH referencedTransactionFullHash
ATTACHMENT_BYTES attachment object
VERSION version
HAS_MESSAGE
HAS_ENCRYPTED_MESSAGE
HAS_PUBLIC_KEY_ANNOUNCEMENT
EC_BLOCK_HEIGHT ecBlockHeight
EC_BLOCK_ID ecBlockId
HAS_ENCRYPTTOSELF_MESSAGE
confirmations
requestProcessingTime

The missing DB fields are DB_ID, which is not needed to uniquely identify a transaction, and the HAS_... booleans which all refer to messages which might be contained in the attachment and which can be determined by examining the attachment. The API has the extra fields signatureHash, which can be obtained from signature, ...RS; the Reed-Solomon versions of IDs, which can be obtained from the IDs; confirmations, which can be computed by comparing height with the last block height on the blockchain; and requestProcessingTime, which is the time to process the API call.

7 A Shell Script to Scan the Blockchain

7.1 Basic Script

Save the following script to a file named block.sh:

#!/bin/bash

function getFields {
 local fields=$(echo $json | jshon -k)
 local field; local type; local name; local names
 for field in $fields; do
   name="$1_$field"; names="$names $name"
   type=$(echo $json | jshon -e $field -t)
   if [[ $type == "string" || $type == "number" ]]; then
    declare -g $name="$(echo $json | jshon -e $field -u)"
   else
    declare -g $name="$(echo $json | jshon -e $field)"
   fi
   #echo $name=${!name} # dump fields
 done
 declare -g "$1_fields"="$names"
}

err=; synched=0
if ((${#1})); then h=$1; else h=0; fi

while ((!synched)) || ((${#b})); do

 json=$(curl -s "http://localhost:7876/nxt?requestType=getBlock&height=$h")
 if ((${#json} == 0)); then err="server not responding"; break; fi
 getFields 'bk'

 if ((${#bk_errorDescription})); then err="$bk_errorDescription"; break; fi
 echo "height=$bk_height, timestamp=$bk_timestamp"

 # do checks here that only refer to the current block

 if ((h != bk_height)); then err="blocks not in sequential height order"; break; fi
  
 if ((synched)); then # do checks here that refer to the previous block

  if ((b != bk_block)); then err="block ID mismatch"; break; fi
  if ((pB != bk_previousBlock)); then err="previousBlock ID mismatch"; break; fi

 fi

 synched=1; ((++h))
 pB=$bk_block; b=$bk_nextBlock
 for var in $bk_fields; do unset $var; done # clear all block fields
 
done

if ((${#err})); then echo "Error: $err"; fi

Make the file executable:

$ chmod +x block.sh

Run it starting from the genesis block:

$ ./block.sh

The height and timestamp of each block beginning with the genesis block is printed on the console:

height=0, timestamp=0
height=1, timestamp=793
height=2, timestamp=1064
height=3, timestamp=1148
...

This simple version of block.sh will scan the entire blockchain in about two hours.

If the "dump fields" line is uncommented

echo $var=${!var} # dump fields

the field variable names and values will be printed for each block.

Now, some explanatory comments about the script.

7.1.1 Call the API

The linux utility curl is used to get a block from the blockchain using the API call getBlock. It returns a JSON-formatted string.

json=$(curl -s "http://localhost:7876/nxt?requestType=getBlock&height=$h")

where $h is the block height and $json is the JSON-formatted block object retrieved.

7.1.2 Loop Control and Command Line Parameter

The variable $h is used to specify block height. It is initialized to the first command line argument $1 (zero default) and incremented near the end of the main block loop, to be checked against the DB field $bk_height in the next loop iteration. To scan the blockchain beginning at height 300000, use:

$ ./block.sh 300000

The block loop terminates when the end of the blockchain is reached. The last block in the chain will have nextBlock null and therefore nextBlock will be missing from the $json string. The value of $bk_nextBlock is stored in $b near the end of the block loop so that when its length is tested at the beginning of the next loop, it will have zero length ${#b} and terminate the loop, since all field variables are cleared at the end of the loop in case one becomes null in the next iteration.

7.1.3 Parsing the JSON Block Object

The getFields function extracts block and transaction fields from the $json string into variables such as $bk_nextBlock and $tx_version. It uses the linux utility program jshon. Lists of variable names $bk_fields and $tx_fields is kept so that the variables can be unset at the end of each loop, in case there are null DB fields omitted from $json in the next iteration.

7.1.4 Handling Errors

In the event of an improper API call, the $json string will contain an errorDescription field instead of block fields. If this is detected, the block loop breaks and an error report is printed.

All errors are handled the same way, by storing an error message in $err, exiting the loop, and finally printing the message.

7.1.5 Blockchain Integrity Checks

If valid block fields are retrieved, integrity checks can be performed in the main body of the block loop. There are two sections for checks. The first is for checks that only refer to the current block, while the second is for checks that refer to the previous block. The synched variable is used to ensure that the second section is only executed after one block has been processed and first-block variables have been stored near the end of the loop for use in the next block.

For example, $bk_block (the block ID of the current block) is stored as pB=$bk_block near the end of the loop, to be compared with $bk_previousBlock (the block ID of the previous block) in the next loop iteration. This is a simple check of blockchain integrity. The field $bk_previousBlock links the current block to the previous block. Likewise, the field $bk_nextBlock (the block ID of the next block) is stored as the variable $b to be compared with $bk_block (the block ID of the current block) in the next loop iteration.

7.2 Additional Blockchain Integrity Checks

The basic shell script can be enhanced to perform additional integrity checks on the blockchain. Each new check slows the program down, sometimes significantly, especially when an external, computationally intensive utility is used. For each check below, add it to the appropriate section. If it belongs in the synched section, store the relevant variables near the end of the block loop before the DB fields are cleared for the next iteration.

7.2.1 Transaction Checks

7.2.1.1 Transaction Number Check
txNum=$(echo $bk_transactions | jshon -l)
if ((txNum != bk_numberOfTransactions)); then err="numberOfTransactions mismatch"; break; fi
7.2.1.2 Transaction Loop
txs=$(echo $bk_transactions | jshon -a -u) # get list of transaction IDs
for tx in $txs; do
 json=$(curl -s "http://localhost:7876/nxt?requestType=getTransaction&transaction=$tx")
 if ((${#json} == 0)); then err="server not responding"; break; fi
 getFields 'tx'
 if ((${#tx_errorDescription})); then err="$tx_errorDescription"; break; fi

 json=$(curl -s "http://localhost:7876/nxt?requestType=getTransactionBytes&transaction=$tx")
 if ((${#json} == 0)); then err="server not responding"; break; fi
 getFields 'txb'
 if ((${#txb_errorDescription})); then err="$txb_errorDescription"; break; fi

 # place transaction checks here

 for var in $tx_fields; do unset $var; done # clear all transaction fields
 for var in $txb_fields; do unset $var; done # clear all transaction bytes fields
done
if ((${#err})); then break; fi

The getTransaction API call is used to get the transaction fields given the transaction ID. The getTransactionBytes API call is used to get the signed and unsigned transaction byte array (in hex form). The alternative to the latter is to pack together the transaction fields as done below with block fields, except that the packing algorithm is more complex.

The following transaction checks should be placed within the transaction loop as indicated except for designated pre-loop initializations and post-loop checks.

7.2.1.3 Amount and Fee Checks

Initialize before the transaction loop:

amt=0; fee=0

Accumulate within the loop:

((amt += tx_amountNQT)); ((fee += tx_feeNQT))

Check after the loop:

if ((amt != bk_totalAmountNQT)); then err="tx amountNQT mismatch"; break; fi
if ((fee != bk_totalFeeNQT)); then err="tx feeNQT mismatch"; break; fi
7.2.1.4 Payload Length and Hash Checks

Initialize before the transaction loop:

payLen=0; payHex=

Accumulate within the loop:

((payLen += ${#txb_transactionBytes})); payHex=$payHex$txb_transactionBytes

Check after the loop:

if ((payLen / 2 != bk_payloadLength)); then err="payload length mismatch"; break; fi
if [[ $(hexToHash $payHex) != $bk_payloadHash ]]; then err="payload hash mismatch"; break; fi
7.2.1.5 Check Other Transaction/Block Linkages
if [[ $tx_block != $bk_block ]]; then err="tx/bk block ID mismatch"; break; fi
if ((tx_blockTimestamp != bk_timestamp)); then err="tx/bk timestamp mismatch"; break; fi
if ((tx_height != bk_height)); then err="tx/bk height mismatch"; break; fi
7.2.1.6 Transaction Version Check
if ((tx_height <= 213000)); then v=0; else v=1; fi
if ((v != tx_version)); then err="tx version mismatch"; break; fi
7.2.1.7 Transaction Hash and ID Checks
sH=$(hexToHash $tx_signature)
if [[ $sH != $tx_signatureHash ]]; then err="tx signature hash mismatch"; break; fi

if ((bk_version < 3)); then
 fH=$(hexToHash $txb_transactionBytes)
else
 fH=$(hexToHash $txb_unsignedTransactionBytes$tx_signatureHash)
fi
txID=$(hashToID $fH)

if [[ $fH != $tx_fullHash ]]; then err="tx full hash mismatch"; break; fi
if [[ $txID != $tx_transaction ]]; then err="tx ID mismatch"; break; fi
7.2.1.8 Transaction Signature Check
if ((bk_height > 0)); then
 v=${tx_signature:0:64}; h1=${tx_signature:64}
 Y=$(java Curve25519 $v $h1 $tx_senderPublicKey $bk_version)
 if (($?)); then err="tx signature invalid"; break; fi
 m=$(hexToHash $txb_unsignedTransactionBytes); h2=$(hexToHash $m$Y)
 if [[ $h1 != $h2 ]]; then err="tx signature not verified"; break; fi
fi

This check makes use of the Curve25519 Utility to verify the signature. The genesis block is skipped to avoid five bad signatures.

7.2.2 Block Version Check

if ((bk_height == 0)); then v=-1
elif ((bk_height <= 30000)); then v=1
elif ((bk_height <= 132000)); then v=2
else v=3; fi
if ((v != bk_version)); then err="incorrect version"; break; fi

7.2.3 previousBlock Hash / ID Check

if ((bk_version > 1)) && (($(hashToID $bk_previousBlockHash) != bk_previousBlock)); then
 err="previousBlockHash doesn't give previousBlock ID"; break
fi

This check makes use of the function

function hashToID { # convert 32-byte hex hash to decimal id
 echo $(hexToDec $(hexToLE8 $1))
}

which in turn makes use of the functions

function hexToLE8 { # convert hex string to 8-byte (maximum) little endian
 echo $1 | sed 's/../&\n/g' | head -8 | tac | tr -d '\n'
}
function hexToDec { # convert hex string (any length) to decimal
 echo "ibase=16; ${1^^}" | bc
}

These functions can be included near the top of the script. They are written to echo a result which can be captured with $(...). $1 is the first function argument, $2 the second.

hexToLE8 converts a hex string to an 8-byte (16 hex digit) maximum little endian (LE) string, meaning that the byte order is reversed. All ID's in Nxt are 8-byte maximum LE unsigned integers, although internally in the Nxt software they are stored as signed integers since that is the only option in Java. Thus only the eight most significant bytes of the 32-byte previousBlockHash are used to compute the previousBlock ID.

hexToDec converts arbitrary size hex strings to decimal numbers. The utility bc (basic calculator) requries the hex sting to be capitalized (accomplished by ${1^^}) but can handle large strings, in contrast to the built-in printf %d 0x$1 which can only handle up to 63 bits.

7.2.4 Block ID Check

blockHex=$(decToLE8 4 $bk_version)$(decToLE8 4 $bk_timestamp)
blockHex=$blockHex$(decToLE8 8 $bk_previousBlock)$(decToLE8 4 $bk_numberOfTransactions)
tA=$bk_totalAmountNQT; tF=$bk_totalFeeNQT; w=8
if ((bk_version < 3)); then ((tA /= 100000000)); ((tF /= 100000000)); w=4; fi
blockHex=$blockHex$(decToLE8 $w $tA)$(decToLE8 $w $tF)
blockHex=$blockHex$(decToLE8 4 $bk_payloadLength)$bk_payloadHash$bk_generatorPublicKey$bk_generationSignature
if ((bk_version > 1)); then blockHex=$blockHex$bk_previousBlockHash; fi 
blockHexNoSignature=$blockHex
blockHex=$blockHex$bk_blockSignature
blockHash=$(hexToHash $blockHex)
blockID=$(hashToID $blockHash)
if ((blockID != bk_block)); then err="computed block ID mismatch"; break; fi

$blockID is computed from the hash of a byte string composed of certain block fields packed in a certain order. Decimal number fields must first be converted to hex strings with the function decToLE8, which in turn uses the function hexToLE8 previously introduced:

function decToLE8 { # convert $1-byte (8-byte maximum) decimal $2 to little-endian hex
 local hex=$(printf %0$(($1 * 2))x $2)
 local len=$(($1 * 2 - ${#hex})); if ((len < 0)); then hex=${hex:0-len}; fi # handles version = -1
 echo $(hexToLE8 $hex)
}

Once the hex string blockHex is ready, it is hashed using the function hexToHash, which relies on the utility sha256sum:

function hexToHash { # convert hex string to 32-byte hex hash
 local hash=($(echo -ne $(echo $1 | sed 's/../\\x&/g') | sha256sum))
 echo ${hash[0]}
}

The hex string is prepared by converting it to a byte string as expected by sha256sum; echo must not append a newline nor treat backslashes as escapes, hence echo -ne.

Finally, the block ID is computed from the hash using the previously mentioned hashToID function. It is then checked against the corresponding block field.

Note that the $blockHex string definition depends on the version of the blockchain. Version 1 introduced the previousBlock field, and version 3 replaced NXT with NQT=10-8 NXT in the amount and fee fields.

7.2.5 Block Signature Check

v=${bk_blockSignature:0:64}; h1=${bk_blockSignature:64}
Y=$(java Curve25519 $v $h1 $bk_generatorPublicKey $bk_version)
if (($?)); then err="block signature invalid"; break; fi
m=$(hexToHash $blockHexNoSignature); h2=$(hexToHash $m$Y)
if [[ $h1 != $h2 ]]; then err="block signature not verified"; break; fi

There is no linux utility available for verifying Curve25519 signatures, so the same Curve25519 Java class used by the Nxt software is used. The $bk_blockSignature field is split in half and each half is passed along with $bk_generatorPublicKey to the customized Curve25519 utility, which returns $Y. $Y is hashed with the hash of $blockHexNoSignature, which is just $blockHex without the trailing $bk_blockSignature field.

7.2.5.1 Curve25519 Utility - Java Version

Curve25519.java can be acquired from the Nxt software directory nxt/src/java/nxt/crypto/Curve25519.java. A main method can be added to the Curve25519 class as follows, after removing the first line (package nxt.crypto;):

final class Curve25519 {

 // Usage: java Curve25519 v h publicKey version
 // signature is (v 32 bytes + h 32 bytes), public key is 32 bytes, version is integer

 public static void main (String[] args) {
  if (args.length != 4) System.exit(1);
  byte [] v = hexStringToByteArray(args[0]);
  byte [] h = hexStringToByteArray(args[1]);
  byte [] publicKey = hexStringToByteArray(args[2]); 
  int version = Integer.parseInt(args[3]);
  if (version >= 3) {
   byte[] signature = new byte[v.length + h.length];
   System.arraycopy(v, 0, signature, 0, v.length);
   System.arraycopy(h, 0, signature, v.length, h.length);
   if(!Curve25519.isCanonicalSignature(signature)) System.exit(2);
   if(!Curve25519.isCanonicalPublicKey(publicKey)) System.exit(3);
  }
  byte[] Y = new byte[32];
  Curve25519.verify(Y, v, h, publicKey);
  System.out.println(bytesToHex(Y));
  System.exit(0);
 }

 public static byte[] hexStringToByteArray(String s) {
  int len = s.length();
  byte[] data = new byte[len / 2];
  for (int i = 0; i < len; i += 2) {
   data[i / 2] = (byte) ((Character.digit(s.charAt(i), 16) << 4) + Character.digit(s.charAt(i+1), 16));
  }
  return data;
 }

 final protected static char[] hexArray = "0123456789ABCDEF".toCharArray();
 public static String bytesToHex(byte[] bytes) {
  char[] hexChars = new char[bytes.length * 2];
  for ( int j = 0; j < bytes.length; j++ ) {
   int v = bytes[j] & 0xFF;
   hexChars[j * 2] = hexArray[v >>> 4];
   hexChars[j * 2 + 1] = hexArray[v & 0x0F];
  }
  return new String(hexChars);
 }
  ...
}

This method provides a command line interface that accepts and returns hex strings. It also checks the input fields for canonical form when version >= 3.

Compile the modified class with:

$ javac Curve25519.java

The resulting Curve25519.class and Curve25519$long10.class files are now ready for use with the java command

$ java Curve25519 v h publicKey version
7.2.5.2 curve25519 Utility - C Version

Curve25519.java was ported to C and is available here. Acquire curve25519.c and compile it with:

$ gcc curve25519.c -o curve25519

This version includes all of the cryptographic methods from the original Java source code, plus a main function that provides a convenient interface allowing key pair generation, signing, verifying and key agreement. Here we only need verification. Everywhere where the Java utility is invoked:

Y=$(java Curve25519 $v $h1 ...PublicKey $bk_version)

the C utility can be invoked instead:

Y=$(./curve25519 -v $v $h1 ...PublicKey)

where the executable is assumed to be in the same directory as block.sh. Note that the C version always checks for canonical form, regardless of version; hence the final argument is omitted.

7.2.6 previousBlockHash Check (Synched)

if ((bk_version > 1)) && [[ $pBH != $bk_previousBlockHash ]]; then err="previousBlockHash mismatch"; break; fi

Insert the following near the end of the main block loop:

pBH=$blockHash

7.2.7 generationSignature Check (Synched)

if ((bk_version == 1)); then
 v=${bk_generationSignature:0:64}; h1=${bk_generationSignature:64}
 Y=$(java Curve25519 $v $h1 $bk_generatorPublicKey $bk_version)
 if (($? > 1)); then err="canonical failure"; break; fi
 m=$(hexToHash $pGS);  h2=$(hexToHash $m$Y)
 if [[ $h1 != $h2 ]]; then err="version == 1 generation signature not verified"; break; fi
else
 generationSignatureHash=$(hexToHash $pGS$bk_generatorPublicKey)
 if [[ $generationSignatureHash != $bk_generationSignature ]]; then
  err="version > 1 generation signature not verified"; break
 fi
fi

Insert the following near the end of the main block loop:

pGS=$bk_generationSignature

When version = 1, $bk_generationSignature is checked exactly like the $bk_blockSignature. With version 2, a simple hash algorithm not involving a signature is used. The change was made because there was a flaw in the algorithm which used generationSignature in forging calculations due to randomness in Curve22519 signature generation.

7.2.8 baseTarget Check (Synched)

bT=$(((pBT * (bk_timestamp - pT)) / 60)); if ((bT > MAX_BASE_TARGET)); then bT=$MAX_BASE_TARGET;fi
if ((bT < pBT / 2)); then ((bT = pBT / 2)); fi
if ((bT == 0)); then bT=1; fi
pBTx2=$((pBT * 2)); if ((pBTx2 > MAX_BASE_TARGET)); then pBTx2=$MAX_BASE_TARGET; fi
if ((bT > pBTx2)); then bT=$pBTx2; fi
if ((bT != bk_baseTarget)); then err="baseTarget mismatch"; break; fi

Insert the following near the end of the main block loop:

pBT=$bk_baseTarget; pT=$bk_timestamp

Insert the following constant initializations before the start of the main block loop:

TWO64=$(echo "ibase=16; 10000000000000000" | bc) # 2^64 in decimal
MAX_BALANCE_NXT=1000000000
GENESIS_BASE_TARGET=$(echo "$TWO64 / (2 * 60 * $MAX_BALANCE_NXT)" | bc) # 153722867
MAX_BASE_TARGET=$((MAX_BALANCE_NXT * GENESIS_BASE_TARGET))

$MAX_BALANCE_NXT is the total amount of NXT in existence, 1 billion. $GENESIS_BASE_TARGET is the initial $bk_baseTarget at height zero. It is used to determine which Nxt node will forge a new block. A hit value is computed for each node based on an effectively random 8-byte (64-bit) number computed from $bk_generationSignature. All nodes with a hit value less than $bk_baseTarget * $tx_effectiveBalance (computed from transaction data) * (time since last block, in seconds) are permitted to submit new blocks to the Nxt network for inclusion in the blockchain. The candidate blockchain with the greatest $bk_cumulativeDifficulty (inversely related to $bk_baseTarget) is accepted by the network. Eventually some node will get a hit because it becomes easier with each passing second.

If all $MAX_BALANCE_NXT NXT in existence were concentrated in one node, $bk_baseTarget * $tx_effectiveBalance would be TWO64 / (2 * 60) which that node's hit value would have a 50% chance of exceeding in 60 attempts. Theoretically then, a hit occurs on average once per minute throughout the network if all NXT tokens are available for forging. In practice the average elapsed time between blocks is somewhat longer.

$bk_baseTarget is dynamically adjusted to maintain a steady blockchain growth rate. If blocks are being generated too slowly, $bk_baseTarget is increased proportional to the time elapsed since the previous block, but is not allowed to more than double nor less than halve with each new block.

7.2.9 cumulativeDifficulty Check (Synched)

cD=$(echo "$pCD + $TWO64 / $bk_baseTarget" | bc)
if [[ $cD != $bk_cumulativeDifficulty ]]; then err="cumulativeDifficulty mismatch"; break; fi

Insert the following near the end of the main block loop:

pCD=$bk_cumulativeDifficulty

The utility bc is necessary, since $TWO64 is out of the normal long integer range and $bk_cumulativeDifficulty grows without limit; it will eventually go out of range.

7.3 Complete Script

#!/bin/bash

function hexToLE8 { # convert hex string to 8-byte (maximum) little endian
 echo $1 | sed 's/../&\n/g' | head -8 | tac | tr -d '\n'
}

function hexToDec { # convert hex string (any length) to decimal
 echo "ibase=16; ${1^^}" | bc
}

function hashToID { # convert 32-byte hex hash to decimal id
 echo $(hexToDec $(hexToLE8 $1))
}

function decToLE8 { # convert $1-byte (8-byte maximum) decimal $2 to little-endian hex
 local hex=$(printf %0$(($1 * 2))x $2)
 local len=$(($1 * 2 - ${#hex})); if ((len < 0)); then hex=${hex:0-len}; fi # handles version = -1
 echo $(hexToLE8 $hex)
}

function hexToHash { # convert hex string to 32-byte hex hash
 local hash=($(echo -ne $(echo $1 | sed 's/../\\x&/g') | sha256sum))
 echo ${hash[0]}
}

TWO64=$(echo "ibase=16; 10000000000000000" | bc) # 2^64 in decimal
MAX_BALANCE_NXT=1000000000
GENESIS_BASE_TARGET=$(echo "$TWO64 / (2 * 60 * $MAX_BALANCE_NXT)" | bc) # 153722867
MAX_BASE_TARGET=$((MAX_BALANCE_NXT * GENESIS_BASE_TARGET))

function getFields {
 local fields=$(echo $json | jshon -k)
 local field; local type; local name; local names
 for field in $fields; do
   name="$1_$field"; names="$names $name"
   type=$(echo $json | jshon -e $field -t)
   if [[ $type == "string" || $type == "number" ]]; then
    declare -g $name="$(echo $json | jshon -e $field -u)"
   else
    declare -g $name="$(echo $json | jshon -e $field)"
   fi
   #echo $name=${!name} # dump fields
 done
 declare -g "$1_fields"="$names"
}

err=; synched=0
if ((${#1})); then h=$1; else h=0; fi

while ((!synched)) || ((${#b})); do

 json=$(curl -s "http://localhost:7876/nxt?requestType=getBlock&height=$h")
 if ((${#json} == 0)); then err="server not responding"; break; fi
 getFields 'bk'

 if ((${#bk_errorDescription})); then err="$bk_errorDescription"; break; fi
 echo "height=$bk_height, timestamp=$bk_timestamp"

 # do checks here that only refer to the current block

 txNum=$(echo $bk_transactions | jshon -l)
 if ((txNum != bk_numberOfTransactions)); then err="numberOfTransactions mismatch"; break; fi

 amt=0; fee=0
 payLen=0; payHex=
 txs=$(echo $bk_transactions | jshon -a -u) # get list of transaction IDs
 for tx in $txs; do
  json=$(curl -s "http://localhost:7876/nxt?requestType=getTransaction&transaction=$tx")
  if ((${#json} == 0)); then err="server not responding"; break; fi
  getFields 'tx'
  if ((${#tx_errorDescription})); then err="$tx_errorDescription"; break; fi

  json=$(curl -s "http://localhost:7876/nxt?requestType=getTransactionBytes&transaction=$tx")
  if ((${#json} == 0)); then err="server not responding"; break; fi
  getFields 'txb'
  if ((${#txb_errorDescription})); then err="$txb_errorDescription"; break; fi

  # place transaction checks here

  ((amt += tx_amountNQT)); ((fee += tx_feeNQT))
  ((payLen += ${#txb_transactionBytes})); payHex=$payHex$txb_transactionBytes

  if [[ $tx_block != $bk_block ]]; then err="tx/bk block ID mismatch"; break; fi
  if ((tx_blockTimestamp != bk_timestamp)); then err="tx/bk timestamp mismatch"; break; fi
  if ((tx_height != bk_height)); then err="tx/bk height mismatch"; break; fi

  if ((tx_height <= 213000)); then v=0; else v=1; fi
  if ((v != tx_version)); then err="tx version mismatch"; break; fi

  sH=$(hexToHash $tx_signature)
  if [[ $sH != $tx_signatureHash ]]; then err="tx signature hash mismatch"; break; fi

  if ((bk_version < 3)); then
   fH=$(hexToHash $txb_transactionBytes)
  else
   fH=$(hexToHash $txb_unsignedTransactionBytes$tx_signatureHash)
  fi
  txID=$(hashToID $fH)

  if [[ $fH != $tx_fullHash ]]; then err="tx full hash mismatch"; break; fi
  if [[ $txID != $tx_transaction ]]; then err="tx ID mismatch"; break; fi

  if ((bk_height > 0)); then
   v=${tx_signature:0:64}; h1=${tx_signature:64}
   Y=$(java Curve25519 $v $h1 $tx_senderPublicKey $bk_version)
   if (($?)); then err="tx signature invalid"; break; fi
   m=$(hexToHash $txb_unsignedTransactionBytes); h2=$(hexToHash $m$Y)
   if [[ $h1 != $h2 ]]; then err="tx signature not verified"; break; fi
  fi

  for var in $tx_fields; do unset $var; done # clear all transaction fields
  for var in $txb_fields; do unset $var; done # clear all transaction bytes fields
 done
 if ((${#err})); then break; fi
 if ((amt != bk_totalAmountNQT)); then err="tx amountNQT mismatch"; break; fi
 if ((fee != bk_totalFeeNQT)); then err="tx feeNQT mismatch"; break; fi
 if ((payLen / 2 != bk_payloadLength)); then err="payload length mismatch"; break; fi
 if [[ $(hexToHash $payHex) != $bk_payloadHash ]]; then err="payload hash mismatch"; break; fi

 if ((h != bk_height)); then err="blocks not in sequential height order"; break; fi

 if ((bk_height == 0)); then v=-1
 elif ((bk_height <= 30000)); then v=1
 elif ((bk_height <= 132000)); then v=2
 else v=3; fi
 if ((v != bk_version)); then err="incorrect version"; break; fi  

 if ((bk_version > 1)) && (($(hashToID $bk_previousBlockHash) != bk_previousBlock)); then
  err="previousBlockHash doesn't give previousBlock ID"; break
 fi

 blockHex=$(decToLE8 4 $bk_version)$(decToLE8 4 $bk_timestamp)
 blockHex=$blockHex$(decToLE8 8 $bk_previousBlock)$(decToLE8 4 $bk_numberOfTransactions)
 tA=$bk_totalAmountNQT; tF=$bk_totalFeeNQT; w=8
 if ((bk_version < 3)); then ((tA /= 100000000)); ((tF /= 100000000)); w=4; fi
 blockHex=$blockHex$(decToLE8 $w $tA)$(decToLE8 $w $tF)
 blockHex=$blockHex$(decToLE8 4 $bk_payloadLength)$bk_payloadHash$bk_generatorPublicKey$bk_generationSignature
 if ((bk_version > 1)); then blockHex=$blockHex$bk_previousBlockHash; fi 
 blockHexNoSignature=$blockHex
 blockHex=$blockHex$bk_blockSignature
 blockHash=$(hexToHash $blockHex)
 blockID=$(hashToID $blockHash)
 if ((blockID != bk_block)); then err="computed block ID mismatch"; break; fi

 v=${bk_blockSignature:0:64}; h1=${bk_blockSignature:64}
 Y=$(java Curve25519 $v $h1 $bk_generatorPublicKey $bk_version)
 if (($?)); then err="block signature invalid"; break; fi
 m=$(hexToHash $blockHexNoSignature); h2=$(hexToHash $m$Y)
 if [[ $h1 != $h2 ]]; then err="block signature not verified"; break; fi

 if ((synched)); then # do checks here that refer to the previous block

  if ((b != bk_block)); then err="block ID mismatch"; break; fi
  if ((pB != bk_previousBlock)); then err="previousBlock ID mismatch"; break; fi

  if ((bk_version > 1)) && [[ $pBH != $bk_previousBlockHash ]]; then err="previousBlockHash mismatch"; break; fi

  if ((bk_version == 1)); then
   v=${bk_generationSignature:0:64}; h1=${bk_generationSignature:64}
   Y=$(java Curve25519 $v $h1 $bk_generatorPublicKey $bk_version)
   if (($? > 1)); then err="canonical failure"; break; fi
   m=$(hexToHash $pGS);  h2=$(hexToHash $m$Y)
   if [[ $h1 != $h2 ]]; then err="version == 1 generation signature not verified"; break; fi
  else
   generationSignatureHash=$(hexToHash $pGS$bk_generatorPublicKey)
   if [[ $generationSignatureHash != $bk_generationSignature ]]; then
    err="version > 1 generation signature not verified"; break
   fi
  fi

  bT=$(((pBT * (bk_timestamp - pT)) / 60)); if ((bT > MAX_BASE_TARGET)); then bT=$MAX_BASE_TARGET;fi
  if ((bT < pBT / 2)); then ((bT = pBT / 2)); fi
  if ((bT == 0)); then bT=1; fi
  pBTx2=$((pBT * 2)); if ((pBTx2 > MAX_BASE_TARGET)); then pBTx2=$MAX_BASE_TARGET; fi
  if ((bT > pBTx2)); then bT=$pBTx2; fi
  if ((bT != bk_baseTarget)); then err="baseTarget mismatch"; break; fi

  cD=$(echo "$pCD + $TWO64 / $bk_baseTarget" | bc)
  if [[ $cD != $bk_cumulativeDifficulty ]]; then err="cumulativeDifficulty mismatch"; break; fi

 fi

 synched=1; ((++h))
 pB=$bk_block; b=$bk_nextBlock; pBH=$blockHash; pGS=$bk_generationSignature
 pBT=$bk_baseTarget; pT=$bk_timestamp; pCD=$bk_cumulativeDifficulty
 for var in $bk_fields; do unset $var; done # clear all block fields
 
done

if ((${#err})); then echo "Error: $err"; fi