null_radix

SRFI-89 support of Guile

2020-08-11T13:45:00Z

Recently I implemented SRFI-89 for Guile. You can check out the code here. If you use Guix you can install it with guix install guile-srfi-89

Why?

The motivation for me was that I wanted to write more portable scheme code. Some say Scheme has a problem with a fractured ecosystem. One way to combat a fractured ecosystem is to write portable scheme code. SRFIs help with this.

SRFI-89 - Optional positional and named parameters

Guile has a rather nice extended define know as define* It is close to the define* of SRFI-89 but there are some differences. Guile's define* allows you to add optional and named parameters to your procedures. It works like this

(define* (optional-proc a b #:optional c d . e) '()
  (list a b c d e))

Here, all parameters after #:optional - that is c d and e - are optional and don't have to be used when the procedure is called. The parameters a b c d are known as positional parameters, since the position of the values used when calling the produced determines what value is bound to which variable. Parameters can also have names. For example:

(define* (sir-yes-sir #:key action how-high)
  (list action how-high))`

Here, any parameter occurring after #:key is a key that can be used as follows:

(sir-yes-sir #:action 'jump #:how-high 10)

The order of the keys doesn't matter, so this is equivalent to the above:

(sir-yes-sir #:how-high 10  #:action 'jump)

Keys can be used in any order and can have default values. This is really convenient when you have a procedure that may have many options.

Guile and SRFI-89 Differences

Named Parameters

SRFI-89's define* has nearly the same capabilities as Guile but there are a few differences.

Instead of using #:keyword <keywords> ... SRFI-89 uses the following syntax

(define* (sir-yes-sir (#:action action) (#:how-high how-high))
  (list action how-high))`

Each keyword is specified by being a keyword object paired with a variable that it will bind to in the body of the defined procedure.

Optional Parameters

Optionals also work differently. For example:

(define* (optional-proc a b (c #f) (d #f) . e) 
  (list a b c d e))

With SRFI-89's every optional must have a corresponding expression. With Guile, optionals always default to #f. This is also true for named parameters.

Rest

The . rest parameter catches any tailing arguments in a procedure. For example:

(define* (test a b . c)
  (list a b c))

(test 1 2 3 4 5 6)
-> (list 1 2 (4 5 6))

When using the rest syntax with named parameters in Guile the named parameters are also bound to the rest variable. For example:

(define* (test a #:key b . c) (list a b c))

(test 1 #:b 'z 2 3)
-> (list 1 'z (#:b 'z 2 3))

SRFI-89 on the other hand does not do this. In SRFI-89's world we would have the following:

(define* (test a (#:b b). c) (list a b c))

(test 1 #:b 'z 2 3)
-> (list 1 'z (2 3))

Forms

Guile's define* has one basic form which is:

(define* proc-name <positional> | <optionals> | <named> | <rest>)

While SRFI-89 has two forms:

(define* proc-name <positional> | <optionals> | <named> | <rest>)

(define* proc-name  <named> | <positional> | <optionals>  | <rest>)

This means you can write a procedure as:

(define* (test a-positional (an-optional-positional 1) (#:named name #f) . rest)
  (list a-positional an-optional-positional name rest))

(define* (test  (#:named name #f) a-positional (an-optional-positional 1) . rest)
  (list a-positional an-optional-positional name rest))

Implementation

To implement SRFI-89 I wanted to be as lazy as possible, performant and hygienic. The original implementation of SRFI-89 was not so performant on Guile compared to the native define* and was not hygienic. To start with I didn't know much about macros so to get up to speed I read Guile's documention and Writing Hygienic Macros in Scheme with Syntax-Case which I highly recommend. The latter has a bunch of nice examples that were very helpful. Also Oleg Kiselyov's page on marcos is quite fun.

To accomplish the goals of performance and being lazy I just reused Guile's define*. Lets see what the macro produces.

Here is an example of the form <positionals>|<named>|<rest>:

(define* (test a (b #f) (#:c c') (#:d d #t) . r)
  (list a b c))

Which results in:

(define test
  (guile:lambda*
    (a #:optional (b #f)
       #:key (c (guile:error "key c is required")) (d #t)
       #:rest t-8ae9909742848ae-ae4)
 
     (let ((r (remove-keywords t-8ae9909742848ae-ae4))
       (c' c)
       (d d))
     
       (list a b c))))

It's a quite straightforward mapping I think. The statement (c (guile:error "key c is required")) prevents c from defaulting to #f which is guile:lambda* default behavior. The let provides the mapping from keywords to internal variable names. And (remove-keywords t-8ae9909742848ae-ae4) removes the keywords from rest.

The other form <named>|<positionals>|<rest> is a bit more tricky. Here is an example:

(define* (test  (#:a a') (#:b b #t) c (d #f) . r)
  (list a b c))

Which results in:

(define test
  (guile:lambda*
    (#:key (a (guile:error key a is required)) (b #t)
     #:rest t-8ae9909742848ae-c77)
       (let ((a' a)
             (b b))
         (apply (guile:lambda* (c #:optional (d #f) #:rest r)
            (list a b c))
            (remove-keywords t-8ae9909742848ae-c77)))))

Since guile:lambda* doesn't support this form at all we need an extra guile:lambda*. The outer guile:lambda* processes the named parameters and leaves the positional parameters which can be accessed the #:rest. The inner guile:lambda* the processes the positional arguments after removing the keywords from rest.

Ambients Applied to Ethereum

2016-02-01T13:22:00Z

Part I

Sometimes Ethereum is compared to a singleton Virtual Machine. While this is correct in some sense; I think it is a bit more. First of all what is a singleton in a distributed system? It is merely a set of values that some threshold of participants have come to consensus on. A Virtual Machine is a computational environment that is isolated from the physical computer and from other environments.

A hypervisor allows the physical machine to be multiplexed into many VMs. According to this definition a common hypervisor is the web browser where webpages are VMs. Another example of a hypervisor would be Ethereum as each contract gets its own isolated computational environment.

There are many differences between the common web browser and Ethereum, but one of the more interesting ones is how VMs communicate and interact with each other. Web browsers don???t provide a way for VMs to directly interact while Ethereum on the other hand provides some simple mechanism for VM interaction; the opcodes CALL, DELEGATECALL, CALLCODE, CREATE. In this post will explore the question; What other rules could exist? Can we generalize VM interactions and provided an abstract framework for these interactions? And from this framework can we reason about distributed hypervisors?

Most of this post will resemble ambient calculus but there are several notable differences from ambient calculus and what is presented here. The diagrams can be thought of as bigraphs but they should also be self explanatory. Part I will describe the rules of ambients and then apply them to Ethereum. Part II will discuss scaling in the terms of ambients as laid out by part I.

What is an Ambient?

An ambient is a bounded place in which computation can occur. A boundary determines what is inside and what is outside an ambient. For ambients we call this boundary a membrane. The area inside an ambient is hierarchical namespace. Objects can exist inside an ambient. The objects are addressable via the namespace. There are three base elements in ambient calculus. Objects, Namespaces and Messages.

Hierarchical Namespaces

One of the most familiar namespace is the file system tree. Namespaces allow us to identify objects with paths or names. Namespaces here have the following properties

For every possible path there exists a null or an object
At any point in the namespace you can move up or down. This is what is implied by hierarchical.
Every path has a root associated with it. The root uniquely identifies the content for all the paths below the root. You can think of the root as a pointer to the content of the path.
Paths can be read from or written to
Messages can be sent along paths to objects

Object Types

What is an object? It is just a value. In real life computing its just some data. This data can be interpreted in several different ways. Any Object can be read as data. The pink circle is some data that exists in the grey ambient.

Objects can also be interpreted as ambients. This allows ambients to have sub-ambients. Here the orange and grey circles are ambients.

Objects can also be interpreted as ports. Two or more ports form a I/O channel. Channels allow messages to be sent to ambients in a different namespaces. Channels can be thought of as tunnels through an ambient???s membrane. Both the entrance and exit ports must exist somewhere in a namespace. Here the green objects represent ports.

Lastly messages can also be considered to be an object. Messages are special since they are defined as objects in motion or thought of as objects with velocity.

To Recap; Objects can be the following types

Objects :: =
     Data
     Port
     Ambient
     Message

Messages

As stated above messages are objects that are in transit. Messages can be sent through a namespace and through channels. Messages have the following properties that are set by the systems message handler. They are not all intrinsically part of the message but as you will see later they make working with messages easier.

To - The path to the destination of the message. This is immutable.
From - The sender of the message. This is immutable.
Type - The type of message. This is immutable.
Data - The message's body. This is immutable.
Heading - The destination relative to its current position. If Heading is null then the message has arrived at its destination and will travel no further. This is not directly encoded in the message but instead set by the systems message handler. This is mutable.
Direction - Which direction the message is traveling. It can either be going ???out??? of the ambient or going ???in??? to the ambient. This is mutable.

Message Types

Message have the following types which have corresponding commands used to send them.

Message :: =
       Set(path, value) - Sets a path to a given value
       Get(path) - Gets a value of the given path
       SetRoot(path, root) - sets the root of `path` to `root`
       GetRoot(path) - Gets the path???s root
       Call(path, data) - Sends a message along the given path
       Connect(to, from, options) - creates a channel between two paths.

Deleting

It might not be immediately obvious how to delete an ambient or other objects. To do this we use the Set and SetRoot message.

The Set message sets the value of a path. Setting a path to null is equivalent to deleting the contents of that path. For example Set(???pinkAmbient???, null) Here the pink ambient is set to null. Note the the orange ambient was not deleted.

The SetRoot message sets the root of a path. If the root is set to null all the path values below the root will become null. For example CopyRoot(???pinkAmbient???, null) will set the pink ambient???s root to null which will also cause the orange ambient be to null.

Of course if we did something like SetRoot(???a???, ???pinkAmbientsRoot???) we would copy the pink Ambient and all of it contents to path ???a???

Iterating the through a Namespace.

In many cases it useful to iterate through all the ambients in a given namespace. One way we could approach this is to get each path in the namespace. But the problem is that most namespaces are infinite. A better way would be to provide an explicit iteration method. Let???s add a message

Message :: =
   Next(path) - Given a path return the next non-null path in the namespace.

This implies that namespaces all must have an order. Also this provides us with a nice way to build more complicated ambient operations like merging two or more ambients. We also need this to build type checking.

Membrane computing

The ambient???s border is its membrane. It can filter message coming into and going out of it. For example the if the grey ambient sends a Set(???blueAmbient???, null) message to the path of the ???blueAmbient??? it will go through the membrane of the orange ambient. The orange ambient can decided whether or not to let the message pass through.

A Membrane API

Lets walk through a small example of what programming ambients might look like.

Ambient A is trying send a message to ambient B but the message has to go through Ambient C. Since A is a sub-ambient of C, C can control this message. Here is what an api for dealing with messages might look like. Let say that we have a function ???onMessage??? that gets ran whenever the ambient gets a message. Here is what C membrane could look like.

/**
* Allow any message to pass through the membrane except messages from Ambient D
* @method onMessage
* @param message - the message that is leaving the ambient
* @retruns Boolean
*/

function onMessage(message) {
  if(Message.sender != ???A??? && Message.direction == ???out???){
    Message.heading = ???D???
  }
}

C filters any messages coming from the path ???A??? that are going out of it. Instead of letting the message go to its intended location C reroutes the message to location ???D???. Notice how C set the heading on the message. If C set Message.heading to null then the message would stop there. C can only decide where to forward the message or to stop it.

The ability of ambients to filter and decide which message can travel through them is an important one. This is also known as Membrane computing. It will allow you to build flexible and easily composable contracts. Especially when it comes to administration of sub-contracts.

Mapping ambients to a Ethereum

Now that we have the basics of ambients let???s apply them to a one of our favorite data structures, the merkle tree. To start you might have already recognized the fact that a contract in Ethereum is like an ambient and the namespace is provided by the merkle tree.

Ambients ::= contracts
Namespace ::=the merkle tree

This could be visualized like this

In Ethereum each ambient has an address that is 20 bytes long and looks like the following 0x1158c3c9a70e85d8358972810ed984c8e6ffcf0f. Ethereum ambients have storage that allow them store store arbitrary values permanently. Storage is accessed and manipulated with the SSTORE and SLOAD opcodes. The equivalent to these are the set and get messages. Also command Call is equivalent.

Set ::= SSTORE
Get ::= SLOAD
Call :: = CALL

SetRoot, GetRoot and Connect do not have equivalents in Ethereum currently. SetRoot and GetRoot would read from and manipulate the underlying mekle trie

Now we are going to deviate from current Ethereum to Ethereum + Ambients. Let us say the contract 0x1158c3c9a70e85d8358972810ed984c8e6ffcf0f sets the value ???doge??? at the addresses ???coin??? which is 636f696e in hex. The address 0x1158c3c9a70e85d8358972810ed984c8e6ffcf0f/636f696e would then contain the value ???doge???. Also ???doge??? could also be interpreted as code if a Call was made to that path.

Personal Accounts

Lets use a personal Ethereum account as an example. For convenience we are going to say the address of the account is ???accountA??? which will be represented as the grey ambient. This ambient would hold the basic signature validation code as seen in the currency and crypto abstraction. If the user wanted to place a spending limits on herself then she could create a ???Savings Account??? which would only permit a certain amount of ether to be spent per day. Furthermore the user could create her own custom Name Reg or other financial apps. The hierarchical nature of the ambients allows you to build up administrative ???zone???. They can make code very modular since the ???saving account??? and other contracts don???t need to have any code dedicated to checking if the user is an admin or checking other credential since that could be done by the accountA???s ambient.

Part II - Scalability

In this section we will explore some ideas about scalability in terms of ambients. The basic idea of scalability is fairly simple. Most methods proposed so far involve these properties:

Separating some part of the state into a shard that is processed independent of the other shards
Some sort of cross validation; where some portion of a shard???s work is checked by other shards which is usually triggered by cross shard communication.

We are also assuming we have a Proof of Stake algorithm like Casper and this algorithm is implemented in a set of ambients. Along with casper we have a currency ambient that tracks the amount of ether each account ambient has. These ambients are grouped together into the system ambient. There maybe many more ambients in the system ambient but for now we will just consider these.

For now we will simply assume that casper works and produces the correct state for the ???Ethereum Ambient???.

Sharding

If Ethereum is successful, the volume of transaction will increase over time. After a while a high volume of transactions will cause the price of gas to increase. At a certain threshold determined by a Threshold function the Casper ambient will produce a shard. It should be noted that only from the casper ambient???s perspective is Ethereum sharded. Everyone else sees Ethereum as one continued namespace extending through many ambients.

There is some threshold that is needed to create a shard in Casper. This is not the focus of this post but we can image some of the parameters it might be based off of. It could use gasPrice to transaction ratio. Or could it use a voting system or a bidding system or combination of all them.

Besides the Threshold function we will assume the following about Casper:

Anyone can contest a state transition.
Validators are randomly assigned to shards. These form a validation group that run Casper for that shard.
Validator may be assigned to more than one shard
New shards must be initially validated by all validators
The total amount in bond in a validation group of a shard should be equivalent to what the shard is worth.

Creation of Shards

For now we will assume that new shards will start out as an empty ambient. But keep in mind this might not always be the case- for example a particularly successfully dapp could perhaps pay the Casper contract enough to make it worthwhile for the validator to create a shard out of it. But for now it is empty.
The new shard would send a getRoot to the abstract system. Then it would use setRoot internally to copy the abstract system its namespace
Part of the threshold function might be pledges from other ambients to move to a new shard once it is created. When the new shard is created, all the accounts that pledged to move are automatically moved to the new shard. This is done after the system ambient is in place. The accounts are also copied with the CopyRoot command.
After they have been copied their original address is replaced by a port (created by the ???Connect??? command) creating a channel to their new account on the new shard.
The currency contract then sets the amount of ether that the shard has to the sum of the accounts that pledge to move.
Lastly the in the new shards currency, the contract is populated by the values of the copied accounts.

Fractal chains?

The end result will be that the top level ambients no longer ???see??? the individual accounts that are in the new shard, instead it only see the value of the sum of the account on the new shard ($82 in the diagram). While the new shard???s currency contract keeps track of the individual accounts in the shard. This resembles a fractal in the way that part of the whole is encoded in every section of the structure.

Also if anyone uses the old address of an ambient that moved, their messages will be forwarded to them via the channels. There are some disadvantages to using the channels; 1) its will be more costly 2) there will be higher latency.

Financial Isolation - Counterfeiting Attacks

The shards can be seen forming a hierarchy; each shard ambient keeping track of its accounts and the sum of the accounts in its children shards.

This creates a strong guarantee of the correctness of account balances. No shard can create counterfeit currency and send it to another shard. Furthermore the security is additive. Meaning that the more shards that a message crosses the stronger the guarantee that it is correct. We are assuming that every validation group will check that transaction going through it. If a transaction is going from shard C to C.A.B then shards C, C.A and C.A.B all will check the transaction and ask the shard C for merkle proof of the sender???s account. If the transaction was found to be invalid after the validator???s approved it then the validators in all three groups would lose their deposits. If accounts were defrauded they would first be refunded from the validators deposits.

Let???s consider a long range counterfeit attack. This is where a validation group on a shard creates an account with an invalid amount of currency associated with it and then they just leave it in the shard. If they ever try to move it from the shard the parent validation group will request a complete transaction log that shows how the accounts got its money. At this point the attack would fail unless the parent validation group was also compromised. And in a long range attack the attackers wait until the parent validation group is compromised. The best way to counter this is to make each validation group responsible for the complete history of its shard and not to release the bonds to unbonded validators after several epochs. This gives the current validation group an incentive to check the previous validation groups work.

One way in which a validation group can check the previous validation group work quickly is to just sum the transaction graph. We can think of all messages that transfer currency as forming a directed graph. Since we know the global amount of currency that the shard has, a validation group just needs to sum up the total amount the accounts had for each block in the previous epoch and check it against the known global amount.

To recap, several properties that can increase security are:

Give the Parent Validation group an incentive to check the work of their children.
Give validator an incentive to check previous work

Validation Group Groups (Hierarchical validation groups)

Validators may have to put up a very high bond to participate in validation. The amount of bond needed is a function of the target number of validators which is a function of the number of shards that exists.

But this poses a problem since if there were a higher number of validators it would be harder to coordinate a bribe attack on a shard but on the other hand Casper can become inefficient when there are large number of validators. One way this might be solved is to have validators themselves composed of validation groups. The validation group would run in a separate ambient on a separate blockchain from Ethereum.

In the validation group ambient, work is further subdivided into smaller chunks. Each individual validator would get assigned several ambients from the shard that validator group was assigned to. This should effectively allow even a small device to participate in validation increasing the total number of participants that briber would have to potentially coordinate with.

Channels outside the Ethereum ambient

To do this the validation group would create a new ambient that was connected by a channel to the validator group???s ambient. You might wonder how it is possible to link to an ambient outside of Ethereum. But underneath its straightforward.

Initially there would only be a validators account controlled by multisig on the Ethereum blockchain. Then the validators would create their own blockchain (represented as an ambient) which would have the same system ambients and Casper ambients as Ethereum. After creation, the validator group would connect the two ambients with a channel. Any message entering or exiting the ports the must be agreed upon by all the validators, so the channel should also be protected by a multisig. The code for the multisig would exist in the ports message handler. The channel could only be followed by those running both sets of ambients. Nodes running just the Ethereum ambient would see the channel but would not be able to follow it.

This provides a pattern that could be elsewhere as it provides a generic way to connect arbitrary ambients to the Ethereum blockchain. These ambients could stand for the state of your personal computer or an arbitrary feed of data. Beyond the examples given here, there are many other design patterns that make thinking in ambients useful. While there are still many lacunae ambients could be a useful model for computational environments. Ambients adds a new dimension to Ethereum???s hypervisor. Quite literally too. It allows for contract to be even more modular and provides for a convenient way to create administrative domains and model many everyday situations.

NOTES and PROBLEMS

SetRoot would have to fail if the root didn???t exist in the current namespace. If SetRoot was explicitly used the parent namespace (../<root>) then that tree would be copied to the namespace. If this happened between shards the tree would be serialized into a transaction.
Message
- All messages are assumed to be async. messages can timeout.
- Messages all have a response. The response need to be recoded as transaction on requesting shard and the responding shard.
- Blocks would need two parts; in transaction and out transactions.
Capture and delete - The sibling ambient sets a value to a path above another sibling with code for to create an ambient that deletes all of its sub-ambients.
- Solution 1 any action that might affect a sibling ambient must go through its message handler
- Solution 2 an ambient could define a message handle for all internal message that explicitly disallowed certain types of messages.
- Solution 3 reintroduce capabilities as presented in ambient calculus

Merklizing ASTs

2015-12-29T13:22:00Z

Special thanks to Juan Benet for mentioning this idea at Devcon 1.

Recently I wrote a draft of EIP 105 which propose using a subset of webassembly as Ethereum???s VM. If you aren???t aware webassembly ???is a new, portable, size- and load-time-efficient format suitable for compilation to the web.??? One interesting note about Webassembly doesn???t compile to linear byte code. Instead it uses an Abstract Syntax Tree (AST). This might not surprise you if you have an experience with LLVM IR. But for me it was a new concept to have bytecode in this form.

What is an AST?

It is just a tree repesentation of some code. Each node of the AST represents an expression. Each function body consists of exactly one expression. Compared to the source code, an AST does not include certain elements, such as inessential punctuation and delimiters (braces, semicolons, parentheses, etc.).

To give you a better idea here is a textual representation of some webassembly code using s-expressions.

  ;; Recursive factorial
  (func $factorial (param $i i64) (result i64)
    (if_else (i64.eq (get_local $i) (i64.const 0))
      (i64.const 1)
      (i64.mul (get_local $i) (call $factorial (i64.sub (get_local $i) (i64.const 1))))))

The AST of the above could be displayed like

figure 1 - an AST

This can then be serialized and sent across the wire.

Why use an AST?

The rationale that webassembly gives is

Trees allow a smaller binary encoding: JSZap[].
Polyfill prototype shows simple and efficient translation to asm.js.

Some auxiliary reason might be:

Effective to JIT
Code Deduplication

The idea has a bit of an interesting history. It appears that Michael Franz first used the idea to compress java bytecode in a paper on Slim Binaries. The slim binaries were also implemented in the Oberon OS

In addition to the Slim Binaries paper here are some more papers if you are interested in the subject.

Merklized ASTs

So another fun thing to do with AST is to merklize them! To review; a merkle tree is just a tree which links its nodes together by using the cryptographic hashes of the nodes.

The result is one root hash that points to the root node. This allows for efficient and secure verification of the contents of large data structures. To turn a AST into Merkle AST all you have walk from the leaf nodes up to the root hashing each node along the path.

Efficiency

If you look at figure one you can count 19 nodes. This is a fairly small program and the number of nodes in a larger program can add up fast. The computation time needed to merklize larger AST might start to add up rather fast. One method to increase efficiency would be to store entire subroutines in a single node and only create merkle tree branches at the point where the AST has a branch condition. For example see figure 3.

figure 3 - nodes contain entire subroutines

In figure 3 the green blocks represent subroutines that are in a single node. Note how the if else still forms a block by itself since it is a branch condition. This also provide an nice opportunity for parallelization. The interpreter or JIT can run the first branch while the next branch is still being fetched.

Why Merkle ASTs?

One reason is code security. It would make DLL hijack impossible. Of course how you would use DLL???s would be a bit different. Instead of open a dynamic link by a name you would reference the root hash of the routine you wanted to use. This way you have 100% confidence that you're getting the code that you want.

Have you ever had a software problem that you googled and found that 100s of forum posts with the same problem as you but none of their solutions worked for you? This is maybe in part caused by the fact you computer is in a different configuration or ???state??? then thiers. If we had secure merklized code this would a lot less common since the state of any given piece of software could be immediately determined or set by the root hash.

Perhaps one the most convincing reason is bandwidth saving and massive code deduplication. How many nearly-the-same versions of libraries you have on your harddrive? Or how many time the same function and routine is duplicated through programs?

Let's say you have you already have the green nodes since they are common subroutines. You would only have to download the orange nodes. Where this bandwidth saving could be very important is things like Ethereum light clients but also for general computation. As Web Pages begin to more and more resemble apps the larger their code size becomes. In the age of ephemeral webapps code is repeatedly downloaded many times. How many times do you think you have downloaded jquery? Couple this with a peer-to-peer distribution method like IPFS and I think you would have a very efficient system.

How to run contracts and create traces with Node.js

2014-08-12T00:00:00Z

A valid state transition is one which comes about through a transaction. Formally: ??_t+1 ??? ??(??_t , T ) where ?? is the Ethereum state transition function. In ???thereum, ??, together with ?? are considerably more powerful then any existing comparable system; ?? allows components to carry out arbitrary computation, while ?? allows components to store arbitrary state between transactions. - The Yellow Paper

At the core of Etheruem is the state transition function. Within the this function lies the ???thereum ???irtual Machine which upon ??????M code runs. In this post we will be running transactions thourgh the VM and looking at the results. A note on Terminology: the VM as descibed in the yellow paper is a subset of the state transition function. With an executioner that manipulates accounts and sets up the enviorment for the VM. Here we will just refer to the state transition function as the VM although this may not be techicanally correct. I did not like having an executioner in my code. It's a bit too barbaric for my taste. Altogether the VM in this sense handles all changes done to the state, which wholly resided in a trie.

This post will explore creating transactions and processing them with the VM. You can find all the code for this post here;

Running Contracts

To get stated we will be using two libraries. async and ethereum-lib
npm install async
npm install ethereum-lib

First import the necessary libraries and initailize some varibles.

var async = require('async'),
    Ethereum = require('ethereum-lib'),
    VM = Ethereum.VM,
    Account = Ethereum.Account,
    Transaction = Ethereum.Transaction,
    Trie = Ethereum.Trie,
    rlp = Ethereum.rlp
    utils = Ethereum.utils;

//creating a trie that just resides in memory
var stateTrie = new Trie();

//create a new VM instance
var vm = new VM(stateTrie);

//we will use this later
var storageRoot;

//the private/public key pare. used to sign the transactions and generate the addresses
var secretKey = '3cd7232cd6f3fc66a57a6bedc1a8ed6c228fff0a327e169c2bcc5e869ed49511';
var publicKey = '0406cc661590d48ee972944b35ad13ff03c7876eae3fd191e8a2f77311b0a3c6613407b5005e63d7d8d76b89d5f900cde691497688bb281e07a5052ff61edebdc0';

Lets set up the state trie. We need to give the sender's account enougth wei to send the transaction and run the code.

//sets up the initial state and runs  the callback when complete
function setup(cb) {
  //the address we are sending from
  var address = utils.pubToAddress(new Buffer(publicKey, 'hex'));

  //create a new account
  var account = new Account();

  //give the account some wei. 
  //This needs to be a `Buffer` or a string. all strings need to be in hex.
  account.balance = 'f00000000000000000';

  //store in the trie
  stateTrie.put(address, account.serialize(), cb);
}

Next we will create a transaction and process it. If you want to know more about generating transaction see this post In this example we will forgo generating a transaction from scratch and just use the some json I generated earlier. Transaction have a toJSON method if you want to do the same. This transaction contains the initializtion code for the name register.

var rawTx = {
  nonce: '00',
  gasPrice: '09184e72a000',
  gasLimit: '2710',
  data: '7f4e616d65526567000000000000000000000000000000000000000000000000003055307f4e616d6552656700000000000000000000000000000000000000000000000000557f436f6e666967000000000000000000000000000000000000000000000000000073661005d2720d855f1d9976f88bb10c1a3398c77f5573661005d2720d855f1d9976f88bb10c1a3398c77f7f436f6e6669670000000000000000000000000000000000000000000000000000553360455560df806100c56000396000f3007f726567697374657200000000000000000000000000000000000000000000000060003514156053576020355415603257005b335415603e5760003354555b6020353360006000a233602035556020353355005b60007f756e72656769737465720000000000000000000000000000000000000000000060003514156082575033545b1560995733335460006000a2600033545560003355005b60007f6b696c6c00000000000000000000000000000000000000000000000000000000600035141560cb575060455433145b1560d25733ff5b6000355460005260206000f3'
};

Since running a transaction in the VM is asynchronous lets wrap this in a function which we will use later. Running this transaction should create a new name register contract. It is important to note that vm.runTx() would also need a Block a the second parameter if the EVM code where to use any opcodes that accessed the block propeties.

//runs a transaction through the vm
function runTx(raw, cb) {
  //create a new transaction out of the json
  
  var tx = new Transaction(raw);
  
  tx.sign(new Buffer(secretKey, 'hex'));

  //run the tx
  vm.runTx(tx, function(err, results) {
    var createdAddress = results.createdAddress;
    //log some results 
    console.log('gas used: ' + results.gasUsed.toString());
    if (createdAddress) console.log('address created: ' + createdAddress.toString('hex'));
    cb(err);
  });
}

vm.runTx 1) checks if the sending account has enough wei to run then 2) runs the code and lastly 3) saves the changed accounts in the trie. It uses a callback to returns the results of running the transaction. We are only going to log some of the results here but if you are curious george you can look at the docs

Now lets actully run this!

//run the asynchronous functions in series
async.series([
    setup,
    async.apply(runTx, rawTx)
]);

So you should see how much gas the transaction used and the addrres of the created contract c8b97e77d29c5ccb5e9298519c707da5fb83c442. But not too exciting yet. Lets make sure that things worked. The VM should have created a new account for the contract in the trie. Lets make sure.

function checkResults(cb) {
  var createdAddress = new Buffer('692a70d2e424a56d2c6c27aa97d1a86395877b3a', 'hex');

  //fetch the new account from the trie.
  stateTrie.get(createdAddress, function(err, val) {

    var account = new Account(val);

    //we will use this later! :)
    storageRoot = account.stateRoot;

    console.log('------results------');
    console.log('nonce: ' + account.nonce.toString('hex'));
    console.log('blance in wei: ' + account.balance.toString('hex'));
    console.log('stateRoot: ' + storageRoot.toString('hex'));
    console.log('codeHash:' + account.codeHash.toString('hex'));
    console.log('-------------------');
    cb(err);
  });
}

Add this to async.series like so

async.series([
    setup,
    async.apply(runTx, rawTx),
    checkResults
]);

And run. We should see the details of the new created contract. Next thing we can do is send a message to our newly created contract. Lets create another transaction to do so. This transaction should register the sending address as "null_radix". The data field here is the RLP of pad_32_bytes('register') and pad_32_bytes('null_radix')

//This transaction should register the sending address as "null_radix"
var rawTx2 = {
  nonce: '01',
  gasPrice: '09184e72a000',
  gasLimit: '2710',
  to: '692a70d2e424a56d2c6c27aa97d1a86395877b3a',
  data: '72656769737465720000000000000000000000000000000000000000000000006e756c6c5f726164697800000000000000000000000000000000000000000000'
};

//run everything
async.series([
    setup,
    async.apply(runTx, rawTx),
    async.apply(runTx, rawTx2),
    checkResults
]);

So if everything went right we should have "null_radix" stored at "0x9bdf9e2cc4dfa83de3c35da792cdf9b9e9fcfabd". To see this we need to print out the name register's storage trie.

//reads and prints the storage of a contract
function readStorage(cb) {
  //we need to create a copy of the state root
  var storageTrie = stateTrie.copy();

  //Since we are using a copy we won't change the root of `stateTrie` 
  storageTrie.root = storageRoot;

  var stream = storageTrie.createReadStream();

  console.log('------Storage------');

  //prints all of the keys and values in the storage trie
  stream.on('data', function(data) {
    console.log('key: ' + data.key.toString('hex'));
    //remove the 'hex' if you want to see the ascii values
    console.log('Value: ' + rlp.decode(data.value).toString('hex'));
  });

  stream.on('end', cb);
}

//and finally 
//run everything
async.series([
    setup,
    async.apply(runTx, rawTx),
    async.apply(runTx, rawTx2),
    checkResults,
    readStorage
]);

After running you should see the contents of the contract's storage.

Debugging with traces

Lastly lets create a trace of the EMV code. This can be very usefully for debugging (and deherbing, don't smoke and code m'kay?) contracts.

The VM provides a simple hook for each step the VM takes while running EVM code.

//runs on each opcode
vm.onStep = function (info, done) {
    //prints the program counter, the current opcode and the amount of gas left 
    console.log('[vm] ' + info.pc + ' Opcode: ' + info.opcode + ' Gas: ' + info.gasLeft.toString());

    //prints out the current stack
    info.stack.forEach(function (item) {
        console.log('[vm]    ' + item.toString('hex'));
    });
    //important! call `done` when your done messing around
    done();
};

Now when you run you should see a complete trace. onStep provides an object that contians all the information on the current state of the VM.

the vm also provides a runBlock function that will process a block as well as a few other usefull functions. And now I'm out of stuff to say.

Creating Ethereum contracts and verifying messages with node.js

2014-06-14T23:39:00Z

In this article I'll be covering ethereum transaction creation and verifying transactions. First of all; whats the rational for creating transaction with with javascript? Well I, as always just want to explore and play around with ethereum. But I imagine this would be usefull in testing compilers and even more usefull, create contracts programmatically. So lets get started! :D

First of all we will be using ethereumjs-lib in this post. You can install it from npm npm install ethereumjs-lib or directly from git npm install git+https://github.com/ethereum/ethereumjs-lib

Creating a transaction

After that let create a new file and put the following in it. For the those how are cut and paste handicap here is the full example.

var Ethereum = require("ethereum-lib");
var Transaction = Ethereum.Transaction;

//create a blank transaction
var tx = new Transaction();

// So now we have created a blank transaction but Its not quiet valid yet. We
// need to add some things to it. Lets start with 
tx.nonce = 0;
tx.gasPrice = 100;
tx.gasLimit = 1000;
tx.to = 0; //Sending to 0, therefore we are creating a contract
tx.value = 0;
tx.data = "7f4e616d65526567000000000000000000000000000000000000000000000000003057307f4e616d6552656700000000000000000000000000000000000000000000000000573360455760415160566000396000f20036602259604556330e0f600f5933ff33560f601e5960003356576000335700604158600035560f602b590033560f60365960003356573360003557600035335700";

Hopefull this is pretty self explanatory. nonce is the number of transaction that the sending account has sent. gasPrice is the amount you are will to pay for gas. gasLimit if the amount of gas you are will to spend. to is the address we are sending to, in this case the zero address which creates a new contract. If we where to put a valid address here this account would become a message instead of a contract. To check which type of transaction you have, you can do

tx.type === "message"
//or
tx.type === "contract"
//type updates every time you change tx.to

value is the is the amount you are sending and data is data that you are sending in this case it is initialization code for a contract. The data is a Name Registrar contract. You can use one of the compilers to generate the code. The two main ones are lllc which is compiled with cpp-ethereum and serpent. Ok now we have a transaction with data. Now we need to sign it. To do this we will need a private key. There are several way to acquire one but for now we are just going to steal one from AlethZero. That way I know that I'm create valid transaction that actually has the ether that this transaction is says that it has. If you have AlethZero running. You can select tools>export selected key, and the copy the private key out. Here is mine.

var privateKey = new Buffer("e331b6d69882b4cb4ea581d88e0b604039a3de5967688d3dcffdd2270c0fd109");
tx.sign(privateKey);

Now we have a signed transaction, but for it to be valid, the account that we signed it with needs to have a certain amount of wei in to. To see how much that this account needs we can use tx.getUpFrontCost

console.log("Total Amount of wei needed:" + tx.getUpFrontCost());

this gives the amount in wei that the account needs to have. If your wondering how this is caculated it is

data lenght in bytes * 5
+ 500 Default transaction fee
+ gasLimit * gasPrice

Lets serialize the transaction

console.log("---Serialized TX----");
console.log(tx.serialize().toString("hex")); //serialize returns a buffer
console.log("--------------------");

Now that we have the serialized transaction we can get AlethZero to except it by selecting debug>inject transaction and pasting in the transaction serialization. It should then show up in the pending transaction pane.

Parsing & Validating transactions

If you have a transaction that you want to verify you can parse it. If you got it directly from the network it will be rlp encoded. You can decode it using the rlp module. After decoding it that you should have something like this.

var rawTx =  [
        "00",
        "09184e72a000",
        "2710",
        "0000000000000000000000000000000000000000",
        "00",
        "7f7465737432000000000000000000000000000000000000000000000000000000600057",
        "1c",
        "5e1d3a76fbf824220eafc8c79ad578ad2b67d01b0c2425eb1f1347e8f50882ab",
        "5bd428537f05f9830e93792f90ea6a3e2d1ee84952dd96edbae9f658f831ab13"
    ];

var tx = new Transaction(rawTx);

Note: rlp.decode will actully produce an array of buffers new Transaction will take either and array of buffers or and array of hex strings. So assuming that you were able to parse the transaction, we will now get the sender's address

console.log("Senders Address" + tx.getSenderAddress());

Cool now we know who sent the tx! Lets verify the signature to make sure it not some poser.

if(tx.verifySignature()){
    console.log("Signature Checks out!");
}

And hopefull its verified. For the transaction to be total valid we would also need to check the account of the sender and see if they have at least tx.getUpFrontCost().

And that is all for now I hope you have enjoyed the read. You can view the documentation for transaction here to see all the exciting things it can do. Let me know if you have any questions or comments.

Validating GeoJson from the command line

2014-05-26T13:22:00Z

This week I add a CLI to geojson validation module and will be going over how to use it in this post. To start, install it npm install geojson-validation -g.

Next you need some geoJSON. We will use this for the example. Now to check if it is valid simple run curl https://raw.githubusercontent.com/wanderer/Detroit-Farm-Map/master/data/map.geojson | gjv. If it was invalid gjv would print out the errors else it will just tell you its valid.

You can also validate local files like this gjv file1 files2 ...

Exploring Ethereum's state trie with Node.js

2014-05-21T13:22:00Z

If you not familiar with ethereum start here. In this post I will attempt to read cpp-ethereum's state db give a root using Node.js and the merkle-patricia-tree module. The merkle-patricia-tree module is hot off the press so there might be bugs. If you find any please let me know. The State DB is like a global ledger; It stores balances of all the accounts was well as the code for all of the contracts and their respective balances.

The global state is stored in a modified merkle patricia tree (trie), which you can read more about in the yellow paper or on the wiki. The important part is understanding what it is suppose to do, which is "The core of the trie, and its sole requirement in terms of the protocol specification is to provide a single 32-byte value that identifies a given set of key-value pairs."

Reading the db

cpp-ethereum stores the trie in a level db. On linux the path to the state db is ~/.ethereum/state. Let try just opening the database and seeing what we get. To do this we are going to need install the levelup module npm install levelup

We are going to look up one of the state roots. You can get them from althezero. In the example I'm using the genesis root hash from the latest cpp code from the dev branch. Looking up the root should give the top node in the trie.

var levelup = require('levelup');
var db = levelup('./.ethereum/state');

//the genesis state root
var root = '12582945fc5ad12c3e7b67c4fc37a68fc0d52d995bb7f7291ff41a2739a7ca16';

//Note: we are doing everything using binary encoding.
db.get(new Buffer(root, 'hex'), {
  encoding: 'binary'
}, function (err, value) {
  console.log(value);
});

We should get a print out of a Buffer... not necessarily usefully. Each node in the trie is rlp encoded, so let decoded the node. To do that we are going to need to install the rlp module npm install rlp. and add it the code

var rlp = require('rlp');

....

db.get(new Buffer(root, 'hex'), {
  encoding: 'binary'
}, function (err, value) {
  var decoded = rlp.decode(value);
  console.log(decoded);
});

Yay we got one node! (hopefully). Now the node decoded should print out an array of buffers. In our case we should get a branch node with is an array of the length 17. 1-16 are more nodes and 17 is a value.

Looking up Values in the Trie

Now lets try to look up a key. After looking up the root node finding a key is as simple as following the key path to the right leaf. I'm skimming over the details, but you can read the specs here. First install npm install merkle-patricia-tree. This module allows you to read and manipulate the trie. Lets try looking up Gav's, our patron saint of cpp programming, account. His address is "8a40bfaa73256b60764c1bf40675a99083efb075".

var Trie = require('merkle-patricia-tree');
var rlp = require('rlp');
var levelup = require('levelup');
var db = levelup('/home/null/.ethereum/state');

//the genesis state root
var root = '12582945fc5ad12c3e7b67c4fc37a68fc0d52d995bb7f7291ff41a2739a7ca16';
var trie = new Trie(db, root);

//gav's address
var gav = new Buffer('8a40bfaa73256b60764c1bf40675a99083efb075', 'hex');

trie.get(gav, function (err, val) {
  var decoded = rlp.decode(val);
  console.log(decoded);
});

The accounts are rlp encoded before they are stored so we need to decoded them after we retrieve them. After decoding them we should get an array of the length 4. Which are the four fields in an account, balance, nonce, stateRoot and codeHash

Streaming the Trie

The Last thing we want to do is to read out the entire trie. This might be handy for debugging or just poking around for fun. To do this we are going to use streams. If your not familiar with node streams here is a really fun place to get stated. Here the on data event returns an objectthat has two propeties; the key and the value. Both should be buffers.

var Trie = require('merkle-patricia-tree'),
  rlp = require('rlp'),
  levelup = require('levelup');

var db = levelup('/home/null/.ethereum/state');
var root = "12582945fc5ad12c3e7b67c4fc37a68fc0d52d995bb7f7291ff41a2739a7ca16"
var trie = new Trie(db, root);

//get a read stream object
var stream = trie.createReadStream();

stream.on('data', function (data) {
  console.log('key:' + data.key.toString('hex'));

  //accouts are rlp encoded
  var decodedVal = rlp.decode(data.value);
  console.log(decodedVal);
});

stream.on('end', function (val) {
  console.log('done reading!');
});

This should read out all the account's address and the account's contents in the trie.

The End?

merkle-patricia-tree module can do more than what I went over here. It can create, update and delete values too! amazing! And if you actually read all the way to the bottom, thanks for reading.