EOLink 0.1.1: Oracle GET request

If you like this content consider tipping in ETH, LINK or an EC20 stablecoin at this address: 0xf9f6849230cBe10200B43dB103511b778898e71C

Note, in this series I am using linux (Ubuntu 20.04) and Python 3.8 – some instructions will be specific to this OS/env

brownie Chainlink-mix

The first milestone I aimed for in the EOLink development journey was to code a smart contract on Ethereum that successfully uses a Chainlink oracle to make a simple http GET request and store the result at the contract address. For now, the actual data returned is arbitrary, the point was just to learn how to instruct a Chainlink oracle to make a GET request to an external API. Later, this will be replaced by a custom API endpoint, but not yet.

To do this, I used the brownie chainlink-mix which provides boilerplate code for making external API calls from smart contracts. The aim is to deploy contracts to the Kovan testnet and then interact with them using either Python scripts or the Brownie console. There is an excellent tutorial on this by Patrick Collins from the Chainlink Hackathon which I found extremely helpful. For this initial milestone, I am not deviating much from the example in the chainlink-mix, which uses an external API to get the 24 hour trading volume of ETH from an external API, because my goal is to understand how the contract and the oracle and the API interact in this simple model system.

The skeleton of the project can be initiated by “baking” the brownie mix in the terminal:

$ brownie bake chainlink-mix

I discarded the files that use pre-existing oracles and only kept the code relevant to external API calls. This yielded the following directory structure:

├── brownie-config.yaml
├── build
│   ├── contracts
│   │   ├── APIConsumer.json
│   │   ├── dependencies
│   │   │   └── smartcontractkit
│   │   │       └── chainlink-brownie-contracts@1.0.2
│   │   │           ├── BasicToken.json
│   │   │           ├── BufferChainlink.json
│   │   │           ├── CBORChainlink.json
│   │   │           ├── ChainlinkClient.json
│   │   │           ├── Chainlink.json
│   │   │           ├── ChainlinkRequestInterface.json
│   │   │           ├── ERC20Basic.json
│   │   │           ├── ERC20.json
│   │   │           ├── ERC677.json
│   │   │           ├── ERC677Receiver.json
│   │   │           ├── ERC677Token.json
│   │   │           ├── LinkTokenInterface.json
│   │   │           ├── LinkTokenReceiver.json
│   │   │           ├── PointerInterface.json
│   │   │           ├── SafeMathChainlink.json
│   │   │           ├── StandardToken.json
│   │   │           
│   │   │           
│   │   ├── LinkToken.json
│   │   ├── MockOracle.json
│   │   ├── MockV3Aggregator.json
│   │   └── VRFCoordinatorMock.json
|   |
│   ├── deployments
│   │   
│   ├── interfaces
│   │   └── LinkTokenInterface.json
|
├── contracts
│   ├── APIConsumer.sol
|
├── interfaces
│   └── LinkTokenInterface.sol
|
├── LICENSE
├── NOTES.txt
├── README.md
├── reports
├── requirements.txt
|
└── tests
    ├── conftest.py
    ├── test_api_consumer.py



brownie-config

brownie-config.yaml contains all the information necessary to configure the development environment for the project. I went through the template version and stripped out all the unneccessary information to make this as easy as possible to understand (I am only deploying on Kovan and in my local Ganache environment for now, so I stripped out everything relating to mainnet, mainnet forks and Binance Smart Chain). I am also using MetaMask to pay transaction gas (ETH) and oracle gas (LINK) and will do so by providing the private key for my test account and an environment variable, so I removed everything relating to mnemonic-based wallet access. In the end, my brownie-config.yaml file looked like this:

# exclude SafeMath when calculating test coverage
# https://eth-brownie.readthedocs.io/en/v1.10.3/config.html#exclude_paths

# first set environment variables including private key 
# (from metamask test account) and Infura Project ID

reports:
  exclude_contracts:
    - SafeMath

# external contracts required for code to function    
dependencies:
  - smartcontractkit/chainlink-brownie-contracts@1.0.2
  - OpenZeppelin/openzeppelin-contracts@3.4.0

# define compiler (remapping ensures prefix @openzeppelin links to
# the version downloaded in dependencies statement above 
# (same for @chainlinK)

compiler:
  solc:
    remappings:
      - '@chainlink=smartcontractkit/chainlink-brownie-contracts@1.0.2'
      - '@openzeppelin=OpenZeppelin/openzeppelin-contracts@3.4.0'


# automatically fetch contract sources from Etherscan
autofetch_sources: True

# enable python .env for setting environment variables
dotenv: .env

# define networks
networks:

  default: development # default to Ganache

  kovan: # contracts already exsting on Kovan network - find using link marketplace
    link_token: '0xa36085F69e2889c224210F603D836748e7dC0088'
    keyhash: '0x6c3699283bda56ad74f6b855546325b68d482e983852a7a82979cc4807b641f4'
    fee: 100000000000000000
    oracle: '0x2f90A6D021db21e1B2A077c5a37B3C7E75D15b7e'
    jobId: '29fa9aa13bf1468788b7cc4a500a45b8'

# only enable private key wallet access
wallets:
  from_key: ${PRIVATE_KEY}

There are a couple of things in this file that are worth explaining.

First there are dependencies which are pretty straightforward- these are just external contract files that are required by some of the project code – we need to define them here so they are available to import. Then there is some configuration relating to the compiler. “solc” refers to “solidity compiler” and here we define some remappings – this effectively just aliases some keywords prepended by the @ symbol to point to specific imports. In this case the tag @chainlink maps to “smartcontractkit/chainlink-brownie-contracts@1.0.2” and the tag @openzeppelin maps to “OpenZeppelin/openzeppelin-contracts@3.4.0”. This provides us with some shorthand that ensures the compiler is internally consistent with versions of the imports from Chainlink and OpenZeppelin .

Autofetch can be toggled on to instruct brownie to automatically search for source code where addresses that have not been explicitly defined in the project. dotenv enables brownie to use Python .env files to set environment variables, a useful alternative to manually setting them in the terminal or running a bash file on project startup.

The next section defines the network that the project will communicate on. I have set the default to “development” which is a local Ganache network (a local blockchain not connected to any public blockchain). As the Ganache blockchain only exsts within the domain of the computer running it, it does not have access to external data oracles, so a full integration test of the project would require “mock oracles” to be deployed locally. Then, in this minimal example I have only defined one other network – Kovan. Kovan is a public Ethereum testnet watched by real Chainlink oracles. The values defined under ‘kovan’ are all addresses for smart contracts living on the Kovan testnet. These smart contracts are those related to the Chainlink oracle making our GET request, and they were found by browsing the Chainlink marketplace.

Finally, under wallet we have the option to generate from a private key or a mnemonic (seed phrase) provided to brownie as environment variables. I have only used the private key option here. This will be covered in more detail below in the section about setting up MetaMask and providing brownie access to the MetaMask account.

For now, all other config options get their default values. The comprehensive list of config variables and their meanings can be found here.

Setting up Metamask

Metamask is a browser extension that allows your existing Web2 browser to transact with blockchains. It provides you with wallets in which to hold digital assets and interact with smart contracts. I am using Firefox – MetaMask also works with Chrome and Brave. To install MetaMask in Firefox use this link. When MetaMask has been successfully installed, a fox-head logo will appear in the upper right of the browser window.

Many people use MetaMask for developing dapps as well as holding and transacting ETH and ERC20 tokens on Ethereum mainnet. This means there is risk of a) accidentally sending real assets to a tesnet address, b) accidentally exposing private keys or mnemonics in project code uploaded to github or other public repositories. Either of these errors could lead to irreversible loss of real assets, so they are really important to avoid.

I started a fresh account inside MetaMask that will only be used to develop this project. I named the account “Kovan Test Wallet” for complete clarity. To do this, open MetaMask and click on the circle-logo in the upper right of the GUI. In the menu, choose “create account”. The new account that is generated will have its own address and its private key can be accessed by opening the three-dot menu positioned to the right of the account name, clicking “account details” >> “export private key”.

This private key is required by brownie – it enables a wallet to send assets and signatures to smart contracts. To make it available to the brownie project, there are several options.

  • Manually provide it as an environment variable

In the terminal, environment variables can be set using the export command. In this project we want to use the following syntax (this is consistent with the brownie-config.yaml “wallet” settings”.

$ export PRIVATE_KEY=0x.....
  • Add it to the project .env file

The same syntax used directly in the terminal above can be used inside a text file saved as .env in the top-level directory. This file must not be included in any git commits or it will become publicly available and vulnerable to hacks. The .env file should be added to .gitignore. Both .gitignore and .env are hidden files in the top level directory by default.

  • Register it locally as a brownie account

This is my preferred method as it password protects the account and gives added functionality within brownie console to query the account properties, such as its ETH balance, and enables simple transfers out of the account into other wallets or smart contracts. To do this, we first start the brownie console, then add the account’s private key to brownie. I chose to name my account “main” in this example.

$ brownie accounts new main

brownie then requests the account private key which can be copy and pasted into the terminal, then a password which will be required to unlock the account. More details are available in the brownie docs here.

Now, the account is ready to transact with smart contracts. Before using the account we must make sure Metamask is connected to the Kovan testnet, not Ethereum mainnet. Then, we need some tokens to fund transactions.

In the brownie console, this account can then be loaded using

# loads main account to var "account"

account = accounts.load('main')

and then queried, for example:

# returns ETH balance in wei

account.balance()

INFURA PROJECT ID

As well as a wallet private key, the project also needs to be connected to an Infura project ID in order to connect to the Kovan network. Getting an Infura ID is well explained here. This project ID then has to be added to the project environment variables in the same way(s) as the wallet private key, using the following syntax:

$export WEB3_INFURA_PROJECT_ID='7bg...'

Getting test eth / test Link

Since this is a training exercise, I don’t want to spend real assets on gas for transactions, but I do want to simulate the gas transactions that will be required if the project is deployed on mainnet. Gas is the term used to describe the cost of changing that state of the blockchain – the more on-chain work is done by a transaction the more gas it costs. On Etheruem, gas is paid in ETH and the cost of a transaction varies according to demand for block-space. At the same time, oracles also require payment to transfer data to the chain. This “oracle gas” is paid in LINK tokens. This oracle-native token allows the Chainlink oracles to be blockchain-agnostic (if they charged gas in ETH they would only work on Ethereum). This all need to be simulated in our test environment. Thankfully, the Kovan testnet has “faucets” for both ETH and LINK, where tokens can be requested that power transactions on the Kovan network but have no real-world value.

Obtaining ETH from the Kovan Eth Faucet requires login via Github. Providing valid Github credentials leads to a page where the wallet private key can be provided. After a minute or two, 2 ETH will appear in the wallet, which can be verified by opening MetaMask or querying the account balance via brownie console.

Obtaining Kovan LINK requires an additional step because the wallet created in MetaMask dos not display LINK by default, it must be added to the wallet using the “add token” button. The Kovan test-link is available at this adress: 0xa36085F69e2889c224210F603D836748e7dC0088 which should be provided in the “add token” menu in MetaMask. Once added, we can grab test LINK from the Kovan LINK faucet by providing the wallet address. 100 LINK will then be added to the account.

Now an account is available, connected to Kovan and known in the brownie environment with plenty of funds to pay gas for transactions with smart contracts.

The contract

The contract to deploy to the blockchain is called “APIConsumer.sol”. This is a Solidity file that encodes all the instructions for sending funds to and receiving data from the oracle. This is also where transactions between accounts will be coded for example when insurance payouts are automated based on the value returned by the oracle. See more details in the Chainlink docs.

Like all Solidity files, the first line is a pragma statement, which defines which compile version to use. The GET request is made via a Chainlink oracle, which requires a specific struct (chainlink.request) that is inherited from ChainLinkClient, which is imported in the second statement in the contract.

Then we define our contract using the syntax “contract <name> is ChainlinkClient”. The keyword “is” is a Solidity inheritance instruction that signifies that the contract we are creating should inherit the properties of the imported ChainLinkClient contract. The request function requires four arguments: oracle address, jobID, fee and link token address. These were all defined in brownie-config.yaml and will be required arguments for transactions sent to the contract after it is deployed. These arguments are defined in the contract with a data type, but not instantiated, immediately before the constructor definition. An additional variable, “volume” is also defined there, which is the variable that will contain the data returned from the oracle – this must have identical data type to that returned by the oracle (this can be checked on the chainlink marketplace for the specific oracle/job linked to in the config).

Having made the necessary imports and defined the relevant variables, the constructor function is defined. A constructor is a function that runs once and only once when the contract operates, and it usually serves to instantiate variables with values passed to the contract by the transaction. In the contract constructor shown below, the LINK token address is instantiated first using one of two functions inherited from ChainLinkClient. There is a conditional statement that checks whether the LINK token address passed to the contract is the “zero address”. In Solidity the zero address is a transaction with all properties set to zero and in this case indicates that no valid LINK address information has been provided. In this case, the LINK address is automatically retrieved for the specific network the contract is being deployed on using the setPublicChainLinkToken() function from ChainLinkClient. Alternatively, if a valid LINK token address has been supplied, it is passed to a different function, setChainlinkToken() – either way the LINK token address for the Kovan network will be ingested into the contract. Then, the variables oracle, jobID and fee are instantiated with the values passed to the contract during a transaction ( we will see this in action in later sections).

// SPDX-License-Identifier: MIT
pragma solidity ^0.6.6;

import "@chainlink/contracts/src/v0.6/ChainlinkClient.sol";

contract APIConsumer is ChainlinkClient {
  
    uint256 public volume;
    
    address private oracle;
    bytes32 private jobId;
    uint256 private fee;
    

    constructor(address _oracle, string memory _jobId, uint256 _fee, address _link) public {
        if (_link == address(0)) {
            setPublicChainlinkToken();
        } else {
            setChainlinkToken(_link);
        }

        oracle = _oracle;
        jobId = stringToBytes32(_jobId);
        fee = _fee;
    }
    

After the contract has been created with the necessary imports and the constructor function defined, the only remaining task is to build the GET request function. This requires functions inherited from ChainLinkClient.

   function requestVolumeData() public returns (bytes32 requestId) 
    {
        Chainlink.Request memory request = buildChainlinkRequest(jobId, address(this), this.fulfill.selector);
        
        // Set the URL to perform the GET request on
        request.add("get", "https://min-api.cryptocompare.com/data/pricemultifull?fsyms=ETH&tsyms=USD");
        
        request.add("path", "RAW.ETH.USD.VOLUME24HOUR");
        
        // Multiply the result by 1000000000000000000 to remove decimals
        int timesAmount = 10**18;
        request.addInt("times", timesAmount);
        
        // Sends the request
        return sendChainlinkRequestTo(oracle, request, fee);
    }
    

    function fulfill(bytes32 _requestId, uint256 _volume) public recordChainlinkFulfillment(_requestId)
    {
        volume = _volume;
    }

    }

First, a function called requestVolumeData() is defined. This is the function that will later be called by a transaction sent to the contract, and it should trigger a GET request via the Chainlink oracle and save the result in the contract where it can be accessed by external users. This needs to be a public function so it can be accessed by external users. The return value is surprisingly not the value we request from the oracle, but the ID number of the request made to the oracle, which is a value of type bytes32. A second function later uses this requestID to get the actual data.

Inside requestVolumeData() the URL where the GET request should point is defined – in this case it is https://min.api.cryptocompare.com/&#8230; but there are many other crypto price oracles with APIs that could have been accessed instead. The syntax “request.add(request_type, URL)” is used to format the request. The URL passed to the request actually returns a large amount of json data, of which we only need one value. The data at the URL can be viewed by pasting the URL into a browser:

To isolate only the required value rather than ingesting all of this json data, the json path is added to the request, essentially “walking” down the structure similar to defining the path to a file in several nest directories in a “cd” bash command. In this case the path is RAW >> ETH >> USD >>VOLUME24HOUR, which is formatted in the request by separating each label with a stop. Now, only the 24 hour trading volume for ETH will be returned.

Solidity does not use decimals and the request will return a value in wei rather than ETH. To convert from wei to ETH the value should be multiplied by 10**18, which is achieved using the request.addInt() function.

Intuitively, this feels like the end of the process because the request has been sent to the oracle and the resulting data converted to a more readable format, but actually all that has been done to this point is to send the request configuration to the oracle – a second function is required to retrieve the resulting data from the oracle and ingest it back into the contract. This is the “fulfill()” contract defined at the bottom of the above code snippet. The requestID is used to identify which of many requests made to the oracle is the one needed by this contract. The resulting data is finally stored in the “volume” variable, where it can be accessed by external users transacting with the contract.

NB for a real contract we would also include some further security a “withdraw” function should also be included in this contract to return unspent LINK tokens back to the senders wallet, otherwise they are permanently locked inside the contract.

Deploying contracts

With the development environment, wallet and test tokens set up and the contract written, it is time to deploy the contract to the blockchain. The chainlink-mix comes with a Python script for this purpose (scripts/deploy_api_consumer.py). This file looks like this:

#!/usr/bin/python3
from brownie import APIConsumer, accounts, config, network

def main():
    dev = accounts.add(config["wallets"]["from_key"])
    return APIConsumer.deploy(
        config["networks"][network.show_active()]["oracle"],
        config["networks"][network.show_active()]["jobId"],
        config["networks"][network.show_active()]["fee"],
        config["networks"][network.show_active()]["link_token"],
        {"from": dev},
        publish_source=False,
    )

This code imports the APIconsumer smart contract along with the account, config and network information we have already defined. Then, inside the “main” function:

  1. grab the active account (the wallet we set up earlier) and assign to var “dev”
  2. use the built-in “deploy” function to migrate the smart contract onto the blockchain

The deployment requires information from our brownie-config file which is accessed by walking down the heirarchy of headings in sequential indexes to the config argument, for example to grab the address of the oracle:

config["networks"][network.show_active()]["oracle"]

We can trace these indexes through our brownie config file – first look for heading “networks”, then we index to network.show_active() which is a function call that returns the name of the network we are connected to – in this case “kovan”, then under “networks>>kovan” we will find an address associated to the variable “oracle”. Here is the same information as it is presented in brownie-config.yaml:

...

networks:

  default: development # default to Ganache

  kovan: # contracts already exsting on Kovan network - find using link marketplace
    link_token: '0xa36085F69e2889c224210F603D836748e7dC0088'
    keyhash: '0x6c3699283bda56ad74f6b855546325b68d482e983852a7a82979cc4807b641f4'
    fee: 100000000000000000
    oracle: '0x2f90A6D021db21e1B2A077c5a37B3C7E75D15b7e'

...

In the deployment script we repeat this process for each of the necessary function arguments: “oracle”, “jobID”, “fee”, “link_token”.

This could also be achieved by coding directly in brownie console rather than scripting it. While scripts are better for re-useability, I found it quite instructive to follow the process step-by-step in the console myself. To do this, first open brownie console and load the wallet account:

$brownie console --network kovan

$account = accounts.load('main')

Since the APIconsumer.sol contract is available in the project’s contracts folder it can be accessed via the console, and its “deploy” function called. The necessary keyword arguments are the same as those defined in the Python script above: “oracle”, “jobID”, “fee”, “link_token” and we also need to define who is sending the transaction, with {“from”:account}.

$APIconsumer.deploy(config["networks"]["kovan"]["oracle"], config["networks"]["kovan"]["jobID"], config["networks"]["kovan"]["fee"], config["networks"]["kovan"]["link_token"], {"from":account})

With this command, we push the smart contract onto the Kovan testnet. If we want to check that it has worked, we first check that some ETH has been spent from our wallet (open MetaMask or use account.balance() in the brownie console) and we can paste the address that the contract was deployed to into the Kovan etherscan to see it “in the wild”.

Funding contract

To deploy the contract gas was automatically paid in ETH, but to make an oracle request we have to make LINK available. This means pre-funding the smart contract with LINK tokens to send to the oracle when the GET request is made. Neglecting to do this will result in a failed interaction with the oracle, and wasted transaction gas.

There are a couple of ways to fund the contract. When we deployed the contract we received its address. We can therefore simply open MetaMask, click on LINK >> send and provide the contract address. The actual fee is provided in the oracle details which we have already copied into our config file, but we can oversupply the contract for now by sending 10 LINK. This will give ample funds for several GET requests while developing the project.

Alternatively, the contract can be funded using brownie console. To do this, the link token address is needed (grab from the config file) along with the contract address (which as displayed when the contract was deployed). Then to transfer the tokens the transfer function defined in the LinkTokenInterface is called as follows:

# define address for link token
# (the link token address could be pasted directly instead of grabbing from config)
link_address = config["networks"]["kovan"]["link_token"]

# define address for contract
contract_address = "0x765...."

# make transfer
interface.LinkTokenInterface(link_address).transfer(contract_address, 1000000000000000000, {"from":account}

Note that the amount of LINK is presented as the desired amount multiplid by 1e18, equivalent to providing wei for an ETH transfer. This process can also be scripted, as in the file scripts/fund_chainlink_api.py

Making GET request

With the contract deployed and funded, it is now ready to receive transactions and make its GET request. Looking back at the APIConsumer.sol file, the function we defined to make the GET request was named “requestVolumeData()”. This is accessible in brownie console after instantiating the deployed contract:

# create instance of Contract class with the deployed address
contract = Contract('0x8aaC2CEa448c8E1BBa80F0e421212d6909cA77D5')

# call requestVolumeData function 
transaction = contract.requestVolumeData({"from":account})

Accessing resulting data

The request made above returns a transaction receipt to the variable “transaction”. This object contains lots of information about the transaction itself, but not its result. For example, “transaction” contains information about how much gas was used in the transaction, which block number it was assigned to, a transactiob ID, etc. A full trace of the transaction is also stored inside this transaction object. It can be explored in brownie console:

$ transaction.block_number
>>> 24976903

$ transaction.gas_used
>>> 133743

$ transaction.traceback
>>> <bound method TransactionReceipt.traceback of <Transaction '0xe8710d344e455e4abb1a3c6eaec6d42c0d96b5cb45ac99d3b301dd4237fec380'>>

The point of the contract is to return the 24 hour transaction volume for ETH. This value has been generated by the oracle, but it is not found in the transaction object, it is found in the contract object. This is because it is the contract that makes the data request to the oracle, and therefore where the oracle sends the data back to. That means the dtaa we requested can be accessed by calling the “volume” method on the contract object:

$ volume = contract.volume()
>>> 1675010120955040000000000

So, this demonstrates that a smart contract was successfully deployed to the Kovan testnet, transactions sent from a local console which initiated a data request to a Chainlink oracle. Gas was paid in ETH for updating the state of the blockchain and in LINK for an oracle job. The data from the oracle was from an external API and it was passed back to the deployed smart contract, where it became accessible from the brownie console.

6 thoughts on “EOLink 0.1.1: Oracle GET request

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s