Keep Network, local setup

This post will guide you how to setup Keep Network’s local environment. It will follow the original guidance but in more details. At the time of writing this guide I used CentOS 8, so it aims that OS, but I tried to cover the other ones too with links.

Install prerequisites

These are Go, Geth, Solidity compiler, Protobuf compiler, protoc-gen-gogoslick, jq, Docker, Docker compose, pyenv, pipenv.

Let’s start with Go. I’m mainly a Go developer so I prefer installing it with the tar.gz. It is much easier to update in the future for different versions. At the moment the version of Go is 1.15.8, but check out newer versions under the link.

wget https://golang.org/dl/go1.15.8.linux-amd64.tar.gz
tar -C /usr/local -xzf go1.15.8.linux-amd64.tar.gz
export PATH=$PATH:/usr/local/go/bin
go version

This will download the tarball and place it in the /usr/local, so we can add the binaries to the path. Keep in mind that you have to add the export to the .bashrc, so it will do the PATH update on each bash start.

That’s easy because Geth has installer for nearly all of the platforms. You can check what you use and pick your installations process. If you use RHEL based system just like me, then compile it. Fortunately we have Go installed on our system.

wget https://github.com/ethereum/go-ethereum/archive/v1.9.25.zip
unzip v1.9.25.zip
cd go-ethereum-1.9.25/
go get # requires gcc
go mod vendor
go install github.com/ethereum/go-ethereum/cmd/geth

At this point you have a compiled Geth, but there is a missing GOPATH bin from the PATH. Go places the GOPATH in your home folder so you can set it up with the following command:

export PATH=$PATH:$HOME/go/bin
geth version

Install docker first because it will be useful in the next section. You can find the installation guides for different OS environments here. I will present the CentOS installation.

sudo yum install -y yum-utils
sudo yum-config-manager \
--add-repo \
https://download.docker.com/linux/centos/docker-ce.repo
sudo yum install docker-ce docker-ce-cli containerd.io
sudo systemctl enable --now docker
# Verify installation
docker version
docker ps # should get the empty list of containers

This is where the pain comes, because solcjs is not solc and in this case we need solc. If you want to run the Keep’s code with solcjs you will get several unexpected errors. The funny part of that is I didn’t find a proper way to get the real solc for CentOS so I hacked it out from the ethereum/solc docker container. Other OS environments installations can be found here.

Keep in mind that the submodules require Solidity compiler 0.5.17, so you have to modify the Dockerfile. For this let’s open the Dockerfile and modify the FROM part to ethereum/solc:0.5.17-alpine.

wget https://github.com/PumpkinSeed/solc-rhel/archive/v0.1.0.zip
unzip v0.1.0.zip
cd solc-rhel-0.1.0
sh install.sh
# Verify installation
solc --version
# Output:
# solc, the solidity compiler commandline interface
# Version: 0.5.17+commit.d19bba13.Linux.g++

Go with an easier one. I’m sure you can find installation for your own environment.

sudo yum -y install epel-release
sudo yum -y install jq
# Verify installation
jq -Version

That won’t be easy, because the local setup requires at least 3.11.4 and I didn’t find RPM for that version. So I had to install it from source, which took time…

sudo yum install -y autoconf automake libtool curl make gcc-c++
wget https://github.com/protocolbuffers/protobuf/releases/download/v3.14.0/protobuf-all-3.14.0.zip
unzip protobuf-all-3.14.0.zip
cd protobuf-3.14.0/
./autogen.sh
./configure
make
make check
sudo make install
sudo ldconfig
# Verify installation
protoc --version
# libprotoc 3.14.0

The hardest parts of the installations are already done.

This is an extension for the Go protobuf which helps faster operation over Go code generation.

go get github.com/gogo/protobuf/protoc-gen-gogoslick

I will show the CentOS based installation, but you can find different installation methods here.

sudo curl -L "https://github.com/docker/compose/releases/download/1.28.2/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-composesudo chmod +x /usr/local/bin/docker-compose# Verify installation
docker-compose --version
# docker-compose version 1.28.2, build 67630359

Python version management. Other OS installation. Keep in mind that the Development Tools quotes can differ from your layout, so the yum can warn you about it.

sudo yum groupinstall -y "Development Tools"sudo yum install -y zlib zlib-devel bzip2-devel openssl-devel sqlite-devel readline-develcurl -L https://github.com/pyenv/pyenv-installer/raw/master/bin/pyenv-installer | bash# Load pyenv automatically by adding
# the following to ~/.bashrc:
export PATH="/root/.pyenv/bin:$PATH"
eval "$(pyenv init -)"
eval "$(pyenv virtualenv-init -)"

I think this is totally unnecessary but I installed it anyways.

sudo yum install -y python3-pip
pip3 install pipenv

At this point we have all of the necessary tools so we can start to bootstrap and setup the Keep Network. Before we jump into this, let’s install one other tool which can help us a lot. Note that it’s only necessary if we want to have control over all parts of the system.

This will help us have control over the different moving parts and the ability to organize the running parts.

sudo yum install -y tmux

Setting up the Keep Network local environment

First let’s get the local-setup repository and start to use it. We have to clone it, because there are steps where we fetch the submodules.

git clone https://github.com/keep-network/local-setup.git
cd local-setup/

Let’s start a tmux session with the tmux command. If you are not familiar with tmux here’s a quick tutorial. It’s not mandatory, you can open more terminals and run the several commands on them or you can send them to the background, but tmux helps you organizing it and you can have control over everything.

So open 3 panes. That should be enough for this session and we will initialize and start a local Ethereum and Bitcoin chain.

./initialize-geth.sh

This script sets up the Ethereum node data location. It will be under $PWD/ethereum/data. By using this variable it will call geth init with the genesis files under $PWD/ethereum/genesis/genesis.json. So the geth data dir is initialized. We can safely start the geth instance. We can do it in the same tmux pane where we had initialized it.

./run-geth.sh

This sets up the same data directory like initialize does. Then it will start the geth long running process. This means that a single-node ethereum runs on our machine. The script starts Websocket and RPC endpoints as well. The mining also starts because this will execute the actual transactions. More details can be found in the shell script.

Since the ethereum is a long running process we move to the second tmux pane and start the bitcoin chain.

./run-bitcoin.sh

At this point I don’t want to go into details but this script starts the docker-compose file under the bitcoin directory. Both Bitcoin daemon and ElectrumX have their own Dockerfile.

So we also get a long running process with this script so we have to move to the third tmux pane.

3 pane of the system

We can run the install shell script like I have done in the picture above. There is a small chance that you haven’t install npm on your system, but it’s required for this script.

sudo yum install -y npm
./install.sh
# ....
# You should see this line at the and with the list of submodules:
# Installation script executed successfully...

This will take time… (probably hours).

We want to check out what happens inside the script. We will see that the first one is updating the corresponding submodules. You can find these submodules in the .gitmodules file. Then it calls the submodule installers. Install keep-core, keep-ecdsa, tbtc, tbtc-dapp and keep-dashboard. Since it compiles solidity and does npm install with a huge amount of C++ compilation, it takes time.

So at this point we have ethereum and bitcoin local nodes and the bootstrapped keep system. We have to start the keep related subsystems.

Setup Keep Network clients locally

Open a new tmux session with 4 panes and start the keep clients. 1 core and 3 ecdsas to make it possible to use its consensus algorithm.

# 1. pane:
./run-core-1.sh
# 2. pane:
./run-ecdsa-1.sh
# 3. pane:
./run-ecdsa-2.sh
# 4. pane:
./run-ecdsa-3.sh
Running clients

That was easy to set up. If they are running without any problem you should be good. Note that keep-ecdsa can burn the CPU. It uses all of my cores with ~90% load.

Keep-ecdsa CPU usage

We have to do a genesis block on Keep network and then we can start the dashboard.

cd keep-coreKEEP_ETHEREUM_PASSWORD="password" ./keep-core --config configs/config.local.1.toml relay genesis

Going back to the root of the repository and run the dashboard.

cd keep-core/solidity/dashboard
npx browserslist@latest --update-db
cd ../..
./run-keep-dashboard.sh

This will start a web application on port 3000. If you open it you should see something like the image below.

Keep dashboard

The last things are the end to end tests. First we have to install and switch to a newer version of node by using the node version manager.

curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.35.3/install.sh | bash
source ~/.bashrc
nvm install 14.3.0cd e2e && npm install
cd ..
./run-e2e-test.sh

Yes, I know the shell scripts also contain the npm install, but it always runs into issues until I run it myself.

Ideally it should pass and basically we can start the development.

It is hard to write a guide like this since there lots of moving parts. If you run into an issue at any point which I haven’t covered in this article, please feel free to message me on twitter.

https://twitter.com/PumpkinSeed_dev

Gopher, Rustacean, Hobby Hacker

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store