Keep Network, local setup
This post will guide you how to setup Keep Network’s local environment. It will follow the original guidance but in more details. At the time of writing this guide I used CentOS 8, so it aims that OS, but I tried to cover the other ones too with links.
Install prerequisites
These are Go, Geth, Solidity compiler, Protobuf compiler, protoc-gen-gogoslick, jq, Docker, Docker compose, pyenv, pipenv.
Go
Let’s start with Go. I’m mainly a Go developer so I prefer installing it with the tar.gz. It is much easier to update in the future for different versions. At the moment the version of Go is 1.15.8, but check out newer versions under the link.
wget https://golang.org/dl/go1.15.8.linux-amd64.tar.gz
tar -C /usr/local -xzf go1.15.8.linux-amd64.tar.gz
export PATH=$PATH:/usr/local/go/bin
go versionThis will download the tarball and place it in the /usr/local, so we can add the binaries to the path. Keep in mind that you have to add the export to the .bashrc, so it will do the PATH update on each bash start.
Geth
That’s easy because Geth has installer for nearly all of the platforms. You can check what you use and pick your installations process. If you use RHEL based system just like me, then compile it. Fortunately we have Go installed on our system.
wget https://github.com/ethereum/go-ethereum/archive/v1.9.25.zip
unzip v1.9.25.zip
cd go-ethereum-1.9.25/
go get # requires gcc
go mod vendor
go install github.com/ethereum/go-ethereum/cmd/gethAt this point you have a compiled Geth, but there is a missing GOPATH bin from the PATH. Go places the GOPATH in your home folder so you can set it up with the following command:
export PATH=$PATH:$HOME/go/bin
geth versionDocker
Install docker first because it will be useful in the next section. You can find the installation guides for different OS environments here. I will present the CentOS installation.
sudo yum install -y yum-utils
sudo yum-config-manager \
--add-repo \
https://download.docker.com/linux/centos/docker-ce.repo
sudo yum install docker-ce docker-ce-cli containerd.io
sudo systemctl enable --now docker# Verify installation
docker version
docker ps # should get the empty list of containers
Solidity compiler
This is where the pain comes, because solcjs is not solc and in this case we need solc. If you want to run the Keep’s code with solcjs you will get several unexpected errors. The funny part of that is I didn’t find a proper way to get the real solc for CentOS so I hacked it out from the ethereum/solc docker container. Other OS environments installations can be found here.
Keep in mind that the submodules require Solidity compiler 0.5.17, so you have to modify the Dockerfile. For this let’s open the Dockerfile and modify the FROM part to ethereum/solc:0.5.17-alpine.
wget https://github.com/PumpkinSeed/solc-rhel/archive/v0.1.0.zip
unzip v0.1.0.zip
cd solc-rhel-0.1.0
sh install.sh# Verify installation
solc --version
# Output:
# solc, the solidity compiler commandline interface
# Version: 0.5.17+commit.d19bba13.Linux.g++
jq
Go with an easier one. I’m sure you can find installation for your own environment.
sudo yum -y install epel-release
sudo yum -y install jq# Verify installation
jq -Version
Protobuf compiler
That won’t be easy, because the local setup requires at least 3.11.4 and I didn’t find RPM for that version. So I had to install it from source, which took time…
sudo yum install -y autoconf automake libtool curl make gcc-c++
wget https://github.com/protocolbuffers/protobuf/releases/download/v3.14.0/protobuf-all-3.14.0.zipunzip protobuf-all-3.14.0.zip
cd protobuf-3.14.0/
./autogen.sh
./configure
make
make check
sudo make install
sudo ldconfig# Verify installation
protoc --version
# libprotoc 3.14.0
The hardest parts of the installations are already done.
protoc-gen-gogoslick
This is an extension for the Go protobuf which helps faster operation over Go code generation.
go get github.com/gogo/protobuf/protoc-gen-gogoslickdocker-compose
I will show the CentOS based installation, but you can find different installation methods here.
sudo curl -L "https://github.com/docker/compose/releases/download/1.28.2/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-composesudo chmod +x /usr/local/bin/docker-compose# Verify installation
docker-compose --version
# docker-compose version 1.28.2, build 67630359
pyenv
Python version management. Other OS installation. Keep in mind that the Development Tools quotes can differ from your layout, so the yum can warn you about it.
sudo yum groupinstall -y "Development Tools"sudo yum install -y zlib zlib-devel bzip2-devel openssl-devel sqlite-devel readline-develcurl -L https://github.com/pyenv/pyenv-installer/raw/master/bin/pyenv-installer | bash# Load pyenv automatically by adding
# the following to ~/.bashrc:
export PATH="/root/.pyenv/bin:$PATH"
eval "$(pyenv init -)"
eval "$(pyenv virtualenv-init -)"
pipenv
I think this is totally unnecessary but I installed it anyways.
sudo yum install -y python3-pip
pip3 install pipenvAt this point we have all of the necessary tools so we can start to bootstrap and setup the Keep Network. Before we jump into this, let’s install one other tool which can help us a lot. Note that it’s only necessary if we want to have control over all parts of the system.
tmux
This will help us have control over the different moving parts and the ability to organize the running parts.
sudo yum install -y tmuxSetting up the Keep Network local environment
First let’s get the local-setup repository and start to use it. We have to clone it, because there are steps where we fetch the submodules.
git clone https://github.com/keep-network/local-setup.git
cd local-setup/Installing the system
Let’s start a tmux session with the tmux command. If you are not familiar with tmux here’s a quick tutorial. It’s not mandatory, you can open more terminals and run the several commands on them or you can send them to the background, but tmux helps you organizing it and you can have control over everything.
So open 3 panes. That should be enough for this session and we will initialize and start a local Ethereum and Bitcoin chain.
./initialize-geth.shThis script sets up the Ethereum node data location. It will be under $PWD/ethereum/data. By using this variable it will call geth init with the genesis files under $PWD/ethereum/genesis/genesis.json. So the geth data dir is initialized. We can safely start the geth instance. We can do it in the same tmux pane where we had initialized it.
./run-geth.shThis sets up the same data directory like initialize does. Then it will start the geth long running process. This means that a single-node ethereum runs on our machine. The script starts Websocket and RPC endpoints as well. The mining also starts because this will execute the actual transactions. More details can be found in the shell script.
Since the ethereum is a long running process we move to the second tmux pane and start the bitcoin chain.
./run-bitcoin.shAt this point I don’t want to go into details but this script starts the docker-compose file under the bitcoin directory. Both Bitcoin daemon and ElectrumX have their own Dockerfile.
So we also get a long running process with this script so we have to move to the third tmux pane.
We can run the install shell script like I have done in the picture above. There is a small chance that you haven’t install npm on your system, but it’s required for this script.
sudo yum install -y npm
./install.sh# ....
# You should see this line at the and with the list of submodules:
# Installation script executed successfully...
This will take time… (probably hours).
We want to check out what happens inside the script. We will see that the first one is updating the corresponding submodules. You can find these submodules in the .gitmodules file. Then it calls the submodule installers. Install keep-core, keep-ecdsa, tbtc, tbtc-dapp and keep-dashboard. Since it compiles solidity and does npm install with a huge amount of C++ compilation, it takes time.
So at this point we have ethereum and bitcoin local nodes and the bootstrapped keep system. We have to start the keep related subsystems.
Setup Keep Network clients locally
Open a new tmux session with 4 panes and start the keep clients. 1 core and 3 ecdsas to make it possible to use its consensus algorithm.
# 1. pane:
./run-core-1.sh# 2. pane:
./run-ecdsa-1.sh# 3. pane:
./run-ecdsa-2.sh# 4. pane:
./run-ecdsa-3.sh
That was easy to set up. If they are running without any problem you should be good. Note that keep-ecdsa can burn the CPU. It uses all of my cores with ~90% load.
We have to do a genesis block on Keep network and then we can start the dashboard.
cd keep-coreKEEP_ETHEREUM_PASSWORD="password" ./keep-core --config configs/config.local.1.toml relay genesis
Going back to the root of the repository and run the dashboard.
cd keep-core/solidity/dashboard
npx browserslist@latest --update-db
cd ../..
./run-keep-dashboard.shThis will start a web application on port 3000. If you open it you should see something like the image below.
The last things are the end to end tests. First we have to install and switch to a newer version of node by using the node version manager.
curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.35.3/install.sh | bash
source ~/.bashrcnvm install 14.3.0cd e2e && npm install
cd ..
./run-e2e-test.sh
Yes, I know the shell scripts also contain the npm install, but it always runs into issues until I run it myself.
Ideally it should pass and basically we can start the development.
It is hard to write a guide like this since there lots of moving parts. If you run into an issue at any point which I haven’t covered in this article, please feel free to message me on twitter.
