Setting Up a Flow Node
This guide is for running a Collection, Consensus, Verification and Execution node. If you are planning to run an Access node then refer to access node setup.
First you'll need to provision a machine or virtual machine to run your node software. Please see follow the node-provisioning guide for it.
Pull the Flow Images
The flow-go
binaries are distributed as container images, and need to be pulled down to your host with your image management tool of choice.
Replace $ROLE
with the node type you are planning to run. Valid options are:
- collection
- consensus
- execution
- verification
- access
_10 # Docker_10 docker pull gcr.io/flow-container-registry/${ROLE}:alpha-v0.0.1_10_10 # Containerd_10 ctr images pull gcr.io/flow-container-registry/${ROLE}:alpha-v0.0.1",
Prepare Your Node to Start
Your nodes will need to boot at startup, and restart if they crash.
If you are running systemd
you can use the service files provided by flow-go
.
Find them in the Flow Go.
If you are using some other system besides Systemd, you need to ensure that the Flow container is started, the appropriate key directories are mounted into the container, and that the container will automatically restart following a crash.
The systemd
files pull runtime settings from /etc/flow/runtime-config.env
and any .env
files under /etc/flow/conf.d
. Examples of these files are also available in the github repo.
You will need to modify the runtime config file later.
Systemd
If you are not using Systemd, you can skip this step
- Ensure that you pulled the latest changes from flow-go repository via
git
_10## Clone the repo if you haven't already done so_10git clone https://github.com/onflow/flow-go_10_10## Get latest changes_10cd flow-go_10git pull origin master
- Copy your respective systemd unit file to:
/etc/systemd/system
- Create directory
sudo mkdir /etc/flow
- Copy the runtime-conf.env file to:
/etc/flow/
- Enable your service
sudo systemctl enable flow-$ROLE.service
(replace$ROLE
with your node role - eg.collection
)
Docker Configuration
If you are not using Systemd, sample commands for running each Docker container are below.
Be sure to replace /path/to/data
and /path/to/bootstrap
with the appropriate paths you are using.
Do not run your node using docker run
command directly without a mechanism for the node
to automatically restart following a crash.
The actual Docker image tag can be found here for appropriate spork.
System Configuration
Flow nodes create connections to other nodes on the network, which are represented as file descriptors by the OS. Depending on the default limits for your machine, you may need to increase the soft limit available to the node software.
Make sure the soft limit is at least 8192
.
You can configure the ulimit for the node's docker container. See the Docker documentation for more details.
Admin Server
Each node can be configured with an admin server, which allows you to control some of the node's configuration, as well as view some of its internal state. You can find a few of the commands in the Admin Server README. Two commands to highlight are:
list-commands
: which returns a list of all of the available commands for your nodeset-log-level
: which allows you to change the log level of your node at runtime
You can enable the admin server by passing the --admin-addr
flag with an interface and port.
⚠️ IMPORANT: The admin server can modify your node's configuration. DO NOT allow access to untrusted clients.
Access
_19docker run --rm \_19 -v /path/to/bootstrap:/bootstrap:ro \_19 -v /path/to/data:/data:rw \_19 --name flow-go \_19 --network host \_19 --ulimit nofile=8192 \_19 gcr.io/flow-container-registry/access:<applicable docker tag> \_19 --nodeid=${FLOW_GO_NODE_ID} \_19 --bootstrapdir=/bootstrap \_19 --datadir=/data/protocol \_19 --secretsdir=/data/secrets \_19 --execution-data-dir=/data/execution_data \_19 --rpc-addr=0.0.0.0:9000 \_19 --http-addr=0.0.0.0:8000 \_19 --admin-addr=0.0.0.0:9002 \_19 --collection-ingress-port=9000 \_19 --script-addr=${FLOW_NETWORK_EXECUTION_NODE} \_19 --bind 0.0.0.0:3569 \_19 --loglevel=error
Collection
_15docker run --rm \_15 -v /path/to/bootstrap:/bootstrap:ro \_15 -v /path/to/data:/data:rw \_15 --name flow-go \_15 --network host \_15 --ulimit nofile=8192 \_15 gcr.io/flow-container-registry/collection:<applicable docker tag> \_15 --nodeid=${FLOW_GO_NODE_ID} \_15 --bootstrapdir=/bootstrap \_15 --datadir=/data/protocol \_15 --secretsdir=/data/secrets \_15 --ingress-addr=0.0.0.0:9000 \_15 --admin-addr=0.0.0.0:9002 \_15 --bind 0.0.0.0:3569 \_15 --loglevel=error
Consensus
_14docker run --rm \_14 -v /path/to/bootstrap:/bootstrap:ro \_14 -v /path/to/data:/data:rw \_14 --name flow-go \_14 --network host \_14 --ulimit nofile=8192 \_14 gcr.io/flow-container-registry/consensus:<applicable docker tag> \_14 --nodeid=${FLOW_GO_NODE_ID} \_14 --bootstrapdir=/bootstrap \_14 --datadir=/data/protocol \_14 --secretsdir=/data/secrets \_14 --admin-addr=0.0.0.0:9002 \_14 --bind 0.0.0.0:3569 \_14 --loglevel=error
Execution
_17docker run --rm \_17 -v /path/to/bootstrap:/bootstrap:ro \_17 -v /path/to/data:/data:rw \_17 --name flow-go \_17 --network host \_17 --ulimit nofile=500000 \_17 gcr.io/flow-container-registry/execution:<applicable docker tag> \_17 --nodeid=${FLOW_GO_NODE_ID} \_17 --bootstrapdir=/bootstrap \_17 --datadir=/data/protocol \_17 --secretsdir=/data/secrets \_17 --triedir=/data/execution \_17 --execution-data-dir=/data/execution_data \_17 --rpc-addr=0.0.0.0:9000 \_17 --admin-addr=0.0.0.0:9002 \_17 --bind 0.0.0.0:3569 \_17 --loglevel=error
For execution nodes, it is recommend to increase the open files limit in your operating system. To do that, add the following to your /etc/security/limits.conf
or the equivalent limits.conf
for your distribution:
_10* hard nofile 500000_10* soft nofile 500000_10root hard nofile 500000_10root soft nofile 500000
Restart your machine to apply these changes. To verify that the new limits have been applied, run:
_10ulimit -n
Verification
_14docker run --rm \_14 -v /path/to/bootstrap:/bootstrap:ro \_14 -v /path/to/data:/data:rw \_14 --name flow-go \_14 --network host \_14 --ulimit nofile=8192 \_14 gcr.io/flow-container-registry/verification:<applicable docker tag> \_14 --nodeid=${FLOW_GO_NODE_ID} \_14 --bootstrapdir=/bootstrap \_14 --datadir=/data/protocol \_14 --secretsdir=/data/secrets \_14 --admin-addr=0.0.0.0:9002 \_14 --bind 0.0.0.0:3569 \_14 --loglevel=error
Start the Node
Now that your node is provisioned and configured, it can be started.
Before starting your node, ensure it is registered and authorized.
Ensure you start your node at the appropriate time. See Spork Process for when to start up a node following a spork. See Node Bootstrap for when to start up a newly registered node.
Systemd
- Check that your
runtime-conf.env
is at/etc/flow/runtime-conf.env
- Update your environment variables:
source /etc/flow/runtime-conf.env
- Start your service:
sudo systemctl start flow
Verify your Node is Running
Here are a few handy commands that you can use to check if your Flow node is up and running
Systemd
- To get Flow logs:
sudo journalctl -u flow-YOUR_ROLE
- To get the status:
sudo systemctl status flow
_10● flow-verification.service - Flow Access Node running with Docker_10Loaded: loaded (/etc/systemd/system/flow-verification.service; enabled; vendor preset: enabled)_10Active: active (running) since Wed 2020-05-20 18:18:13 UTC; 1 day 6h ago_10Process: 3207 ExecStartPre=/usr/bin/docker pull gcr.io/flow-container-registry/verification:${FLOW_GO_NODE_VERSION} (code=exited, status=0/SUCCESS)_10Main PID: 3228 (docker)_10Tasks: 10 (limit: 4915)_10Memory: 33.0M_10CGroup: /system.slice/flow-verification.service_10 └─3228 /usr/bin/docker run --rm -v /var/flow/bootstrap:/bootstrap:ro -v /var/flow/data:/data:rw --rm --name flow-go --network host gcr.io/flow-container-registry/verification:candidate8 --nodeid=489f8a4513d5bd8b8b093108fec00327b683db545b37b4ea9153f61b2c0c49dc --bootstrapdir=/bootstrap --datadir=/data/protocol --alpha=1 --bind 0.0.0.0:3569 --loglevel=error
Docker
- To get Flow logs:
sudo docker logs flow-go
- To get the status:
sudo docker ps
_10$ sudo docker ps_10CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES_101dc5d43385b6 gcr.io/flow-container-registry/verification:candidate8 \"/bin/app --nodeid=4…\" 30 hours ago Up 30 hours flow-go
Monitoring and Metrics
This is intended for operators who would like to see what their Flow nodes are currently doing. Head over to Monitoring Node Health to get setup.
Node Status
The metrics for the node should be able to provide a good overview of the status of the node. If we want to get a quick snapshot of the status of the node, and if it's properly participating in the network, you can check the consensus_compliance_finalized_height
or consensus_compliance_sealed_height
metric, and ensure that it is not zero and strictly increasing.
_10curl localhost:8080/metrics | grep consensus_compliance_sealed_height_10_10# HELP consensus_compliance_sealed_height the last sealed height_10# TYPE consensus_compliance_sealed_height gauge_10consensus_compliance_sealed_height 1.132054e+06