Nym Node Troubleshooting

If you have problems running a nym-node you are likely to find a solution here.

ℹ️

Our documentation often refer to syntax annotated in <> brackets. We use this expression for variables that are unique to each user (like path, local moniker, versions etcetra). Any syntax in <> brackets needs to be substituted with your correct name or version, without the <> brackets. If you are unsure, please check our table of essential parameters and variables (opens in a new tab).

Binary Build Problems

I am trying to build from the GitHub archive files and the build fails

GitHub automatically includes .zip and tar.gz files of the Nym repository in its release. You cannot extract these and build - you'll see something like this:

  process didn't exit successfully: `/build/nym/src/nym-0.12.1/target/release/build/nym-socks5-client-c1d0f76a8c7d7e9a/build-script-build` (exit status: 101)
  --- stderr
  thread 'main' panicked at 'failed to extract build metadata: could not find repository from '/build/nym/src/nym-0.12.1/clients/socks5'; class=Repository (6); code=NotFound (-3)', clients/socks5/build.rs:7:31
  note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
warning: build failed, waiting for other jobs to finish...
error: build failed

Why does this happen?

We have scripts which automatically include the Git commit hash and Git tag in the binary for easier debugging later. If you download a .zip and try building from that, it's not a Git repository and build will fail as above.

What to do?

Follow the instructions in the Binaries section to build nym from source or download precompiled binaries
To upgrade, follow the upgrade instructions

General Node Config

Where can I find my private and public keys and config?

All config and keys files are stored in a directory named after your node local identifier (<ID>) which you chosen (if not, default one is default-nym-node), and can be found at the following PATH: $HOME/.nym/nym-node/<NODE_ID> where $HOME is a home directory of the user (your current user in this case) that launched the node or client.

The directory structure for each node will be roughly as follows:

~/.nym/nym-nodes/
└── default-nym-node
    ├── config
    │   └── config.toml
    └── data
        ├── aes128ctr_ipr_ack
        ├── aes128ctr_nr_ack
        ├── clients.sqlite
        ├── cosmos_mnemonic
        ├── description.toml
        ├── ed25519_identity
        ├── ed25519_identity.pub
        ├── ed25519_ipr_identity
        ├── ed25519_ipr_identity.pub
        ├── ed25519_nr_identity
        ├── ed25519_nr_identity.pub
        ├── ipr_gateways_info_store.sqlite
        ├── nr_gateways_info_store.sqlite
        ├── nr_persistent_reply_store.sqlite
        ├── x25519_ipr_dh
        ├── x25519_ipr_dh.pub
        ├── x25519_noise
        ├── x25519_noise.pub
        ├── x25519_nr_dh
        ├── x25519_nr_dh.pub
        ├── x25519_sphinx
        └── x25519_sphinx.pub

💡

If you cat the public_sphinx.pem key, the output will be different from the public key you will see on Nym dashboard (opens in a new tab). The reason for this is that .pem files are encoded in base64, however on the web they are in base58. Don't be confused if your keys look different. They are the same keys, just with different encoding.

Accidentally killing your node process on exiting session

When you close your current terminal session, you need to make sure you don't kill the Mix Node process! There are multiple ways on how to make it persistent even after exiting your ssh session, the easiest solution is to use tmux or nohup, and the more elegant solution is to run the node with systemd. Read the automation manual here.

What is `verloc` and do I have to configure my Nym Node to implement it?

verloc is short for verifiable location. Mix Nodes and Gateways now measure speed-of-light distances to each other, in an attempt to verify how far apart they are. In later releases, this will allow us to algorithmically verify node locations in a non-fake-able and trustworthy manner.

You don't have to do any additional configuration for your node to implement this, it is a passive process that runs in the background of the mixnet.

Node Functionality Specific Troubleshooting

Choose the mode that you want to troubleshoot.

Gateways Mode

My `exit-gateway` is running but appears offline in the explorer

Let your Gateway run and follow these steps:

1. Check if your firewall configuration

If ufw is active and if the necessary ports are open / allowed, including the ones for Swagger page and Reversed proxy/WSS if this is your case.

2. See if the Gateway is not on the list of blacklisted Gateways (opens in a new tab)

3. If it's blacklisted, check out the point below

My Gateway is blacklisted

Nym API measures performance by routing traffic through the Mixnet. If the average of a Gateway's routing score in past 24h is less than 50%, the Gateway gets blacklisted and it remains so until its performance is higher than 50%.

In case your Gateway appeared on the blacklist (opens in a new tab), it's because there is some flaw in the configuration. The most common sources of problems are:

Outdated version of nym-node
Bonding before starting the node/service
Bonding before opening needed ports
VPS restarted without operator having a systemd automation or some alert notification flow setup (so the operator doesn't know the node was stopped)
IP address or host is incorrectly configured
Process logs grew too big
Node is wrapped in systemd service and the operator forgot to run systemctl daemon-reload after last changes

What to do

Begin with a sanity check by opening harbourmaster.nymtech.net (opens in a new tab) and check your node there. To see IPv4 and IPv6 routing in real time (harbourmaster can have a cache up to 90 min), run Gateway Probe CLI.

Then follow these steps:

1. Make sure your node is on the latest version and it's running . Do not stop it if there is no need!

2. Open all needed ports

3. Check your `config.toml` - often people have filled `hostname` without the domain being registered to `nym-node` IP, or a wrong IP address after moving their node.

4. Check Gateway Connectivity

5. See logs of your Gateway and search for errors - if you find any unusual one, you can ask in the Element Node Operators (opens in a new tab) channel

If your logs show that your Node has cover down: 0.00 that means that the embedded IPR and NR is not sending any cover traffic.

6. Check out if your `syslog`s aren't eating all your disk space and prune them

7. When all problems are addressed: Restart the node service

Don't forget systemctl daemon-reload) and wait until your node gets above 50% of performance (average of last 24h) - this will likely take 24-48 hours. During this time your node is tested by nym-api and every positive response picks up your routing score.

8. If your node doesn't pick up the routing score within 24h try changing mode

If it stayed on zero performance and you succeed in all the checks above, while your running in --mode exit-gateway, run it as --mode entry-gateway. When your node is above 75% performance (past 24h), switch back to --mode exit-gateway.

💡

Do not repeatedly restart your Nym Node without reason, your routing score will only get worse!

Check Gateway connectivity

Here are a few steps to check whether your Gateway is actually connecting.

1. Check out the API endpoints

Start with checking if your Gateway IPR and NR is active. To determine which mode your node is running, you can check the :8080/api/v1/roles endpoint. For example:

# for http
http://<PUBLIC_IP>:8080/api/v1/roles
# or
http://<PUBLIC_IP>/api/v1/roles
 
# for reversed proxy/WSS
https://<HOSTNAME>/api/v1/roles

Everything necessary will exist on your node by default. For instance, if you're running a mixnode, you'll find that a NR (Network Requester) and IPR (IP Packet Router) address exist, but they will be ignored in mixnode mode.

For more information about available endpoints and their status, you can refer to:

# for http
http://<PUBLIC_IP>:8080/api/v1/swagger/#/
# or
http://<PUBLIC_IP>/api/v1/swagger/#/
 
# for reversed proxy/WSS
https://<HOSTNAME>/api/v1/swagger/#/

2. Configure IPv4 and IPv6 tables and rules

In case you haven't lately, follow the steps in the node configuration chapter connectivity test and configurastion.

3. Test connectivity

Telnet - from your local machine try to connect to your VPS bu running:

telnet <PUBLIC_IP> <PORT>

Websocket wcat (opens in a new tab):

Install on your local machine:

sudo apt install node-ws

Run wscat pointing to the IP of your VPS with port 9000:

wscat -c ws://<PUBLIC_IP>:<PORT>

My Exit Gateway "is still not online..."

The Nyx chain epoch takes up to 60 min. To prevent the Gateway getting blacklisted, it's essential to start it before the bonding process and let it running. In case it already got blacklisted check the steps above.

Last updated on March 19, 2025

Troubleshooting VPS Setup Validators