305 lines
9.7 KiB
Markdown
305 lines
9.7 KiB
Markdown
# P4 Testbed
|
|
|
|
In this repository we're have scripts to setup a network testbed with
|
|
2 nodes and a Tofino Wedge 100BF-32X 100GBE Baremetal Switch.
|
|
|
|
## Prerequisites
|
|
|
|
- Hardware
|
|
- Tofino Wedge 100BF-32X 100GBE Baremetal Switch
|
|
- 2x 2m 40G QSFP+ to 4x 10G SFP+ Breakout Cable
|
|
- 2x Server with each 4 usable SFP+ Ports
|
|
- Software
|
|
- Barefoot SDE
|
|
- SSH config for nodes
|
|
|
|
## Setup and Usage
|
|
|
|
The setup looks as follows.
|
|
Depending on the setup of the QSFP Breakout cable the port numbers or the $n$
|
|
may differ for $31/n$ and $32/n$ and the order may be different.
|
|
|
|
The ports shown in this setup are only the connections between the wedge and
|
|
the nodes (data plane).
|
|
A separate network connection is needed to SSH into the wedge/nodes (control plane).
|
|
|
|
For this testbed we're setting up a simple switch which forwards all traffic
|
|
according to rules in the ingress.
|
|
Look at [l1switch.p4](../l1switch/l1switch.p4) for more information.
|
|
|
|
<!-- markdownlint-disable MD013 -->
|
|
|
|
```ascii
|
|
+-----------------+
|
|
| WEDGE | +----------------------------------------------------------+
|
|
+-----------------+ | Node 1 : IP | MAC | Namespace |
|
|
| Port 32 | +----------------------------------------------------------+
|
|
| 32/2 | 266 |--------+ | enp6s0f0np0: 10.0.1.1 | 64:9d:99:b1:0a:88 | tb_node1_if0 |
|
|
| 32/3 | 267 |--------+ | enp6s0f1np1: 10.0.1.2 | 64:9d:99:b1:0a:89 | tb_node1_if1 |
|
|
| 32/1 | 265 |--------+ | enp6s0f2np2: 10.0.1.3 | 64:9d:99:b1:0a:8a | tb_node1_if2 |
|
|
| 32/0 | 264 |--------+ | enp6s0f3np3: 10.0.1.4 | 64:9d:99:b1:0a:8b | tb_node1_if3 |
|
|
| | +----------------------------------------------------------+
|
|
| |
|
|
| | +----------------------------------------------------------+
|
|
| | | Node 2 : IP | MAC | Namespace |
|
|
| Port 31 | +----------------------------------------------------------+
|
|
| 31/0 | 256 |--------+ | enp6s0f0np0: 10.0.2.1 | 64:9d:99:b1:0a:a5 | tb_node1_if0 |
|
|
| 31/3 | 259 |--------+ | enp6s0f1np1: 10.0.2.2 | 64:9d:99:b1:0a:a4 | tb_node1_if1 |
|
|
| 31/1 | 257 |--------+ | enp6s0f2np2: 10.0.2.3 | 64:9d:99:b1:0a:a4 | tb_node1_if2 |
|
|
| 31/2 | 258 |--------+ | enp6s0f3np3: 10.0.2.4 | 64:9d:99:b1:0a:a4 | tb_node1_if3 |
|
|
| | +----------------------------------------------------------+
|
|
+-----------------+
|
|
```
|
|
|
|
<!-- markdownlint-enable MD013 -->
|
|
|
|
### 00-wedge_on_reboot.sh
|
|
|
|
> This script needs to be run on the Tofino Wedge.
|
|
|
|
When the wedge has just restarted for the first time we'll need to run this script
|
|
on the wedge to load the necessary kernel module.
|
|
|
|
### 01-wedge_compile_code.sh
|
|
|
|
> This script needs to be run on the Tofino Wedge.
|
|
|
|
### 02-run_switch_program.sh
|
|
|
|
> This script needs to be run on the Tofino Wedge.
|
|
|
|
### 03-wedge_port_configure.sh
|
|
|
|
> This script does not need to be run.
|
|
|
|
This script is just a placeholder and needs not to be run.
|
|
Instead look at the [./03-wedge_port_configure.command](./03-wedge_port_configure.command)
|
|
script.
|
|
|
|
### 03-wedge_port_configure.command
|
|
|
|
> This script needs to be run with a special command on the Tofino Wedge.
|
|
|
|
Enter the configuration shell of the current switch program and setup the ports:
|
|
|
|
- 31/0
|
|
- 31/1
|
|
- 31/2
|
|
- 31/3
|
|
- 32/0
|
|
- 32/1
|
|
- 32/2
|
|
- 32/3
|
|
|
|
That script needs to be run like this:
|
|
`bfshell -f ./03-wedge_port_configure.command`
|
|
|
|
### 04-wedge_route_configure.py
|
|
|
|
> This script needs to be run with a special command on the Tofino Wedge.
|
|
|
|
This script needs to be run on the Tofino Wedge.
|
|
|
|
Configure the ingress of the switch program. Maps the `D_P` ports to the
|
|
respective IP.
|
|
|
|
This script is run with `bfshell -b $(pwd)/04-wedge_route_configure.py`
|
|
|
|
### 10-host*setup*\*.sh
|
|
|
|
> These scripts need to be run on the respective nodes.
|
|
|
|
This script executes the helper scripts to
|
|
|
|
- Setup Namespaces
|
|
- Setup Interfaces
|
|
- Setup IP Addresses
|
|
- Setup ARP Table Entries
|
|
on the respective node.
|
|
|
|
./10-host_setup_node1.sh needs to be executed on node1.
|
|
|
|
./10-host_setup_node2.sh needs to be executed on node2.
|
|
|
|
### 19-cleanup.sh
|
|
|
|
> This script needs to be run on a node
|
|
|
|
Removes all namespaces.
|
|
|
|
### 30/31/... Scripts
|
|
|
|
> These are helper scripts
|
|
|
|
These scripts are executed by [./10-host_setup.sh](./10-host_setup.sh).
|
|
|
|
### Run traffic
|
|
|
|
From here on out you can run traffic between the hosts.
|
|
|
|
Example:
|
|
|
|
```sh
|
|
# On node1
|
|
sudo ip netns exec tb_node1_if0 python3 receiver.py 10.0.2.1 30123 30123
|
|
```
|
|
|
|
```sh
|
|
# On node2
|
|
sudo ip netns exec tb_node2_if0 python3 sender.py 10.0.1.1 30123 30123 100 128
|
|
```
|
|
|
|
## Appendix
|
|
|
|
### Node and Wedge Ports
|
|
|
|
How the port setup etc. looks like on the nodes and ports after everything is
|
|
set up.
|
|
|
|
<!-- markdownlint-disable MD013 -->
|
|
|
|
#### ncs-node1
|
|
|
|
Control: enp4s0f2
|
|
|
|
Data:
|
|
|
|
| Interface | IP | MAC | Namespace | Wedge Port | Wedge D_P |
|
|
| ----------- | -------- | ----------------- | ------------ | ---------- | --------- |
|
|
| enp6s0f0np0 | 10.0.1.1 | 64:9d:99:b1:0a:88 | tb_node1_if0 | 32/2 | 266 |
|
|
| enp6s0f1np1 | 10.0.1.2 | 64:9d:99:b1:0a:89 | tb_node1_if1 | 32/3 | 267 |
|
|
| enp6s0f2np2 | 10.0.1.3 | 64:9d:99:b1:0a:8a | tb_node1_if2 | 32/1 | 265 |
|
|
| enp6s0f3np3 | 10.0.1.4 | 64:9d:99:b1:0a:8b | tb_node1_if3 | 32/0 | 264 |
|
|
|
|
#### ncs-node2
|
|
|
|
Control: enp4s0f2
|
|
|
|
Data:
|
|
|
|
| Interface | IP | MAC | Namespace | Wedge Port | Wedge D_P |
|
|
| ----------- | -------- | ----------------- | ------------ | ---------- | --------- |
|
|
| enp6s0f0np0 | 10.0.2.1 | 64:9d:99:b1:0a:a4 | tb_node2_if0 | 31/0 | 256 |
|
|
| enp6s0f1np1 | 10.0.2.2 | 64:9d:99:b1:0a:a5 | tb_node2_if1 | 31/3 | 259 |
|
|
| enp6s0f2np2 | 10.0.2.3 | 64:9d:99:b1:0a:a6 | tb_node2_if2 | 31/1 | 257 |
|
|
| enp6s0f3np3 | 10.0.2.4 | 64:9d:99:b1:0a:a7 | tb_node2_if3 | 31/2 | 258 |
|
|
|
|
#### ncs-wedge
|
|
|
|
| PORT | MAC | D_P | P/PT | SPEED | FEC | AN | KR | RDY | ADM | OPR | LPBK | FRAMES RX | FRAMES TX | E |
|
|
| ---- | ---- | --- | ---- | ----- | ---- | --- | --- | --- | --- | --- | ---- | --------- | --------- | --- |
|
|
| 31/0 | 48/0 | 256 | 2/ 0 | 10G | NONE | Ds | Au | YES | ENB | UP | NONE | 62 | 0 | |
|
|
| 31/1 | 48/1 | 257 | 2/ 1 | 10G | NONE | Ds | Au | YES | ENB | UP | NONE | 62 | 0 | |
|
|
| 31/2 | 48/2 | 258 | 2/ 2 | 10G | NONE | Ds | Au | YES | ENB | UP | NONE | 62 | 0 | |
|
|
| 31/3 | 48/3 | 259 | 2/ 3 | 10G | NONE | Ds | Au | YES | ENB | UP | NONE | 62 | 0 | |
|
|
| 32/0 | 50/0 | 264 | 2/ 8 | 10G | NONE | Ds | Au | YES | ENB | UP | NONE | 62 | 0 | |
|
|
| 32/1 | 50/1 | 265 | 2/ 9 | 10G | NONE | Ds | Au | YES | ENB | UP | NONE | 62 | 0 | |
|
|
| 32/2 | 50/2 | 266 | 2/10 | 10G | NONE | Ds | Au | YES | ENB | UP | NONE | 65 | 0 | |
|
|
| 32/3 | 50/3 | 267 | 2/11 | 10G | NONE | Ds | Au | YES | ENB | UP | NONE | 65 | 0 | |
|
|
|
|
<!-- markdownlint-enable MD013 -->
|
|
|
|
### Adding new node
|
|
|
|
We want to add a new node: =ncs-node3= to our network.
|
|
|
|
We'll connect the 4 sfp+ ports of our node via a qsfp+ breakout cable to the
|
|
Tofino Wedge on port 30.
|
|
|
|
Given that the 4 interfaces on =ncs-node3= have the names:
|
|
|
|
- enp6s1 with 00:00:00:00:00:01 (MAC) and 10.0.3.1 (IP)
|
|
- enp6s2 with 00:00:00:00:00:02 (MAC) and 10.0.3.2 (IP)
|
|
- enp6s3 with 00:00:00:00:00:03 (MAC) and 10.0.3.3 (IP)
|
|
- enp6s4 with 00:00:00:00:00:04 (MAC) and 10.0.3.4 (IP)
|
|
|
|
Find and replace the actual MAC addresses of the actual interfaces by running
|
|
`ip a s` on the node.
|
|
|
|
#### Adjust ./03-wedge_port_configure.command
|
|
|
|
Add:
|
|
|
|
```command
|
|
port-add 30/- 10G NONE
|
|
an-set 30/- 2
|
|
port-enb 30/-
|
|
```
|
|
|
|
- port-add: Add the port
|
|
- an-set: Set auto negotiation
|
|
- port-enb: Enable the port
|
|
|
|
To the command script before the
|
|
|
|
```command
|
|
exit
|
|
exit
|
|
```
|
|
|
|
When running this take note of the `D_P` column of the new ports. You'll need
|
|
to enter them as `egress_port` in the next script.
|
|
|
|
#### Adjust ./04-wedge_route_configure.py
|
|
|
|
Add the following:
|
|
|
|
Replace `???` by the actual port number for the respective interface.
|
|
|
|
```py
|
|
|
|
bfrt.l1switch.pipe.SwitchIngress.t_l1_forwarding.add_with_send(
|
|
egress_port=???, dst_addr="10.0.3.1"
|
|
)
|
|
bfrt.l1switch.pipe.SwitchIngress.t_l1_forwarding.add_with_send(
|
|
egress_port=???, dst_addr="10.0.3.2"
|
|
)
|
|
bfrt.l1switch.pipe.SwitchIngress.t_l1_forwarding.add_with_send(
|
|
egress_port=???, dst_addr="10.0.3.3"
|
|
)
|
|
bfrt.l1switch.pipe.SwitchIngress.t_l1_forwarding.add_with_send(
|
|
egress_port=???, dst_addr="10.0.3.4"
|
|
)
|
|
```
|
|
|
|
- egress_port: Value at the `D_P` column at the output of the last script.
|
|
|
|
#### Add ./10-host_setup_node3.sh
|
|
|
|
Create a new script
|
|
|
|
```sh
|
|
#!/bin/bash
|
|
|
|
# node3
|
|
sudo ./30-add_namespaces.sh tb_node3_if0
|
|
sudo ./30-add_namespaces.sh tb_node3_if1
|
|
sudo ./30-add_namespaces.sh tb_node3_if2
|
|
sudo ./30-add_namespaces.sh tb_node3_if3
|
|
|
|
sudo ./31-configure_interfaces.sh tb_node3_if0 enp6s1 10.0.4.1/16
|
|
sudo ./31-configure_interfaces.sh tb_node3_if1 enp6s2 10.0.4.2/16
|
|
sudo ./31-configure_interfaces.sh tb_node3_if2 enp6s3 10.0.4.3/16
|
|
sudo ./31-configure_interfaces.sh tb_node3_if3 enp6s4 10.0.4.4/16
|
|
|
|
sudo ./32-setup_arp.sh tb_node3_if0 enp6s1
|
|
sudo ./32-setup_arp.sh tb_node3_if1 enp6s2
|
|
sudo ./32-setup_arp.sh tb_node3_if2 enp6s3
|
|
sudo ./32-setup_arp.sh tb_node3_if3 enp6s4
|
|
```
|
|
|
|
Adjust [./32-setup_arp.sh](./32-setup_arp.sh) before running this script.
|
|
|
|
#### Adjust ./32-setup_arp.sh
|
|
|
|
Add the following section for your new interfaces like this:
|
|
|
|
```sh
|
|
ip netns exec "$NS" arp -i "$IF" -s 10.0.3.1 00:00:00:00:00:01
|
|
ip netns exec "$NS" arp -i "$IF" -s 10.0.3.2 00:00:00:00:00:02
|
|
ip netns exec "$NS" arp -i "$IF" -s 10.0.3.3 00:00:00:00:00:03
|
|
ip netns exec "$NS" arp -i "$IF" -s 10.0.3.4 00:00:00:00:00:04
|
|
```
|
|
|
|
Replace the placeholder mac address with the actual MAC addresses of the interfaces.
|