# P4 Testbed In this repository we're have scripts to setup a network testbed with 2 nodes and a Tofino Wedge 100BF-32X 100GBE Baremetal Switch. You can run the script [./99-deployment.sh](./99-deployment.sh) for a sample deployment. ## Prerequisites - Hardware - Tofino Wedge 100BF-32X 100GBE Baremetal Switch - 2x 2m 40G QSFP+ to 4x 10G SFP+ Breakout Cable - 2x Server with each 4 usable SFP+ Ports - Software - Barefoot SDE - SSH config for nodes ## Setup and Usage The setup looks as follows. Depending on the setup of the QSFP Breakout cable the port numbers or the $n$ may differ for $31/n$ and $32/n$ and the order may be different. The ports shown in this setup are only the connections between the wedge and the nodes (data plane). A separate network connection is needed to SSH into the wedge/nodes (control plane). For this testbed we're setting up a simple switch which forwards all traffic according to rules in the ingress. Look at [l1switch.p4](../l1switch/l1switch.p4) for more information. ```ascii +-----------------+ | WEDGE | +----------------------------------------------------------+ +-----------------+ | Node 1 : IP | MAC | Namespace | | Port 32 | +----------------------------------------------------------+ | 32/2 | 266 |--------+ | enp6s0f0np0: 10.0.1.1 | 64:9d:99:b1:0a:88 | tb_node1_if0 | | 32/3 | 267 |--------+ | enp6s0f1np1: 10.0.1.2 | 64:9d:99:b1:0a:89 | tb_node1_if1 | | 32/1 | 265 |--------+ | enp6s0f2np2: 10.0.1.3 | 64:9d:99:b1:0a:8a | tb_node1_if2 | | 32/0 | 264 |--------+ | enp6s0f3np3: 10.0.1.4 | 64:9d:99:b1:0a:8b | tb_node1_if3 | | | +----------------------------------------------------------+ | | | | +----------------------------------------------------------+ | | | Node 2 : IP | MAC | Namespace | | Port 31 | +----------------------------------------------------------+ | 31/0 | 256 |--------+ | enp6s0f0np0: 10.0.2.1 | 64:9d:99:b1:0a:a5 | tb_node1_if0 | | 31/3 | 259 |--------+ | enp6s0f1np1: 10.0.2.2 | 64:9d:99:b1:0a:a4 | tb_node1_if1 | | 31/1 | 257 |--------+ | enp6s0f2np2: 10.0.2.3 | 64:9d:99:b1:0a:a4 | tb_node1_if2 | | 31/2 | 258 |--------+ | enp6s0f3np3: 10.0.2.4 | 64:9d:99:b1:0a:a4 | tb_node1_if3 | | | +----------------------------------------------------------+ +-----------------+ ``` ### 00-wedge_on_reboot.sh > This script needs to be run on the Tofino Wedge. When the wedge has just restarted for the first time we'll need to run this script on the wedge to load the necessary kernel module. ### 01-wedge_compile_code.sh > This script needs to be run on the Tofino Wedge. ### 02-run_switch_program.sh > This script needs to be run on the Tofino Wedge. ### 03-wedge_port_configure.sh > This script does not need to be run. This script is just a placeholder and needs not to be run. Instead look at the [./03-wedge_port_configure.command](./03-wedge_port_configure.command) script. ### 03-wedge_port_configure.command > This script needs to be run with a special command on the Tofino Wedge. Enter the configuration shell of the current switch program and setup the ports: - 31/0 - 31/1 - 31/2 - 31/3 - 32/0 - 32/1 - 32/2 - 32/3 That script needs to be run like this: `bfshell -f ./03-wedge_port_configure.command` ### 04-wedge_route_configure.py > This script needs to be run with a special command on the Tofino Wedge. This script needs to be run on the Tofino Wedge. Configure the ingress of the switch program. Maps the `D_P` ports to the respective IP. This script is run with `bfshell -b $(pwd)/04-wedge_route_configure.py` ### 10-host*setup*\*.sh > These scripts need to be run on the respective nodes. This script executes the helper scripts to - Setup Namespaces - Setup Interfaces - Setup IP Addresses - Setup ARP Table Entries on the respective node. ./10-host_setup_node1.sh needs to be executed on node1. ./10-host_setup_node2.sh needs to be executed on node2. ### 19-cleanup.sh > This script needs to be run on a node Removes all namespaces. ### 30/31/... Scripts > These are helper scripts These scripts are executed by [./10-host_setup.sh](./10-host_setup.sh). ### Run traffic From here on out you can run traffic between the hosts. Example: ```sh # On node1 sudo ip netns exec tb_node1_if0 python3 receiver.py 10.0.2.1 30123 30123 ``` ```sh # On node2 sudo ip netns exec tb_node2_if0 python3 sender.py 10.0.1.1 30123 30123 100 128 ``` ## Appendix ### Node and Wedge Ports How the port setup etc. looks like on the nodes and ports after everything is set up. #### ncs-node1 Control: enp4s0f2 Data: | Interface | IP | MAC | Namespace | Wedge Port | Wedge D_P | | ----------- | -------- | ----------------- | ------------ | ---------- | --------- | | enp6s0f0np0 | 10.0.1.1 | 64:9d:99:b1:0a:88 | tb_node1_if0 | 32/2 | 266 | | enp6s0f1np1 | 10.0.1.2 | 64:9d:99:b1:0a:89 | tb_node1_if1 | 32/3 | 267 | | enp6s0f2np2 | 10.0.1.3 | 64:9d:99:b1:0a:8a | tb_node1_if2 | 32/1 | 265 | | enp6s0f3np3 | 10.0.1.4 | 64:9d:99:b1:0a:8b | tb_node1_if3 | 32/0 | 264 | #### ncs-node2 Control: enp4s0f2 Data: | Interface | IP | MAC | Namespace | Wedge Port | Wedge D_P | | ----------- | -------- | ----------------- | ------------ | ---------- | --------- | | enp6s0f0np0 | 10.0.2.1 | 64:9d:99:b1:0a:a4 | tb_node2_if0 | 31/0 | 256 | | enp6s0f1np1 | 10.0.2.2 | 64:9d:99:b1:0a:a5 | tb_node2_if1 | 31/3 | 259 | | enp6s0f2np2 | 10.0.2.3 | 64:9d:99:b1:0a:a6 | tb_node2_if2 | 31/1 | 257 | | enp6s0f3np3 | 10.0.2.4 | 64:9d:99:b1:0a:a7 | tb_node2_if3 | 31/2 | 258 | #### ncs-wedge | PORT | MAC | D_P | P/PT | SPEED | FEC | AN | KR | RDY | ADM | OPR | LPBK | FRAMES RX | FRAMES TX | E | | ---- | ---- | --- | ---- | ----- | ---- | --- | --- | --- | --- | --- | ---- | --------- | --------- | --- | | 31/0 | 48/0 | 256 | 2/ 0 | 10G | NONE | Ds | Au | YES | ENB | UP | NONE | 62 | 0 | | | 31/1 | 48/1 | 257 | 2/ 1 | 10G | NONE | Ds | Au | YES | ENB | UP | NONE | 62 | 0 | | | 31/2 | 48/2 | 258 | 2/ 2 | 10G | NONE | Ds | Au | YES | ENB | UP | NONE | 62 | 0 | | | 31/3 | 48/3 | 259 | 2/ 3 | 10G | NONE | Ds | Au | YES | ENB | UP | NONE | 62 | 0 | | | 32/0 | 50/0 | 264 | 2/ 8 | 10G | NONE | Ds | Au | YES | ENB | UP | NONE | 62 | 0 | | | 32/1 | 50/1 | 265 | 2/ 9 | 10G | NONE | Ds | Au | YES | ENB | UP | NONE | 62 | 0 | | | 32/2 | 50/2 | 266 | 2/10 | 10G | NONE | Ds | Au | YES | ENB | UP | NONE | 65 | 0 | | | 32/3 | 50/3 | 267 | 2/11 | 10G | NONE | Ds | Au | YES | ENB | UP | NONE | 65 | 0 | | ### Adding new node We want to add a new node: =ncs-node3= to our network. We'll connect the 4 sfp+ ports of our node via a qsfp+ breakout cable to the Tofino Wedge on port 30. Given that the 4 interfaces on =ncs-node3= have the names: - enp6s1 with 00:00:00:00:00:01 (MAC) and 10.0.3.1 (IP) - enp6s2 with 00:00:00:00:00:02 (MAC) and 10.0.3.2 (IP) - enp6s3 with 00:00:00:00:00:03 (MAC) and 10.0.3.3 (IP) - enp6s4 with 00:00:00:00:00:04 (MAC) and 10.0.3.4 (IP) Find and replace the actual MAC addresses of the actual interfaces by running `ip a s` on the node. #### Adjust ./03-wedge_port_configure.command Add: ```command port-add 30/- 10G NONE an-set 30/- 2 port-enb 30/- ``` - port-add: Add the port - an-set: Set auto negotiation - port-enb: Enable the port To the command script before the ```command exit exit ``` When running this take note of the `D_P` column of the new ports. You'll need to enter them as `egress_port` in the next script. #### Adjust ./04-wedge_route_configure.py Add the following: Replace `???` by the actual port number for the respective interface. ```py bfrt.l1switch.pipe.SwitchIngress.t_l1_forwarding.add_with_send( egress_port=???, dst_addr="10.0.3.1" ) bfrt.l1switch.pipe.SwitchIngress.t_l1_forwarding.add_with_send( egress_port=???, dst_addr="10.0.3.2" ) bfrt.l1switch.pipe.SwitchIngress.t_l1_forwarding.add_with_send( egress_port=???, dst_addr="10.0.3.3" ) bfrt.l1switch.pipe.SwitchIngress.t_l1_forwarding.add_with_send( egress_port=???, dst_addr="10.0.3.4" ) ``` - egress_port: Value at the `D_P` column at the output of the last script. #### Add ./10-host_setup_node3.sh Create a new script ```sh #!/bin/bash # node3 sudo ./30-add_namespaces.sh tb_node3_if0 sudo ./30-add_namespaces.sh tb_node3_if1 sudo ./30-add_namespaces.sh tb_node3_if2 sudo ./30-add_namespaces.sh tb_node3_if3 sudo ./31-configure_interfaces.sh tb_node3_if0 enp6s1 10.0.4.1/16 sudo ./31-configure_interfaces.sh tb_node3_if1 enp6s2 10.0.4.2/16 sudo ./31-configure_interfaces.sh tb_node3_if2 enp6s3 10.0.4.3/16 sudo ./31-configure_interfaces.sh tb_node3_if3 enp6s4 10.0.4.4/16 sudo ./32-setup_arp.sh tb_node3_if0 enp6s1 sudo ./32-setup_arp.sh tb_node3_if1 enp6s2 sudo ./32-setup_arp.sh tb_node3_if2 enp6s3 sudo ./32-setup_arp.sh tb_node3_if3 enp6s4 ``` Adjust [./32-setup_arp.sh](./32-setup_arp.sh) before running this script. #### Adjust ./32-setup_arp.sh Add the following section for your new interfaces like this: ```sh ip netns exec "$NS" arp -i "$IF" -s 10.0.3.1 00:00:00:00:00:01 ip netns exec "$NS" arp -i "$IF" -s 10.0.3.2 00:00:00:00:00:02 ip netns exec "$NS" arp -i "$IF" -s 10.0.3.3 00:00:00:00:00:03 ip netns exec "$NS" arp -i "$IF" -s 10.0.3.4 00:00:00:00:00:04 ``` Replace the placeholder mac address with the actual MAC addresses of the interfaces.