p4_testbed/scripts

P4 Testbed

In this repository we're have scripts to setup a network testbed with 2 nodes and a Tofino Wedge 100BF-32X 100GBE Baremetal Switch.

Prerequisites

• Hardware
  • Tofino Wedge 100BF-32X 100GBE Baremetal Switch
  • 2x 2m 40G QSFP+ to 4x 10G SFP+ Breakout Cable
  • 2x Server with each 4 usable SFP+ Ports
• Software
  • Barefoot SDE
  • SSH config for nodes

Setup and Usage

The setup looks as follows. Depending on the setup of the QSFP Breakout cable the port numbers or the n may differ for 31/n and 32/n and the order may be different.

The ports shown in this setup are only the connections between the wedge and the nodes (data plane). A separate network connection is needed to SSH into the wedge/nodes (control plane).

For this testbed we're setting up a simple switch which forwards all traffic according to rules in the ingress. Look at l1switch.p4 for more information.

+-----------------+
|     WEDGE       |          +----------------------------------------------------------+
+-----------------+          | Node 1     : IP       | MAC               | Namespace    |
|  Port 32        |          +----------------------------------------------------------+
|  32/2 | 266     |--------+ | enp6s0f0np0: 10.0.1.1 | 64:9d:99:b1:0a:88 | tb_node1_if0 |
|  32/3 | 267     |--------+ | enp6s0f1np1: 10.0.1.2 | 64:9d:99:b1:0a:89 | tb_node1_if1 |
|  32/1 | 265     |--------+ | enp6s0f2np2: 10.0.1.3 | 64:9d:99:b1:0a:8a | tb_node1_if2 |
|  32/0 | 264     |--------+ | enp6s0f3np3: 10.0.1.4 | 64:9d:99:b1:0a:8b | tb_node1_if3 |
|                 |          +----------------------------------------------------------+
|                 |
|                 |          +----------------------------------------------------------+
|                 |          | Node 2     : IP       | MAC               | Namespace    |
|  Port 31        |          +----------------------------------------------------------+
|  31/0 | 256     |--------+ | enp6s0f0np0: 10.0.2.1 | 64:9d:99:b1:0a:a5 | tb_node1_if0 |
|  31/3 | 259     |--------+ | enp6s0f1np1: 10.0.2.2 | 64:9d:99:b1:0a:a4 | tb_node1_if1 |
|  31/1 | 257     |--------+ | enp6s0f2np2: 10.0.2.3 | 64:9d:99:b1:0a:a4 | tb_node1_if2 |
|  31/2 | 258     |--------+ | enp6s0f3np3: 10.0.2.4 | 64:9d:99:b1:0a:a4 | tb_node1_if3 |
|                 |          +----------------------------------------------------------+
+-----------------+

00-wedge_on_reboot.sh

This script needs to be run on the Tofino Wedge.

When the wedge has just restarted for the first time we'll need to run this script on the wedge to load the necessary kernel module.

01-wedge_compile_code.sh

This script needs to be run on the Tofino Wedge.

02-run_switch_program.sh

This script needs to be run on the Tofino Wedge.

03-wedge_port_configure.sh

This script does not need to be run.

This script is just a placeholder and needs not to be run. Instead look at the ./03-wedge_port_configure.command script.

03-wedge_port_configure.command

This script needs to be run with a special command on the Tofino Wedge.

Enter the configuration shell of the current switch program and setup the ports:

• 31/0
• 31/1
• 31/2
• 31/3
• 32/0
• 32/1
• 32/2
• 32/3

That script needs to be run like this: bfshell -f ./03-wedge_port_configure.command

04-wedge_route_configure.py

This script needs to be run with a special command on the Tofino Wedge.

This script needs to be run on the Tofino Wedge.

Configure the ingress of the switch program. Maps the D_P ports to the respective IP.

This script is run with bfshell -b $(pwd)/04-wedge_route_configure.py

10-host_setup.sh

This script needs to be run on the respective nodes with the section of the hosts commented in and the other sections commented out.

This script executes multiple scripts which setup the hosts. In our setup we'll only have two hosts, but this can be extended to other hosts as well. Information needed to do so is:

• The interface name which is connected to the switch
• MAC address of the interface
• Unused IP address
• And take a look at the appendix under "Adding new node"

19-cleanup.sh

This script needs to be run on a node

Removes all namespaces.

30/31/... Scripts

These are helper scripts

These scripts are executed by ./10-host_setup.sh.

Run traffic

From here on out you can run traffic between the hosts.

Appendix

Node and Wedge Ports

How the port setup etc. looks like on the nodes and ports after everything is set up.

ncs-node1

Control: enp4s0f2

Data:

Interface	IP	MAC	Namespace	Wedge Port	Wedge D_P
enp6s0f0np0	10.0.1.1	64:9d:99:b1:0a:88	tb_node1_if0	32/2	266
enp6s0f1np1	10.0.1.2	64:9d:99:b1:0a:89	tb_node1_if1	32/3	267
enp6s0f2np2	10.0.1.3	64:9d:99:b1:0a:8a	tb_node1_if2	32/1	265
enp6s0f3np3	10.0.1.4	64:9d:99:b1:0a:8b	tb_node1_if3	32/0	264

ncs-node2

Control: enp4s0f2

Data:

Interface	IP	MAC	Namespace	Wedge Port	Wedge D_P
enp6s0f0np0	10.0.2.1	64:9d:99:b1:0a:a4	tb_node2_if0	31/0	256
enp6s0f1np1	10.0.2.2	64:9d:99:b1:0a:a5	tb_node2_if1	31/3	259
enp6s0f2np2	10.0.2.3	64:9d:99:b1:0a:a6	tb_node2_if2	31/1	257
enp6s0f3np3	10.0.2.4	64:9d:99:b1:0a:a7	tb_node2_if3	31/2	258

ncs-wedge

PORT	MAC	D_P	P/PT	SPEED	FEC	AN	KR	RDY	ADM	OPR	LPBK	FRAMES RX	FRAMES TX	E
31/0	48/0	256	2/ 0	10G	NONE	Ds	Au	YES	ENB	UP	NONE	62	0
31/1	48/1	257	2/ 1	10G	NONE	Ds	Au	YES	ENB	UP	NONE	62	0
31/2	48/2	258	2/ 2	10G	NONE	Ds	Au	YES	ENB	UP	NONE	62	0
31/3	48/3	259	2/ 3	10G	NONE	Ds	Au	YES	ENB	UP	NONE	62	0
32/0	50/0	264	2/ 8	10G	NONE	Ds	Au	YES	ENB	UP	NONE	62	0
32/1	50/1	265	2/ 9	10G	NONE	Ds	Au	YES	ENB	UP	NONE	62	0
32/2	50/2	266	2/10	10G	NONE	Ds	Au	YES	ENB	UP	NONE	65	0
32/3	50/3	267	2/11	10G	NONE	Ds	Au	YES	ENB	UP	NONE	65	0

Adding new node

We want to add a new node: =ncs-node3= to our network.

We'll connect the 4 sfp+ ports of our node via a qsfp+ breakout cable to the Tofino Wedge on port 30.

Given that the 4 interfaces on =ncs-node3= have the names:

• enp6s1 with 00:00:00:00:00:01 (MAC) and 10.0.3.1 (IP)
• enp6s2 with 00:00:00:00:00:02 (MAC) and 10.0.3.2 (IP)
• enp6s3 with 00:00:00:00:00:03 (MAC) and 10.0.3.3 (IP)
• enp6s4 with 00:00:00:00:00:04 (MAC) and 10.0.3.4 (IP)

Find and replace the actual MAC addresses of the actual interfaces by running ip a s on the node.

Adjust ./03-wedge_port_configure.command

Add:

port-add 30/- 10G NONE
an-set 30/- 2
port-enb 30/-

• port-add: Add the port
• an-set: Set auto negotiation
• port-enb: Enable the port

To the command script before the

exit
exit

When running this take note of the D_P column of the new ports. You'll need to enter them as egress_port in the next script.

Adjust ./04-wedge_route_configure.py

Add the following:

Replace ??? by the actual port number for the respective interface.

bfrt.l1switch.pipe.SwitchIngress.t_l1_forwarding.add_with_send(
    egress_port=???, dst_addr="10.0.3.1"
)
bfrt.l1switch.pipe.SwitchIngress.t_l1_forwarding.add_with_send(
    egress_port=???, dst_addr="10.0.3.2"
)
bfrt.l1switch.pipe.SwitchIngress.t_l1_forwarding.add_with_send(
    egress_port=???, dst_addr="10.0.3.3"
)
bfrt.l1switch.pipe.SwitchIngress.t_l1_forwarding.add_with_send(
    egress_port=???, dst_addr="10.0.3.4"
)

• egress_port: Value at the D_P column at the output of the last script.

Adjust ./10-host_setup.sh

Add a section for you new node like this:

# node3
sudo ./30-add_namespaces.sh tb_node3_if0
sudo ./30-add_namespaces.sh tb_node3_if1
sudo ./30-add_namespaces.sh tb_node3_if2
sudo ./30-add_namespaces.sh tb_node3_if3

sudo ./31-configure_interfaces.sh tb_node3_if0 enp6s1 10.0.4.1/16
sudo ./31-configure_interfaces.sh tb_node3_if1 enp6s2 10.0.4.2/16
sudo ./31-configure_interfaces.sh tb_node3_if2 enp6s3 10.0.4.3/16
sudo ./31-configure_interfaces.sh tb_node3_if3 enp6s4 10.0.4.4/16

sudo ./32-setup_arp.sh tb_node3_if0 enp6s1
sudo ./32-setup_arp.sh tb_node3_if1 enp6s2
sudo ./32-setup_arp.sh tb_node3_if2 enp6s3
sudo ./32-setup_arp.sh tb_node3_if3 enp6s4

Adjust ./32-setup_arp.sh before running this script.

Adjust ./32-setup_arp.sh

Add the following section for your new interfaces like this: Replace the placeholder mac address with the actual MAC addresses of the interfaces.

ip netns exec "$NS" arp -i "$IF" -s 10.0.3.1 00:00:00:00:00:01
ip netns exec "$NS" arp -i "$IF" -s 10.0.3.2 00:00:00:00:00:02
ip netns exec "$NS" arp -i "$IF" -s 10.0.3.3 00:00:00:00:00:03
ip netns exec "$NS" arp -i "$IF" -s 10.0.3.4 00:00:00:00:00:04