Turbo¶
Turbo Drone is an Ostinato add-on that provides 10Gbps, 25Gbps and 40Gbps line rate traffic generation capability. It is a drop-in replacement for the Drone shipped along with the standard Ostinato package.

System requirements¶
- Linux only (for the Drone agent); the GUI controller can run on any platform
- One of the following distros -
- CentOS 8
- Ubuntu 18.04 LTS
- Ubuntu 20.04 LTS
- Ubuntu 22.04 LTS
- Linux Kernel 5.3.0 or higher (check using
uname -a
) - the default Linux Kernel versions with the above distros may be older. Your NIC might also need a higher kernel version. In both cases you will need to upgrade to a higher Kernel version - Intel i7/i9/Xeon multi-core CPU; number of cores required depends upon the number of ports on the host -
No. of Ports | No. of CPU cores |
---|---|
1 port | 3 CPU cores |
2 ports | 5 CPU cores |
4 ports | 9 CPU cores |
n ports | 2*n+1 cores |
Note: cores refer to actual cores, not hyper-threaded cores
- Memory - 2GB or more
- XDP supported NIC - see list below alongwith the required Linux Kernel version. Use
ethtool -i <intf-name>
to get your NIC driver name. Intel NICs are recommended.
Driver | Linux Kernel version |
---|---|
Amazon ena |
5.6 or higher |
Broadcom bnxt_en |
5.3 or higher |
Cavium thunderx |
5.3 or higher |
QLogic (Cavium) qed* |
5.3 or higher |
Freescale dpaa2 |
5.3 or higher |
Intel i40e |
5.3 or higher |
Intel ice |
5.5 or higher |
Intel igb |
5.10 or higher |
Intel ixgbe* |
5.3 or higher |
Marvell mvneta |
5.5 or higher |
mlx4 , mlx5 |
|
Microsoft hv_netvsc |
5.6 or higher |
Netronome nfp |
5.3 or higher |
Socionext netsec |
5.3 or higher |
Solarflare sfc |
5.5 or higher |
TI cpsw |
5.3 or higher |
tap , veth , virtio_net |
5.3 or higher |
xen-netfront |
5.9 or higher |
- Always-on Internet access for license validation
Limitations¶
- Rates can be guaranteed only on bare-metal. VM (with SR-IOV) may also get full line rate, but cannot be guaranteed - VM introduces too many performance related variables
- The max packet size that can be generated is between 3000 to 3500 bytes, depending on the NIC and driver (this is a limitation of the AF_XDP technology)
- Ostinato/Wireshark cannot capture Turbo Drone generated packets (this is a limitation of the AF_XDP technology)
How to Install¶
-
Install the deb or rpm package as applicable to your system
sudo apt install ./ostinato-turbo-agent-<VERSION>.deb sudo yum --nogpgcheck install ./ostinato-turbo-agent-<VERSION>.rpm
-
Edit
/etc/xdg/Ostinato/drone.ini
to specify the 10/25/40G ports on your system and your license keys (each port needs a unique license key)[Turbo] License/size=2 License/1/port=eth1 License/1/speed=10G License/1/key=<license-key1> License/2/port=eth2 License/2/speed=25G License/2/key=<license-key2>
If a port is not explicitly specified in the Turbo
section alongwith it's speed and license key, it will fall back to work as a normal Drone port (without the full line rate support).
How to Activate your license¶
To activate your license(s), make sure you fill in the port name, speed and license key details in drone.ini
as specified above and then run -
turbo-drone -a
Follow the instructions given by the above command.
How to Run¶
Make sure you have activated your licenses before proceeding further.
sudo turbo-drone
From the Ostinato GUI, goto File|Add New PortGroup
and enter the hostname (or IP address) of the host running turbo-drone. Turbo ports will show up in the GUI annotated with (Turbo) -
Create a stream with packets/sec
as 0 (top speed) and Next
as Goto First to see what is the max rate that turbo drone can send on your system.
If it's not full line rate allowed by your port and your Turbo license, follow the below performance tips and if that doesn't help, reach out to support with the info listed in debug tips
Performance tips¶
- Disable hyper-threading in the BIOS
- Change system profile for performance (requires
tuned
)sudo tuned-adm profile throughput-performance
- Tune the system for high Ostinato performance
NOTE: It's highly reccomended to run
sudo turbo-tune config <list-of-space-separated-eth-ports>
turbo-tune
every time before runingturbo-drone
to ensure the system is setup and configured/tuned correctly forturbo-drone
.
Debug tips¶
- Run
sudo turbo-drone -d
to enable debug logs; save them in a file and provide for analysis - Also provide output of
sudo turbo-tune info <list-of-space-separated-eth-ports>
alongwith drone logs
If you are not getting the expected performance, in addition to the above, do the following and provide the logs -
-
Make sure you have the following utilities (if not, install them) -
timeout mpstat turbostat perfstat perf
-
Download and load this standard test stream in Ostinato and start traffic before running the below commands -
-
Find CPU utilization -
From the above output, note down the CPUs with %idle < 90% (i.e. non-idle CPUs)$ mpstat -P ALL 5 3 Linux 5.3.0-62-generic (sirius) Wednesday 05 May 2021 _x86_64_ (3 CPU) . . . Average: CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle Average: all 0.33 0.00 26.27 0.02 0.00 7.13 0.00 0.00 0.00 66.24 Average: 0 0.00 0.00 78.75 0.00 0.00 21.25 0.00 0.00 0.00 0.00 <<<<<< Average: 1 0.73 0.00 0.07 0.00 0.00 0.13 0.00 0.00 0.00 99.07 Average: 2 0.27 0.00 0.13 0.00 0.00 0.00 0.00 0.00 0.00 99.60
-
For each of those CPUs, run the following commands replacing
with the actual CPU number (Each command takes about 20-30 seconds each) - Make sure you run the above commands for each non-idle CPU.sudo turbostat -c <cpu> --quiet -n 3 | grep -v "-" sudo timeout 20 perf stat -C <cpu> -I 5000 sudo perf top -C <cpu> -E 10 --stdio # press 'q' after 10 seconds
Provide all the logs for analysis