Skip to content

Turbo

Turbo Drone is an Ostinato add-on that provides 10Gbps, 25Gbps and 40Gbps line rate traffic generation capability. It is a drop-in replacement for the Drone shipped along with the standard Ostinato package.

10Gbps transmit rate

System requirements

  • Linux only (for the Drone agent); the GUI controller can run on any platform
  • One of the following distros -
    • CentOS 8
    • Ubuntu 18.04 LTS
    • Ubuntu 20.04 LTS
    • Ubuntu 22.04 LTS
  • Linux Kernel 5.3.0 or higher (check using uname -a) - the default Linux Kernel versions with the above distros may be older. Your NIC might also need a higher kernel version. In both cases you will need to upgrade to a higher Kernel version
  • Intel i7/i9/Xeon multi-core CPU; number of cores required depends upon the number of ports on the host -
No. of Ports No. of CPU cores
1 port 3 CPU cores
2 ports 5 CPU cores
4 ports 9 CPU cores
n ports 2*n+1 cores

Note: cores refer to actual cores, not hyper-threaded cores

  • Memory - 2GB or more
  • XDP supported NIC - see list below alongwith the required Linux Kernel version. Use ethtool -i <intf-name> to get your NIC driver name. Intel NICs are recommended.
Driver Linux Kernel version
Amazon ena 5.6 or higher
Broadcom bnxt_en 5.3 or higher
Cavium thunderx 5.3 or higher
QLogic (Cavium) qed* 5.3 or higher
Freescale dpaa2 5.3 or higher
Intel i40e 5.3 or higher
Intel ice 5.5 or higher
Intel igb 5.10 or higher
Intel ixgbe* 5.3 or higher
Marvell mvneta 5.5 or higher
Mellanox mlx4, mlx5 5.3 or higher
Microsoft hv_netvsc 5.6 or higher
Netronome nfp 5.3 or higher
Socionext netsec 5.3 or higher
Solarflare sfc 5.5 or higher
TI cpsw 5.3 or higher
tap, veth, virtio_net 5.3 or higher
xen-netfront 5.9 or higher
  • Always-on Internet access for license validation

Limitations

  • Rates can be guaranteed only on bare-metal. VM (with SR-IOV) may also get full line rate, but cannot be guaranteed - VM introduces too many performance related variables
  • The max packet size that can be generated is between 3000 to 3500 bytes, depending on the NIC and driver (this is a limitation of the AF_XDP technology)
  • Ostinato/Wireshark cannot capture Turbo Drone generated packets (this is a limitation of the AF_XDP technology)

How to Install

  • Install the deb or rpm package as applicable to your system 

    sudo apt install ./ostinato-turbo-agent-<VERSION>.deb
sudo yum --nogpgcheck install ./ostinato-turbo-agent-<VERSION>.rpm

  • Edit /etc/xdg/Ostinato/drone.ini to specify the 10/25/40G ports on your system and your license keys (each port needs a unique license key) 

    [Turbo]
License/size=2
License/1/port=eth1
License/1/speed=10G
License/1/key=<license-key1>
License/2/port=eth2
License/2/speed=25G
License/2/key=<license-key2>

If a port is not explicitly specified in the Turbo section alongwith it's speed and license key, it will fall back to work as a normal Drone port (without the full line rate support).

How to Activate your license

To activate your license(s), make sure you fill in the port name, speed and license key details in drone.ini as specified above and then run - 

turbo-drone -a

Follow the instructions given by the above command.

How to Run

Make sure you have activated your licenses before proceeding further.

sudo turbo-drone

From the Ostinato GUI, goto File|Add New PortGroup and enter the hostname (or IP address) of the host running turbo-drone. Turbo ports will show up in the GUI annotated with (Turbo) - Turbo port in GUI

Create a stream with packets/sec as 0 (top speed) and Next as Goto First to see what is the max rate that turbo drone can send on your system.

If it's not full line rate allowed by your port and your Turbo license, follow the below performance tips and if that doesn't help, reach out to support with the info listed in debug tips

Performance tips

  • Disable hyper-threading in the BIOS
  • Change system profile for performance (requires tuned) 
    sudo tuned-adm profile throughput-performance
  • Tune the system for high Ostinato performance 
    sudo turbo-tune config <list-of-space-separated-eth-ports>
    NOTE: It's highly reccomended to run turbo-tune every time before runing turbo-drone to ensure the system is setup and configured/tuned correctly for turbo-drone.

Debug tips

  • Run sudo turbo-drone -d to enable debug logs; save them in a file and provide for analysis
  • Also provide output of sudo turbo-tune info <list-of-space-separated-eth-ports> alongwith drone logs

If you are not getting the expected performance, in addition to the above, do the following and provide the logs -

  • Make sure you have the following utilities (if not, install them) - 

    timeout
mpstat
turbostat
perfstat
perf

  • Download and load this standard test stream in Ostinato and start traffic before running the below commands -

  • Find CPU utilization - 

    $ mpstat -P ALL 5 3
Linux 5.3.0-62-generic (sirius)         Wednesday 05 May 2021   _x86_64_        (3 CPU)
    . . .
Average: CPU   %usr  %nice   %sys %iowait   %irq  %soft %steal %guest %gnice  %idle
Average: all   0.33   0.00  26.27    0.02   0.00   7.13   0.00   0.00   0.00  66.24
Average:   0   0.00   0.00  78.75    0.00   0.00  21.25   0.00   0.00   0.00   0.00 <<<<<<
Average:   1   0.73   0.00   0.07    0.00   0.00   0.13   0.00   0.00   0.00  99.07
Average:   2   0.27   0.00   0.13    0.00   0.00   0.00   0.00   0.00   0.00  99.60
    From the above output, note down the CPUs with %idle < 90% (i.e. non-idle CPUs)

  • For each of those CPUs, run the following commands replacing with the actual CPU number (Each command takes about 20-30 seconds each) - 

    sudo turbostat -c <cpu> --quiet -n 3 | grep -v "-"
sudo timeout 20 perf stat -C <cpu> -I 5000
sudo perf top -C <cpu> -E 10 --stdio # press 'q' after 10 seconds
    Make sure you run the above commands for each non-idle CPU.

Provide all the logs for analysis