Skip to content

Turbo

Note

Achievable performance depends on host CPU, NIC, and tuning configuration

Turbo Transmit extends Ostinato with advanced high-speed transmit capabilities, supporting up to 10G, 100G, and 400G line-rate traffic generation on optimized host systems.

10Gbps transmit rate

Turbo Software

A separate turbo-drone binary is no longer required.

Please use the base Ostinato for Linux drone binary with a Turbo license.

Use your Turbo license key to download the turbo-drone binary for one of the following supported Linux distros -

  • CentOS 8
  • Rocky 9
  • Ubuntu 20.04 LTS
  • Ubuntu 22.04 LTS
  • Ubuntu 24.04 LTS

System requirements

  • Linux Kernel 5.4.0 or higher (check using uname -a). Some NICs may require a higher kernel version
  • Intel/AMD (x86) multi-core CPU (more cores => better performance at smaller packet sizes)
  • Memory - 8GB or more
  • For best performance, Intel NICs are highly recommended

Limitations

  • Rates can be guaranteed only on bare-metal. VMs may also get full line rate, but be mindful that the hyper-visor may introduce performance restrictions due to resource-sharing
  • The max packet size that can be generated is between 3000 to 3500 bytes, depending on the NIC and driver (this is a limitation of the AF_XDP technology)
  • Ostinato/Wireshark cannot capture Turbo Drone generated packets on the same host (this is a limitation of the AF_XDP technology)

How to Install

  • Ostinato 2.0 for Linux includes the Turbo code, so no separate software installation is necessary

  • However, additional privileges are required - so assign those 

    sudo setcap cap_bpf,cap_sys_resource,cap_net_raw,cap_net_admin=eip `which drone`

  • Edit /etc/Ostinato/drone.ini to specify the 10/100/400G ports on your system and your license keys (each port needs a unique license key) 

    [Turbo]
Ports/size=2
Ports/1/name=eth1
Ports/1/speed=10G
Ports/1/key=<license-key1>
Ports/2/name=eth2
Ports/2/speed=100G
Ports/2/key=<license-key2>

  • Install the deb or rpm package as applicable to your system 

    sudo apt install ./ostinato-turbo-agent-<VERSION>.deb
OR
sudo yum --nogpgcheck install ./ostinato-turbo-agent-<VERSION>.rpm

  • Assign required privileges to turbo-drone 

    sudo setcap cap_bpf,cap_sys_resource,cap_net_raw,cap_net_admin=eip `which turbo-drone`

  • Edit /etc/xdg/Ostinato/drone.ini to specify the 10/40/100G ports on your system and your license keys (each port needs a unique license key) 

    [Turbo]
License/size=2
License/1/port=eth1
License/1/speed=10G
License/1/key=<license-key1>
License/2/port=eth2
License/2/speed=25G
License/2/key=<license-key2>

Info

If a port is not explicitly specified in the Turbo section in drone.ini along with it's speed and license key, it will fall back to work as a normal Drone port (without the accelerated high-speed support).

Activate your license

To activate your license(s), make sure you fill in the port name, speed and license key details in drone.ini as specified above and then run - 

drone -a
Follow the instructions given by the above command.

To activate your license(s), make sure you fill in the port name, speed and license key details in drone.ini as specified above and then run - 

turbo-drone -a
Follow the instructions given by the above command.

Tune your system

Please tune your system before running drone/turbo-drone.

Note

The following are general-purpose tuning guidelines. Because COTS hardware can vary significantly, if you find that these settings don’t deliver the desired performance, please contact support for host-specific fine-tuning assistance.

  • Disable hyper-threading in the BIOS
  • Change system profile for high performance (requires tuned) 
    sudo tuned-adm profile throughput-performance
  • Tune the system for peak Ostinato Turbo performance 
    sudo turbo-tune config <list-of-space-separated-eth-ports>

Tip

It's strongly recommended to run turbo-tune every time before runing turbo-drone to ensure the system is setup and configured/tuned correctly for turbo-drone.

How to Run

Run drone/turbo-drone.

From the Ostinato GUI, goto File|Add New PortGroup and enter the hostname (or IP address) of the host running drone/turbo-drone. Turbo ports will show up in the GUI annotated with (Turbo*n) where n is the number of CPU cores used for high-speed traffic generation -

Turbo port in GUI

Run a speed test

Open the speed-test streams in Ostinato and check what is the max rate that turbo drone can achieve on your system.

Turbo Test streams

There are multiple streams for different packet sizes. You can test the performance for a particular packet size by enabling that stream and disabling all the other streams.

If you don't get the desired performance (capped by your Turbo license speed), try to change the number of CPU cores used for Tx as shown below.

Adjust number of CPU cores

To achieve higher Tx rates, we need to use multiple CPU cores. The number of CPU cores used for Tx is determined as follows.

We first calculate the max packets/sec rate for the port's line rate (the max packet rate is for 64 byte packets). This Port Max packet rate is divided by the Max Tx Packets/sec per core rate (Ostinato uses a default of 20Mpps per core, but this can be changed).

Port Line Rate Port Max Packet Rate MaxPacketRate/core Cores used for Tx
10 Gbps 14,880,952 pkts/sec 20,000,000 1
100 Gbps 148,809,523 pkts/sec 20,000,000 8
400 Gbps 595,238,095 pkts/sec 20,000,000 32

You can change the Max Tx Packet Rate per core in drone.ini (count the zeroes and don't use commas) - 

[Turbo]
MaxTxPpsPerCore=10000000
...

Tip

Lower the value of MaxTxPpsPerCore, more no of CPU cores are used for Tx. Making it too low would mean you run out of cores!

Restart required

You MUST restart drone/turbo-drone after changing any setting in drone.ini file for it to take effect

Restart drone/turbo-drone and double-check the no of Tx cores used is as expected. Run a speed-test again and note the performance changes.

If your experiments with no of cores don’t deliver the desired performance, please contact support with the info listed in debug tips for assistance.

Debug tips

  • Run drone -d (or turbo-drone -d) to enable debug logs; save them in a file and provide for analysis
  • Also provide output of sudo turbo-tune info <list-of-space-separated-eth-ports> along with drone logs

Advanced logs

Only if support requests for it, please collect and provide these advanced logs -

  • Make sure you have the following utilities (if not, install them) - 

    timeout
mpstat
turbostat
perfstat
perf

  • Download and load this standard test stream in Ostinato and start traffic before running the below commands -

  • Find CPU utilization - 

    $ mpstat -P ALL 5 3
Linux 5.3.0-62-generic (sirius)         Wednesday 05 May 2021   _x86_64_        (3 CPU)
    . . .
Average: CPU   %usr  %nice   %sys %iowait   %irq  %soft %steal %guest %gnice  %idle
Average: all   0.33   0.00  26.27    0.02   0.00   7.13   0.00   0.00   0.00  66.24
Average:   0   0.00   0.00  78.75    0.00   0.00  21.25   0.00   0.00   0.00   0.00 <<<<<<
Average:   1   0.73   0.00   0.07    0.00   0.00   0.13   0.00   0.00   0.00  99.07
Average:   2   0.27   0.00   0.13    0.00   0.00   0.00   0.00   0.00   0.00  99.60
    From the above output, note down the CPUs with %idle < 90% (i.e. non-idle CPUs)

  • For each of those CPUs, run the following commands replacing with the actual CPU number (Each command takes about 20-30 seconds each) - 

    sudo turbostat -c <cpu> --quiet -n 3 | grep -v "-"
sudo timeout 20 perf stat -C <cpu> -I 5000
sudo perf top -C <cpu> -E 10 --stdio # press 'q' after 10 seconds
    Make sure you run the above commands for each non-idle CPU.

Provide all the logs for analysis

Back to top