Turbo Drone is an Ostinato add-on that provides 10Gbps, 25Gbps and 40Gbps line rate traffic generation capability. It is a drop-in replacement for the Drone shipped along with the standard Ostinato package.
- Linux only (for the Drone agent); the GUI controller can run on any platform
- One of the following distros -
- CentOS 8
- Ubuntu 18.04 LTS
- Ubuntu 20.04 LTS
- Linux Kernel 5.3.0 or higher (check using
uname -a) - the default Linux Kernel versions with the above distros may be older and in such cases you will need to upgrade to a supported Kernel version
- CPU - Intel i7/i9/Xeon with one core for base system plus 2 cores per port i.e. for a single port, you need minimum 1+1x2=3 cores; for 2 ports, you need 1+2x2=5 cores; for 4 ports, you need 1+4x2=9 cores (Note: cores refer to actual cores, not hyper-threaded cores).
- Memory - 2GB or more
- NIC - Turbo drone uses the new Linux technologies of XDP and AF_XDP. Run
ethtool -i <intf-name>to get the corresponding driver name and check it against Linux drivers supporting XDP to see if your NIC is supported
- Always-on Internet access for license validation
- Rates can be guaranteed only on bare-metal. VM (with SR-IOV) may also get full line rate, but cannot be guaranteed - VM introduces too many performance related variables
- Ostinato/Wireshark cannot capture Turbo Drone generated packets (this is a limitation of the AF_XDP technology)
How to Install
Install the deb or rpm package as applicable to your system
sudo apt install ./ostinato-turbo-agent-<VERSION>.deb sudo yum --nogpgcheck install ./ostinato-turbo-agent-<VERSION>.rpm
/etc/xdg/Ostinato/drone.inito specify the 10/25/40G ports on your system and your license keys (each port needs a unique license key)
[Turbo] License/size=2 License/1/port=eth1 License/1/speed=10G License/1/key=<license-key1> License/2/port=eth2 License/2/speed=25G License/2/key=<license-key2>
If a port is not explicitly specified in the
Turbo section alongwith it's speed and license key, it will fall back to work as a normal Drone port (without the full line rate support).
How to Activate your license
To activate your license(s), make sure you fill in the port name, speed and license key details in
drone.ini as specified above and then run -
Follow the instructions given by the above command.
How to Run
Make sure you have activated your licenses before proceeding further.
From the Ostinato GUI, goto
File|Add New PortGroup and enter the hostname (or IP address) of the host running turbo-drone. Turbo ports will show up in the GUI annotated with (Turbo) -
Create a stream with
packets/sec as 0 (top speed) and
Next as Goto First to see what is the max rate that turbo drone can send on your system.
- Disable hyper-threading in the BIOS
- Change system profile for performance (requires
sudo tuned-adm profile throughput-performance
- Tune the system for high Ostinato performance
NOTE: It's highly reccomended to run
sudo turbo-tune config <list-of-space-separated-eth-ports>
turbo-tuneevery time before runing
turbo-droneto ensure the system is setup and configured/tuned correctly for
sudo turbo-drone -dto enable debug logs; save them in a file and provide for analysis
- Also provide output of
sudo turbo-tune info <list-of-space-separated-eth-ports>alongwith drone logs
If you are not getting the expected performance, in addition to the above, do the following and provide the logs -
Make sure you have the following utilities (if not, install them) -
timeout mpstat turbostat perfstat perf
Download and load this standard test stream in Ostinato and start traffic before running the below commands -
Find CPU utilization -
$ mpstat -P ALL 5 3 Linux 5.3.0-62-generic (sirius) Wednesday 05 May 2021 _x86_64_ (3 CPU) . . . Average: CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle Average: all 0.33 0.00 26.27 0.02 0.00 7.13 0.00 0.00 0.00 66.24 Average: 0 0.00 0.00 78.75 0.00 0.00 21.25 0.00 0.00 0.00 0.00 <<<<<< Average: 1 0.73 0.00 0.07 0.00 0.00 0.13 0.00 0.00 0.00 99.07 Average: 2 0.27 0.00 0.13 0.00 0.00 0.00 0.00 0.00 0.00 99.60
From the above output, note down the CPUs with %idle < 90% (i.e. non-idle CPUs)
For each of those CPUs, run the following commands replacing
with the actual CPU number (Each command takes about 20-30 seconds each) -
sudo turbostat -c <cpu> --quiet -n 3 | grep -v "-" sudo timeout 20 perf stat -C <cpu> -I 5000 sudo perf top -C <cpu> -E 10 --stdio # press 'q' after 10 seconds
Make sure you run the above commands for each non-idle CPU.
Provide all the logs for analysis