Turbo

Turbo Drone is an Ostinato add-on that provides 10Gbps, 25Gbps and 40Gbps line rate traffic generation capability. It is a drop-in replacement for the Drone shipped along with the standard Ostinato package.

10Gbps transmit rate

System requirements

  • Linux only (for the Drone agent); the GUI controller can run on any platform
  • One of the following distros -
    • CentOS 8
    • Ubuntu 18.04 LTS
    • Ubuntu 20.04 LTS
  • Linux Kernel 5.3.0 or higher (check using uname -a) - the default Linux Kernel versions with the above distros may be older and in such cases you will need to upgrade to a supported Kernel version
  • CPU - Intel i7/i9/Xeon with one core for base system plus 2 cores per port i.e. for a single port, you need minimum 1+1x2=3 cores; for 2 ports, you need 1+2x2=5 cores; for 4 ports, you need 1+4x2=9 cores (Note: cores refer to actual cores, not hyper-threaded cores).
  • Memory - 2GB or more
  • NIC - Turbo drone uses the new Linux technologies of XDP and AF_XDP. Run ethtool -i <intf-name> to get the corresponding driver name and check it against Linux drivers supporting XDP to see if your NIC is supported
  • Always-on Internet access for license validation

Limitations

  • Rates can be guaranteed only on bare-metal. VM (with SR-IOV) may also get full line rate, but cannot be guaranteed - VM introduces too many performance related variables
  • Ostinato/Wireshark cannot capture Turbo Drone generated packets (this is a limitation of the AF_XDP technology)

How to Install

  • Install the deb or rpm package as applicable to your system

    sudo apt install ./ostinato-turbo-agent-<VERSION>.deb
    sudo yum --nogpgcheck install ./ostinato-turbo-agent-<VERSION>.rpm
    
  • Edit /etc/xdg/Ostinato/drone.ini to specify the 10/25/40G ports on your system and your license keys (each port needs a unique license key)

    [Turbo]
    License/size=2
    License/1/port=eth1
    License/1/speed=10G
    License/1/key=<license-key1>
    License/2/port=eth2
    License/2/speed=25G
    License/2/key=<license-key2>
    

If a port is not explicitly specified in the Turbo section alongwith it's speed and license key, it will fall back to work as a normal Drone port (without the full line rate support).

How to Activate your license

To activate your license(s), make sure you fill in the port name, speed and license key details in drone.ini as specified above and then run -

turbo-drone -a

Follow the instructions given by the above command.

How to Run

Make sure you have activated your licenses before proceeding further.

sudo turbo-drone

From the Ostinato GUI, goto File|Add New PortGroup and enter the hostname (or IP address) of the host running turbo-drone. Turbo ports will show up in the GUI annotated with (Turbo) - Turbo port in GUI

Create a stream with packets/sec as 0 (top speed) and Next as Goto First to see what is the max rate that turbo drone can send on your system.

If it's not full line rate allowed by your port and your Turbo license, follow the below performance tips and if that doesn't help, reach out to support with the info listed in debug tips

Performance tips

  • Disable hyper-threading in the BIOS
  • Change system profile for performance (requires tuned)
    sudo tuned-adm profile throughput-performance
    
  • Tune the system for high Ostinato performance
    sudo turbo-tune config <list-of-space-separated-eth-ports>
    
    NOTE: It's highly reccomended to run turbo-tune every time before runing turbo-drone to ensure the system is setup and configured/tuned correctly for turbo-drone.

Debug tips

  • Run sudo turbo-drone -d to enable debug logs; save them in a file and provide for analysis
  • Also provide output of sudo turbo-tune info <list-of-space-separated-eth-ports> alongwith drone logs

If you are not getting the expected performance, in addition to the above, do the following and provide the logs -

  • Make sure you have the following utilities (if not, install them) -

    timeout
    mpstat
    turbostat
    perfstat
    perf
    
  • Download and load this standard test stream in Ostinato and start traffic before running the below commands -

  • Find CPU utilization -

    $ mpstat -P ALL 5 3
    Linux 5.3.0-62-generic (sirius)         Wednesday 05 May 2021   _x86_64_        (3 CPU)
      . . .
    Average: CPU   %usr  %nice   %sys %iowait   %irq  %soft %steal %guest %gnice  %idle
    Average: all   0.33   0.00  26.27    0.02   0.00   7.13   0.00   0.00   0.00  66.24
    Average:   0   0.00   0.00  78.75    0.00   0.00  21.25   0.00   0.00   0.00   0.00 <<<<<<
    Average:   1   0.73   0.00   0.07    0.00   0.00   0.13   0.00   0.00   0.00  99.07
    Average:   2   0.27   0.00   0.13    0.00   0.00   0.00   0.00   0.00   0.00  99.60
    

    From the above output, note down the CPUs with %idle < 90% (i.e. non-idle CPUs)

  • For each of those CPUs, run the following commands replacing with the actual CPU number (Each command takes about 20-30 seconds each) -

    sudo turbostat -c <cpu> --quiet -n 3 | grep -v "-"
    sudo timeout 20 perf stat -C <cpu> -I 5000
    sudo perf top -C <cpu> -E 10 --stdio # press 'q' after 10 seconds
    

    Make sure you run the above commands for each non-idle CPU.

Provide all the logs for analysis

results matching ""

    No results matching ""