Multi-Threaded File Search and Performance Tuning Options

VX Search is optimized for modern multi-core and multi-CPU systems and is capable of searching files stored on multiple disks, directories or network shares in parallel using all CPUs installed in the computer. VX Search provides a number of different performance optimization options allowing one to tune the file search operations for user-specific hardware and storage configurations.

In order to customize the file search performance optimization options, open the file search operation dialog, press the 'Options' button and select the 'Advanced' tab. The 'File Search Threads' option controls how many parallel threads are used to search files. The 'Directories Scanning Threads' option controls how many parallel threads are used to scan input disks, directories and network shares. In the 'Fault-Tolerant' directory scanning mode, VX Search uses an individual processing thread for each input disk, directory or network share, but limits the maximum number of parallel scanning threads to the specified value. In the high-performance directory scanning mode, VX Search always uses the specified number of parallel directory scanning threads even when processing a single input disk, directory or network share.

For example, when searching files stored on a high-speed NVMe SSD disk, VX Search reaches up to 53,000 files/sec using a single search thread. With two parallel search threads, the performance scales up to 88,000 files/sec and with four parallel search threads, the performance increases up to 129,000 files/sec showing a very good level of multi-threaded performance scalability. With six processing threads the file search performance reaches up to 155,000 files/sec and with eight processing threads the file search performance increases up to 169,000 files/sec allowing one to quickly search huge numbers of files.

When searching files stored on regular SATA SSD drives, which are significantly slower than NVMe SSD drives, the performance of the file search process reaches up to 25,000 files/sec using a single process thread and scales up to 62,000 files/sec with four parallel file search threads. With eight parallel file search threads, the performance reaches up to 72,000 files/sec, which allows to search large numbers of files very fast.

Searching file stored on a NAS storage device via a network is a more complicated task because the user needs to take into account the speed and the latency of the network. If the computer, on which VX Search is installed, is connected to the NAS storage device via a high-speed, low-latency network, the performance of the file search operations may reach up to 7,300 files/sec with one file search thread, scale up to 22,400 files/sec with four parallel file search threads and increase up to 32,100 files/sec with eight parallel search threads.

On the other hand, if VX Search will need to access network shares via the Internet or via a long-distance, high-latency network, the performance of the file search operations will be relatively slow. One of the options to increase the performance of the file search operations in such configurations is to set the 'High-Performance' directory scanning mode and increase the number of parallel file search threads to 16 or even 32 disregarding how many CPUs are actually installed on the computer.

Modern USB flash drives provide plenty of the storage space and are reasonably fast allowing one to store vast amounts of data for backup purposes. When searching files stored on a USB flash drive, VX Search can reach up to 9,500 files/sec with a single search thread. With two parallel search threads, the performance increases up to 10,700 files/sec, with four parallel threads the performance increases up to 12,900 files/sec and with six parallel file search threads the performance scales up to 13,500 files/sec.

Today, modern IT environments widely deploy virtual servers and/or virtual workstations. Most of the popular virtualization platforms provide a high level of performance, but depending on the target hardware and software platforms, significant performance degradations are inevitable when a file search operation is executed on a guest virtual machine compared to the same file search operation executed directly on the host computer.

For example, when a virtual machine with 4 virtual CPUs is stored on an SSD disk and searching files stored on a virtual local disk drive, which is physically stored on the same SSD disk, the performance of the file search operations reaches up to 3,200 files/sec using a single search thread. With two parallel search threads, the performance of the file search operations scales up to 5,800 files/sec and with four parallel search threads, the performance of the file search operations increases up to 10,100 files/sec.