diff options
Diffstat (limited to 'fbench')
-rw-r--r-- | fbench/README | 95 | ||||
-rwxr-xr-x | fbench/util/plot.pl | 82 | ||||
-rwxr-xr-x | fbench/util/pretest.sh | 10 | ||||
-rwxr-xr-x | fbench/util/runtests.sh | 92 | ||||
-rwxr-xr-x | fbench/util/separate.pl | 29 |
5 files changed, 0 insertions, 308 deletions
diff --git a/fbench/README b/fbench/README index 8d217ad0dc5..17807cb3931 100644 --- a/fbench/README +++ b/fbench/README @@ -20,16 +20,8 @@ The above installation provides the follwing vespa-fbench executables: /opt/vespa/bin/vespa-fbench /opt/vespa/bin/vespa-fbench-filter-file /opt/vespa/bin/vespa-fbench-geturl - /opt/vespa/bin/vespa-fbench-result-filter.pl /opt/vespa/bin/vespa-fbench-split-file -Additional utilities referenced in this document can be fetched from -https://github.com/vespa-engine/vespa/tree/master/fbench/util: - plot.pl - pretest.sh - runtests.sh - separate.pl - It is also possible to use Docker to directly execute vespa-fbench by using the pre-built Vespa docker image: docker run --entrypoint /opt/vespa/bin/vespa-fbench \ @@ -251,90 +243,3 @@ results. This section will explain what each of these numbers mean. utilization will drop since the client has 'wasted' the time spent on the failed request. - - -6 Running test series ---------------------- - -For more complete benchmarking you will want to combine the results -from several test runs and present them together in a graph or maybe a -spreadsheet. The perl script vespa-fbench-result-filter.pl may be used to convert -the output from vespa-fbench into a single line of numbers. Lines of numbers -produced from several test runs may then be concatenated into the same -text file and used to plot a graph with gnuplot or imported into an -application accepting structured text files (like Excel). - -The task described above is performed by the runtests.sh script. It -runs vespa-fbench several times with varying client count and cycle -time. Between each test run, the script pretest.sh (located in the bin -directory) is run. The pretest.sh script should make sure that the -server you want to benchmark is in the same state before each of the -test runs. This typically means that the caches should be cleared. The -supplied pretest.sh file does nothing, and should therefore be -modified to fit your needs before you start benchmarking with the -runtests.sh script. NOTE: 'runtests.sh' must be run from the vespa-fbench -install directory in order to find the scripts and programs it depends -on. (vespa-fbench is run as 'bin/vespa-fbench' etc.). - -| usage: runtests.sh [-o] [-l] <minClients> <maxClients> <deltaClients> -| <minCycle> <maxCycle> <deltaCycle> [vespa-fbench options] <hostname> <port> -| -| The number of clients varies from <minClients> to <maxClients> with -| <deltaClients> increments. For each client count, the cycle time will -| vary in the same way according to <minCycle>, <maxCycle> and <deltaCycle>. -| vespa-fbench is run with each combination of client count and cycle time, and -| the result output is filtered with the 'vespa-fbench-result-filter.pl' script. -| If you want to save the results you should redirect stdout to a file. -| -| -o : change the order in which the tests are performed so that client -| count varies for each cycle time. -| -l : output a blank line between test subseries. If -o is not specified this -| will output a blank line between test series using different client count. -| If -o was specified this will output blank lines between test series -| using different cycle time. -| -| [vespa-fbench options] <hostname> <port>: These arguments are passed to vespa-fbench. -| There are 2 things to remember: first; do not specify either of the -n -| or -c options since they will override the values for client count and -| cycle time generated by this script. secondly; make sure you specify -| the correct host and port number. See the vespa-fbench usage (run vespa-fbench -| without parameters) for more info on how to invoke vespa-fbench. - -Example: You want to see how well fastserver performs with varying -client count and cycle time. Assume that you have already prepared 200 -query files. To test with client count -from 10 to 200 with intervals of 10 clients and cycle time from 0 to -5000 milliseconds with 500 ms intervals you may do the following: - -$ bin/runtests.sh 10 200 10 0 5000 500 <host> <port> - -The duration of each test run will be 60 seconds (the default). This -may be a little short. You will also get all results written directly -to your console. Say you want to run each test run for 5 minutes and -you want to collect the results in the file 'results.txt'. You may -then do the following: - -$ bin/runtests.sh 10 200 10 0 5000 500 -s 300 <host> <port> > result.txt - -The '-s 300' option will be given to vespa-fbench causing each test run to -have a duration of 300 seconds = 5 minutes. The standard output is -simply redirected to a file to collect the results for future use. - -The perl utility scripts separate.pl and plot.pl may be used to create -graphs using gnuplot. - -| usage: separate.pl <sepcol> -| Separate a tabular numeric file into chunks using a blank -| line whenever the value in column 'sepcol' changes. - -| usage: plot.pl [-h] [-x] <plotno> -| Plot the contents of 'result.txt'. -| -h This help -| -x Output to X11 window (default PS-file 'graph.ps') -| plotno: 1: Response Time Percentiles by NumCli -| 2: Rate by NumCli -| 3: Response Time Percentiles by Rate - -Note that the separate.pl script does the same thing as the -l option -of runtests.sh; it inserts blank lines into the result to let gnuplot -interpret each chunk as a separate dataseries. diff --git a/fbench/util/plot.pl b/fbench/util/plot.pl deleted file mode 100755 index 45311d9f97c..00000000000 --- a/fbench/util/plot.pl +++ /dev/null @@ -1,82 +0,0 @@ -#!/usr/bin/perl -s -# Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. - -# TODO -# - parameter for input and output file name -# - more graphs - -sub usage { - die qq{usage: plot.pl [-h] [-x] <plotno> <format> -Plot the contents of 'result.txt' to 'graph.<format>'. - -h This help - -x Output to X11 window - plotno: 1: Response Time Percentiles by NumCli - 2: Rate by NumCli - 3: Response Time Percentiles by Rate - format: png (default), ps -}; -} - -$plotno = shift || die usage; -$term = shift || "png"; - -if ($h) { - usage; -} - -# setup the output -if ($x) { - # X11 output - open(PLOTSCRIPT, "| gnuplot -persist"); - print PLOTSCRIPT "set term X11\n"; - -} else { - open(PLOTSCRIPT, "| gnuplot"); - if ("$term" eq "ps") { - print PLOTSCRIPT "set term postscript\n"; - print PLOTSCRIPT "set output \"graph.ps\"\n"; - } - else { - print PLOTSCRIPT "set term png transparent small medium enhanced\n"; - print PLOTSCRIPT "set output \"graph.png\"\n"; - } -} -select(PLOTSCRIPT); - - - -# choose the graph -if ($plotno == 1) { - # Cli Percentile - print qq{ -set data style lines -set title "Response Time Percentiles by NumCli" -set xlabel "Number of clients" -set ylabel "Response time (msec)" -set key left top -plot 'result.txt' using 1:10 title "max", 'result.txt' using 1:17 title "99 %", 'result.txt' using 1:16 title "95 %", 'result.txt' using 1:15 title "90 %", 'result.txt' using 1:14 title "75 %", 'result.txt' using 1:13 title "50 %", 'result.txt' using 1:12 title "25 %", 'result.txt' using 1:9 title "min" - }; - -} elsif ($plotno == 2) { - # Cli Rate - print qq{ -set data style lines -set title "Rate by NumCli" -set xlabel "Number of clients" -set ylabel "Rate (queries/sec)" -set nokey -plot 'result.txt' using 1:18 - }; -} elsif ($plotno == 3) { - # Rate Percentile - print qq{ -set data style lines -set title "Response Time Percentiles by Rate" -set xlabel "Rate (queries/sec)" -set ylabel "Response time (msec)" -set key left top -plot 'result.txt' using 18:17 title "99 %", 'result.txt' using 18:16 title "95 %", 'result.txt' using 18:15 title "90 %", 'result.txt' using 18:14 title "75 %", 'result.txt' using 18:13 title "50 %", 'result.txt' using 18:12 title "25 %" - }; -} - -close(PLOTSCRIPT); diff --git a/fbench/util/pretest.sh b/fbench/util/pretest.sh deleted file mode 100755 index 1bf2a0372f8..00000000000 --- a/fbench/util/pretest.sh +++ /dev/null @@ -1,10 +0,0 @@ -#!/bin/sh -# Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. - -# -# This script will be run by the 'runtests.sh' script before -# each individual test run. - -# do not produce any output, log error messages to 'pretest.err' -exec > /dev/null 2>>pretest.err - diff --git a/fbench/util/runtests.sh b/fbench/util/runtests.sh deleted file mode 100755 index 1c5c7583e6e..00000000000 --- a/fbench/util/runtests.sh +++ /dev/null @@ -1,92 +0,0 @@ -#!/bin/sh -# Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. - -opt_o=false -opt_l=false - -opt_error=false - -while getopts "ol" option; do - case $option in - "o") opt_o=true;; - "l") opt_l=true;; - "*") opt_error=true;; - esac -done - -shift $(($OPTIND - 1)) -if [ $# -lt 8 ] || [ "$opt_error" = "true" ]; then - echo "usage: runtests.sh [-o] [-l] <minClients> <maxClients> <deltaClients>" - echo " <minCycle> <maxCycle> <deltaCycle> [vespa-fbench options] <hostname> <port>" - echo "" - echo "The number of clients varies from <minClients> to <maxClients> with" - echo "<deltaClients> increments. For each client count, the cycle time will" - echo "vary in the same way according to <minCycle>, <maxCycle> and <deltaCycle>." - echo "vespa-fbench is run with each combination of client count and cycle time, and" - echo "the result output is filtered with the 'vespa-fbench-result-filter.pl' script." - echo "If you want to save the results you should redirect stdout to a file." - echo "" - echo " -o : change the order in which the tests are performed so that client" - echo " count varies for each cycle time." - echo " -l : output a blank line between test subseries. If -o is not specified this" - echo " will output a blank line between test series using different client count." - echo " If -o was specified this will output blank lines between test series" - echo " using different cycle time." - echo "" - echo "[vespa-fbench options] <hostname> <port>: These arguments are passed to vespa-fbench." - echo " There are 2 things to remenber: first; do not specify either of the -n" - echo " or -c options since they will override the values for client count and" - echo " cycle time generated by this script. secondly; make sure you specify" - echo " the correct host and port number. See the vespa-fbench usage (run vespa-fbench" - echo " without parameters) for more info on how to invoke vespa-fbench." - exit 1 -fi - -minClients=$1; shift -maxClients=$1; shift -deltaClients=$1; shift -minCycle=$1; shift -maxCycle=$1; shift -deltaCycle=$1; shift - -if [ ! $deltaClients -gt 0 ]; then - echo "error: deltaClients must be greater than 0 !" - exit 1 -fi - -if [ ! $deltaCycle -gt 0 ]; then - echo "error: deltaCycle must be greater than 0 !" - exit 1 -fi - -echo "# vespa-fbench results collected by 'runtests.sh'." -echo "#" -echo "#1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20" -echo "#clients duration cycle lowlimit skip fail ok overtime min max avg 25% 50% 75% 90% 95% 99% rate util zerohit" -echo "#--------------------------------------------------------------------------------------------------" - -if [ "$opt_o" = "true" ]; then - cycle=$minCycle - while [ ! $cycle -gt $maxCycle ]; do - clients=$minClients - while [ ! $clients -gt $maxClients ]; do - test -f pretest.sh && ./pretest.sh > /dev/null 2>&1 - vespa-fbench -n $clients -c $cycle $@ | vespa-fbench-result-filter.pl - clients=$(($clients + $deltaClients)) - done - [ "$opt_l" = "true" ] && echo "" - cycle=$(($cycle + $deltaCycle)) - done -else - clients=$minClients - while [ ! $clients -gt $maxClients ]; do - cycle=$minCycle - while [ ! $cycle -gt $maxCycle ]; do - test -f pretest.sh && ./pretest.sh > /dev/null 2>&1 - vespa-fbench -n $clients -c $cycle $@ | vespa-fbench-result-filter.pl - cycle=$(($cycle + $deltaCycle)) - done - [ "$opt_l" = "true" ] && echo "" - clients=$(($clients + $deltaClients)) - done -fi diff --git a/fbench/util/separate.pl b/fbench/util/separate.pl deleted file mode 100755 index 418733bde7c..00000000000 --- a/fbench/util/separate.pl +++ /dev/null @@ -1,29 +0,0 @@ -#!/usr/bin/perl -# Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. - -$sepcol = shift; - -if ($sepcol eq "") { - die qq{usage: separate.pl <sepcol> - Separate a tabular numeric file into chunks using a blank - line whenever the value in column 'sepcol' changes. -}; -} - -$oldval = -2; -$newval = -2; - -while (<>) { - if (/^#/) { - print; - } else { - chomp; - @vals = split; - $newval = $vals[$sepcol]; - if ($newval != $oldval) { - print "\n"; - $oldval = $newval; - } - print "@vals\n"; - } -} |