Adding Apdex (Application Performance inDEX) on reporting.

The Apdex is a numerical measure of user satisfaction, it is based on three zones of application responsiveness:

• Satisfied: The user is fully productive. This represents the time value (T seconds) below which users are not impeded by application response time.
• Tolerating: The user notices performance lagging within responses greater than T, but continues the process.
• Frustrated: Performance with a response time greater than 4*T seconds is unacceptable, and users may abandon the process.

By default T is set to 1.5s this means that response time between 0 and 1.5s the user is fully productive, between 1.5 and 6s the responsivness is tolerating and above 6s the user is frustrated.

The T variable can be set using the fl-build-report option --apdex-T.

The Apdex score converts many measurements into one number on a uniform scale of 0-to-1 (0 = no users satisfied, 1 = all users satisfied).

To ease interpretation the Apdex score can be converted to a rating:

• Unacceptable represented in gray for a score between 0 and 0.5
• Poor represented in red for a score between 0.5 and 0.7
• Fair represented in yellow for a score between 0.7 and 0.85
• Good represented in green for a score between 0.85 and 0.94
• Excellent represented in blue for a score between 0.94 and 1

By looking at the Page stats in the report you can take the maximum throughput with a Good/Excellent rating as an overall performance score.

Note that this new metric can be generated with old funkload results file.

Here is a report example: http://funkload.nuxeo.org/report-example/test_seam_java6/#page-stats

Visit http://www.apdex.org/ for more information on Apdex.

The selection of the best cycle to sort slowest requests in the report is found using the product of throuthput and apdex score instead of simple maximum.

fl-build-report is now able to create a report using many result files. The goal of this feature is to create bench report for a distributed bench. To be merged the result files must have the same cycles and cycle duration.

The hostname and python version is now logged in the result file.

Try to use psyco if installed. Note that psyco works only on 32-bit system.

Miscellaneous speed improvments: removing minimal sleep time between actions, removing useless logs.

New --as-fast-as-possible or -f option to fl-run-bench to remove sleep any times between requests and test cases, this is equivalent to -m0 -M0 -t0.

New --enable-debug-server option for fl-run-bench command. This option run a debug HTTP server which exposes an interface using which parameters can be modified at run-time.

Currently supported parameters:

• http://localhost:8000/cvu?inc=<integer> to increase the number of CVUs,
• http://localhost:8000/cvu?dec=<integer> to decrease the number of CVUs,
• http://localhost:8000/getcvu returns number of CVUs.

You can load a server and be able to change the number of concurrent users during the run without pre-specifying it through 'cycles' argument, this is usefull during debugging or profiling.

This feature has been provided by Goutham Bhat.

Switch to gnuplot 4.2 for charts generation. Gnuplot script and data are kept with the report enabling charts customization. No more gdchart charting woes.

Enhanced charts displaying more information, view a report example.

New diff report option, comparing 2 reports is a long and error prone task, fl-build-report has been enhanced to produce a differencial report. This new feature requires gnuplot.

To produce a diff report:

fl-build-report --diff path/to/report-reference path/to/report-challanger

View a diff report example.

Post method handles custom data and content type. For example to post a xmlrpc by hand

from funkload.utils import Data
...
    data = Data('text/xml', """<?xml version='1.0'?>
<methodCall>
<methodName>getStatus</methodName>
<params>
</params>
</methodCall>
""")
    self.post(server_url, data, description="Post user data")

The recorder translates properly application/xml or any content type using the new Data class (see above exemple).

New test script provided with fl-install-demo to bench the JBoss Seam Booking application.

# 1284: TestCase: support of doctest

There is a new FunkLoadDocTest class that ease usage of FunkLoad from doctest:

>>> from funkload.FunkLoadDocTest import FunkLoadDocTest
>>> fl = FunkLoadDocTest()
>>> response = fl.get('http://localhost/')
>>> 'HTML' in response.body
True
>>> response
<response url="http://127.0.0.1:80//" code="200" message="OK" />

If you use python2.4, the test runner fl-run-test is able launch doctest from a plain text file or embedded in python docstring:

$ fl-run-test -v doctest_dummy.txt
Doctest: doctest_dummy.txt          Ok
-----------------------------------------------------
Ran 1 test in 0.077s
OK

And the --debug option makes doctests verbose:

$ fl-run-test -d doctest_dummy.txt
...
Trying:
    fl = FunkLoadDocTest()
Expecting nothing
ok
Trying:
    fl.get('http://localhost/')
Expecting:
    <response url="http://127.0.0.1:80//" code="200" message="OK" />
ok
Ok
----------------------------------------------------------------------
Ran 1 test in 0.051s
OK

Test runner can use a negative regex to select tests. For example if you want to launch all tests that does not ends with 'foo'

fl-run-test myFile.py -e '!foo$'

# 1282: TestRunner: more verbosity

The new fl-run-test option --debug-level=2 will produce debug output on each link (images or css) fetched.

Improve firefox view in real time by using approriate file extention for the content type.

CPSTestCase is up to date for 3.4.0, use CPS338TestCase for a CPS 3.3.8.

New --loop-on-pages option for fl-run-test.

This option enable to check response time of some specific pages inside a test without changing the script, which make easy to tune a page in a complex context. Use the debug option to find the page numbers. For example:

fl-run-test myfile.py MyTestCase.testSomething -l 3 -n 100

Run MyTestCase.testSomething, reload one hundred time the page 3 without concurrency and as fast as possible. Output response time stats. You can loop on many pages using slice -l 2:4.

New --accept-invalid-links option for fl-run-test and fl-run-bench

Don't fail if css/image links are not reachable.

New --list option for fl-run-test to list the test names without running them.

# 936: TestRunner: use regexp to load test

New --regex or -e option for fl-run-test to filter test names that match a regular expression.

# 939: Browser: Provide an option to disable image and links load

New --simple-fetch option for fl-run-test and fl-run-bench.

# 937: TestRunner: Add an immediate fail option

New --stop-on-fail option for fl-run-test that stops tests on first failure or error.

# 933: Report: Add global info

Adding total number of tests, pages and requests during the bench.

CPSTestCase.listDocumentHref is renamed into cpsListDocumentHref

FunkLoadTestCase.xmlrpc_call is renamed into xmlrpc (xmlrpc_call is still working)

Some code cleaning, cheesecake index 460/560 ~82%.

New epydoc API documentation.

fl-run-test is now able to run standard unittest.TestCase.

# 944: Recorder: replace TestMaker recorder.

Providing a fl-record command that drive a TCPWatch proxy. See INSTALL to setup TCPWatch.

# 1041: Browser: implement an addHeader method.

FunkLoadTestCase provides new methods setUserAgent, addHeader and clearHeaders.

# 1088: TestRunner / BenchRunner: use compatible command line option

• All fl-* executables have a --version option to display the FunkLoad version.
• All fl-run-* are now in color mode by default. Use --no-color options for monochrome output. You need to remove the -c option for fl-run-test and -C for fl-run-bench in your scripts.
• Changing fl-run-bench short option -d into -D for duration, keeping -d for debug mode.
• Removing fl-run-test short option -D to not conflict with new -D option of fl-run-bench, you now have to use the long format --dump-directory.