Kea will support several different database backends, using both popular databases (like MySQL or SQLite) and custom-developed solutions (such as an in-memory database). To aid in the choice of backend, the BIND 10 source code features a set of performance microbenchmarks. Written in C/C++, these are small tools that simulate expected DHCP server behaviour and evaluate the performance of considered databases. As implemented, the benchmarks are not really simulating DHCP operation, but rather use set of primitives that can be used by a real server. For this reason, they are called micro-benchmarks.

Although there are many operations and data types that server could store in a database, the most frequently used data type is lease information. Although the information held for IPv4 and IPv6 leases differs slightly, it is expected that the performance differences will be minimal between IPv4 and IPv6 lease operations. Therefore each test uses the lease4 table (in which IPv4 leases are stored) for performance measurements.

All benchmarks are implemented as single threaded applications that take advantage of a single database connection.

Those benchmarks are stored in tests/tools/dhcp-ubench directory of the BIND 10 source tree. This directory contains simplified prototypes for the various database back-ends that are planned or considered as a possibly for BIND10 DHCP. These benchmarks are expected to evolve into useful tools that will allow users to measure performance in their specific environment.

Currently the following benchmarks are implemented:

As the benchmarks require additional (sometimes heavy) dependencies, they are not built by default. Actually, their build system is completely separate from that of the rest of BIND 10. It will be eventually merged with the main BIND 10 build system.

All benchmarks will follow the same pattern:

Although this approach does not attempt to simulate actual DHCP server operation that has mix of all steps, it answers the questions about basic database strengths and weak points. In particular it can show what is the impact of specific database optimizations, such as changing engine, optimizing for writes/reads etc.

The framework attempts to do the same amount of work for every backend thus allowing fair comparison between them.

The MySQL backend requires the MySQL client development libraries. It uses the mysql_config tool (similar to pkg-config) to discover required compilation and linking options. To install required packages on Ubuntu, use the following command:

$ sudo apt-get install mysql-client mysql-server libmysqlclient-dev

Make sure that MySQL server is running. Make sure that you have your setup configured so there is a user that is able to modify used database.

Before running tests, you need to initialize your database. You can use mysql.schema script for that purpose.

WARNING: It will drop existing Kea database. Do not run this on your production server.

Assuming your MySQL user is "kea", you can initialize your test database by:

$ mysql -u kea -p < mysql.schema

After the database is initialized, you are ready to run the test:

$ ./mysql_ubench

or

$ ./mysql_ubench > results-mysql.txt

Redirecting output to a file is important, because for each operation there is a single character printed to show progress. If you have a slow terminal, this may considerably affect test performance. On the other hand, printing something after each operation is required as poor database settings may slow down operations to around 20 per second. (The observant user is expected to note that the initial dots are printed too slowly and abort the test.)

Currently all default parameters are hardcoded. Default values can be overridden using command line switches. Although all benchmarks take the same list of parameters, some of them are specific to a given backend. To get a list of supported parameters, run the benchmark with the "-h" option:

$ ./mysql_ubench -h

Synchronous operation requires database backend to physically store changes to disk before proceeding. This property ensures that no data is lost in case of the server failure. Unfortunately, it slows operation considerably. Asynchronous mode allows database to write data at a later time (usually controlled by the database engine on OS disk buffering mechanism).

> show engines;

The SQLite backend requires both the sqlite3 development and run-time packages. Their names may vary from system to system, but on Ubuntu 12.04 they are called sqlite3 libsqlite3-dev. To install them, use the following command:

> sudo apt-get install sqlite3 libsqlite3-dev

Before running the test the database has to be created. Use the following command for that:

> cat sqlite.schema | sqlite3 sqlite.db

A new database called sqlite.db will be created. That is the default name used by sqlite_ubench test. If you prefer other name, make sure you update sqlite_ubench.cc accordingly.

Once the database is created, you can run tests:

> ./sqlite_ubench

or

> ./sqlite_ubench > results-sqlite.txt

The memfile backend is a custom backend that somewhat mimics operation of ISC DHCP4. It implements in-memory storage using standard C++ and boost mechanisms (std::map and boost::shared_ptr<>). All database changes are also written to a lease file, which is strictly write-only. This approach takes advantage of the fact that file append operation is faster than modifications introduced in the middle of the file (as it often requires moving all data after modified point, effectively requiring writing large parts of the whole file, not just changed fragment).

There are no preparatory steps required for memfile benchmark. The only requirement is the ability to create and write specified lease file (dhcpd.leases in the current directory). The tests can be run as follows:

> ./memfile_ubench

or

> ./memfile_ubench > results-memfile.txt

This section contains sample results for backend performance measurements, taken using microbenchmarks. Tests were conducted on reasonably powerful machine:

CPU: Quad-core Intel(R) Core(TM) i7-2600K CPU @ 3.40GHz (8 logical cores)
HDD: 1,5TB Seagate Barracuda ST31500341AS 7200rpm, ext4 partition
OS: Ubuntu 12.04, running kernel 3.2.0-26-generic SMP x86_64
compiler: g++ (Ubuntu/Linaro 4.6.3-1ubuntu5) 4.6.3
MySQL version: 5.5.24
SQLite version: 3.7.9sourceid version is 2011-11-01 00:52:41 c7c6050ef060877ebe77b41d959e9df13f8c9b5e

The benchmarks were run without using precompiled statements. The code was compiled with the -O0 flag (no code optimizations). Each run was executed once.

Two series of measures were made, synchronous and asynchronous. As those modes offer radically different performances, synchronous mode was conducted for one thousand repetitions and asynchronous mode was conducted for one hundred thousand repetitions.

The following parameters were measured for asynchronous mode. MySQL and SQLite were run with one hundred thousand repetitions.

The presented performance results can be converted into operations per second metrics. It should be noted that due to large differences between various operations (sometimes over three orders of magnitude), it is difficult to create a simple, readable chart with that data.

Graphical representation of the basic performance results presented in table Table 3.3, “Estimated basic performance”.

This section contains sample results for backend performance measurements, taken using microbenchmarks. Tests were conducted on reasonably powerful machine:

CPU: Quad-core Intel(R) Core(TM) i7-2600K CPU @ 3.40GHz (8 logical cores)
HDD: 1,5TB Seagate Barracuda ST31500341AS 7200rpm, ext4 partition
OS: Ubuntu 12.04, running kernel 3.2.0-26-generic SMP x86_64
compiler: g++ (Ubuntu/Linaro 4.6.3-1ubuntu5) 4.6.3
MySQL version: 5.5.24
SQLite version: 3.7.9sourceid version is 2011-11-01 00:52:41 c7c6050ef060877ebe77b41d959e9df13f8c9b5e

The benchmarks were run with precompiled statements enabled. The code was compiled with the -Ofast flag (optimize compilation for speed). Each run was repeated three times and measured values were averaged.

Again the benchmarks were run in two series, synchronous and asynchronous. As those modes offer radically different performances, synchronous mode was conducted for one thousand repetitions and asynchronous mode was conducted for one hundred thousand repetitions.

The following parameters were measured for asynchronous mode. MySQL and SQLite were run with one hundred thousand repetitions.

The presented performance results can be converted into operations per second metrics. It should be noted that due to large differences between various operations (sometime over three orders of magnitude), it is difficult to create a simple, readable chart with the data.

Graphical representation of the optimized performance results presented in table Table 3.6, “Estimated optimized performance”.

Improvements gained by introducing support for precompiled statements in MySQL is somewhat disappointing - between 6 and 29%. On the other hand, the improvement in SQLite is surprisingly high - the efficiency is more than doubled.

Compiled statements do not have any measureable impact on synchronous operations. That is as expected, because the major bottleneck is the disk performance.

Compilation flags yield surprisingly high improvements for C++ STL code. The memfile backend is in some operations is almost twice as fast.

If synchronous operation is required, the current performance results are likely to be deemed inadequate. The limiting factor here is a disk access time. Even migrating to high performance 15,000 rpm disk is expected to only roughly double number of leases per second, compared to the current results. The reason is that to write a file to disk, at least two disk sector writes are required: the new content and i-node modification of the file. The easiest way to boost synchronous performance is to switch to SSD disks. Memory-backed RAM disks are also a viable solution. However, care should be taken to properly engineer backup strategy for such RAM disks.

While the custom made backend (memfile) provides the best perfomance, it carries over all the limitations existing in the ISC DHCP4 code: there are no external tools to query or change database, the maintenance requires deep knowledge etc. Those flaws are not shared by usage of a proper database backend, like MySQL and SQLite. They both offer third party tools for administrative tasks, they are well documented and maintained. However, SQLite support for concurrent access is limiting in certain cases. Since all three investigated backends more than meet expected performance results, it is recommended to use MySQL as a first concrete database backend. Should this choice be rejected for any reason, the second recommended choice is SQLite.

It should be emphasized that obtained measurements indicate only database performance and they cannot be directly translated to expected leases per second or queries per second performance by an actual server. The DHCP server must do much more than just query the database to properly process client's message. The provided results should be considered as only rough estimates. They can also be used for relative comparisons between backends.

For basic measurements the code was compiled with -g -O0 flags. For optimized measurements the benchmarking code was compiled with -Ofast (optimize for speed). In both cases, the same backend (MySQL or SQLite) library was used. It may be useful to recompile the libraries (or the whole server in case of MySQL) with -Ofast.

There are many MySQL parameters that various sources recommend to improve performance. They were not investigated further.

Currently all operations are conducted on one by one basis. Each operation is treated as a separate transaction. Grouping N operations together will potentially bring almost N fold increase in synchronous operations. Such a feature is present in ISC DHCP4 and is called cache-threshold. Extension for this benchmark in this regard should be considered. That affects only write operations (insert, update and delete). Read operations (search) are expected to be barely affected.

Multi-threaded or multi-process benchmark may be considered in the future. It may be somewhat difficult as only some backends support concurrent access.

Evaluation of the performance of a DHCP server requires that it be tested under varying traffic loads. perfdhcp is a testing tool with the capability to create traffic loads and generate statistics from the results. Additional features, such as the ability to send customised DHCP packets, allow it to be used in a wide range of functional testing.

perfdhcp has a number of command line switches to control DHCP message exchanges. Currently they fall into the following categories:

The following "help" output from the tool describes the command line switches. This summary also lists the tool's possible exit codes as well as describing the error counters printed when the test is complete:

$ perfdhcp -h
perfdhcp [-hv] [-4|-6] [-r<rate>] [-t<report>] [-R<range>] [-b<base>]
    [-n<num-request>] [-p<test-period>] [-d<drop-time>] [-D<max-drop>]
    [-l<local-addr|interface>] [-P<preload>] [-a<aggressivity>]
    [-L<local-port>] [-s<seed>] [-i] [-B] [-c] [-1]
    [-T<template-file>] [-X<xid-offset>] [-O<random-offset]
    [-E<time-offset>] [-S<srvid-offset>] [-I<ip-offset>]
    [-x<diagnostic-selector>] [-w<wrapped>] [server]

The [server] argument is the name/address of the DHCP server to
contact.  For DHCPv4 operation, exchanges are initiated by
transmitting a DHCP DISCOVER to this address.

For DHCPv6 operation, exchanges are initiated by transmitting a DHCP
SOLICIT to this address.  In the DHCPv6 case, the special name 'all'
can be used to refer to All_DHCP_Relay_Agents_and_Servers (the
multicast address FF02::1:2), or the special name 'servers' to refer
to All_DHCP_Servers (the multicast address FF05::1:3).  The [server]
argument is optional only in the case that -l is used to specify an
interface, in which case [server] defaults to 'all'.

The default is to perform a single 4-way exchange, effectively pinging
the server.
The -r option is used to set up a performance test, without
it exchanges are initiated as fast as possible.

Options:
-1: Take the server-ID option from the first received message.
-4: DHCPv4 operation (default). This is incompatible with the -6 option.
-6: DHCPv6 operation. This is incompatible with the -4 option.
-a<aggressivity>: When the target sending rate is not yet reached,
    control how many exchanges are initiated before the next pause.
-b<base>: The base mac, duid, IP, etc, used to simulate different
    clients.  This can be specified multiple times, each instance is
    in the <type>=<value> form, for instance:
    (and default) mac=00:0c:01:02:03:04.
-d<drop-time>: Specify the time after which a request is treated as
    having been lost.  The value is given in seconds and may contain a
    fractional component.  The default is 1 second.
-E<time-offset>: Offset of the (DHCPv4) secs field / (DHCPv6)
    elapsed-time option in the (second/request) template.
    The value 0 disables it.
-h: Print this help.
-i: Do only the initial part of an exchange: DO or SA, depending on
    whether -6 is given.
-I<ip-offset>: Offset of the (DHCPv4) IP address in the requested-IP
    option / (DHCPv6) IA_NA option in the (second/request) template.
-l<local-addr|interface>: For DHCPv4 operation, specify the local
    hostname/address to use when communicating with the server.  By
    default, the interface address through which traffic would
    normally be routed to the server is used.
    For DHCPv6 operation, specify the name of the network interface
    via which exchanges are initiated.
-L<local-port>: Specify the local port to use
    (the value 0 means to use the default).
-O<random-offset>: Offset of the last octet to randomize in the template.
-P<preload>: Initiate first <preload> exchanges back to back at startup.
-r<rate>: Initiate <rate> DORA/SARR (or if -i is given, DO/SA)
    exchanges per second.  A periodic report is generated showing the
    number of exchanges which were not completed, as well as the
    average response latency.  The program continues until
    interrupted, at which point a final report is generated.
-R<range>: Specify how many different clients are used. With 1
    (the default), all requests seem to come from the same client.
-s<seed>: Specify the seed for randomization, making it repeatable.
-S<srvid-offset>: Offset of the server-ID option in the
    (second/request) template.
-T<template-file>: The name of a file containing the template to use
    as a stream of hexadecimal digits.
-v: Report the version number of this program.
-w<wrapped>: Command to call with start/stop at the beginning/end of
    the program.
-x<diagnostic-selector>: Include extended diagnostics in the output.
    <diagnostic-selector> is a string of single-keywords specifying
    the operations for which verbose output is desired.  The selector
    keyletters are:
   * 'a': print the decoded command line arguments
   * 'e': print the exit reason
   * 'i': print rate processing details
   * 'r': print randomization details
   * 's': print first server-id
   * 't': when finished, print timers of all successful exchanges
   * 'T': when finished, print templates
-X<xid-offset>: Transaction ID (aka. xid) offset in the template.

DHCPv4 only options:
-B: Force broadcast handling.

DHCPv6 only options:
-c: Add a rapid commit option (exchanges will be SA).

The remaining options are used only in conjunction with -r:

-D<max-drop>: Abort the test if more than <max-drop> requests have
    been dropped.  Use -D0 to abort if even a single request has been
    dropped.  If <max-drop> includes the suffix '%', it specifies a
    maximum percentage of requests that may be dropped before abort.
    In this case, testing of the threshold begins after 10 requests
    have been expected to be received.
-n<num-request>: Initiate <num-request> transactions.  No report is
    generated until all transactions have been initiated/waited-for,
    after which a report is generated and the program terminates.
-p<test-period>: Send requests for the given test period, which is
    specified in the same manner as -d.  This can be used as an
    alternative to -n, or both options can be given, in which case the
    testing is completed when either limit is reached.
-t<report>: Delay in seconds between two periodic reports.

Errors:
- tooshort: received a too short message
- orphans: received a message which doesn't match an exchange
   (duplicate, late or not related)
- locallimit: reached to local system limits when sending a message.

Exit status:
The exit status is:
0 on complete success.
1 for a general error.
2 if an error is found in the command line arguments.
3 if there are no general failures in operation, but one or more
  exchanges are not successfully completed.

        

In order to run a performance test, at least two separate systems have to be installed: client and server. The first one must have perfdhcp installed, and the latter must be running the DHCP server (either v4 or v6). If only single system is available the client and server can be run on virtual machines (running on the same physical system) but in this case performance data may be heavily impacted by the overhead involved in running such the virtual machines.

Currently, perfdhcp is seen from the server perspective as relay agent. This simplifies its implementation: specifically there is no need to receive traffic sent to broadcast addresses. However, it does impose a requirement that the IPv4 address has to be set manually on the interface that will be used to communicate with the server. For example, if the DHCPv4 server is listening on the interface connected to the 172.16.1.0 subnet, the interface on client machine has to have network address assigned from the same subnet, e.g.

#ifconfig eth3 172.16.1.2. netmask 255.255.255.0 up

As DHCP uses low port numbers (67 for DHCPv4 relays and 547 for DHCPv6), running perfdhcp with non-root privileges will usually result in the error message similar to this:

$ perfdhcp -4 -l eth3 -r 100 all
Error running perfdhcp: Failed to bind socket 3 to 172.16.1.2/port=67
        

The '-L' command line switch allows the use of a custom local port. However, although the following command line will work:

$ perfdhcp -4 -l eth3 -r 100 -L 10067 all

in the standard configuration no responses will be received from the DHCP server because the server responds to default relay port 67. A way to overcome this issue is to run perfdhcp as root.

In this section, a number of perfdhcp command line examples are presented as a quick start guide for new users. For the detailed list of command line options refer to Section 4.3, “Command line options”.

# perfdhcp 172.16.1.1
***Rate statistics***
Rate: 206.345

***Statistics for: DISCOVER-OFFER***
sent packets: 21641
received packets: 350
drops: 21291
orphans: 0

min delay: 9.022 ms
avg delay: 143.100 ms
max delay: 259.303 ms
std deviation: 56.074 ms
collected packets: 30

***Statistics for: REQUEST-ACK***
sent packets: 350
received packets: 268
drops: 82
orphans: 0

min delay: 3.010 ms
avg delay: 152.470 ms
max delay: 258.634 ms
std deviation: 56.936 ms
collected packets: 0
          

# perfdhcp -l eth3

# perfdhcp -l 172.16.1.2

# perfdhcp -l eth3 -p 60 -r 300
***Rate statistics***
Rate: 256.683 exchanges/second, expected rate: 300 exchanges/second

***Statistics for: DISCOVER-OFFER***
sent packets: 17783
received packets: 15401
drops: 2382
orphans: 0

min delay: 0.109 ms
avg delay: 75.756 ms
max delay: 575.614 ms
std deviation: 60.513 ms
collected packets: 11

***Statistics for: REQUEST-ACK***
sent packets: 15401
received packets: 15317
drops: 84
orphans: 0

min delay: 0.536 ms
avg delay: 72.072 ms
max delay: 576.749 ms
std deviation: 58.189 ms
collected packets: 0
          

# perfdhcp -l eth3 -p 60 -r 100 -R 30
***Rate statistics***
Rate: 99.8164 exchanges/second, expected rate: 100 exchanges/second

***Statistics for: DISCOVER-OFFER***
sent packets: 5989
received packets: 5989
drops: 0
orphans: 0

min delay: 0.023 ms
avg delay: 2.198 ms
max delay: 181.760 ms
std deviation: 9.429 ms
collected packets: 0

***Statistics for: REQUEST-ACK***
sent packets: 5989
received packets: 5989
drops: 0
orphans: 0

min delay: 0.473 ms
avg delay: 2.355 ms
max delay: 189.658 ms
std deviation: 5.876 ms
collected packets: 0
          

# perfdhcp -6 -l eth3 -r 100 -R 20 -T solicit.hex -T request6.hex -O 21 -E 84 -S 22 -I 40 servers
***Rate statistics***
Rate: 99.5398 exchanges/second, expected rate: 100 exchanges/second

***Statistics for: SOLICIT-ADVERTISE***
sent packets: 570
received packets: 569
drops: 1
orphans: 0

min delay: 0.259 ms
avg delay: 0.912 ms
max delay: 6.979 ms
std deviation: 0.709 ms
collected packets: 0

***Statistics for: REQUEST-REPLY***
sent packets: 569
received packets: 569
drops: 0
orphans: 0

min delay: 0.084 ms
avg delay: 0.607 ms
max delay: 6.490 ms
std deviation: 0.518 ms
collected packets: 0