SourceLabs AMP Stack Test Results
Here is a document that we've found extremely helpful for
determining server scaling, and optimum CPU and memory for
enterprise servers. I'm posting it here as it is no longer available
Summary
As part of its effort to provide greater confidence to enterprises who wish to use open source infrastructure
software, SourceLabs is releasing theSourceLabs AMP Stack v1.0. This stack, consisting of Apache Web
Server, MySQL, and PHP, has been tested for functionality (unit testing), hardened for improved security,
and profiled for scalability according to three of the seven dependability criteria addressed in SourceLabs
CERT7 methodology. Key findings include:
? The stack passes our acceptance tests as described in this document.
? A number of potential security vulnerabilities exist in default configurations of components of the
AMP stack. SourceLabs configured its stack to protect against these vulnerabilities.
? Apache Web Server scalability was limited by bandwidth in all cases tested, and not CPU or
memory capacity.
? PHP processing scaled linearly with respect to CPU capacity. For capacity planning purposes, this
means that the desired number of servers running Apache Web Server/PHP for a given application
can be calculated based on the processing speed of an individual user scenario. For this, an
Unit and Functional Testing
The individual unit tests were run for each individual component and are composed of the tests that are
downloaded with each individual package. The acceptance criteria included the need for these entire tests
to pass.
Results: PASS. The tests for Apache and MySQL were all run without any failures. The tests for PHP
were run, and it was noted that there were 12 test cases that failed. Each of these test cases were
individually examined and it was determined that the reason for these failures were not related to the
build or the package, and were therefore determined to be false positives. Examples of these failures
include items such as tests for issues that were closed by design/bogus, or the string that was being
compared to determine test success was different from what was expected (but the results were
examined to show the test passed). SourceLabs verified this conclusion with the PHP developer
community and will be working over time to improve the accuracy of these tests.
It was determined that this section of the acceptance testing succeeded.
Burn-in Testing
A rigorous burn-in test under load was performed on the system for a continuous period of 72 hours. These
tests were composed of a subset of the CERT7 scalability tests (static HTML, computationally intensive
PHP, database intensive functionality) described below.
CERT7 Security Hardening
Security testing was performed on the stack using two widely supported open source projects called Nessus
(http://www.nessus.org) and Nikto (http://www.cirt.net/code/nikto.shtml). Both of these tools use a plug-in
architecture to continually expand and test for new issues. The latest versions and complete list of plug-ins
were used for this portion of the test.
Initial results: Running these tests revealed several potential issues related to the configuration of the
AMP stack.
SourceLabs addressed these issues with the security hardening measures below:
1. The configuration of Apache web server is modified such that the user/group that owns the
components is different from the default of the ? nobody ? user. The ? nobody ? user has an underlying
association with the root user. As a result, a new and unprivileged user is created called ? webserv ?
for the server application. In addition, separate groups of ? webadmin ? and ? webdev ? are used for
other non-binary areas of the web server.
2. Removal of most ?f the def?ult files. These files include the default HTML files that are typically
setup in the htdocs directory, the manual files contained off the htdocs directory, and the default
cgi files that are added to the cgi-bin directory. The purpose for removing these files is to reduce
the amount of information that can be obtained from the server by someone maliciously looking at
known areas. These files are all available online on the Apache website.
The CERT7 scalability tests were broken into three main categories: static HTML delivery, computational
PHP, and database testing.
Static HTML Tests
To perform the static HTML tests, a series of static web pages were constructed of known sizes, and a
number of distinct users were modeled requesting these pages through standard HTTP calls. The tests
measured the throughput of the return data. The number of users was increased to determine what the load
SourceLabs AMP Stack Test Results March 25, 2005
Page 4
The AS3AP multi-user tests model four different workloads:
1. Information Retrieval (IR) Throughput with IR background: All users select a random row from the
same table using the primary key.
2. IR Throughput with OLTP Background: One user executes the same query as in the IR Throughput
with IR background test, while all other (background) users update a value in a random row from
the same table using the primary key. The column that is updated is not indexed.
3. Cross-section with IR Background: One user executes a cross-section of update and select queries
while all other users execute the same query as in the IR Throughput with IR background test.
4. Cross-section with OLTP Background: One user executes a cross-section of update and select
queries while all other users execute the same query as the background users in the IR Throughput
with OLTP Background test.
The AS3AP specification provides details about the database structure and the queries that are used in the
tests. A development version of the benchmark is available from MySQL
http://www.mysqlusers.com/msg/4161.html ). This version was used to generate data for the 4GB MySQL
database that was used in all tests. All tables use the MyISAM table engine. Of the five tables that are
generated, 3 are used by the multi-user tests: updates, hundred, and tiny.
The AS3AP specification indicates the type of index that should be defined for each index. The possible
Page 5
1. Start up apache and MySQL.
2. Load indexes for the updates and hundred tables into the index cache.
3. Start up 6 users that execute the ir_select query repeatedly.
4. Run IR Throughput with IR Background test. After a 15 minute warm-up, measure IR throughput for
one of the users executing the ir_select query for 5 minutes (while the other 5 users continue to
execute the ir_select query).
5. Run Cross-section with IR Background test. Measure execution time for one user running the
cross-section queries (while the other 5 users continue to execute the ir_select query).
6. Restore the updates table.
RAM Amount 4 x 512 (2 GB) 2 x 512 (1 GB)
4 x 512 (2 GB)
4 x 512 + 2 x 1 GB (4 GB)
Hard Drives 2 X 40GB SATA No RAID 2 X 40GB SATA No RAID
HD Capacity 80 GB 80 GB
NIC e1000 e1000
Net Speed 100 Mbps 100 Mbps
Distro RHEL 3.2 RHEL 3.2
Kernel 2.4.21-27 2.4.21-27
Architecture x86_64 x86_64
The following configurations were used in our testing:
? 1 CPU box 2GB of memory
? 2 CPU box 2GB of memory
? 2 CPU box with 1 GB of memory
? 2 CPU box with 2 GB of memory
? 2 CPU box with 4 GB of memory
Each configuration executes the same set of tests. The tests vary only in a number of parallel threads
spawned (from 1 to 70). Each thread was running a loop with 1000 iterations, each iteration
downloads the same html file (1150 Bytes long).
http://jakarta.apache.org/jmeter
JMeter [ ] was used to run the tests and gather results.
The attached Excel spread sheet shows that the throughput is a linear function of number of threads.
It also shows that hardware differences have virtually no impact on the throughput.
Our experiments show that throughput peaks at 70 threads, then drops slightly, peaks again at 100
threads and then plateau. CPU utilization was virtually the same at 2-3%, regardless of the hardware.
Since we intended to avoid bandwidth saturation, we focused on running tests with a small file and
fewer threads. When we ran the tests with either larger files (100KB and up) or with number of
threads above 100, we saturated bandwidth very quickly (e.g. with 100KB file 100 Http gets per
second saturated network).
This chart shows the resulting data when run on various platforms. The data shows a near linear
progression with the number of threads making requests. This progression occurs until the network
bandwidth is consumed.
3500
0 1 2 4
Complete Static HTML Test Results:
Test Avg (ms) Min (ms) Max (ms) Rate/sec % of Baseline
2 2 CPU/2 GB/1 Thread 0 0 36 857.60 83.30%
2 2 CPU/2 GB/2 Threads 0 0 30 1668.50 162.07%
2 2 CPU/2 GB/4 Threads 1 0 31 2704.70 262.72%
2 2 CPU/2 GB/1 Threads 0 0 35 779.60 88.46%
2 2 CPU/2 GB/2 Threads 1 0 38 1504.30 170.69%
SourceLabs AMP Stack Test Results March 25, 2005
Page 7
2 CPU/2 GB/1 Thread 0 0 36 992.70 96.43%
2 CPU/2 GB/2 Threads 0 0 28 1967.50 191.11%
2 CPU/2 GB/4 Threads 0 0 29 3039.30 295.22%
2 CPU/2 GB/1 Threads 0 0 36 913.40 103.64%
2 CPU/2 GB/2 Threads 0 0 30 1690.70 191.84%
2 CPU/2 GB/4 Threads 1 0 30 2563.40 290.87%
1 CPU/2 GB/1 Thread 0 0 36 1029.50 100.00%
1 CPU/2 GB/2 Threads 0 0 41 2076.30 201.68%
1 CPU/2 GB/4 Threads 0 0 37 3253.40 316.02%
1 CPU/2 GB/1 Threads 0 0 40 881.30 100.00%
1 CPU/2 GB/2 Threads 0 0 27 1647.90 186.99%
1 CPU/2 GB/4 Threads 1 0 28 2568.20 291.41%
2 CPU/1 GB/1 Thread 0 0 41 1050.00 101.99%
2 CPU/1 GB/2 Threads 0 0 37 1979.80 192.31%
2 CPU/1 GB/4 Threads 0 0 27 3216.20 312.40%
2 CPU/1 GB/1 Threads 0 0 35 885.40 100.47%
2 CPU/1 GB/2 Threads 0 0 39 1771.20 200.98%
SourceLabs AMP Stack Test Results March 25, 2005
Page 8
Several different hardware configurations were used to test the scalability of PHP. The following is a
table explaining the hardware used:
Configuration CPU Memory Notes
cert7-base-1-2 1 CPU 2 GB
cert7-lb-lc-2-2-2 2 CPU 2 GB 2 nodes load balanced using least connections
cert7-lb-lc-4-2-2 2 CPU 2 GB 4 nodes load balanced using least connections
cert7-base-1-2 phpbench-6 6128.9 4737 6725 1.0052
cert7-base-1-2 phpbench-7 7207.4286 4632 9644 0.9861
cert7-lb-lc-2-2-2 phpbench-10 2383.12 1047 3357 4.0991
cert7-lb-lc-2-2-2 phpbench-100 24855.471 1878 50220 3.9543
cert7-lb-lc-2-2-2 phpbench-2 1050.6 1034 1078 1.7967
cert7-lb-lc-2-2-2 phpbench-20 4814.5875 1042 10467 4.0512
cert7-lb-lc-2-2-2 phpbench-3 1054.4667 1039 1086 2.6661
cert7-lb-lc-2-2-2 phpbench-4 1127.3 1040 2085 3.1653
cert7-lb-lc-2-2-2 phpbench-5 1363.43 1034 2317 2.6681
cert7-lb-lc-2-2-2 phpbench-50 12286.519 1272 25052 3.969
cert7-lb-lc-2-2-2 phpbench-6 1594.8083 1037 2343 3.2518
cert7-lb-lc-2-2-2 phpbench-7 1720.8786 1035 2339 3.6525
cert7-lb-lc-4-2-2 phpbench-10 1439.535 1031 2117 6.1772
cert7-lb-lc-4-2-2 phpbench-100 12309.53 1048 26414 7.8967
cert7-lb-lc-4-2-2 phpbench-2 1051.825 1031 1074 1.7847
cert7-lb-lc-4-2-2 phpbench-20 2486.13 1032 5566 6.9238
cert7-lb-lc-4-2-2 phpbench-3 1058.4 1038 1333 2.7171
cert7-noht-1-2 phpbench-2 2036.95 1600 2401 0.9827
cert7-noht-1-2 phpbench-20 17361.8925 1040 288123 0.9628
cert7-noht-1-2 phpbench-3 3076.05 2697 3335 0.9888
cert7-noht-1-2 phpbench-4 4113.8375 3792 4436 0.9963
cert7-noht-1-2 phpbench-5 5164.77 4467 5544 0.9928
cert7-noht-1-2 phpbench-50 47326.188 1055 529987 0.9589
cert7-noht-1-2 phpbench-6 6091.1083 1074 21942 0.9832
cert7-noht-1-2 phpbench-7 7244.6 5454 7938 0.9914
cert7-noht-2-2 phpbench-10 5009.075 1190 16034 1.9469
cert7-noht-2-2 phpbench-100 46021.137 1120 402085 1.8937
cert7-noht-2-2 phpbench-2 1075.725 1044 1336 1.7746
cert7-noht-2-2 phpbench-20 10045.26 1450 33014 1.9113
cert7-noht-2-2 phpbench-3 1484.6333 1063 1993 1.9378
cert7-noht-2-2 phpbench-4 1992.4875 1174 2432 1.9754
cert7-noht-2-2 phpbench-5 2485.11 1079 3957 1.9414
cert7-noht-2-2 phpbench-50 24023.2 1055 131017 1.9126
cert7-smp-2-2 phpbench-3 1425.6 1027 2133 1.8612
cert7-smp-2-2 phpbench-4 1866.5125 1610 2119 2.1175
cert7-smp-2-2 phpbench-5 2280.6 1840 2966 2.169
cert7-smp-2-2 phpbench-50 23696.817 1748 44275 2.0687
cert7-smp-2-2 phpbench-6 2741.85 1649 3739 2.1794
cert7-smp-2-2 phpbench-7 3205.6643 1037 5712 2.0897
cert7-smp-2-4 phpbench-10 4607.04 1116 6440 2.1843
cert7-smp-2-4 phpbench-100 47699.098 1840 73602 2.0182
cert7-smp-2-4 phpbench-2 1037.575 1027 1048 1.8075
cert7-smp-2-4 phpbench-20 9406.2325 2365 20410 2.0932
cert7-smp-2-4 phpbench-3 1383.5667 1032 1798 2.0367
cert7-smp-2-4 phpbench-4 1841.1875 1496 2088 2.162
cert7-smp-2-4 phpbench-5 2302.55 1250 2947 1.8074
cert7-smp-2-4 phpbench-50 23814.032 1915 34353 2.0499
SourceLabs AMP Stack Test Results March 25, 2005
IR throughput was 95% lower with an OLTP background than with an IR background due to the
locking method MySQL uses for MyISAM tables. In order to make an update to a MyISAM table, an
exclusive table lock must be obtained. New update and select requests are queued while an update
is being processed. By default, updates have priority over selects, so a select query will wait until
there are no more updates to process. When IR throughput (of single-row selects) is measured with
an OLTP background, the background threads are repeatedly requesting single row updates. IR
throughput is low because the single-row selects must wait until there are no updates waiting. The
results for the IR Throughput with OLTP Background test with 1 CPU and 2GB RAM showed that
selects were nearly starved, with only 1 iteration in 635 seconds.
With 2 GB RAM, increasing from 1 CPU to 2 CPU resulted in a 7% increase in IR throughput with an
IR background. As discussed above, the IR throughput with an OLTP background was virtually 0 with
1 CPU. At least 2 CPUs are required to run this test with 5 background threads.
It is more difficult to make conclusions about the cross-section tests because the set of queries was
only run once per test and the each test was only run once on each hardware configuration. Even
with this small sample size, performance improved in most cases when increasing from 2 GB to 4 GB
RAM with 2 CPUs. Excluding the simple_report , heckmod_100_seq , and checkmod_100_rand
queries, the cross-section test with an IR background performed 12-99% faster. Excluding the
o_mode_100k and checkmod_100_rand queries, the cross-section test with an OLTP background
performed 33-99% faster.
Cross-section with IR Background:
The second test shows the results from Cross-section with IR Background. In this test, one user
executes a cross-section of insert, update, and select queries. During this test, 5 other users
SourceLabs AMP Stack Test Results March 25, 2005
Page 11
repeatedly execute the ir_select query. The ir_select query selects a random row from a table using
the primary key. See below for details about the queries used in this test.
IR Throughput:
#ir_select_queries 10539 6467 11297 17298 12444
queries/Second 35.1566 21.5666 37.6865 57.7354 41.5069
Minimum Time (sec) 0.0003 0.0003 0.0003 0.0003 0.0007
Maximum Time (sec) 0.1911 0.2011 0.1615 0.1367 0.1423
Mean Time (sec) 0.0284 0.0464 0.0265 0.0173 0.0241
Total Time (sec) 300 300 300 300 300
Cross-Section Test:
o_mode_tiny 0.0361 0.0812 0.0910 0.0513 0.0741
o_mode_100k 85.8731 105.5086 106.9898 93.5280 75.9425
sel_1_ncl 0.1921 0.5707 0.1828 0.0734 0.1999
simple_report 231.6333 998.8098 212.3199 252.0597 207.1404
sel_100_seq 1.3583 8.8804 1.1388 0.0093 1.5805
sel_100_rand 0.1989 0.7891 0.1624 0.0169 0.0885
mod_100_seq 1.4978 3.4777 1.4688 0.0290 1.9684
Page 12
CPUs 1 2 2 2
Memory 2GB 1G 2G 4G
Background Type OLTP OLTP OLTP OLTP
#Background Threads 5 5 5 5
IR Throughput:
#ir_select_queries 1 383 453 620
queries/Second 0.0016 1.2767 1.5047 2.0643
Minimum Time (sec) 635.3523 0.0406 0.0369 0.0181
Maximum Time (sec) 635.3523 3.8636 3.8059 3.7908
Queries:
The following query is used for the Information Retrieval (IR) and the Mixed IR tests:
ir_select:
select col_key, col_code, col_date, col_signed, col_name from updates where col_key =
random_number
The following query is used for the On-line transaction processing (OLTP) and the Mixed OLTP tests:
oltp_update :
update updates set col_signed = col_signed + 1 where col_key = random_number
The following cross-section queries are used for Mixed IR and Mixed OLTP tests:
o_mode_tiny:
select * from tiny
o_mode_100k:
select * from hundred where col_key <= 1000 sel_1_ncl: 980000000) sel_100_seq: insert into sel100seq (select * from updates where updates.col_key between 1001 and 1100) sel_100_rand: insert into sel100rand (select * from updates where updates.col_int between 1001 and 1100) mod_100_seq: update updates set col_double = col_double+100000000 where col_key between 1001 and 1100 mod_100_rand: update updates set col_double = col_double+100000000 where col_int between 1001 and 1100 unmod_100_seq: This query ? unmodifies ? the effects of mod_100_seq . update updates set col_double = col_double-100000000 where col_key between 1001 and 1100 unmod_100_rand : This query ? unmodifies ? the effects of mod_100_rand . update updates set col_double = col_double-100000000" where col_int between 1001 and 1100. with mod_100_rand , we use ? where col_int between 1001 and 1100 ? . checkmod_100_seq : ? ? This query verifies that unmod_100_rand correctly unmodifed the effects ofmod_100_seq. select count(*) from updates, sel100seq where updates.col_key = sel100seq.col_key and abs(updates.col_double - sel100seq.col_double) > 0.000001
Note: The specification uses the constraint ? not updates.double = sel100seq.double ? . Due to
roundoff errors subtracting and adding 100000000, some ? unmodified ? values do not equal the
original values. For this reason, we check for any values that differ from the original values by more
than 0.000001 rather than checking for equality.
checkmod_100_rand:
This query verifies that unmod_100_rand correctly ? unmodifed ? the effects ofmod_100_seq.
select count(*) from updates, sel100rand where updates.col_int = sel100rand.col_int and
abs(updates.col_double - sel100rand.col_double) > 0.000001
Note: The specification uses the constraint ? not updates.double = sel100rand.double ? . Due to
roundoff errors subtracting and adding 100000000, some ? unmodified ? values do not equal the
original values. For this reason, we check for any values that differ from the original values by more
than 0.000001 rather than checking for equality.
determining server scaling, and optimum CPU and memory for
enterprise servers. I'm posting it here as it is no longer available
Summary
As part of its effort to provide greater confidence to enterprises who wish to use open source infrastructure
software, SourceLabs is releasing theSourceLabs AMP Stack v1.0. This stack, consisting of Apache Web
Server, MySQL, and PHP, has been tested for functionality (unit testing), hardened for improved security,
and profiled for scalability according to three of the seven dependability criteria addressed in SourceLabs
CERT7 methodology. Key findings include:
? The stack passes our acceptance tests as described in this document.
? A number of potential security vulnerabilities exist in default configurations of components of the
AMP stack. SourceLabs configured its stack to protect against these vulnerabilities.
? Apache Web Server scalability was limited by bandwidth in all cases tested, and not CPU or
memory capacity.
? PHP processing scaled linearly with respect to CPU capacity. For capacity planning purposes, this
means that the desired number of servers running Apache Web Server/PHP for a given application
can be calculated based on the processing speed of an individual user scenario. For this, an
Unit and Functional Testing
The individual unit tests were run for each individual component and are composed of the tests that are
downloaded with each individual package. The acceptance criteria included the need for these entire tests
to pass.
Results: PASS. The tests for Apache and MySQL were all run without any failures. The tests for PHP
were run, and it was noted that there were 12 test cases that failed. Each of these test cases were
individually examined and it was determined that the reason for these failures were not related to the
build or the package, and were therefore determined to be false positives. Examples of these failures
include items such as tests for issues that were closed by design/bogus, or the string that was being
compared to determine test success was different from what was expected (but the results were
examined to show the test passed). SourceLabs verified this conclusion with the PHP developer
community and will be working over time to improve the accuracy of these tests.
It was determined that this section of the acceptance testing succeeded.
Burn-in Testing
A rigorous burn-in test under load was performed on the system for a continuous period of 72 hours. These
tests were composed of a subset of the CERT7 scalability tests (static HTML, computationally intensive
PHP, database intensive functionality) described below.
CERT7 Security Hardening
Security testing was performed on the stack using two widely supported open source projects called Nessus
(http://www.nessus.org) and Nikto (http://www.cirt.net/code/nikto.shtml). Both of these tools use a plug-in
architecture to continually expand and test for new issues. The latest versions and complete list of plug-ins
were used for this portion of the test.
Initial results: Running these tests revealed several potential issues related to the configuration of the
AMP stack.
SourceLabs addressed these issues with the security hardening measures below:
1. The configuration of Apache web server is modified such that the user/group that owns the
components is different from the default of the ? nobody ? user. The ? nobody ? user has an underlying
association with the root user. As a result, a new and unprivileged user is created called ? webserv ?
for the server application. In addition, separate groups of ? webadmin ? and ? webdev ? are used for
other non-binary areas of the web server.
2. Removal of most ?f the def?ult files. These files include the default HTML files that are typically
setup in the htdocs directory, the manual files contained off the htdocs directory, and the default
cgi files that are added to the cgi-bin directory. The purpose for removing these files is to reduce
the amount of information that can be obtained from the server by someone maliciously looking at
known areas. These files are all available online on the Apache website.
The CERT7 scalability tests were broken into three main categories: static HTML delivery, computational
PHP, and database testing.
Static HTML Tests
To perform the static HTML tests, a series of static web pages were constructed of known sizes, and a
number of distinct users were modeled requesting these pages through standard HTTP calls. The tests
measured the throughput of the return data. The number of users was increased to determine what the load
SourceLabs AMP Stack Test Results March 25, 2005
Page 4
The AS3AP multi-user tests model four different workloads:
1. Information Retrieval (IR) Throughput with IR background: All users select a random row from the
same table using the primary key.
2. IR Throughput with OLTP Background: One user executes the same query as in the IR Throughput
with IR background test, while all other (background) users update a value in a random row from
the same table using the primary key. The column that is updated is not indexed.
3. Cross-section with IR Background: One user executes a cross-section of update and select queries
while all other users execute the same query as in the IR Throughput with IR background test.
4. Cross-section with OLTP Background: One user executes a cross-section of update and select
queries while all other users execute the same query as the background users in the IR Throughput
with OLTP Background test.
The AS3AP specification provides details about the database structure and the queries that are used in the
tests. A development version of the benchmark is available from MySQL
http://www.mysqlusers.com/msg/4161.html ). This version was used to generate data for the 4GB MySQL
database that was used in all tests. All tables use the MyISAM table engine. Of the five tables that are
generated, 3 are used by the multi-user tests: updates, hundred, and tiny.
The AS3AP specification indicates the type of index that should be defined for each index. The possible
Page 5
1. Start up apache and MySQL.
2. Load indexes for the updates and hundred tables into the index cache.
3. Start up 6 users that execute the ir_select query repeatedly.
4. Run IR Throughput with IR Background test. After a 15 minute warm-up, measure IR throughput for
one of the users executing the ir_select query for 5 minutes (while the other 5 users continue to
execute the ir_select query).
5. Run Cross-section with IR Background test. Measure execution time for one user running the
cross-section queries (while the other 5 users continue to execute the ir_select query).
6. Restore the updates table.
RAM Amount 4 x 512 (2 GB) 2 x 512 (1 GB)
4 x 512 (2 GB)
4 x 512 + 2 x 1 GB (4 GB)
Hard Drives 2 X 40GB SATA No RAID 2 X 40GB SATA No RAID
HD Capacity 80 GB 80 GB
NIC e1000 e1000
Net Speed 100 Mbps 100 Mbps
Distro RHEL 3.2 RHEL 3.2
Kernel 2.4.21-27 2.4.21-27
Architecture x86_64 x86_64
The following configurations were used in our testing:
? 1 CPU box 2GB of memory
? 2 CPU box 2GB of memory
? 2 CPU box with 1 GB of memory
? 2 CPU box with 2 GB of memory
? 2 CPU box with 4 GB of memory
Each configuration executes the same set of tests. The tests vary only in a number of parallel threads
spawned (from 1 to 70). Each thread was running a loop with 1000 iterations, each iteration
downloads the same html file (1150 Bytes long).
http://jakarta.apache.org/jmeter
JMeter [ ] was used to run the tests and gather results.
The attached Excel spread sheet shows that the throughput is a linear function of number of threads.
It also shows that hardware differences have virtually no impact on the throughput.
Our experiments show that throughput peaks at 70 threads, then drops slightly, peaks again at 100
threads and then plateau. CPU utilization was virtually the same at 2-3%, regardless of the hardware.
Since we intended to avoid bandwidth saturation, we focused on running tests with a small file and
fewer threads. When we ran the tests with either larger files (100KB and up) or with number of
threads above 100, we saturated bandwidth very quickly (e.g. with 100KB file 100 Http gets per
second saturated network).
This chart shows the resulting data when run on various platforms. The data shows a near linear
progression with the number of threads making requests. This progression occurs until the network
bandwidth is consumed.
3500
0 1 2 4
Complete Static HTML Test Results:
Test Avg (ms) Min (ms) Max (ms) Rate/sec % of Baseline
2 2 CPU/2 GB/1 Thread 0 0 36 857.60 83.30%
2 2 CPU/2 GB/2 Threads 0 0 30 1668.50 162.07%
2 2 CPU/2 GB/4 Threads 1 0 31 2704.70 262.72%
2 2 CPU/2 GB/1 Threads 0 0 35 779.60 88.46%
2 2 CPU/2 GB/2 Threads 1 0 38 1504.30 170.69%
SourceLabs AMP Stack Test Results March 25, 2005
Page 7
2 CPU/2 GB/1 Thread 0 0 36 992.70 96.43%
2 CPU/2 GB/2 Threads 0 0 28 1967.50 191.11%
2 CPU/2 GB/4 Threads 0 0 29 3039.30 295.22%
2 CPU/2 GB/1 Threads 0 0 36 913.40 103.64%
2 CPU/2 GB/2 Threads 0 0 30 1690.70 191.84%
2 CPU/2 GB/4 Threads 1 0 30 2563.40 290.87%
1 CPU/2 GB/1 Thread 0 0 36 1029.50 100.00%
1 CPU/2 GB/2 Threads 0 0 41 2076.30 201.68%
1 CPU/2 GB/4 Threads 0 0 37 3253.40 316.02%
1 CPU/2 GB/1 Threads 0 0 40 881.30 100.00%
1 CPU/2 GB/2 Threads 0 0 27 1647.90 186.99%
1 CPU/2 GB/4 Threads 1 0 28 2568.20 291.41%
2 CPU/1 GB/1 Thread 0 0 41 1050.00 101.99%
2 CPU/1 GB/2 Threads 0 0 37 1979.80 192.31%
2 CPU/1 GB/4 Threads 0 0 27 3216.20 312.40%
2 CPU/1 GB/1 Threads 0 0 35 885.40 100.47%
2 CPU/1 GB/2 Threads 0 0 39 1771.20 200.98%
SourceLabs AMP Stack Test Results March 25, 2005
Page 8
Several different hardware configurations were used to test the scalability of PHP. The following is a
table explaining the hardware used:
Configuration CPU Memory Notes
cert7-base-1-2 1 CPU 2 GB
cert7-lb-lc-2-2-2 2 CPU 2 GB 2 nodes load balanced using least connections
cert7-lb-lc-4-2-2 2 CPU 2 GB 4 nodes load balanced using least connections
cert7-base-1-2 phpbench-6 6128.9 4737 6725 1.0052
cert7-base-1-2 phpbench-7 7207.4286 4632 9644 0.9861
cert7-lb-lc-2-2-2 phpbench-10 2383.12 1047 3357 4.0991
cert7-lb-lc-2-2-2 phpbench-100 24855.471 1878 50220 3.9543
cert7-lb-lc-2-2-2 phpbench-2 1050.6 1034 1078 1.7967
cert7-lb-lc-2-2-2 phpbench-20 4814.5875 1042 10467 4.0512
cert7-lb-lc-2-2-2 phpbench-3 1054.4667 1039 1086 2.6661
cert7-lb-lc-2-2-2 phpbench-4 1127.3 1040 2085 3.1653
cert7-lb-lc-2-2-2 phpbench-5 1363.43 1034 2317 2.6681
cert7-lb-lc-2-2-2 phpbench-50 12286.519 1272 25052 3.969
cert7-lb-lc-2-2-2 phpbench-6 1594.8083 1037 2343 3.2518
cert7-lb-lc-2-2-2 phpbench-7 1720.8786 1035 2339 3.6525
cert7-lb-lc-4-2-2 phpbench-10 1439.535 1031 2117 6.1772
cert7-lb-lc-4-2-2 phpbench-100 12309.53 1048 26414 7.8967
cert7-lb-lc-4-2-2 phpbench-2 1051.825 1031 1074 1.7847
cert7-lb-lc-4-2-2 phpbench-20 2486.13 1032 5566 6.9238
cert7-lb-lc-4-2-2 phpbench-3 1058.4 1038 1333 2.7171
cert7-noht-1-2 phpbench-2 2036.95 1600 2401 0.9827
cert7-noht-1-2 phpbench-20 17361.8925 1040 288123 0.9628
cert7-noht-1-2 phpbench-3 3076.05 2697 3335 0.9888
cert7-noht-1-2 phpbench-4 4113.8375 3792 4436 0.9963
cert7-noht-1-2 phpbench-5 5164.77 4467 5544 0.9928
cert7-noht-1-2 phpbench-50 47326.188 1055 529987 0.9589
cert7-noht-1-2 phpbench-6 6091.1083 1074 21942 0.9832
cert7-noht-1-2 phpbench-7 7244.6 5454 7938 0.9914
cert7-noht-2-2 phpbench-10 5009.075 1190 16034 1.9469
cert7-noht-2-2 phpbench-100 46021.137 1120 402085 1.8937
cert7-noht-2-2 phpbench-2 1075.725 1044 1336 1.7746
cert7-noht-2-2 phpbench-20 10045.26 1450 33014 1.9113
cert7-noht-2-2 phpbench-3 1484.6333 1063 1993 1.9378
cert7-noht-2-2 phpbench-4 1992.4875 1174 2432 1.9754
cert7-noht-2-2 phpbench-5 2485.11 1079 3957 1.9414
cert7-noht-2-2 phpbench-50 24023.2 1055 131017 1.9126
cert7-smp-2-2 phpbench-3 1425.6 1027 2133 1.8612
cert7-smp-2-2 phpbench-4 1866.5125 1610 2119 2.1175
cert7-smp-2-2 phpbench-5 2280.6 1840 2966 2.169
cert7-smp-2-2 phpbench-50 23696.817 1748 44275 2.0687
cert7-smp-2-2 phpbench-6 2741.85 1649 3739 2.1794
cert7-smp-2-2 phpbench-7 3205.6643 1037 5712 2.0897
cert7-smp-2-4 phpbench-10 4607.04 1116 6440 2.1843
cert7-smp-2-4 phpbench-100 47699.098 1840 73602 2.0182
cert7-smp-2-4 phpbench-2 1037.575 1027 1048 1.8075
cert7-smp-2-4 phpbench-20 9406.2325 2365 20410 2.0932
cert7-smp-2-4 phpbench-3 1383.5667 1032 1798 2.0367
cert7-smp-2-4 phpbench-4 1841.1875 1496 2088 2.162
cert7-smp-2-4 phpbench-5 2302.55 1250 2947 1.8074
cert7-smp-2-4 phpbench-50 23814.032 1915 34353 2.0499
SourceLabs AMP Stack Test Results March 25, 2005
IR throughput was 95% lower with an OLTP background than with an IR background due to the
locking method MySQL uses for MyISAM tables. In order to make an update to a MyISAM table, an
exclusive table lock must be obtained. New update and select requests are queued while an update
is being processed. By default, updates have priority over selects, so a select query will wait until
there are no more updates to process. When IR throughput (of single-row selects) is measured with
an OLTP background, the background threads are repeatedly requesting single row updates. IR
throughput is low because the single-row selects must wait until there are no updates waiting. The
results for the IR Throughput with OLTP Background test with 1 CPU and 2GB RAM showed that
selects were nearly starved, with only 1 iteration in 635 seconds.
With 2 GB RAM, increasing from 1 CPU to 2 CPU resulted in a 7% increase in IR throughput with an
IR background. As discussed above, the IR throughput with an OLTP background was virtually 0 with
1 CPU. At least 2 CPUs are required to run this test with 5 background threads.
It is more difficult to make conclusions about the cross-section tests because the set of queries was
only run once per test and the each test was only run once on each hardware configuration. Even
with this small sample size, performance improved in most cases when increasing from 2 GB to 4 GB
RAM with 2 CPUs. Excluding the simple_report , heckmod_100_seq , and checkmod_100_rand
queries, the cross-section test with an IR background performed 12-99% faster. Excluding the
o_mode_100k and checkmod_100_rand queries, the cross-section test with an OLTP background
performed 33-99% faster.
Cross-section with IR Background:
The second test shows the results from Cross-section with IR Background. In this test, one user
executes a cross-section of insert, update, and select queries. During this test, 5 other users
SourceLabs AMP Stack Test Results March 25, 2005
Page 11
repeatedly execute the ir_select query. The ir_select query selects a random row from a table using
the primary key. See below for details about the queries used in this test.
IR Throughput:
#ir_select_queries 10539 6467 11297 17298 12444
queries/Second 35.1566 21.5666 37.6865 57.7354 41.5069
Minimum Time (sec) 0.0003 0.0003 0.0003 0.0003 0.0007
Maximum Time (sec) 0.1911 0.2011 0.1615 0.1367 0.1423
Mean Time (sec) 0.0284 0.0464 0.0265 0.0173 0.0241
Total Time (sec) 300 300 300 300 300
Cross-Section Test:
o_mode_tiny 0.0361 0.0812 0.0910 0.0513 0.0741
o_mode_100k 85.8731 105.5086 106.9898 93.5280 75.9425
sel_1_ncl 0.1921 0.5707 0.1828 0.0734 0.1999
simple_report 231.6333 998.8098 212.3199 252.0597 207.1404
sel_100_seq 1.3583 8.8804 1.1388 0.0093 1.5805
sel_100_rand 0.1989 0.7891 0.1624 0.0169 0.0885
mod_100_seq 1.4978 3.4777 1.4688 0.0290 1.9684
Page 12
CPUs 1 2 2 2
Memory 2GB 1G 2G 4G
Background Type OLTP OLTP OLTP OLTP
#Background Threads 5 5 5 5
IR Throughput:
#ir_select_queries 1 383 453 620
queries/Second 0.0016 1.2767 1.5047 2.0643
Minimum Time (sec) 635.3523 0.0406 0.0369 0.0181
Maximum Time (sec) 635.3523 3.8636 3.8059 3.7908
Queries:
The following query is used for the Information Retrieval (IR) and the Mixed IR tests:
ir_select:
select col_key, col_code, col_date, col_signed, col_name from updates where col_key =
random_number
The following query is used for the On-line transaction processing (OLTP) and the Mixed OLTP tests:
oltp_update :
update updates set col_signed = col_signed + 1 where col_key = random_number
The following cross-section queries are used for Mixed IR and Mixed OLTP tests:
o_mode_tiny:
select * from tiny
o_mode_100k:
select * from hundred where col_key <= 1000 sel_1_ncl: 980000000) sel_100_seq: insert into sel100seq (select * from updates where updates.col_key between 1001 and 1100) sel_100_rand: insert into sel100rand (select * from updates where updates.col_int between 1001 and 1100) mod_100_seq: update updates set col_double = col_double+100000000 where col_key between 1001 and 1100 mod_100_rand: update updates set col_double = col_double+100000000 where col_int between 1001 and 1100 unmod_100_seq: This query ? unmodifies ? the effects of mod_100_seq . update updates set col_double = col_double-100000000 where col_key between 1001 and 1100 unmod_100_rand : This query ? unmodifies ? the effects of mod_100_rand . update updates set col_double = col_double-100000000" where col_int between 1001 and 1100. with mod_100_rand , we use ? where col_int between 1001 and 1100 ? . checkmod_100_seq : ? ? This query verifies that unmod_100_rand correctly unmodifed the effects ofmod_100_seq. select count(*) from updates, sel100seq where updates.col_key = sel100seq.col_key and abs(updates.col_double - sel100seq.col_double) > 0.000001
Note: The specification uses the constraint ? not updates.double = sel100seq.double ? . Due to
roundoff errors subtracting and adding 100000000, some ? unmodified ? values do not equal the
original values. For this reason, we check for any values that differ from the original values by more
than 0.000001 rather than checking for equality.
checkmod_100_rand:
This query verifies that unmod_100_rand correctly ? unmodifed ? the effects ofmod_100_seq.
select count(*) from updates, sel100rand where updates.col_int = sel100rand.col_int and
abs(updates.col_double - sel100rand.col_double) > 0.000001
Note: The specification uses the constraint ? not updates.double = sel100rand.double ? . Due to
roundoff errors subtracting and adding 100000000, some ? unmodified ? values do not equal the
original values. For this reason, we check for any values that differ from the original values by more
than 0.000001 rather than checking for equality.
posted by Milton Keynes Web Design at 5:48 PM
0 Comments:
Post a Comment
<< Home