Case: SEMA - Swedish Emergency Management Agency
How to verify, test and improve performance of a web site ahead of an upcoming, expected traffic peak load
The Swedish Emergency Management Agency – SEMA (Krisberedskapsmyndigheten (KBM), (http://www.krisberedskapsmyndigheten.se/) is a governmental body with the task to coordinate crisis preparedness in the Swedish society.
During the Spring of 2008, SEMA carried out a vast communications exercise called SAMÖ2008. The purpose of the exercise was to try the preparedness of financial systems against a possible IT-based attack. The exercise drew appox. 5000 participants from the Swedish Government, other central governmental bodies, the counties, organizations and companies as well as a large society network.
The subject of the exercise was organized IT-attacks which would threaten or bring down the financial systems and which could lead to a crises of confidence in the society; in a case like this, the traditional payment systems normal functions are affected by very severe distortions.
The exercise was a simulated crises scenario, which played out with the participants in the exercise and was headed up by SEMA, where approx. 100 participants were active in the so called counter scenario which brought the exercise game forward.
"Large communications exercises is an important means to strengthen the capacity to handle crises and to drill coordination between different sectors which work with crises preparedness in the society", said Helena Lindberg, leader of the exercise SAMÖ2008 and General Director of SEMA in a press release.
Technical challenge
The exercise had a central web site - www.samö2008.se – which delivered continuously updated information during the exercise for all participants. The performance of this site was of ultimate importance as the thousands of participants in the exercise periodically all logged in during a few minutes to update themselves on their exercise status.
The issues at hand for SEMA was, amongst others, “can the site handle 5000 logged on users during the one and same minute?” and “how should we best scale up our web application and server environment to be able to avoid bottlenecks?”.
The technical supervisor with SEMA, Per Söderström, designed together with Apica a load test of the production environment to verify the maximum performance of the site. The test was designed to simulate a scenario where 1,000 up to 10,000 users logs on to the site during the one and same minute and gets information from a number of sub-pages. Test results were aggregated regarding:
- No. of active users
- Data from server-CPU and web server
- Response times for the scenario as well as URL’s
Test results and actions
The result of the first test was that the site without any kind of trimming could cope with approx. 4000 concurrent logged in users. The response times were however to long.
This result did not certify that the site could live up to the expected size of traffic. One of the actions taken which had an impact was to make the size of the landing page smaller. But this was not sufficient.
The solution was to put a separate front-end cache before SAMÖ:s web site. This meant that the ordinary web site was offloaded all static traffic. For larger web-sites this is usually called a CDN (Content Delivery Network).
But why cannot the ordinary cache on a typical web server handle this?
The answer is that it depends on which kind of cache is implemented in the web application. A separate front end cache typically often delivers much better performance than the cache built in an ordinary web server. It is also very important that the no of inquiries per second to the web server goes down drastically, since the cache treats all static content. The CPU load is thus drastically lowered.
It is however worth mentioning that even if the content is flagged as cached in a web server, the actual no of hits per second is a separate problem when you reach high volumes of inquiries to the site.
It is impossible to have a general opinion on how a web cluster/web server will handle high load. The only way to be certain is to load test the production environment of the system and analyze how the separate components of the site reacts.
A front-end cache based on Varnish gives a much better throughput for static content than an equivalent web server. The most simple way of explaining it is that design and structure in the code is optimized specifically for delivery of images and not to generate complex web pages or all the rest of the functionality that comes with a modern web server.
On a source code level, Varnish is optimized to deliver maximum data per instruction. That type of optimization is impossible to achieve on a conventional web server.
|
|
|
|
|
Without Varnish 4000 users
|
|
Test
|
AV Time
|
<= 90 %
|
AV Size
|
URL
|
|
[1]
|
1'888 ms
|
5'161 ms
|
15'126 bytes
|
GET http://www.samo2008.se:80/
|
|
[2]
|
637 ms
|
1'547 ms
|
852 bytes
|
GET http://www.samo2008.se:80/styles/samo.css
|
|
[3]
|
641 ms
|
1'624 ms
|
1'739 bytes
|
GET http://www.samo2008.se:80/styles/reset.css
|
|
[4]
|
654 ms
|
1'629 ms
|
2'635 bytes
|
GET http://www.samo2008.se:80/styles/specialpages.css
|
|
[5]
|
1'910 ms
|
4'811 ms
|
15'145 bytes
|
GET http://www.samo2008.se:80/styles/global.css
|
|
[6]
|
2'681 ms
|
6'725 ms
|
23'615 bytes
|
GET http://www.samo2008.se:80/scripts/DOMAssistant.js
|
|
[7]
|
661 ms
|
1'675 ms
|
1'735 bytes
|
GET http://www.samo2008.se:80/scripts/global.js
|
|
[8]
|
2'992 ms
|
7'476 ms
|
21'622 bytes
|
GET http://www.samo2008.se:80/WebResource.axd?d=1jbvyOWjrSv4Om
|
|
[9]
|
2'965 ms
|
7'572 ms
|
22'186 bytes
|
GET http://www.samo2008.se:80/WebResource.axd?d=eUGjkbaL2CxyowU
|
|
[10]
|
4'516 ms
|
10'915 ms
|
34'821 bytes
|
GET http://www.samo2008.se:80/util/javascript/episerverscriptmanager.js
|
|
[11]
|
694 ms
|
1'904 ms
|
1'954 bytes
|
GET http://www.samo2008.se:80/images/samo_logo.gif
|
|
[12]
|
625 ms
|
1'674 ms
|
873 bytes
|
GET http://www.samo2008.se:80/images/bg_mainnavigation_separator.png
|
|
[13]
|
642 ms
|
1'812 ms
|
789 bytes
|
GET http://www.samo2008.se:80/images/bg_menutopspacer.png
|
|
[14]
|
4'707 ms
|
11'381 ms
|
38'411 bytes
|
GET http://www.samo2008.se:80/images/top.png
|
|
[15]
|
590 ms
|
1'662 ms
|
824 bytes
|
GET http://www.samo2008.se:80/images/bg_mainnavigation.png
|
|
[16]
|
603 ms
|
1'669 ms
|
1'158 bytes
|
GET http://www.samo2008.se:80/images/bg_contentwrapper.png
|
|
[17]
|
607 ms
|
1'676 ms
|
776 bytes
|
GET http://www.samo2008.se:80/images/bg_footer.png
|
|
Total
|
28'013 ms
|
61'430 ms
|
184'261 bytes
|
17 URLs
|
|
Samö
|
|
|
|
With Varnish 4000 users
|
|
[1]
|
12 ms
|
9 ms
|
15'126 bytes
|
GET http://www.samo2008.se:80/
|
|
[2]
|
1 ms
|
2 ms
|
852 bytes
|
GET http://www.samo2008.se:80/styles/samo.css
|
|
[3]
|
1 ms
|
2 ms
|
1'739 bytes
|
GET http://www.samo2008.se:80/styles/reset.css
|
|
[4]
|
1 ms
|
2 ms
|
2'635 bytes
|
GET http://www.samo2008.se:80/styles/specialpages.css
|
|
[5]
|
3 ms
|
3 ms
|
15'145 bytes
|
GET http://www.samo2008.se:80/styles/global.css
|
|
[6]
|
3 ms
|
3 ms
|
23'615 bytes
|
GET http://www.samo2008.se:80/scripts/DOMAssistant.js
|
|
[7]
|
1 ms
|
2 ms
|
1'735 bytes
|
GET http://www.samo2008.se:80/scripts/global.js
|
|
[8]
|
5 ms
|
7 ms
|
21'622 bytes
|
GET http://www.samo2008.se:80/WebResource.axd?d=1jbvyOWjrSv4Om
|
|
[9]
|
5 ms
|
7 ms
|
22'186 bytes
|
GET http://www.samo2008.se:80/WebResource.axd?d=eUGjkbaL2CxyowU
|
|
[10]
|
3 ms
|
3 ms
|
34'821 bytes
|
GET http://www.samo2008.se:80/util/javascript/episerverscriptmanager.js
|
|
[11]
|
1 ms
|
2 ms
|
1'954 bytes
|
GET http://www.samo2008.se:80/images/samo_logo.gif
|
|
[12]
|
1 ms
|
2 ms
|
873 bytes
|
GET http://www.samo2008.se:80/images/bg_mainnavigation_separator.png
|
|
[13]
|
1 ms
|
2 ms
|
789 bytes
|
GET http://www.samo2008.se:80/images/bg_menutopspacer.png
|
|
[14]
|
3 ms
|
3 ms
|
38'411 bytes
|
GET http://www.samo2008.se:80/images/top.png
|
|
[15]
|
1 ms
|
2 ms
|
824 bytes
|
GET http://www.samo2008.se:80/images/bg_mainnavigation.png
|
|
[16]
|
1 ms
|
2 ms
|
1'158 bytes
|
GET http://www.samo2008.se:80/images/bg_contentwrapper.png
|
|
[17]
|
1 ms
|
2 ms
|
776 bytes
|
GET http://www.samo2008.se:80/images/bg_footer.png
|
|
Total
|
44 ms
|
47 ms
|
184'261 bytes
|
17 URLs
|
For more information on the case above, please contact:
Sven Hammar
CEO, Apica AB
Phone: 08 – 555 830 30
Mobile: 0735 - 05 90 68
E mail: sven.hammar@apicasystem.com
and
Per Söderström
IT-manager, KBM
Phone: 08 - 593 71 000
E-mail: Per.Soderstrom@kbm-sema.se
Cache and acceleration of web traffic
So called HTTP-acceleration efficiently quickens up your web traffic and in a cost efficient way. Apica offers a packaged solution based on Varnish (Apica Acceleration) which is the market leading HTTP-acceleration solution.
No difficult integration is necessary as Apica delivers a pre-configured solution on optimized hardware with tested Varnish parameters.
Apicas http acceleration is a complete solution. The hardware is delivered on a rental basis. Hardware guarantee as well as continuous upgrades of Varnish and operating systems is included. Apica verifies the installation with a load test and are able to guarantee a performance enhancement. Availability and 24/7monitoring of the server is included in the as an optional service, Apica also offers response time measurement with Apica Availability.
Front-End Cache
All web pages normally consist of dynamic and static content. Static content is defined as data which does not change or is made unique for each visitor. Images and style sheets (i e formats css, gif and jpg) are typical examples of static content. Static content is excellent for caching on a separate server, a so called http-accelerator
Of the total bandwidth a web page needs to be loaded by one single user, the static content typically uses well over 50 % of the bandwidth and CPU-capacity of the web server. The point of a front-end cache is to take load of “unnecessary” tasks of the web servers behind (that is, to serve static content) and thereby improve the performance – often increasing the performance by factor 5-10 compared to a standard installation.
Varnish
Varnish is a leading software for http-cache based on open source, with absolute top class performance. Varnish is today used by several leading newspapers’ web sites and other large sites, and has a rapidly growing community. Apica possesses expert competence in Varnish configuration.
Case: SEMA - Swedish Emergency Management Agency
How to verify, test and improve performance of a web site ahead of an upcoming, expected traffic peak load
The Swedish Emergency Management Agency – SEMA (Krisberedskapsmyndigheten (KBM), (http://www.krisberedskapsmyndigheten.se/) is a governmental body with the task to coordinate crisis preparedness in the Swedish society.
During the Spring of 2008, SEMA carried out a vast communications exercise called SAMÖ2008. The purpose of the exercise was to try the preparedness of financial systems against a possible IT-based attack. The exercise drew appox. 5000 participants from the Swedish Government, other central governmental bodies, the counties, organizations and companies as well as a large society network.
The subject of the exercise was organized IT-attacks which would threaten or bring down the financial systems and which could lead to a crises of confidence in the society; in a case like this, the traditional payment systems normal functions are affected by very severe distortions.
The exercise was a simulated crises scenario, which played out with the participants in the exercise and was headed up by SEMA, where approx. 100 participants were active in the so called counter scenario which brought the exercise game forward.
"Large communications exercises is an important means to strengthen the capacity to handle crises and to drill coordination between different sectors which work with crises preparedness in the society", said Helena Lindberg, leader of the exercise SAMÖ2008 and General Director of SEMA in a press release.
Technical challenge
The exercise had a central web site - www.samö2008.se – which delivered continuously updated information during the exercise for all participants. The performance of this site was of ultimate importance as the thousands of participants in the exercise periodically all logged in during a few minutes to update themselves on their exercise status.
The issues at hand for SEMA was, amongst others, “can the site handle 5000 logged on users during the one and same minute?” and “how should we best scale up our web application and server environment to be able to avoid bottlenecks?”.
The technical supervisor with SEMA, Per Söderström, designed together with Apica a load test of the production environment to verify the maximum performance of the site. The test was designed to simulate a scenario where 1,000 up to 10,000 users logs on to the site during the one and same minute and gets information from a number of sub-pages. Test results were aggregated regarding:
- No. of active users
- Data from server-CPU and web server
- Response times for the scenario as well as URL’s
Test results and actions
The result of the first test was that the site without any kind of trimming could cope with approx. 4000 concurrent logged in users. The response times were however to long.
This result did not certify that the site could live up to the expected size of traffic. One of the actions taken which had an impact was to make the size of the landing page smaller. But this was not sufficient.
The solution was to put a separate front-end cache before SAMÖ:s web site. This meant that the ordinary web site was offloaded all static traffic. For larger web-sites this is usually called a CDN (Content Delivery Network).
But why cannot the ordinary cache on a typical web server handle this?
The answer is that it depends on which kind of cache is implemented in the web application. A separate front end cache typically often delivers much better performance than the cache built in an ordinary web server. It is also very important that the no of inquiries per second to the web server goes down drastically, since the cache treats all static content. The CPU load is thus drastically lowered.
It is however worth mentioning that even if the content is flagged as cached in a web server, the actual no of hits per second is a separate problem when you reach high volumes of inquiries to the site.
It is impossible to have a general opinion on how a web cluster/web server will handle high load. The only way to be certain is to load test the production environment of the system and analyze how the separate components of the site reacts.
A front-end cache based on Varnish gives a much better throughput for static content than an equivalent web server. The most simple way of explaining it is that design and structure in the code is optimized specifically for delivery of images and not to generate complex web pages or all the rest of the functionality that comes with a modern web server.
On a source code level, Varnish is optimized to deliver maximum data per instruction. That type of optimization is impossible to achieve on a conventional web server.
|
|
|
|
|
Without Varnish 4000 users
|
|
Test
|
AV Time
|
<= 90 %
|
AV Size
|
URL
|
|
[1]
|
1'888 ms
|
5'161 ms
|
15'126 bytes
|
GET http://www.samo2008.se:80/
|
|
[2]
|
637 ms
|
1'547 ms
|
852 bytes
|
GET http://www.samo2008.se:80/styles/samo.css
|
|
[3]
|
641 ms
|
1'624 ms
|
1'739 bytes
|
GET http://www.samo2008.se:80/styles/reset.css
|
|
[4]
|
654 ms
|
1'629 ms
|
2'635 bytes
|
GET http://www.samo2008.se:80/styles/specialpages.css
|
|
[5]
|
1'910 ms
|
4'811 ms
|
15'145 bytes
|
GET http://www.samo2008.se:80/styles/global.css
|
|
[6]
|
2'681 ms
|
6'725 ms
|
23'615 bytes
|
GET http://www.samo2008.se:80/scripts/DOMAssistant.js
|
|
[7]
|
661 ms
|
1'675 ms
|
1'735 bytes
|
GET http://www.samo2008.se:80/scripts/global.js
|
|
[8]
|
2'992 ms
|
7'476 ms
|
21'622 bytes
|
GET http://www.samo2008.se:80/WebResource.axd?d=1jbvyOWjrSv4Om
|
|
[9]
|
2'965 ms
|
7'572 ms
|
22'186 bytes
|
GET http://www.samo2008.se:80/WebResource.axd?d=eUGjkbaL2CxyowU
|
|
[10]
|
4'516 ms
|
10'915 ms
|
34'821 bytes
|
GET http://www.samo2008.se:80/util/javascript/episerverscriptmanager.js
|
|
[11]
|
694 ms
|
1'904 ms
|
1'954 bytes
|
GET http://www.samo2008.se:80/images/samo_logo.gif
|
|
[12]
|
625 ms
|
1'674 ms
|
873 bytes
|
GET http://www.samo2008.se:80/images/bg_mainnavigation_separator.png
|
|
[13]
|
642 ms
|
1'812 ms
|
789 bytes
|
GET http://www.samo2008.se:80/images/bg_menutopspacer.png
|
|
[14]
|
4'707 ms
|
11'381 ms
|
38'411 bytes
|
GET http://www.samo2008.se:80/images/top.png
|
|
[15]
|
590 ms
|
1'662 ms
|
824 bytes
|
GET http://www.samo2008.se:80/images/bg_mainnavigation.png
|
|
[16]
|
603 ms
|
1'669 ms
|
1'158 bytes
|
GET http://www.samo2008.se:80/images/bg_contentwrapper.png
|
|
[17]
|
607 ms
|
1'676 ms
|
776 bytes
|
GET http://www.samo2008.se:80/images/bg_footer.png
|
|
Total
|
28'013 ms
|
61'430 ms
|
184'261 bytes
|
17 URLs
|
|
Samö
|
|
|
|
With Varnish 4000 users
|
|
[1]
|
12 ms
|
9 ms
|
15'126 bytes
|
GET http://www.samo2008.se:80/
|
|
[2]
|
1 ms
|
2 ms
|
852 bytes
|
GET http://www.samo2008.se:80/styles/samo.css
|
|
[3]
|
1 ms
|
2 ms
|
1'739 bytes
|
GET http://www.samo2008.se:80/styles/reset.css
|
|
[4]
|
1 ms
|
2 ms
|
2'635 bytes
|
GET http://www.samo2008.se:80/styles/specialpages.css
|
|
[5]
|
3 ms
|
3 ms
|
15'145 bytes
|
GET http://www.samo2008.se:80/styles/global.css
|
|
[6]
|
3 ms
|
3 ms
|
23'615 bytes
|
GET http://www.samo2008.se:80/scripts/DOMAssistant.js
|
|
[7]
|
1 ms
|
2 ms
|
1'735 bytes
|
GET http://www.samo2008.se:80/scripts/global.js
|
|
[8]
|
5 ms
|
7 ms
|
21'622 bytes
|
GET http://www.samo2008.se:80/WebResource.axd?d=1jbvyOWjrSv4Om
|
|
[9]
|
5 ms
|
7 ms
|
22'186 bytes
|
GET http://www.samo2008.se:80/WebResource.axd?d=eUGjkbaL2CxyowU
|
|
[10]
|
3 ms
|
3 ms
|
34'821 bytes
|
GET http://www.samo2008.se:80/util/javascript/episerverscriptmanager.js
|
|
[11]
|
1 ms
|
2 ms
|
1'954 bytes
|
GET http://www.samo2008.se:80/images/samo_logo.gif
|
|
[12]
|
1 ms
|
2 ms
|
873 bytes
|
GET http://www.samo2008.se:80/images/bg_mainnavigation_separator.png
|
|
[13]
|
1 ms
|
2 ms
|
789 bytes
|
GET http://www.samo2008.se:80/images/bg_menutopspacer.png
|
|
[14]
|
3 ms
|
3 ms
|
38'411 bytes
|
GET http://www.samo2008.se:80/images/top.png
|
|
[15]
|
1 ms
|
2 ms
|
824 bytes
|
GET http://www.samo2008.se:80/images/bg_mainnavigation.png
|
|
[16]
|
1 ms
|
2 ms
|
1'158 bytes
|
GET http://www.samo2008.se:80/images/bg_contentwrapper.png
|
|
[17]
|
1 ms
|
2 ms
|
776 bytes
|
GET http://www.samo2008.se:80/images/bg_footer.png
|
|
Total
|
44 ms
|
47 ms
|
184'261 bytes
|
17 URLs
|
For more information on the case above, please contact:
Sven Hammar
CEO, Apica AB
Phone: 08 – 555 830 30
Mobile: 0735 - 05 90 68
E mail: sven.hammar@apicasystem.com
and
Per Söderström
IT-manager, KBM
Phone: 08 - 593 71 000
E-mail: Per.Soderstrom@kbm-sema.se
Cache and acceleration of web traffic
So called HTTP-acceleration efficiently quickens up your web traffic and in a cost efficient way. Apica offers a packaged solution based on Varnish (Apica Acceleration) which is the market leading HTTP-acceleration solution.
No difficult integration is necessary as Apica delivers a pre-configured solution on optimized hardware with tested Varnish parameters.
Apicas http acceleration is a complete solution. The hardware is delivered on a rental basis. Hardware guarantee as well as continuous upgrades of Varnish and operating systems is included. Apica verifies the installation with a load test and are able to guarantee a performance enhancement. Availability and 24/7monitoring of the server is included in the as an optional service, Apica also offers response time measurement with Apica Availability.
Front-End Cache
All web pages normally consist of dynamic and static content. Static content is defined as data which does not change or is made unique for each visitor. Images and style sheets (i e formats css, gif and jpg) are typical examples of static content. Static content is excellent for caching on a separate server, a so called http-accelerator
Of the total bandwidth a web page needs to be loaded by one single user, the static content typically uses well over 50 % of the bandwidth and CPU-capacity of the web server. The point of a front-end cache is to take load of “unnecessary” tasks of the web servers behind (that is, to serve static content) and thereby improve the performance – often increasing the performance by factor 5-10 compared to a standard installation.
Varnish
Varnish is a leading software for http-cache based on open source, with absolute top class performance. Varnish is today used by several leading newspapers’ web sites and other large sites, and has a rapidly growing community. Apica possesses expert competence in Varnish configuration.