Performance

Bitrix24 Load Testing

How do you know if software is reliable? That’s right, you don’t - until you test it. Much like students at a university, any software requires regular testing to see if it’s up to the highest standards.

At Bitrix24, we take load testing very seriously. Being a software developer and distributor, we make sure to test our products regularly to ensure the best performance and reliability for our clients.

Every day, millions of people around the world depend on Bitrix24 working properly and even the slightest downtime is a risk we cannot take. This is especially true for our self-hosted version, which is often chosen by companies with serious requirements for the system performance.

Here are the detailed results of our most recent load tests performed to assess the Bitrix24 system performance in a large-scale business environment with a substantial number of employees and high load.

2022 Load Testing

In November 2022, we conducted load testing of the Bitrix24 Self-hosted (Enterprise edition) with a deployed CRM solution to assess its performance.

General information

During the load test, we deployed a CRM system in a clustered solution, filled it with a large amount of test data and subjected it to a load that imitated the behavior of real-life users. The operations performed for load testing purposes included standard CRM scenarios: system authorization, customer base handling, lead management, deal processing and deal status updating, instant messaging and document uploading.
Based on the needs of large-scale corporate segment, we have determined the key objectives for such testing:

  • simulate the operation of a CRM system deployed at a large corporation (tens of thousands of employees, thousand CRM users + large amount of demo data, with high intensity information flowrates)
  • provide a testing methodology that’s as close as possible to user behavior in real life
  • demonstrate the stable operation of inbuilt Bitrix24 scaling and fail-safety technologies on available equipment
  • ensure the response time of deployed solution does not exceed 1 second.

Test Conditions

Equipment

Dedicated virtual servers were selected for performing the test. This allowed to emulate a full-scale cluster solution with simultaneous overhead reduction for infrastructure deployment and support. Similar to the previous testing, we have employed equipment from the mid-price segment available on the market.
The hardware provided for the test featured three configurations of servers:

  • Database server (2 pcs.): Intel Xeon (Skylake, IBRS): 2294 MHz (10 cores), 48 GB RAM, 300 GB HDD
  • Application Server (2 pcs.): Intel Core (Haswell, noTSX, IBRS): 1995 MHz (6 core), 24 GB RAM, 200 GB HDD
  • Load generator (1 pcs.): Intel Xeon (Skylake, IBRS): 2294 MHz (10 core), 24 GB RAM, 300 GB HDD.

Cluster solution and web environment

perf_2022_1

pfweb1, pfweb2 Cluster featuring two application servers (web servers): CentOS 7.9, Nginx 1.20.2, Apache 2.4.6, PHP 8.0.19.
Balancing was performed using Nginx at pfweb1
pfdb1, pfdb2 Cluster featuring two database servers: CentOS 7.9, Percona Server (MySQL) 8.0.29, Master / Slave configuration.
pfjm Load generating server with Jmeter 5.3.3 InfluxDB for Jmeter and Grafana data retrieval and storage to test result display and visualization
data processes and threads (1-7)
  1. Test traffic
  2. Load balancing
  3. Cluster cache
  4. Cluster sessions
  5. File synchronization between Lsync nodes
  6. Database replication
  7. Core database connections for product performance

We have deployed a cluster featuring 2 database servers and 2 application servers. This specific configuration is standard and most widespread, chosen to ensure high cluster performance while also providing high availability.
Server software was configured using the Bitrix24 Virtual Appliance for Linux 7.5.2 and the Web Cluster module enabled to create a cluster solution.
After conducting initial "trial" tests, server configurations were updated.

List of web environment updates:

  • updated operation system parameters:
    • echo 1024 65000 > /proc/sys/net/ipv4/ip_local_port_range
    • sysctl -w net.ipv4.tcp_tw_recycle=1
    • sysctl -w net.ipv4.tcp_tw_reuse=1
  • sync_binlog increased to 1000
  • innodb_buffer_pool_size increased to 34 GB
  • max_heap_table_size set as 256 MB
  • max_heap_table_size, max_heap_table_size set as 90 MB
  • myisam_sort_buffer_size increased to 64 MB
  • max_connections increased to 305

Intranet portal test solution

Cluster solution had a deployed Bitrix24 Self-hosted (Enterprise edition), version 22.375 (Main module version).

  1. Standard Bitrix24 Self-hosted (Enterprise edition), version 22.375 with the latest updates and the Web Cluster module enabled to create a cluster solution.
  2. Demo content at the start of the final test:
      • 500,000 companies
      • 1,515,000 contacts
      • 995,000 leads
      • 500,000 deals
      • 144,000 catalog products
      • 9,391,000 CRM events
      • 4,522,000 CRM timeline records
      • 36,000 Feed messages
  3. The number of employees in the portal database at the time of the start of the final test is 22,198 (distributed across 14 structural divisions).

Load generation method

Testing was performed using two main CRM scenarios:

  • The "Quick leads" scenario with a large number of managers (up to 500 and more) that manage the flow of incoming leads for subsequent lead qualification as lost or won/conversion to a deal for purchase of products/services.
  • The "Longterm deals" scenario with a large number of managers (up to 100 and more) that handle deals, limited per 1 manager. Each day, a manager adds new statuses and updates to handled deals: schedules meetings, uploads documents, comments, calls and corrects specification. Key feature: large number of products in an individual deal.

The test envisaged the company using both scenarios simultaneously, associated with several business directions.
Selected load scenario was quite resource-consuming, allowing to check performance for both Bitrix24 portal and CRM feature within the framework of complex projects.

Test scenarios details are provided below:

Scenario (load %)ActionWeight, %
Quick leads (80%) Authorization 100
Lead list100
Repeated lead list100
Lead100
Product upload100
Product upload 2100
Lost lead18
Won lead2
Update of kanban, leads, push100
Lead list100
Authorization (20%) Authorization 100
Lead list100
Deal list100
Opened deal100
Payment document upload100
Adding a call5
Companies (1%)Authorization 100
Lead list100
Company list100

The load was created using the JMeter tool, version 5.5. The test data was recorded via InfluxDB, a high-performance database designed to handle high query and record loads. Analysts used Grafana for visualization. The servers were monitored using the Telegraf application.

Load generation method

Bitrix24 CRM (Enterprise edition) performing within test scenarios emulating a CRM-based solution for a large company, deployed on a cluster of 4 physical servers in a 24-hour test has ensured stable concurrent CRM operation for 3000 employees.
The monitoring system demonstrated 423 874 system queries per 24-hours with average response time 0.713 seconds in 95% of requests.


Test summary: number of flows, dynamics for number of queries per second, total number of queries and errors:

perf_2022_2

Response time. 95 percentiles for specific pages and their traffic within a test:

perf_2022_3

CPU use at database and app servers:

perf_2022_4

Conclusion

The results of the load test of the Bitrix24 Self-hosted (Enterprise edition) confirmed the high performance of the platform and stability in high load conditions. The demo CRM solution, deployed in a cluster solution of 4 servers ensured the simultaneous operation of 3000 users, which corresponds to the approximate load profile for a large corporation.

At the same time, the system provided fast (even by the standards of Internet projects) response not exceeding 1 second for the majority of requests, which ensures a smooth user experience. The web cluster technology has once again proven its performance and reliability.

The testing methodology, scenarios, and load profiles as well as the use of physical user profiles allowed us to get the test conditions that were as close to real life as possible. This means that the results of this test can be used as a reference for designing new projects.

We are ready to provide the testing methodology, portal configuration, and test environment upon request to our corporate customers and integrators to serve as a basis for conducting full-scale testing of custom-developed portals or their prototypes, testing hypotheses, calculating the safety factor of our solution under peak loads. The test environment can be further customized - e.g., by adding integrations with a company’s corporate systems to ensure more accurate results.

Please, contact us to obtain any additional testing details.

2021 Load Testing

In January 2021, we conducted load testing of the self-hosted version of Bitrix24 (Enterprise edition) to assess the performance of a large company’s intranet portal built on its basis.

General information

During the load test, an intranet portal deployed in a clustered solution and filled with a large amount of test data was subjected to a load that imitated the behavior of real-life users.

Virtual users performed sets of operations that resembled typical work scenarios:

  • logging in/out
  • reading news
  • opening workgroups
  • browsing the Feed
  • creating individual and group chats
  • exchanging instant messages
  • setting tasks
  • downloading documents
  • launching business processes

Each user completed one of the typical scenarios and finished their work on the portal.

Test objectives

  • simulate the operation of an intranet portal of a large corporation (100,000 employees + a large amount of demo data)
  • provide a testing methodology that’s as close as possible to user behavior in real life
  • demonstrate the stable operation of the portal without errors on available equipment (in a situation when ⅓ of all company’s employees (at least 30,000) are using the portal simultaneously)
  • ensure the response time of the portal does not exceed 1 second (for 95% of requests)

Test Conditions

Equipment

The hardware provided for the test featured physical servers in two configurations:

  • Database server: Intel Xeon W-2255 3.7 GHz (10 cores), 128 GB DDR4, 2 x 960 GB NVMe + 2 x 8000 GB HDD
  • Application Server: Intel Xeon E-2236 3.4 GHz (6 cores), 32 GB DDR4, 2 x 480 GB SSD

A cluster featuring 2 database servers and 3 application servers (web servers) was deployed. This particular configuration was chosen to ensure high cluster performance while also providing high availability.

Test intranet portal parameters

1. Standard self-hosted version Bitrix24 (Enterprise edition), version 20.x.x with the latest updates and the Web Cluster module enabled to create a cluster solution.

2. Demo content at the start of the final test:

  • 590,000 messages in the Feed
  • 540,000comments
  • 40,000 news posts
  • 180,000 tasks
  • 415,000 instant messages

3. The number of employees in the portal database at the time of the start of the final test is 111,304 (distributed across 67 structural divisions).

Load generation method

The load was created using the JMeter tool, version 5.3.3. The test data was recorded via InfluxDB, a high-performance database designed to handle high query and record loads. Analysts used Grafana for visualization. The servers were monitored using the Zabbix monitoring system.

For testing purposes, 29 scenarios were selected from 13 blocks characteristic of a typical intranet portal:

  • Authorization
  • Feed
  • Search
  • Chats (group and one-on-one)
  • Tasks
  • Calendar
  • Drive
  • News posts
  • Picture gallery
  • Employees
  • Profile
  • Business processes
  • Workgroups

For each test, a special load distribution system was used taking into account the work of various users on the portal and the share of each of the blocks in the total load.

Unlike the previous load test in 2015, we used a new method of generating user load. Instead of one virtual user randomly performing sets of standard actions, the load was generated by a large number of different users each with their own account on the portal.

The load generator logged each of these users into the system and then executed a different set of user scenarios under that account. The timeout between the start of each new script was from 20 seconds to 10 minutes. This was done to emulate the most realistic behavior of a company’s employees who logged into the intranet portal and work inside it during the day.

Test results

An intranet Bitrix24 test portal (Enterprise edition) for 111,000 users installed on a cluster of 5 physical servers was able to provide smooth operation for 30,000 employees simultaneously. The maximum response time did not exceed 0,9 seconds in 95% of requests.

During the course of 1 hour, the aforementioned number of users generated:

  • 39 news posts
  • 4,614 messages in the Feed
  • 4,685 comments
  • 440 chats
  • 3,386 instant messages
  • 2,297 business processes
  • 912 tasks
  • 367 documents
  • 92 workgroups and projects
  • 291 events in Calendar
  • 2,385 notifications

During the course of 24 hours, there will be generated:

  • 110,736 new entries in the Feed
  • 10,560 individual and group chats
  • 2,208 workgroups and projects
  • 112,440 comments
  • 936 news posts
  • 6,984 events in Calendar
  • 21,888 tasks and assignments
  • 8,808 documents
  • 81,264 instant messages
  • 57,240 notifications
  • 55,128 business processes

Conclusion

The results of the load test of the self-hosted version of Bitrix24 (Enterprise edition) confirmed the high performance of the platform and stability in high load conditions. The demo portal deployed in a cluster solution of 5 servers ensured the simultaneous operation of 30 thousand users, which corresponds to the approximate load profile for a large corporation that employs 100-200 thousand employees.

At the same time, the system provided fast (even by the standards of Internet projects) response not exceeding 1 second for the majority of requests, which ensures a smooth user experience. The web cluster technology has once again proven its performance and reliability.

The testing methodology, scenarios, and load profiles as well as the use of physical user profiles allowed us to get the test conditions that were as close to real life as possible. This means that the results of this test can be used as a reference for designing new projects.

It’s also worth noting that our goal was not to maximize the amount of generated content on the portal during the test (news, tasks, documents, messages, and others). The goal was to create an accurate simulation of real-life user behavior. Still, the speed of adding information to the portal was also very high, which should be more than enough to satisfy the needs of even the most demanding clients.

We are ready to provide the testing methodology, portal configuration, and test environment upon request to our corporate customers and integrators to serve as a basis for conducting full-scale testing of custom-developed portals or their prototypes, testing hypotheses, calculating the safety factor of our solution under peak loads.

The test environment can be further customized - e.g., by adding integrations with a company’s corporate systems to ensure more accurate results.

2015 Load Testing

In December 2015, Bitrix24 performed load testing of the then-latest version of Bitrix24 to assess the system performance in a large-scale business environment with a substantial number of employees and high load.

General information

Load testing was carried out for full 24 hours on a deployed portal populated with a large amount of test data. During that period, the portal had been continuously engaged with incoming data imitating the behavior of real human visitors.

The virtual users were tasked to do many standard operations, including:

  • logging on and off
  • posting to and commenting in Feed
  • using tasks and documents
  • adding records to CRM
  • sending instant messages and more

Each of the virtual users was assigned a predefined role consisting of many operations carried out in an infinite loop, with a slight delay between iterations.

The test was suited to model the real-world scenario when all the employees are enrolled in the portal database and a good deal of them are using the portal throughout the day performing a number of typical actions.

Test conditions

Test software:

  • Bitrix24 version: 15.0.4
  • Hardware: one server; a cluster of 2 and 3 commonly used servers.

Parameters:

  • A ready-to-use, out-of-the-box Bitrix24 template

Demo data:

  • 15,683 employees
  • 38,146 posts in the Feed
  • 10,460 tasks
  • 194,368 instant messages
  • 2,286 files on Bitrix24.Drive
  • 4,966 leads and 697 companies in CRM

Test environment:

  • Hardware: Intel Xeon E3-1270v3 3.5 HHz, 32 GB RAM, 2 х 240 GB SSD.
  • Server environment configured using Bitrix Virtual Appliance. PHP updated to 5.6.9.

Test results

An out-of-the-box Bitrix24 portal (BizPace Enterprise edition) installed on a single commonly used server with test data including 15,683 employees was capable of serving 5,000 employees simultaneously. The maximum response time did not exceed 1,065 seconds.

During the course of 24 hours, this number of employees performed the following actions:

  • sent 19,887 instant messages
  • posted 987 messages to the Feed
  • created 1,645 tasks
  • added 987 documents to Bitrix24.Drive
  • added 384 leads and 384 companies to CRM

An out-of-the-box Bitrix24 portal (BizPace Enterprise edition) installed on a cluster of two commonly used servers with test data including 15,683 employees was capable of serving 7,500 employees simultaneously (+50%). The maximum response time did not exceed 1.224 seconds.

During the course of 24 hours, this number of employees performed the following actions:

  • sent 28,836 instant messages
  • posted 1,473 messages to the Feed
  • created 2,455 tasks
  • added 1,473 documents to Bitrix24.Drive
  • added 571 leads and 571 companies to CRM

An out-of-the-box Bitrix24 portal installed on a cluster of three commonly used servers with test data including 15,683 employees was capable of serving 9,500 employees simultaneously (+90%). The maximum response time did not exceed 0.949 seconds.

During the course of 24 hours, this number of employees performed the following actions:

  • sent 36,702 instant messages
  • posted 3,124 messages to the Feed
  • created 2,455 tasks
  • added 3,124 documents to Bitrix24.Drive
  • added 729 leads and 729 companies to CRM

Optimal Performance

Depending on the expected load and number of daily users accessing the self-hosted version of Bitrix24, your servers should meet the following hardware requirements.

Minimum hardware requirements

  • 2 GHz 32-bit (x86) or 64-bit (x64) processor, or faster
  • 8 GB RAM (32-bit or 64-bit), or more
  • 500 GB available hard disk space, or more

Medium-load project hardware requirements

  • 4 GHz 32-bit (x86) or 64-bit (x64) processor, or faster
  • 64 GB RAM (32-bit or 64-bit), or more
  • 2 TB available hard disk space, or higher

High-load project hardware requirements

  • 4 GHz 32-bit (x86) or 64-bit (x64) processor, or faster
  • 256 GB RAM (32-bit or 64-bit), or more
  • 5 TB available hard disk space, or more