Who offers the best value for money?

Comparison of public cloud offers
Cloud Benchmark 2021: Who offers the best value?



By Michael Herman *

providers on the subject

Infrastructure-as-a-Service (IaaS) sets the pulse for digitization: Powerful processors (vCPUs) and virtual machines (VMs) from the public cloud are an integral part of many companies’ IT strategies. Providers are constantly adapting technologies and services. The performance of cloud services varies accordingly.

CPU, networking and storage technologies are constantly evolving, resulting in price / performance differences between the top four cloud cloud service providers.

(Image: Igor Link – stock.adobe.com)

As in 2020, Cloud Mercato tested a number of comparable 16 VM VCPUs and storage blocks from Microsoft Azure, Amazon Web Services (AWS), Google Cloud and Open Telekom Cloud, developed by European data centers. In addition to this basic test panel, analysts also tested additional VM scaling for specific use cases. The benchmark focused on key parameters such as computing power, memory, bandwidth, OpenSSL encryption, and relational databases. The research was commissioned by T-Systems in July 2021.

Computing power: Geekbench has differences in performance

MIPS (millions of commands per second) or FLOPS (floating point per second operations) as reported by CPU vendors are not sufficient to fully understand CPU performance and performance differences in real cloud workloads. Therefore, analysts used the Geekbench 5 reference suite.

Most virtual machines on the base control panel with comparable VM 16 vCPUs yielded an output value of approximately 8,000 for multiple threads, processing multiple threads simultaneously in one process. From the second panel with additional VMs, the AWS m6i achieved a performance increase of more than 10,000, while Azure and Google Cloud scored just 8,000. However, the escalator VMs surpassed all virtual machines in the Open Telekom Cloud: they ended up with prices above 12,000 and, therefore, a performance increase of more than 50 percent compared to the average.

Local bandwidth on the network: High bandwidth

To measure the local bandwidth on the network, the analysts used the iPerf 3 tool with the TCP (Transmission Control Protocol). For the test setup, each used two identical VMs in the same data center. The number of threads to be processed corresponded to the number of CPUs in order to create maximum performance.

Google Cloud showed performance above 29 Gbit / s, but with high fluctuations, especially compared to the theoretical maximum bandwidth. Network-optimized VMs from AWS (c5n, m5n, r5n) provided a significant increase in performance over general-purpose variants from AWS by more than doubling the network bandwidth from about 10 to about 24 Gbit / s. However, AWS was still below the standard level of the corresponding c4 variant of the Open Telekom Cloud, which reaches about 24.5 Gbit / s. Azure VMs had the lowest network performance scores.

Significantly different encryption speed

Data encryption contributes significantly to secure, compatible cloud usage and affects service performance: How many bytes can be encrypted or decrypted with VM 16 vCPUs per second? To find out, Cloud Mercato used the OpenSSL Speed ​​open source tool, which analyzes encryption and decryption using various algorithms and block sizes.

Test results show that Open Telekom Cloud is well ahead of the top three for both AES-256-CBC and SHA-512 encryption – with about 60 percent higher performance, regardless of algorithm and size of the block. The difference is particularly large with a block size of 16,384 bytes and AES-256-CBC encryption: Here the performance range ranges from the standard Azure E16 with about 220,000 encrypted bytes per second to the s3 variant of Open Telekom Cloud, which with about 2, 7 million bytes, has a speed that is more than 10 times faster.

RAM performance and storage blocks with significant range

To evaluate the main memory (RAM) performance, the analyzers tested the bandwidth between the CPU and the memory to read and write access with Sysbench. The AWS c5 variant showed the best test panel write-off performance, while the Azure E16-4 showed the lowest performance. With more than 71,000 units, the c5 also delivered good reading performance – surpassing Google’s n2 standard (72,000) and the Open Telekom Cloud m4 variant (75,000). None of the VMs from the special purpose add-on set (see diagram) achieved these general purpose VM values.

To measure permanent block storage, analysts used the Flexible I / O Tester (FIO) to determine the number of IOPS (input / output per second). Google topped with more than 50,000 IOPS, with write-in performance exceeding read-by-10,000 IOPS. In second place here is the Open Telekom Cloud with 20,000 IOPS.

Relationship database services

Cloud Mercato tested the performance of relational database services with the Sysbench OLTP benchmarking tool. To explore different uses and scenarios, Cloud Mercato scaled from 1 to 128 clients using read-only, read-only, and read / write functions. Google did well in the sign-up mode, but showed significant reductions in high-scale performance. Open Telekom Cloud showed the best read performance – especially with a large number of customers – with around 195,000 read per second with 128 customers, followed by AWS with almost 150,000.

Performance, price and utility: who performs best?

To realistically evaluate the usefulness of cloud services, it is important to consider not only performance but also price: How much performance do users get for a cent? Cloud Mercato determined this price / performance ratio based on provider prices as of August 1, 2021.

Like last year, a virtual machine from Open Telekom Cloud had the best performance. At the time of testing, the s3 variant offered about a 15 percent higher return on money compared to the next best offer on the market. Meanwhile, the escalators overtook and improved the price-performance ratio compared to the Open Telekom Cloud.

However, analyzes show that additional VMs for specific uses are not automatically the best choice. General purpose VMs, which are suitable for all purposes, can also deliver specific workloads at a favorable price-performance ratio. As last year, Mercato Cloud analysts concluded: “Open Telekom Cloud offerings are well-balanced for all applications and are on par with both escalators in terms of performance and price-performance ratio – and indeed outperform the competition. This is especially true in the areas of encryption and CPU performance. “Therefore, users do not have to worry about choosing the right VM variant:” Open Telekom Cloud s3 and c4 is a good and consistent choice for almost every case the s3 plus 15 percent compared to the next best offer in terms of value for money. ”

You can download the full Cloud Mercato study with all the measurement data for the examined offers and all the price lists after your registration.

Even with AI at eye level

Another study by Mercato Cloud shows that virtual machines from Open Telekom Cloud also withstand artificial intelligence (AI) compared to off-scale ones. At the Performance & Benchmark of AI IaaS Providers, analysts looked at offers from Google, Azure, AWS and Open Telekom Cloud. In many artificial intelligence applications, such as object recognition, graphics card (GPU) performance is vital to image processing and resolution. Therefore, in addition to CPU and RAM, the focus of the research was mainly on virtual GPUs.

Result: The benchmark shows that providers differ only slightly in the performance of their respective GPUs. Open Telekom Cloud is on par with escalators that offer similar performance and similar price-performance ratios. “With a focus on GDPR compliance and the ModelArts suite, the Open Telekom Cloud is a powerful European alternative to the development, training and operation of artificial intelligence,” concludes Cloud Mercato’s study.

* Author Michael Hermann works as a freelance journalist and writer and writes mainly on ICT issues on behalf of companies and institutions.

(ID: 48402660)

Leave a Comment