CyberRatings: Cloud Firewalls Have Gaping Holes

In the release of remarkable firewall test results today, independent nonprofit testing firm CyberRatings.org revealed wild variability in network and cloud firewall efficacy, with special concerns about the firewall instances running in the major public clouds, which seemed not to work very well at all.

In the release of the CyberRatings Q1 2025 Comparative Test Report on Cloud Network Firewalls (CNFWs), many traditional firewalls performed quite well with efficacy ranging at almost 100%. Third-party firewalls from Check Point, Fortinet, Juniper Networks, Palo Alto Networks, and Versa Networks demonstrated the highest security effectiveness blocking exploits and evasion tactics. Results ranged from 99.61% to 100%.

But move into the public cloud, and you get a different story. Some native firewalls from Amazon Web Services (AWS), Google Cloud Platform (GCP), and Microsoft Azure received a 0% Security Effectiveness score as they allowed attacks to bypass existing defenses. In addition, Cisco’s Secure Firewall Defense didn’t receive high ratings, with a 54.5% effectiveness rating and the highest costs per bit of traffic in the bunch.

Firewalls on Fire?

Many CNFWs in public clouds are actually resold versions of the firewalls from the major firewall vendors, so at first this might be a headscratcher. CyberRatings said the discrepancy in the performance of third-party firewalls and those hosted on the public clouds comes down to versioning, in which the same firewalls hosted on public clouds may not be the same up-to-date versions of the firewalls purchased through the third parties.

For example, in looking at the versions of Palo Alto Networks firewalls purchased from Palo Alto or hosted in a public cloud, CyberRatings concluded these were in fact different solutions:

“We attribute the differences in security effectiveness and performance results between the two platforms to each provider independently selecting and deploying different software versions based on their own criteria.”

Ian Foo, the CTO and EVP of product for CyberRatings, explained it further: "If you look at our test data, there is a versioning difference between the Palo Alto Networks products [in different clouds]. If they are using an OEM vendor, they don't know what they are getting."

Foo added that many of the cloud-based firewalls didn't perform well on basic tests. For example, the Google firewalls failed basic evasions, AWS "didn't do well on exploits or evasions," and Azure "couldn't block exploits... they don't support TLS or SSL."

In these dramatic results, CyberRatings went so far as to tell end users to avoid using the firewalls provided by public cloud services altogether.

“Until cloud service provider native firewalls provide better protection, customers should be looking to third parties for their cloud security needs,” said Vikram Phatak, CEO of CyberRatings.org, in a company statement. “Traditional third-party security vendors have demonstrated that they bring significant value to customers.”

In speaking to some security experts here at KubeCon in London, I found out that this may be a well-known secret among those in the security community, though not as well known by cloud users.

"It's amazing they released this," Anirban Sengupta, the CTO of cloud networking and security vendor Aviatrix told me when I asked him about the results. "These gaps between third-party and cloud versions are known in the security community, but most customers don't know that. It's better that they are aware of the shortcomings and they can choose what they want."

Behind the Test Methodology

In the Q1 test, CyberRatings.org used a battery of samples and attacks to test the firewalls. These included :

False Positives: 2,760 samples from various business-critical files and applications, ensuring security measures did not disrupt legitimate traffic.
Exploits: 2,028 attack samples from widely exploited vulnerabilities in enterprise environments.
Evasion Techniques: 2,500 attacks spanning 27 evasion techniques tested across multiple network layers to bypass firewall defenses.
Performance Metrics: 46 different stress and capacity tests under diverse workloads.
Stability & Reliability: Seven extended tests simulating prolonged real-world attack and operational scenarios.

The methodology tested the firewalls at three separate layers of the Open Systems Interconnection (OSI) model, specifically Layers 3, 4, and 7. In the scoring system, missing lower-layer evasions had the greatest impact on the overall score because CyberRatings said a lower-layer compromise undermines the firewall’s primary protective function. Points were deducted based on the firewall’s ability—or inability—to detect evasions.

In other findings, it was clear that the firewalls had highly variable costs on handling traffic. Cisco’s Secure Firewall Threat Defense Virtual received a “caution” rating due to low effectiveness as well as high relative cost at $10.62 per Mbps, while most firewall costs were in the $2-$4 per Mbps range. The overall results can be seen below.