If you run anything network-heavy on EC2 (Elastic Compute Cloud) — Kafka in my case — you have probably met this counter:

$ ethtool -S ens34 | grep allowance_exceeded
     bw_in_allowance_exceeded: 0
     bw_out_allowance_exceeded: 39
     pps_allowance_exceeded: 0
     conntrack_allowance_exceeded: 0
     linklocal_allowance_exceeded: 0

The Nitro card exposes these through the ENA (Elastic Network Adapter) driver, and bw_out_allowance_exceeded is the one that shows up on Kafka broker dashboards when things get busy. Node exporter exposes the metric as node_ethtool_bw_out_allowance_exceeded.

The question I could never answer from the docs was simple: When that number goes up, what actually happened?

There are two different things AWS (Amazon Web Services) shapes on the way out, and they are easy to confuse. Both are covered in the EC2 instance network bandwidth docs:

1. The instance aggregate allowance — everything the instance sends, added up. This is your top-level ENA bandwidth allowance on any given instance type. Some instances get burst credits; those are out of scope here.
2. The 5 Gbps per-flow cap — a silent ceiling per TCP (Transmission Control Protocol) connection that Nitro applies to any traffic outside a cluster placement group.

That second one catches people out. It does not matter that your instance can push 25 Gbps; one connection still gets 5.

So when the counter climbs, is it “the box is saturated” or “one greedy consumer is brushing the flow cap”? Those call for completely different responses, and I had been reading the same graph as if it meant both. An AWS support engineer told me it increments in both cases. The public docs, read closely, only ever define the counters against instance-level maximums. Someone was wrong.

Let’s test it and see what actually happens

I will show you how to test being throttled by the per-flow cap vs the ENA network-level cap, and the implications of either or both.

The trick is picking an instance where the two limits do not overlap.

An m8g.xlarge has a 1.875 Gbps baseline and can burst to 12.5 Gbps on network credits. The per-flow cap is 5 Gbps. That leaves a useful gap: a single flow can sit pinned at 5 Gbps while the instance still has aggregate headroom to spare.

That isolates the two mechanisms. If I saturate one flow and the counter stays still, per-flow shaping is invisible to it. Then I open enough flows to blow past 12.5 Gbps and check if the counter moves at all, so I know the rig works.

Two boxes in the same AZ (Availability Zone), iperf3, and a stopwatch. Using iperf you can also specify your congestion control mechanism.

Test A: one flow, BBR

Read the counter, run a single BBR (Bottleneck Bandwidth and Round-trip propagation time) flow for 60 seconds, read it again.

sudo ethtool -S ens34 | grep allowance_exceeded
iperf3 -c <receiver> -t 60
sudo ethtool -S ens34 | grep allowance_exceeded

Throughput sat at 4.9–5.0 Gbit/s in every single interval, averaging 4.88 Gbit/s. That flat line at almost exactly 5 is the flow cap doing its job, and it is nowhere near the 12.5 Gbps the instance could have burst to.

Retransmits: 0. The cap is enforced by pacing and queueing, not by dropping.

Counter before: 39. Counter after: 39.

Nothing. But I did not trust it yet.

Test A2: one flow, Cubic

The obvious objection: BBR measures the bottleneck and paces itself to fit. Of course the shaper never complained — BBR handed it perfectly compliant traffic. A congestion control that just piles on until something breaks might get a different answer.

iperf3 can set congestion control per socket, so this needs no system changes:

iperf3 -c <receiver> -C cubic -t 60

Same plateau: 4.97 Gbit/s. Retransmits: 0 again. Counter: unchanged.

But something interesting turned up. Cubic’s congestion window grew and grew, all the way to 13.6 MB. For two boxes in the same AZ at 5 Gbps, the bandwidth-delay product (BDP) is about 125 KB. Cubic had roughly a hundred times more data in flight than the path could hold.

This is what you would expect from Cubic. Cubic grows the window until it sees loss, the flow cap never drops anything, so Cubic never gets the hint. All that excess ends up sitting in the shaper’s queue, quietly inflating RTT (round-trip time).

The reason it did not overshoot harder is worth knowing. Since kernel 3.16, the kernel computes sk_pacing_rate for every congestion control, not just BBR, and the fq (Fair Queue) qdisc (queueing discipline) honours it. On an fq host there is no such thing as truly unpaced Cubic. Every box I care about runs fq, so this is the realistic case — but it is the one caveat on the result. I have not tested a genuinely unpaced host.

Test B1: enough flows to actually hurt

Sanity check. Eight parallel flows across two receivers, each capable of ~5 Gbps on its own, so the offered load is far above the 12.5 Gbps aggregate.

iperf3 -c <receiver-1> -P 4 -t 60 &
iperf3 -c <receiver-2> -P 4 -t 60
wait

The shaper carved up the budget unevenly — 3.0, 3.1, 1.1 and 1.1 Gbit/s on one half. Retransmits showed up: 58.

And the counter went from 39 to 10,404,842.

Ten and a half million in one minute. The rig can absolutely trip this counter. So the two null results are real, not a broken measurement.

Counter ticks are not packet loss

Look at those two numbers together, because I had been reading them wrong for months.

10.4 million counter ticks. 58 retransmits.

The docs say the counters record packets “queued or dropped” — and queued and dropped are counted identically. So essentially every one of those ten million packets was delayed, not lost.

A big number here means the shaper is engaged. It does not mean you are losing data. If you want to know whether anything was actually damaged, you have to look at retransmits separately.

What the docs do and do not say

Once I knew the answer, the documentation read differently. AWS defines exactly five allowances: aggregate bandwidth in, aggregate bandwidth out, PPS (packets per second), conntrack (connection tracking), and link-local PPS. The per-flow cap is not one of them. The docs agreed with the experiment the whole time; they just never say so out loud.

Also useful: allowances are enforced at sub-millisecond granularity, roughly proportionally. A 10 Gbps instance gets about 10 Mb (megabits) per millisecond. That is why sub-second microbursts tick these counters while your average utilisation graph looks half-asleep.

What is not public: the token bucket parameters, the burst tolerance, the shaper’s queue depth, and everything about the per-flow shaper. It has no counters and no queue visibility at all. You can only infer it from a throughput plateau near 5 Gbps plus rising RTT.

The cheat sheet

The practical upshot for me: on production brokers running 50 Gbps-class instances, every bw_out_allowance_exceeded spike is genuine aggregate throttling. Given those boxes average nowhere near 50 Gbps, that almost always means microbursts, not sustained saturation. And it is never one hot flow at 5 Gbps.

That also forced me to correct my own story about an earlier incident, where I had blamed a runaway counter on a flow hitting the cap. It cannot have been that. The fix I applied at the time still holds up, but the mechanism I wrote down was wrong.

The uncomfortable part is that flow-cap events remain invisible. There is no counter for the thing that silently halves your throughput. If it matters to you, you need to watch per-peer send rates yourself and look for one that is suspiciously pinned just under 5 Gbps.

Other interesting/related things:

1. When I first encountered these issues ENA express was not yet fully supported in cross AZ clusters. Now it is. Maybe I will write a bit more about that when we migrate.
2. You have the option of rate pacing in BBR. You can always pace your per flow traffic. Be aware how aggressively you pace and its impact on your clients and downstream systems.