Iperf test through ASA

[AirSail]

Hello Folks, 

I have a pair of ASA5516 with HA mode, ISP provides a high-speed WAN 500Mbps up/download, 

while doing some speed tests recently we noticed that all our tests from the inside network are not bypassing 100mbps, 

I checked all inside/outside interfaces "show run inter x/y" and all are showing 1G negotiation, I checked also all interfaces going all the way to the terminals and also to the ISP router. ALL SHOWING 1G negotiation, 

to dig deeper, I moved onsite I spoke to ISP to set an Iperf server in his side and give us the pubic IP, 

and I set Iperf client on my PC, 
* disconnect the outside interface and connect it to my PC (set public IP on my PC) I'm getting full WAN speed, 

* disconnect inside interface and connect it to my PC (set an IP address LAN) I'm getting 100Mbps 

I'm confused here, there is no bandwidth cap on my configuration,   

ASA Gurus, need your assistance? 

[MHM Cisco World]

Show interface IN/OUT <- share the show interface of both in and out interface 

Are asa use pppoe with ISP?

MHM

[AirSail]

no pppoe setup in the ASA fo internet connection 

will share that as soon as get access to the ASA 

[AirSail]

@MHM Cisco World here you go 

 show interface inside
Interface GigabitEthernet1/2 "inside", is up, line protocol is up
Hardware is Accelerator rev01, BW 1000 Mbps, DLY 10 usec
Auto-Duplex(Full-duplex), Auto-Speed(1000 Mbps)
Input flow control is unsupported, output flow control is off
MAC address f80b.cbc4.b727, MTU 1500
IP address ----------, subnet mask ------------
7700798351 packets input, 5786718525856 bytes, 0 no buffer
Received 71406311 broadcasts, 0 runts, 0 giants
304 input errors, 0 CRC, 304 frame, 0 overrun, 0 ignored, 0 abort
0 pause input, 0 resume input
0 L2 decode drops
5786834857 packets output, 3959138579284 bytes, 0 underruns
0 pause output, 0 resume output
0 output errors, 0 collisions, 0 interface resets
0 late collisions, 0 deferred
0 input reset drops, 24 output reset drops
input queue (blocks free curr/low): hardware (1971/1819)
output queue (blocks free curr/low): hardware (2047/1606)
Traffic Statistics for "inside":
7061788943 packets input, 5658458772219 bytes
5786834857 packets output, 3852074986797 bytes
68356019 packets dropped
1 minute input rate 7877 pkts/sec, 7548226 bytes/sec
1 minute output rate 5694 pkts/sec, 3754264 bytes/sec
1 minute drop rate, 10 pkts/sec
5 minute input rate 7365 pkts/sec, 6673943 bytes/sec
5 minute output rate 6282 pkts/sec, 4110732 bytes/sec
5 minute drop rate, 12 pkts/sec

 show interface outside
Interface GigabitEthernet1/1 "outside", is up, line protocol is up
Hardware is Accelerator rev01, BW 1000 Mbps, DLY 10 usec
Auto-Duplex(Full-duplex), Auto-Speed(1000 Mbps)
Input flow control is unsupported, output flow control is off
MAC address f80b.cbc4.b726, MTU 1500
IP address ---------, subnet mask -----------
5734447821 packets input, 4160222411621 bytes, 0 no buffer
Received 30835512 broadcasts, 0 runts, 0 giants
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
0 pause input, 0 resume input
0 L2 decode drops
6697368727 packets output, 5676416488542 bytes, 0 underruns
0 pause output, 0 resume output
0 output errors, 0 collisions, 0 interface resets
0 late collisions, 0 deferred
0 input reset drops, 0 output reset drops
input queue (blocks free curr/low): hardware (2019/1819)
output queue (blocks free curr/low): hardware (2047/1699)
Traffic Statistics for "outside":
5706615466 packets input, 4056768933014 bytes
6697368727 packets output, 5555512002686 bytes
43919763 packets dropped
1 minute input rate 5645 pkts/sec, 4116990 bytes/sec
1 minute output rate 7238 pkts/sec, 7048921 bytes/sec
1 minute drop rate, 72 pkts/sec
5 minute input rate 6245 pkts/sec, 4425410 bytes/sec
5 minute output rate 7280 pkts/sec, 6804177 bytes/sec
5 minute drop rate, 26 pkts/sec

[MHM Cisco World]

68356019 packets dropped

This huge drop and interface is full-duplex and overrun counter is zero

So 

Show asp drop

Do this two or three times 

Check which drop is increasing rapidly

Share the asp drop here if you can

I think tcp out of order make this issue' but let check that 

MHM

[AirSail]

@MHM Cisco World Attached, 
do you think a packet drop may limit the through put ? 
comparing show inter out/in from yesterday and today, it showing a very few drops 

[MHM Cisco World]

VPN conflict <<- why there are high VPN conflict? how many VPN you run 
FW L2 ACL <<- this need to check if you run router mode then there is no L2 ACL I will more check this point  
QoS drop <<-
show service-policy police 
show service-policy shape
show priority-queue statis OUT

share above 

MHM
 

[AirSail]

@MHM Cisco World - man I think you chased something important here, 

#show service-policy police

Interface outside:
Service-policy: <NAME HERE>
Class-map: bandwidth
Input police Interface outside:
cir 100000000 bps, bc 50000 bytes
conformed 12029323552 packets, 9018979062185 bytes; actions: transmit
exceeded 3492948 packets, 4926824082 bytes; actions: drop
conformed 50560648 bps, exceed 18760 bps
Output police Interface outside:
cir 100000000 bps, bc 50000 bytes
conformed 13984694007 packets, 11943019308317 bytes; actions: transmit
exceeded 15878951 packets, 23058164314 bytes; actions: drop
conformed 58548416 bps, exceed 30976 bps

that CIR 100000000 bps could be the root cause? 

[MHM Cisco World]

It sure can be the issue here'

Do test again and monitor the drop

MHM

[AirSail]

@MHM Cisco World  what could be the reason behind placing this bandwidth limit in the outside interface? protecting from burst traffic? what do you think?  

to remove the bandwidth should I do only the below: #NO service-policy <NAME HERE> interface outside
nothing else will be impacted?

[balaji.bandi]

is the ASA just simple config or any IPSec , what other function handling by ASA

how about you iperf test inside and outside of ASA only ?

what ASA code running ?

some troubleshoot tips :

https://www.cisco.com/c/en/us/support/docs/security/asa-5500-x-series-next-generation-firewalls/113393-asa-troubleshoot-throughput-00.html

[AirSail]

this ASA is used mainly as a VPN concentrator so we use a lot of IPSEC termination 

test inside the network is fine, 1000Gbps is all around 

test outside while keeping ASA behind us is fine we can hit the 1G 

iperf client in the inside and iperf server at the ISP side(outside) is showing a cap of 100mbps. 

code is ASA 9.12 

[balaji.bandi]

1000Gbps is all around  ?? 

1. you mentioned directly connected to ISP you getting right throughput.

2. connecting PC behind ASA you get low throughput.

3. how is your network diagram looks like - where is the PC inside connected to switch ? 

4 what i was suggesting test to confirm ASA to bottle neck here

PC----switch---ASA---switch---PC what is the outcome ?

5. hope you are not testing inside to remote vpn clients.

6. again if the device in production, what is the utilization of ASA outside or inside interface in the production ?

7. what is use case to test iperf now ?

8. as any remote users to transfer files to inside network, what is the outcome ?

9. last is how is ASA configured 9 show run will help here)

[AirSail]

@balaji.bandi  nice and tidy set of questions  

1. correct - direct to ISP - getting right values 

2/3. correct - connect PC DIRECTLY to ASA inside, getting values around 100mbps 

4. PC(iperfclient) --> ASA INSIDE --> edge SWITCH --> ISP Router --> ISP IPERF server    **> getting 100mbps

5. of course not

6. how I can determine that ? 

7/8. did get the question  

9. pretty hard to cleanup the config it super long  

[Sheraz.Salim]

Thats interesting issue. I would recommand you to takle this issue in this manner.

* disconnect inside interface and connect it to my PC (set an IP address LAN) I'm getting 100Mbps

here run the wireshark at start of the Ipref on your PC and once the Ipref test completed save the wireshark capture.

now when you connect your inside interface to ASA prior to this setup a capture on ASA inside interface with source Ipref server and destination your pc. once then off load it and share the both captures. I think most probably the TCP handshake and the streams are handled differently which cause you to see the different output. There could be a different number of reasons exmaple. service policy setting on ASA etc.