cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
1244
Views
5
Helpful
16
Replies

FRP/ASA high RAM usage

cisco.13
Level 1
Level 1

Hello,
I upgraded an standalone FRP 2130 ASA appliance mode from version 9.15(1)15 to 9.18(3). Used for remote vpn, everything is working correctly,..But I noticed after 4 days that the memory usage is higher (previously 20%, now +75%). and I see in our monitoring tool which is based on snmp the status is: UNKNOWN, Even though I can retrieve the information successfully :
$ snmpwalk -v 2c -c comm ip sysdesc
SNMPv2-MIB::sysDescr.0 = STRING: Cisco Adaptive Security Appliance Version 9.18(3)

Any ideas, please?

# sh memory
Free memory: 3305546696 bytes (23%)
Used memory: 11062033464 bytes (77%)
------------- ------------------
Total memory: 14367580160 bytes (100%)

Thank you

1 Accepted Solution

Accepted Solutions

Only this chunk type stands out, SNMP-related (SNMP logging):

Chunk type: 563, name: "NsConf log"
pool_type: 1 (MEMPOOL_GLOBAL_SHARED) flags: 00000401 ( DYNAMIC NO_LIMIT)
addr: 0x000000555d1d8f50, data start: 0x000000555d1dd010, end: 0x000000555d225010
next: 0x000000555d47c600, next_sibling: 0x000000fcfc7b9c10, prev_sibling: 0x0000000000000000
real_max: 2048, real_size: 144, first free index: 0
total_inuse: 70198585, total_free: 711, inuse_hwm: 70198587
alignment: 0, base alignment: 0
chunk size: 311507, chunk data size: 0, num chunks: 34277, total size: 2087590847
Per-chunk statistics: siblings created 35497, siblings trimmed 1221

So, it appears the issue is caused by SNMP. The leak is very fast, because you probably poll the device at very high rate. Do you?

There are two bugs which match: CSCwe65245 and internal CSCwf70825. Both were fixed in 9.18.4, but I would still upgrade to 9.18.4.8. Of course, you need to work with TAC to confirm my findings. And also check disk0:, if you have SNMP log files of big size: "dir /r".

HTH

 

View solution in original post

16 Replies 16

cisco.13
Level 1
Level 1

Hello @MHM Cisco World
Thanks for your feedback

I don't think it's related to this bug !
9.18(3) not mentioned, fixed in : https://bst.cisco.com/bugsearch/bug/CSCvx54562
+ I use ASA code not FTD.

Thank you.

This high memory usage unusual 

And I think 90% it relate to bug 

One more bug about high memory usage 

https://bst.cisco.com/bugsearch/bug/CSCwi20848?rfs=qvlogin

MHM

tvotna
Spotlight
Spotlight

If this is a leak, you will see something unusual in the "show memory top" output: memory allocation for blocks of the same size will stand out.

HTH

 

cisco.13
Level 1
Level 1

Hello @tvotna 
Thanks for your feedback
Attached "show memory top" output, can you please check?
Thank you

Google is best friend here 

Write in search your fpr ver and asa and high memory usage and check all bug you get.

There is nothing here you can do.

Believe me.

Goodluck friend 

MHM

@MHM Cisco World, picking up random bugs from Google search is not the best strategy when people ask for help. The fact that you don't know how to troubleshoot memory leak issues doesn't mean that others cannot do this.

@cisco.13, this indeed looks like a bug. From the output it's evident that almost all heap memory is allocated by the same function 0xaaaf81994c in 262144 byte blocks. It's obvious that the memory was allocated by chunk manager process, hence we need "show chunkstat" output. As the "show chunkstat" output is huge, write it to flash: "show chunkstat | redir disk0:/chunkstat.txt".

Also, please provide "show vpn-sessiondb" and "show counters".

There are few memory leak bugs which were fixed in 9.18.3, but it's impossible to identify an exact one, until "show chunkstat" is decoded. Most likely the issue is caused by VPN subsystem or SNMP. If memory utilization becomes critical, open TAC case and provide above outputs, or I can try to decode them myself if you don't have TAC support. In general, you'll probably need to upgrade to the latest 9.18 interim, which is 9.18.4.8, if I'm not mistaken.

 

 

 

cisco.13
Level 1
Level 1

Hello @tvotna
Thank you very much for your feedback and help.
Yes, I opened a TAC case, they are also looking.
Currently
Free memory: 1660771944 bytes (12%)
Used memory: 12706808216 bytes (88%)
------------- ------------------
Total memory: 14367580160 bytes (100%)

Here are the files
Thank you.

Only this chunk type stands out, SNMP-related (SNMP logging):

Chunk type: 563, name: "NsConf log"
pool_type: 1 (MEMPOOL_GLOBAL_SHARED) flags: 00000401 ( DYNAMIC NO_LIMIT)
addr: 0x000000555d1d8f50, data start: 0x000000555d1dd010, end: 0x000000555d225010
next: 0x000000555d47c600, next_sibling: 0x000000fcfc7b9c10, prev_sibling: 0x0000000000000000
real_max: 2048, real_size: 144, first free index: 0
total_inuse: 70198585, total_free: 711, inuse_hwm: 70198587
alignment: 0, base alignment: 0
chunk size: 311507, chunk data size: 0, num chunks: 34277, total size: 2087590847
Per-chunk statistics: siblings created 35497, siblings trimmed 1221

So, it appears the issue is caused by SNMP. The leak is very fast, because you probably poll the device at very high rate. Do you?

There are two bugs which match: CSCwe65245 and internal CSCwf70825. Both were fixed in 9.18.4, but I would still upgrade to 9.18.4.8. Of course, you need to work with TAC to confirm my findings. And also check disk0:, if you have SNMP log files of big size: "dir /r".

HTH

 

cisco.13
Level 1
Level 1

Hello @tvotna,
Thank you very much for your analysis, I will try v9.18.4.8
Possible to share a document how do you diagnose memory problems please (if possible)?
Thank you.

Perfect Q 

I am also interested what number indicates that SNMP is consume memory.????

MHM

@cisco.13, unfortunately, I don't have one. The commands "show memory", "show memory detail", "show memory top" and sometimes "show chunkstat" are the most helpful. If you upgrade, you'll find "show chunkstat top" helpful too. Then look for something unusual, e.g. abnormally high number of memory allocations by the same pc (program counter) or for the same chunk type. Collect stats few times and check if numbers increase. Then apply common sense. Disable suspected subsystems, e.g. SNMP, and check again.

If the leak is fast, it can only be caused by some function which is invoked frequently. You mentioned VPN and SNMP in the very beginning, but the number of VPN sessions is very small, so most likely we can rule out VPN. SNMP, on the other hand, is known to cause many issues as of 9.14, including memory leaks. It's unlikely that regular firewall functions, e.g. connections or NAT can cause leaks like this, simply because corresponding code is old and pretty stable. Of course, there is always a possibility that you configured something very specific on the box, which is rarely used and hence rarely tested by dev team. So it's worth to analyze config from this angle too.

There are few other methods, but they require decoding of memory addresses, which can only be done by TAC, as we don't have symbol files. Also, exact bug is always difficult to spot, because bug descriptions created by TAC are always poorly written.

@MHM Cisco World, "NsConf" is abbreviation for Netsnmp Configuration.

 

Friend let me answer you 
as I mention it is bug you need to check and it smart step you open TAC. 
now why suspect about chunk NsConf Log <<- not need to be SNMP issue
the memory is divide into room and there is specific room for each process 
the total_inuse room for this process is so high and there is so little free left 

total_inuse: 70198585, total_free: 711, inuse_hwm: 70198587

you mention you upgrade to 9.18(3) so you can check 9.18(4) and later ver. the bug solved (check each bug one by one)
each bug give you some detail you can use it to see if it relate to your issue or not.

sorry I can not help you so much here 
thanks a lot 
have a nice day
MHM
 

cisco.13
Level 1
Level 1

Hello @tvotna @MHM Cisco World, Thanks for your feedback
I imagine that TACs have tools to analyze memory leak problems, otherwise, it's complicated and takes a lot of time!
I tested the v9.18.4.8 (+9.18.4.x & 9.18.3.x), it has another bug : loss of 'smart license' after reboot, another similar case has been reported to TAC. only on FPR-2130, not on ASAv.
Thank you