Ever wondered all the precious memory installed on your server has gone? Among many other reasons, if you are running Solaris 10 and use ZFS file system then there may be your answer.
ZFS Adaptive Replacement Cache (ARC) tends to use up to 75% of the installed physical memory on servers with 4GB or less and upto everything except 1GB of memory on servers with more than 4GB of memory to cache data in a bid to improve performance.
This can significantly affect performance on mission critical servers running Databases etc.
To identify how much memory uses:
# kstat -m zfs | grep size data_size 18935877120 hdr_size 66041496 l2_hdr_size 0 l2_size 0 other_size 11310112 size 19013228728
Here “19013228728” (approx 18G) indicates the total memory used by ZFS.
Alternatively, the following mdb command show ZFS ARC usage:
# echo "::arc" | mdb -k|grep size size = 2048 MB hdr_size = 12493584 data_size = 2048608256 other_size = 86475456 l2_size = 0 l2_hdr_size = 0
It makes sense to cap the maximum ZFS ARC can use on servers where memory requirement for other services is more.
To set the maximum limit for ZFS ARC, edit /etc/system file and add the following line
where 2147483648 restricts the usage to a maximum of 2GB physical memory. Unfortunately, this requires a reboot for the setting to take effect and cannot be dynamically changed.
1 thought on “Missing memory in Solaris 10 with ZFS”
Below is the Server specifications:
SPARC Enterprise M4000 Server
Memory size: 65536 Megabytes
root@p # kstat -m zfs | grep size
root@p # echo “::arc” | mdb -k|grep size
size = 16340 MB
buf_size = 135 MB
data_size = 15189 MB
other_size = 1015 MB
l2_hdr_size = 0 MB
root@p # /usr/local/bin/top |grep Memory
Memory: 64G phys mem, 3603M free mem, 65G total swap, 65G free swap
root@p # cat /etc/system |grep zfs
Kindly advise if zfs:zfs_arc_max is ok for the current server usage and specifications ?