Oracle ACFS File System: Inode Limitations Explained

INTRO

Recently, we encountered an issue of running out of inodes on a directory mounted from the ACFS file system after a large-sized table export operation took place, which triggered an alert email from Exadata.

What is ACFS in short?

ACFS (Oracle Automatic Storage Management Cluster File System) is a file system designed for Oracle Real Application Clusters (RAC) and Oracle ASM (Automatic Storage Management) environments and it is a feature of Oracle Grid Infrastructure. It can be used in any Oracle ASM environment, whether on Exadata or other platforms, to manage both database and general-purpose files. It offers management of both database files and non-database files in a single system.

What are Inodes?

Inodes are data structures on a file system that store information about files, such as their size, permissions and location on disk. Every file and directory on the file system requires one inode. If there are many small files, the system can run out of inodes even if there is sufficient free space on the file system. To resolve this issue, unused files or directories should be removed. I was confident that only a small number of files were located in the ACFS filesystem , so we should not be encountering this issue.

This is the email received from the Exadata alert notification system. Since the content is not well-formatted, it is a bit misleading.

Description : The following file systems are running out of inodes with over 80% inodes used:
Filesystem	Inodes allocated	Inodes used	Inodes available	Percentage used
/u02		104857600		88438086	16419514		85.00

When I checked the actual inode counts using the following command, it showed that the inodes were exhausted. I also verified the file and folder counts, and there weren’t many. How is this possible?

[oracle@exa01 daily]$ df /u02 -i
Filesystem			Inodes 		IUsed		IFree     IUse% 	Mounted on
/dev/asm/volume_acfs-308	104857600	88438086        16419514  85.00		/u02


[oracle@exa01 daily]$ find /u02 -xdev -printf '%h\n' | uniq -c | sort -nr | head
  20021 /u02/sth/backup/x/y
     90 /u02/sth/backup/asm_metadata
     77 /u02/sth/backup/z
     17 /u02/sth/backup/k

[oracle@exa01 daily]$ df -hm /u02
Filesystem			1M-blocks 	Used	  Available       Use% 	   Mounted on
/dev/asm/volume_acfs-308	409600		345461    64139 	  85%	   /u02

When I did a quick search, I found an interesting document: Does ACFS Use Inode Architecture? (Doc ID 2026700.1). According to the document, ACFS filesystems do not use the inode architecture and do not have a pre-allocated inode table. Therefore, the ‘df -i’ command returns the number of inodes that are theoretically possible, given the remaining space.

In ACFS, Oracle assumes that one inode is used for every 4KB of space.

Since 345,461 * 1024 = 353,752,064 KB is used, this means 88,438,086 inodes are used. This summarizes the issue we have encountered.

Hope it helps.