Problem(Abstract)
IBM® Informix® Server™ version11.70.FC5W1 and higher will produce a warning message in the online.log and to STDERR, when you have RESIDENT enabled in the onCONFIG file and either KAIO or DIRECT_IO enabled. This warning is only displayed for Informix when running on AIX® and is not displayed in earlier versions.
Symptom
An example of the warning as displayed in the online.log:
- 11:17:09 WARNING: Shared-memory residency is enabled while direct I/O and KAIO (kernel asynchronous I/O) are active.
This configuration could lead to runtime KAIO errors, which might shut down the instance or set chunks offline.
Set the RESIDENT configuration parameter to 0 to turn off shared-memory residency, and then restart the database server. Refer to the machine notes and APAR IC76872 for more information.
Cause
APAR "IC82623 : on AIX, WE SHOULD CHECK FOR THE SETTING OF RESIDENT BEFORE BRINGING THE INSTANCE onLINE"
On AIX, setting RESIDENT to -1 may lead to the instance running into an AIX related APAR:
- APAR IC76872 : AIX: HARD TO DIAGNOSE KAIO ERRORS 22 (EINVAL) WHEN SYSTEM RUNNING LOW on PINNABLE MEMORY PAGES.
On AIX systems with a lot of allocated pinnable ("resident") memory and KAIO or DIRECT_IO being used by Informix, it is possible that KAIO read or write calls fail with errno 22 (EINVAL), potentially leading to down dbspaces or system aborts.
Sample message log error:
04:30:40 KAIO: error in kaio_WRITE, kaiocbp = 0x22b620d0, errno= 22
04:30:40 fildes = 258 (gfd 3), buf = 0x700000122b64000, nbytes= 4096, offset = 130785280
The reason for these EINVAL errors usually is the OS running low on 'pinnable' memory pages (by default 80% of the available RAM). This can be caused by Informix having a lot of shared memory segments allocated as "resident" plus pinned OS kernel memory plus KAIO resources.
This solution is included within Informix Server 11.70.FC6 and the Post Interim Drop 11.70.FC5W1.
Resolving the problem
Changes are required to the onCONFIG -> RESIDENT parameter. For reference, here are the comments from an onconfig.std file.
###################################################################
# Shared Memory Configuration Parameters
###################################################################
# RESIDENT - Controls whether shared memory is resident.
# Acceptable values are:
# 0 off (default)
# 1 lock the resident segment only
# n lock the resident segment and the next n-1
# virtual segments, where n < 100
# -1 lock all resident and virtual segments
Recommendation :
- Turn RESIDENT off (i.e. 0). Optionally set the AIX parameters recommended by AIX Technical Support (below)
or - Set RESIDENT to a positive integer so that only the initial memory segment(s) are pinned and that amount of memory does not exceed approximately 80% of the physical memory of the machine.
Q: Will there be a performance penalty?
A: AIX Technical Support assures us (the Informix team) that the performance penalty for not using RESIDENT in a machine dedicated to the database server is negligible. There are some AIX parameters that can be set to simulate the behavior of RESIDENT while using DIRECT_IO or Kernel AIO without incurring the problems that RESIDENT can cause.
Q: Tell me more about the AIX parameters that can be set to simulate the behavior of RESIDENT while using DIRECT_IO or Kernel AIO without incurring the problems that RESIDENT can cause.
A: Use larger memory page sizes. For example, a 64K page size can be enabled for the stack, data and text pages of oninit processes using this command when starting the Informix Server:LDR_CNTRL=DATAPSIZE=64K@STACKPSIZE=64K@TEXTPSIZE=64K@SHMPSIZE=64K oninit
Q: How does LDR_CNTRL compare to IFX_LARGE_PAGES?
A: First, they are two separate settings and therefore can be used independently. The current view is that LDR_CNTRL will give a small to negligible performance gain in addition to IFX_LARGE_PAGES. For some customers, since IFX_LARGE_PAGES is used with 16MB pages it may be that a LDR_CNTRL of 64K yields no discernible performance improvement.
The OS should always have sufficient pinned memory available for its own operations. For example, if there is insufficient memory errno 22 EINVAL may occur with KAIO operations (APAR IC76872 / 228003).
It is not safe to set RESIDENT to -1 no matter how much memory a system has as it can eventually be consumed and Informix corruption may result.
The amount of pinned memory currently in use may be displayed with the command svmon -G.
Example of setting pinned memory via the operating system.
- Run svmon -G and look at "L" row within the "PageSize / PoolSize" section of the output.
- export IFX_LARGE_PAGES=1
- vmo -p -o lgpg_regions=3200 -o lgpg_size=16777216
- vmo -p -o v_pinshm=1
- Start the engine.
You may wish to consult with AIX Technical Support for further information regarding use of the vmo command on your system.
http://www-01.ibm.com/support/docview.wss?uid=swg21608334