pajama

The underlaying problem is that the default settings on AIX for the LPA mapping config file are

such that only root can read the file. Crypt() function in IDS is run on a CPU VP which is non-root and AIX requires it to be run as root if using LPA, a workaround is to use PAM (which will use MSC VP for authentication )

Use PAM to configure LPA on AIX

Following example instructions to set it: 

1) Add to /etc/pam.conf: 
idslogin auth required pam_aix 
idslogin account required pam_aix 

2) Define DBSERVERALIAS ids_pam_srv 

3) Add to SQLHOSTS: 
ids_pam_srv onsoctcp <host> <port> 
s=4,pam_serv=idslogin,pamauth=password 

Don't forget to add an appropriate entry to /etc/services (if needed). 

4) check /etc/security/login.cfg for: 
auth_type = STD_AUTH 
pwd_algorithm = sblowfish 


5) and final IDS check: 

bash-3.2$ dbaccess - - 
> connect to "test@ids_pam_srv" user "tester"; 
ENTER PASSWORD: 

Connected.

AIX has made the /etc/security/pwdalg.cfg file unreadable to normal users (users other than root), hence -attempting to authenticate as user informix does not work.

You can use PAM to get around this problem, but if that is not an immediate option and re-writing applications to handle PAM can not be completed immediately - you can use the following work-around:

Warning: You need to ensure you are in a secure environment to use this work-around - i.e. your Informix Server machine is secure and you really trust user 'informix'. If this is true, then you can proceed.

Add user 'informix' to the 'security' group on AIX - this will allow the /etc/security/pwdalg.cfg to be read by user informix and authentication should work without throwing an error.

Note - using PAM is the preferred method for handling this situation, so use this as a temporary work-around only while taking steps to properly remedy the issue.

In baract.log file you'd see XBSA errors like the following:

XBSA Error: (BSACreateObject) A system error occurred. Aborting XBSA session.

XBSA Error: (BSAEndTxn) The transaction was aborted.

(-43296) WARNING: BAR_TIMEOUT Storage Manager Progress may be stalled.

The current backup would be aborted after such error (and possibly retried depending on BAR_RETRY setting).

This has been identified as an AIX product defect under APAR IV22133

IBM Informix onbar on AIX. The problem might be more likely on AIX lpars using virtual network adapters.

When hitting such XBSA errors, determine whether AIX OS is affected by APAR IV22133 or its siblings.

Apply latest AIX fixpacks.

An example of the warning as displayed in the online.log:

11:17:09  WARNING: Shared-memory residency is enabled while direct I/O and KAIO (kernel asynchronous I/O) are active.

This configuration could lead to runtime KAIO errors, which might shut down the instance or set chunks offline.

Set the RESIDENT configuration parameter to 0 to turn off shared-memory residency, and then restart the database server. Refer to the machine notes and APAR IC76872 for more information.

APAR "IC82623 : on AIX, WE SHOULD CHECK FOR THE SETTING OF RESIDENT BEFORE BRINGING THE INSTANCE onLINE"

On AIX, setting RESIDENT to -1 may lead to the instance running into an AIX related APAR:

APAR IC76872 : AIX: HARD TO DIAGNOSE KAIO ERRORS 22 (EINVAL) WHEN SYSTEM RUNNING LOW on PINNABLE MEMORY PAGES. 

On AIX systems with a lot of allocated pinnable ("resident") memory and KAIO or DIRECT_IO being used by Informix, it is possible that KAIO read or write calls fail with errno 22 (EINVAL), potentially leading to down dbspaces or system aborts. 

Sample message log error: 

04:30:40  KAIO: error in kaio_WRITE, kaiocbp = 0x22b620d0, errno= 22
04:30:40  fildes = 258 (gfd 3), buf = 0x700000122b64000, nbytes= 4096, offset = 130785280 

The reason for these EINVAL errors usually is the OS running low on 'pinnable' memory pages (by default 80% of the available RAM). This can be caused by Informix having a lot of shared memory segments allocated as "resident" plus pinned OS kernel memory plus KAIO resources. 

Changes are required to the onCONFIG -> RESIDENT parameter. For reference, here are the comments from an onconfig.std file.

###################################################################
# Shared Memory Configuration Parameters
###################################################################
# RESIDENT         - Controls whether shared memory is resident.
#                    Acceptable values are:
#                    0 off (default)
#                    1 lock the resident segment only
#                    n lock the resident segment and the next n-1
#                       virtual segments, where n < 100
#                    -1 lock all resident and virtual segments 

1. Turn RESIDENT off (i.e. 0). Optionally set the AIX parameters recommended by AIX Technical Support (below)
   or
2. Set RESIDENT to a positive integer so that only the initial memory segment(s) are pinned and that amount of memory does not exceed approximately 80% of the physical memory of the machine.

Q: Will there be a performance penalty?
A: AIX Technical Support assures us (the Informix team) that the performance penalty for not using RESIDENT in a machine dedicated to the database server is negligible. There are some AIX parameters that can be set to simulate the behavior of RESIDENT while using DIRECT_IO or Kernel AIO without incurring the problems that RESIDENT can cause.

Q: Tell me more about the AIX parameters that can be set to simulate the behavior of RESIDENT while using DIRECT_IO or Kernel AIO without incurring the problems that RESIDENT can cause.
A: Use larger memory page sizes. For example, a 64K page size can be enabled for the stack, data and text pages of oninit processes using this command when starting the Informix Server:LDR_CNTRL=DATAPSIZE=64K@STACKPSIZE=64K@TEXTPSIZE=64K@SHMPSIZE=64K oninit

Q: How does LDR_CNTRL compare to IFX_LARGE_PAGES?
A: First, they are two separate settings and therefore can be used independently. The current view is that LDR_CNTRL will give a small to negligible performance gain in addition to IFX_LARGE_PAGES. For some customers, since IFX_LARGE_PAGES is used with 16MB pages it may be that a LDR_CNTRL of 64K yields no discernible performance improvement.
The OS should always have sufficient pinned memory available for its own operations. For example, if there is insufficient memory errno 22 EINVAL may occur with KAIO operations (APAR IC76872 / 228003).
It is not safe to set RESIDENT to -1 no matter how much memory a system has as it can eventually be consumed and Informix corruption may result.

The amount of pinned memory currently in use may be displayed with the command svmon -G.

Example of setting pinned memory via the operating system.

• Run svmon -G and look at "L" row within the "PageSize / PoolSize" section of the output.
• export IFX_LARGE_PAGES=1
• vmo -p -o lgpg_regions=3200 -o lgpg_size=16777216
• vmo -p -o v_pinshm=1
• Start the engine.
  

PROBLEM

When using Kernel Asynchronous I/O (KAIO) with IBM Informix Dynamic Servers, you might experience the following message in the database server message log:

KAIO out of resource errno=11
or
KAIO out of resource errno=35

Note: Check Related information for more detail. 

• All configurable kernel parameters must be specified using an integer value or a formula consisting of a valid integer ｅxpression.

• The maximum and/or default values of certain parameters may change between releases or to vary between 32-bit and 64-bit processors.

• Before altering the value of any configurable kernel parameter, be sure you know the implications of making the change.
• Never set any system parameter to a value outside the allowable range for that parameter (SAM refuses to store values outside of the allowable range).
• Many parameters interact, and their values must be selected in a balanced way.

PROBLEM

Database performance is poor because of input/output (I/O) requests. The IBM® Informix® Dynamic Server™ message log might display one of these error messages:

KAIO out of OS resources, errno = 11

or   

KAIO out of OS resources, errno = 35

Limits of the  IFMX_HPKAIO_NUM_REQ  environment variable.
Minimum Value	10
Maximum Value	5000
Default Value	1000

Note: These values may change in future versions of the operating system or database server.

Example:

This example shows how to increase the number of concurrent KAIO requests to 2300 from the default of 1000 using Korn shell:

$ IFMX_HPKAIO_NUM_REQ=2300 
$ export IFMX_HPKAIO_NUM_REQ

• output of oninit -v:
  
  Checking group membership to determine server run mode...succeeded
  Reading configuration file '/opt/informix/etc/onconfig.test'...succeeded
  Creating /INFORMIXTMP/.infxdirs...succeeded
  Checking config parameters...succeeded
  Allocating and attaching to shared memory...FAILED
  
• messages in online.log:
  
  21:45:30 IBM Informix Dynamic Server Started.
  21:45:30 size of resident + virtual segments 10443875920 + 4831838208 > 9663676416
  total allowed by configuration parameter SHMTOTAL

The size of the resident and virtual portions of shared memory exceed the shared memory limit set by the onCONFIG parameter SHMTOTAL.

Resident portion of shared memory 
The resident portion of the database server shared memory stores the following data structures that do not change in size while the database server is running:

• Shared-memory header
• Buffer pool
• Logical-log buffer
• Physical-log buffer
• Lock table
  

• Internal tables
• Big buffers
• Session data
• Thread data (stacks and heaps)
• Data-distribution cache
• Dictionary cache
• SPL routine cache
• SQL statement cache
• Sorting pool
• Global pool
  

Estimating the Size of the Resident Portion of Shared Memory 
Estimating the Size of the Virtual Portion of Shared Memory

• Increase the value of the onCONFIG parameter SHMTOTAL if possible.
• Decrease the value of one or more of these onCONFIG parameters:
  
  - BUFFERPOOL ( the buffers value )
  - LOCKS
  - SHMVIRTSIZE

Once the ping timeout is written to the online.log file of the Primary/Secondary instance (see the sample output below), user sessions return to normal work.

11:27:57  DR: ping timeout                                             
11:27:57  DR: Receive error                                            
11:27:57  ASF Echo-Thread Server: asfcode = -25582: oserr = 4: errstr =
: Network connection is broken.                                        
11:27:57  DR_ERR set to -1                                             
11:27:59  DR: Turned off on primary server    

When data replication is established, primary and secondary regularly exchange ping messages. If the ping acknowledge is not received by the time when DRTIMEOUT is elapsed, a server re-sends ping message three more times and then reports ping timeout and turns off the DR subsytem. From this, the time span between first ping and the "DR: ping timeout" message can be as large as (DRTIMEOUT x 4).

For example, if DRTIMEOUT is set to be 180 second, it will take 12 minutes before DR is turned off.

Scenario #1:
Although with asynchronous replication transactions do not wait for acknowledgement from HDR secondary after the logical log record was put in DR buffer, when there is a transmission failure, the DR buffer may fill up pretty quickly (the time required for that depends on DRTIMEOUT value, LOGBUFF value and the activity that the instance is having). Until DR is not turned off, a user session has to wait until DR buffer has enough space for the logical log record.

Scenario #2:
In addition to the above scenario, a checkpoint can be requested on Primary between the first ping failure and the time when the "DR: ping timeout" message is reported. The checkpoints are synchronous between Primary and Secondary regardless of the DRINTERVAL value. once checkpoint is requested, it will prevent any threads from entering the critical section. The instance will remain blocked until checkpoint acknowledgment is received from the Secondary or until DR is turned off.

For scenario #1 check if the corresponding user thread demonstrates a stack similar to the following:

Stack for thread: 73 sqlexec                  
base: 0x0700000011abc000                      
len: 69632                                    
pc: 0x00000001000370f4                        
tos: 0x0700000011acafe0                       
state: sleeping                               
vp: 8                                         
                                              
0x00000001000370f4 (oninit)yield_processor_mvp
0x0000000100041f30 (oninit)mt_yield           
0x000000010076a5ac (oninit)cdrTimerWait
0x0000000100716908 (oninit)dr_buf_deq_int
0x00000001001fe3c0 (oninit)dr_logcopy         
0x00000001001f2d0c (oninit)logwrite
0x000000010011b7c4 (oninit)log_put            
0x0000000100121384 (oninit)logm_write         
0x00000001001f3e68 (oninit)logputx            
0x000000010017137c (oninit)rscommit           
0x000000010022b70c (oninit)iscommit           
0x00000001002865e4 (oninit)sqiscommit         
0x0000000100533d38 (oninit)committx           
0x0000000100536480 (oninit)commitcmd          
0x000000010053b01c (oninit)excommand          
0x000000010042893c (oninit)sq_execute         
0x000000010026becc (oninit)sqmain             
0x00000001002d51a4 (oninit)listen_verify      
0x00000001002d33b8 (oninit)spawn_thread       
0x0000000100e0b59c (oninit)startup       

For scenario #2 check the 'onstat -g ath' output and see if the user threads are having "cond wait cp" status.

To resolve the problem it may be required to:

1) Fix any problems that can cause data transmission issues between Primary and HDR Secondary (e.g. increase network reliability and throughput)

2) Decrease the value of DRTIMEOUT configuration parameter.

Note: increasing the LOGBUFF may also help to reduce the blockage time, however having a large logical log buffer may result in data loss in case of the Primary failure.

ping timeout,received error,send error

ping timeout will occur if "DR_MSG_PING" can't flow between primary and secondary,or ack duration exceed 4*DRTIMEOUT."ping timeout" is a message type in DR BUFFER QUEUE,so it require waiting for dr buffer space,therefor PING TIMEOUT maybe occur due to dr buffer is full or its priority is too lower than logical log buffer.

Logical log buffer can't be transfer maybe lead to the 'ping timeout'.

HDR environment

DR BUFFER size is same as logical log buffer,and "DR_MSG_PING" save in dr buffer,so we can configure LOGBUFF to adjust DR BUFFER.

Primary server send logical log to Secondary server to keep consistent data as following description.
1.primary : logical log buffer -> dr buffer
2.primary : dr_prsend thread send these logical log to dr buffer on secondary server across network using TCP/IP.
3.secondary:dr_secrecv thread received those logical log in secondary.

HDR primary server and secondary server will ping each other and must waiting for a acknowledgment during a appointed times ,otherwise HDR relation will be broken due to "ping timeout" error .The ack duration is 4 times as DRTIMEOUT value.

To avoid "ping timeout" occur according to following mention.

1.Increasing LOGBUFF value to adjust dr buffer size and lay more signal message.
2.Secondary server hang maybe lead to the "ping timeout" due to dr buffer can't be received immediately or DR_MSG_PING lower priority.
3.Long checkpoint duration in secondary server.

Error message: "FAILED: Fetch statement failed: Encoding or code set not supported. "

error -79783

JDBC version 3.50.JC1 introduced the following APAR:

IC49877 - 
JDBC DRIVER ALLOWS INSERTION OF INVALID CHARACTERS FOR CHARACTER SET. 
http://www-01.ibm.com/support/docview.wss?uid=swg1IC49877 

While the behavior for this APAR is correct, it might lead to problems for those customers who use Native Multibyte data on the application side but store them into en_us locale on the server side. 

In some case, for customer who are using zh_tw.big5 locale on the server side; but stored illegal characters in the table will also be effected.

Error message happened right after upgrade.

Since version JDBC 3.50.JC5, user can set a flag IFX_USE_STRENC to switch to old style of encoding.

Here's an example on how to use it: 
"jdbc:informix-sqli://inst:port:dbname:informixserver=XXX;user=informix;password=XX;DB_LOCALE=en_us.819;IFX_USE_STRENC=true;"