728x90

Informix 서비스가 온라인 상태에서 어떻게 백업이 수행되는지 조사해봤습니다. 오랜기간 Informix를 경험했으면서도 아직도 내부 아키텍처에 정확히 알지 못하고 있었네요.

 

Informix 데이터베이스가 동시에 여러 트랜잭션에 의해 변경되는 중에도 백업을 수행할 수 있는 이유는 Physical Log가 있기 때문입니다. 백업을 시작한 이후부터 데이터가 저장된 청크의 페이지에 변경이 일어나면, 변경된 데이터의 기존 이미지(before-images)를 Physical Log에 기록합니다. 그리고 다시 temp table에 before-images를 복사하여 최종적으로는 백업 미디어(tape)에 기록한다고 하네요. 아래는 그 내용을 도식화한 그림입니다.

 

Informix Dynamic Server Internal Architecture and Advanced Administration - On-Line Archives and Log Backups

백업이 시작된 이후부터는 데이터 청크의 변경된 페이지는 무시합니다. before image를 가지고 있고 최종적으로는 Logical Log로 복구하기 때문이겠죠? 그리고 tape은 연속적으로 기록되어야하므로, before image는 백업 수행 도중에 기록되는 것 같습니다.

 

백업 수행중에 데이터를 변경할 때, before image가 temp table로 생성되는지 간단한 테스트를 해봤습니다.

ontape 백업이 실행중인 상태에서, 데이터를 증식하여 입력해보니 temporary dbspace에 temp table이 생기는 것을 확인할 수 있었습니다.

$ cat q1.sql
insert into customer_copy select * from customer connect by level < 6;
insert into customer_copy select * from customer connect by level < 7;
insert into customer_copy select * from customer connect by level < 8;
$ nohup dbaccess stores_demo q1.sql &
$ oncheck -pe tempdbs1
DBspace Usage Report: tempdbs1            Owner: informix  Created: 07/03/2013
 Chunk Pathname                             Pagesize(k)  Size(p)  Used(p)  Free(p)
     3 /informix/dbspace/tempdbs1                     4   250000   131125   118875
 Description                                                   Offset(p)  Size(p)
 ------------------------------------------------------------- -------- --------
 RESERVED PAGES                                                       0        2
 CHUNK FREELIST PAGE                                                  2        1
 (0x300001) tempdbs1:'informix'.TBLSpace                              3       50
 (0x300002) stores_demo:'informix'._temptable                        53   131072
 FREE                                                            131125   118875

해당 temp table(_temptable)은 트랜잭션이 종료되면 oncheck 명령으로 보았을 때 사라지는 걸로 봐선 곧바로 백업 파일에 기록되는 것 같습니다. temp dbspace 공간이 부족해서 백업이 실패하는 경우도 간혹 있으니 참고하세요~

728x90
728x90

안녕하세요. IBM 커뮤니티에서 인포믹스의 통합 백업 암호화(Integrated Backup Encryption) 기능에 대한 질문이 있어서 소개하려고 합니다.

질문의 요지는 암호화된 백업 파일의 압축이 잘 되는가 하는 것인데, PGP로 암호화된 파일은 압축이 잘 안되었던 모양입니다. 그래서 몇가지 상황을 가정하고 백업 암호화 기능의 작동 방식을 테스트 해보았습니다.

 

인포믹스의 통합 백업 암호화 기능은 14.10 버전부터 제공됩니다.

매뉴얼에서는 통합 백업 암호화 기능을 사용한다면 원격지의 키서버를 통해 백업 암호화 키를 생성할 것을 권장하고 있으나 여기서는 편의상 로컬 암호화 키를 사용하겠습니다. 로컬 암호화 키를 생성하는 방법은 IBM Knowledge Center의 아래 링크를 참고했습니다.

https://www.ibm.com/support/knowledgecenter/SSGU8G_14.1.0/com.ibm.bar.doc/ids_bar_local_encryption_key.htm

-- 1. 통합 백업 암호화 기능을 설정하지 않고 ontape 백업 수행
[informix@db2 backups]$ onstat -c | egrep '^BACKUP_FILTER|^RESTORE_FILTER|^BAR_ENC'
BACKUP_FILTER
RESTORE_FILTER
BAR_ENCRYPTION
[informix@db2 backups]$ ontape -s -L 0
10 percent done.
20 percent done.
30 percent done.
40 percent done.
100 percent done.
File created: /work1/informix/ids1410fc3/backups/db2_0_L0
Please label this tape as number 1 in the arc tape sequence.
This tape contains the following logical logs:
 247
Program over.
[informix@db2 backups]$ ls -la
total 266788
drwxrwxr-x.  3 informix informix        33 Mar 31 09:54 .
drwxr-xr-x. 28 informix informix      4096 Mar 21 22:30 ..
-rw-rw----.  1 informix informix 273186816 Mar 31 09:54 db2_0_L0
drwxrwxr-x.  2 informix informix       182 Mar 31 09:51 tmp
-- 2. 통합 백업 암호화 기능을 설정하고 ontape 백업 수행
[informix@db2 backups]$ openssl rand -base64 24 > /work1/informix/ids1410fc3/etc/l_key192
[informix@db2 backups]$ onstat -c | egrep '^BACKUP_FILTER|^RESTORE_FILTER|^BAR_ENC'
BACKUP_FILTER
RESTORE_FILTER
BAR_ENCRYPTION keyfile=/work1/informix/ids1410fc3/etc/l_key192,cipher=aes192
[informix@db2 backups]$ ontape -s -L 0
The backup volume will be encrypted.
10 percent done.
20 percent done.
30 percent done.
40 percent done.
100 percent done.
File created: /work1/informix/ids1410fc3/backups/db2_0_L0
Please label this tape as number 1 in the arc tape sequence.
This tape contains the following logical logs:
 247
Program over.
[informix@db2 backups]$ ls -la
total 533572
drwxrwxr-x.  3 informix informix        65 Mar 31 09:55 .
drwxr-xr-x. 28 informix informix      4096 Mar 21 22:30 ..
-rw-rw----.  1 informix informix 273186816 Mar 31 09:54 db2_0_20200331_095432_L0
-rw-rw----.  1 informix informix 273186816 Mar 31 09:55 db2_0_L0
drwxrwxr-x.  2 informix informix       182 Mar 31 09:51 tmp
-- 3. 백업 파일 내용 확인
[informix@db2 backups]$ strings db2_0_20200331_095432_L0 | head -30
Archive Backup TapeIBM Informix Dynamic Server Version 14.10.FC3DETue Mar 31 09:54:32 2020informix/dev/pts/0
/work1/informix/ids1410fc3/backups/
rootdbs                                                                                                                         plog                                                                                                                            llog                                                                                                                            datadbs1                                                                                                                        datadbs2                                                                                                                        datadbs3                                                                                                                        data8dbs1                                                                                                                       data8dbs2                                                                                                                       data8dbs3                                                                                                                       sbspace1
IBM Informix Dynamic Server Copyright 2001, 2019  IBM Corporation
14.10.xC2
        F"01-2020
ROOTNAME rootdbs
ROOTPATH /work1/informix/ids1410fc3/storage/rootdbs
ROOTOFFSET 0
ROOTSIZE 157696
MIRROR 0
MIRRORPATH /work1/informix/ids1410fc3/tmp/demo_on.root_mirror
MIRROROFFSET 0
DBSERVERNAME ol_informix1410
SERVERNUM 0
MSGPATH /work1/informix/ids1410fc3/ol_informix1410.log
TAPEDEV /work1/informix/ids1410fc3/backups
TAPESIZE 0
TAPEBLK 32
LTAPEDEV /dev/null
LTAPESIZE 0
LTAPEBLK 32
PHYSFILE 65430
PHYSBUFF 512
LOGFILES 20
LOGSIZE 6144
LOGBUFF 256
DYNAMIC_LOGS 2
LTXHWM 70
LTXEHWM 80
[informix@db2 backups]$ strings db2_0_L0 | head -30
Archive Backup TapeIBM Informix Dynamic Server Version 14.10.FC3DETue Mar 31 09:55:07 2020informix/dev/pts/0
/work1/informix/ids1410fc3/backups/
aes192
DQZQ
QX9p
a5b(
H%*g
wecI
)rO0"
z\$b
#_t;
azXX
aW`g
$9s~f
{Rus
T<.u
(;*&
[!wS
{'dT
g9c%
k#|~
G)9)1
>X2B
CnSq
]SJ~
kmt9
ol!n
>JMsy
b!9~`
[C4R
-- 4. 통합 백업 암호화 기능 및 BACKUP_FILTER, RESTORE_FILTER를 설정하고 ontape 백업 수행
[informix@db2 backups]$ vi $INFORMIXDIR/etc/$ONCONFIG
[informix@db2 backups]$ onstat -c | egrep '^BACKUP_FILTER|^RESTORE_FILTER|^BAR_ENC'
BACKUP_FILTER /bin/gzip
RESTORE_FILTER /bin/gunzip
BAR_ENCRYPTION keyfile=/work1/informix/ids1410fc3/etc/l_key192,cipher=aes192
[informix@db2 backups]$ ontape -s -L 0
The backup volume will be encrypted.
Using the backup and restore filter /bin/gzip.
10 percent done.
20 percent done.
30 percent done.
40 percent done.
100 percent done.
File created: /work1/informix/ids1410fc3/backups/db2_0_L0
Please label this tape as number 1 in the arc tape sequence.
This tape contains the following logical logs:
 247
Program over.
[informix@db2 backups]$ ls -la
total 548100
drwxrwxr-x.  3 informix informix        97 Mar 31 09:56 .
drwxr-xr-x. 28 informix informix      4096 Mar 21 22:30 ..
-rw-rw----.  1 informix informix 273186816 Mar 31 09:54 db2_0_20200331_095432_L0
-rw-rw----.  1 informix informix 273186816 Mar 31 09:55 db2_0_20200331_095507_L0
-rw-rw----.  1 informix informix  14876672 Mar 31 09:56 db2_0_L0
drwxrwxr-x.  2 informix informix       182 Mar 31 09:51 tmp
-- 5. 통합 백업 암호화 기능 및 BACKUP_FILTER, RESTORE_FILTER를 설정하고 ontape 리스토어 수행
[informix@db2 backups]$ onmode -ky
[informix@db2 backups]$ onstat -c | egrep '^BACKUP_FILTER|^RESTORE_FILTER|^BAR_ENC'
BACKUP_FILTER /bin/gzip
RESTORE_FILTER /bin/gunzip
BAR_ENCRYPTION keyfile=/work1/informix/ids1410fc3/etc/l_key192,cipher=aes192
[informix@db2 backups]$ ontape -r
Restore will use level 0 archive file /work1/informix/ids1410fc3/backups/db2_0_L0. Press Return to continue ...
The volume to restore is encrypted.
Using the backup and restore filter /bin/gunzip.
Archive Tape Information
Tape type:      Archive Backup Tape
Online version: IBM Informix Dynamic Server Version 14.10.FC3DE
Archive date:   Tue Mar 31 09:56:06 2020
User id:        informix
Terminal id:    /dev/pts/0
Archive level:  0
Tape device:    /work1/informix/ids1410fc3/backups/
Tape blocksize (in k): 32
Tape size (in k): system defined for directory
Tape number in series: 1
Backup filter:  /bin/gzip
Spaces to restore:1 [rootdbs                                                                                                                         ]
2 [plog                                                                                                                            ]
3 [llog                                                                                                                            ]
4 [datadbs1                                                                                                                        ]
5 [datadbs2                                                                                                                        ]
6 [datadbs3                                                                                                                        ]
7 [data8dbs1                                                                                                                       ]
8 [data8dbs2                                                                                                                       ]
9 [data8dbs3                                                                                                                       ]
10 [sbspace1                                                                                                                        ]
Archive Information
IBM Informix Dynamic Server Copyright 2001, 2019  IBM Corporation
Initialization Time       01/08/2020 22:26:54
System Page Size          2048
Version                   31
Index Page Logging        OFF
Archive CheckPoint Time   03/31/2020 09:56:05
Dbspaces
number   flags    fchunk   nchunks  flags    owner                            name
1        10000001 1        2        N   AE   informix                         rootdbs                                                                                                        
2        11000001 2        1        N P AE   informix                         plog                                                                                                           
3        10000001 3        1        N   AE   informix                         llog                                                                                                           
4        10000001 4        1        N   AE   informix                         datadbs1                                                                                                       
5        10000001 5        1        N   AE   informix                         datadbs2                                                                                                       
6        10000001 6        1        N   AE   informix                         datadbs3                                                                                                       
7        10000001 7        1        N   AE   informix                         data8dbs1                                                                                                      
8        10000001 8        1        N   AE   informix                         data8dbs2                                                                                                      
9        10000001 9        1        N   AE   informix                         data8dbs3                                                                                                      
10       10002001 10       1        N T AE   informix                         tmpdbspace                                                                                                     
11       10008001 11       1        N S AE   informix                         sbspace1                                                                                                       
12       1000a001 12       1        N U AE   informix                         tmpsbspace                                                                                                     
Chunks
chk/dbs offset   size     free     bpages   flags pathname
1   1   0        78848    9236              PO--- /work1/informix/ids1410fc3/storage/rootdbs
2   2   0        32768    0                 PO--E /work1/informix/ids1410fc3/storage/ol_informix1410_2_plog_p_1
3   3   0        70244    4791              PO--E /work1/informix/ids1410fc3/storage/ol_informix1410_2_llog_p_1
4   4   0        32768    21236             PO--E /work1/informix/ids1410fc3/storage/ol_informix1410_2_datadbs1_p_1
5   5   0        32768    32715             PO--E /work1/informix/ids1410fc3/storage/ol_informix1410_2_datadbs2_p_1
6   6   0        32768    32715             PO--E /work1/informix/ids1410fc3/storage/ol_informix1410_2_datadbs3_p_1
7   7   0        32768    32556             PO--E /work1/informix/ids1410fc3/storage/ol_informix1410_2_data8dbs1_p_1
8   8   0        32768    32556             PO--E /work1/informix/ids1410fc3/storage/ol_informix1410_2_data8dbs2_p_1
9   9   0        32768    32556             PO--E /work1/informix/ids1410fc3/storage/ol_informix1410_2_data8dbs3_p_1
10  10  0        429812   429376            PO--E /work1/informix/ids1410fc3/storage/ol_informix1410_2_tmpdbspace_p_1
11  11  0        32768    8466              POS-- /work1/informix/ids1410fc3/storage/ol_informix1410_2_sbspace1_p_1
12  12  0        32768    8466              POS-- /work1/informix/ids1410fc3/storage/ol_informix1410_2_tmpsbspace_p_1
13  1   0        60314    72                PO--E /work1/informix/ids1410fc3/storage/ol_informix1410_rootdbs_p_1
Continue restore? (y/n)y
Do you want to back up the logs? (y/n)n
The volume to restore is encrypted.
Using the backup and restore filter /bin/gunzip.
Warning: Parameter's user-configured value was adjusted. (DS_MAX_SCANS)
Warning: Parameter's user-configured value was adjusted. (ONLIDX_MAXMEM)
Restore a level 1 archive (y/n) n
Do you want to restore log tapes? (y/n)n
/work1/informix/ids1410fc3/bin/onmode -sy
Program over.
[informix@db2 backups]$ onstat -m
IBM Informix Dynamic Server Version 14.10.FC3DE -- Quiescent -- Up 00:00:26 -- 566920 Kbytes
Message Log File: /work1/informix/ids1410fc3/ol_informix1410.log
10:11:58  Physical Recovery Complete: 0 Pages Examined, 0 Pages Restored.
10:11:58  Clearing encrypted primary chunk 10 before initialization...
10:11:58  Clearing encrypted primary chunk 12 before initialization...
10:11:58  Logical Recovery Started.
10:11:58  72 recovery worker threads will be started.
10:12:00  Logical Recovery has reached the transaction cleanup phase.
10:12:00  Logical Recovery Complete.
          0 Committed, 0 Rolled Back, 0 Open, 0 Bad Locks
10:12:01  Bringing system to Quiescent Mode with no Logical Restore.
10:12:02  Quiescent Mode
10:12:02  Checkpoint Completed:  duration was 0 seconds.
10:12:02  Tue Mar 31 - loguniq 247, logpos 0x1ee018, timestamp: 0xeb6d094 Interval: 264
10:12:02  Maximum server connections 0
10:12:02  Checkpoint Statistics - Avg. Txn Block Time 0.000, # Txns blocked 0, Plog used 0, Llog used 1
10:12:04  Defragmenter cleaner thread now running
10:12:04  Defragmenter cleaner thread cleaned:0 partitions

테스트 결과로 보면 암호화된 백업본이라도 압축이 잘 되는 것을 확인할 수 있습니다.

 

참고 사이트 :

https://www.ibm.com/support/knowledgecenter/SSGU8G_14.1.0/com.ibm.bar.doc/ids_bar_integrated_encryption.htm

 

728x90
728x90

iiug.org에서 onbar에서 사용되는 XBSA API에 대한 설명을 발췌하였습니다. 

onbar 명령으로 백업을 수행하면 onbar 프로세스는 백업 솔루션의 프로세스와 통신하게 되는데, 그 과정에서 아래의 API들을 사용하게 됩니다.

onbar로 백업을 하고 계신 곳에서는 백업 로그에서 아래의 일부 API 이름을 보셨을 겁니다.


Onbar can communicate with multiple storage managers. The reason we can pull this off is because recently a standard api has been developed so that different vendors can communicate with each other. This standard/language is know as XBSA. Below are the XBSA calls that onbar uses in order to backup and restore dbspaces to/from the online engine. The functions below are the complete set of XBSA call. It is mearly the calls that onbar uses.


  • BSABeginTxn()- starts an XBSA API transaction.
  • BSACreateObject()- Creates an XBSA object. This basically tells that storage manager to make room for a new backup. The storage manager passes back the copyid (inside the objectdescriptor) that is the unique identifier that onbar uses when it need to get that back/object.
  • BSAEndData()-Tells the storage manager that there is no more data to send. (we are finished sending data associated with that particular object)
  • BSAEndTxn()- Tell the storage manager that we are finished with a backup transaction.
  • BSAGetData()- This function returns a single buffer of data from that storage manager. We pass it the copyid (in the object descriptor) and the storage manager is responsible for finding the requested data.
  • BSAGetObject()- Called to gather information about a particular storage manager object (ie, date created , size, owner etc).
  • BSAInit()- connects to the storage manager and starts a session. This function will return a 'handle' that is a unique identifier associated to that session.
  • BSAQueryApiVersion()- Called to validate which version of the XBSA libraries onbar is talking to.
  • BSASendData()- Sends a buffer full of data to the storage manager.
  • BSATerminate()- Terminates a session with the storage manager. It also frees up and resources that were allocated in the BSAinit call.


Below is a simplistic diagram on how an XBSA application goes. onbar (which is an XBSA applicalion in it self) follows this flow chart as well when creating/restoring archives. onbar will loop calling BSASendData (and BSAGetDATA) until the is no more data to process.




출처 : http://www.iiug.org/onbar/xbsa.html

728x90
728x90

인포믹스 데이터 파일 내용을 조작하여 백업 테스트를 수행해보았습니다.


## 테스트 환경

1. 테스트서버 : AIX 6.1

2. DBSPACE 현황 (청크개수) : rootdbs (1), datadbs (3)

3. onstat -d 결과

IBM Informix Dynamic Server Version 7.31.UD9     -- Quiescent -- Up 00:29:45 -- 18112 Kbytes


Dbspaces

address  number   flags    fchunk   nchunks  flags    owner    name

4005a150 1        1        1        1        N        informix rootdbs

4005a888 2        1        2        1        N        informix plogdbs

4005a948 3        1        3        1        N        informix llogdbs

4005aa08 4        1        4        3        N        informix datadbs

 4 active, 2047 maximum


Chunks

address  chk/dbs offset   size     free     bpages   flags pathname

4005a210 1   1   0        5000     3009              PO-   /work2/INFORMIX/ids731/dbspaces/rootdbs

4005a388 2   2   0        250000   249947            PO-   /work2/INFORMIX/ids731/dbspaces/plogdbs

4005a488 3   3   0        250000   249707            PO-   /work2/INFORMIX/ids731/dbspaces/llogdbs

4005a588 4   4   0        250000   0                 PO-   /work2/INFORMIX/ids731/dbspaces/datadbs

4005a688 5   4   0        25000    0                 PO-   /work2/INFORMIX/ids731/dbspaces/datadbs1

4005a788 6   4   0        25000    0                 PO-   /work2/INFORMIX/ids731/dbspaces/datadbs2

 6 active, 2047 maximum


 

 

## 테스트 절차

1. 데이터 파일 조작 (6 청크 파일)

dd if=/dev/random seek=10 bs=1000 count=10 of=/work2/INFORMIX/ids731/dbspaces/datadbs2 conv=notrunc

 

2. ontape 백업

ontape -s -L 0

 

3. 백업 online.log 파일 내용 확인

17:37:39  Assert Failed: Archive detects that page 0x600003 is corrupt.

17:37:39  IBM Informix Dynamic Server Version 7.31.UD9

17:37:39   Who: Session(23, informix@pilma01, 23134372, 1074292452)

                Thread(60, arcbackup1, 4005e334, 1)

                File: rsarcbu.c Line: 2600

17:37:39  stack trace for pid 23396402 written to /tmp/af.424e052

17:37:39   See Also: /tmp/af.424e052

17:37:39  Archive on rootdbs, datadbs Completed with 2 corrupted pages detected.


백업중 발생한 오류 메시지에서 나타난 페이지 번호(0x600003)는 DBSPace 번호가 아닌 Chunk 번호를 의미하는 것으로 보입니다.

728x90
728x90

To start ISM on UNIX for the first time, issue the ism_startup -init command. The following shell script is an example of a simple setup for making on–Bar storage-space and logical-log backups to disk files:


#!/usr/bin/ksh

# setup for ISM storage manager

INFORMIXDIR=/data/informix

export INFORMIXDIR


echo "ISM setup begins"


# the user must edit these two lines to select

# appropriate disk locations for storage manager use.

SM_DISKDEV1=/backups/dbspaces

SM_DISKDEV2=/backups/logfiles

export SM_DISKDEV1

export SM_DISKDEV2


# start ISM server

echo "initalize ISM server, must be done as root"

$INFORMIXDIR/bin/ism_startup -init


$INFORMIXDIR/bin/ism_add -admin informix@server (server=hostname)


Important: To make informix the ISM user and thereby ensure informix has access to conduct ISM operations during daily routines, you must run the following command after any ism_startup -init or ISM installation:


# create some devices and media

# the ISM user, defined in ism_add -admin, can do these tasks

echo "create and mount ISM devices and pools"

ism_add -device $SM_DISKDEV1 -type file

ism_add -device $SM_DISKDEV2 -type file

ism_op -label $SM_DISKDEV1 -pool ISMDiskData -volume ISMData

ism_op -label $SM_DISKDEV2 -pool ISMDiskLogs -volume ISMLogs

ism_op -mount $SM_DISKDEV1

ism_op -mount $SM_DISKDEV2

echo "end of ISM setup"



Set ISM environment variables and onCONFIG parameters

If you use ISM, you can specify the volume pool names for storage spaces and logical logs in the ISM_DATA_POOL and ISM_LOG_POOL parameters in the onconfig file. The ISM_DATA_POOL configuration parameter specifies the volumepool that you use for backing up storage spaces. The ISM_LOG_POOL configuration parameter specifies the volume pool that you use for backing up
logical logs.

e.g.
ISM_DATA_POOL ISMDiskData
ISM_LOG_POOL ISMDiskLogs

If you do not set these configuration parameters, they default to the volume pool names ISMData and then ISMLogs.


LTAPEDEV parameter
If you specify a tape device in the LTAPEDEV configuration parameter, ON-Bar ignores the value. on-Bar also ignores the value of the LTAPEBLK, LTAPESIZE, TAPEDEV, TAPEBLK, and TAPESIZE parameters. Consider leaving these parameter values blank when you use on-Bar because the storage manager sets these values.

Important: Set LTAPEDEV to /dev/null or leave it blank on UNIX or NUL on Windows only if you do not want to back up the logical logs. The on-Bar activity log shows a warning and return code 152. Because the database server marks the logical logs as backed up when they are no longer current, on-Bar cannot find logical logs to back up. All transactions in those logs are lost, and you are not able to restore them.

728x90

+ Recent posts