10/01/2008 ---à the original copy.
08/26/2009 ---à the updated copy --- 1st revision.
10/05/2009 ---à the updated copy --- 2nd revision.
01/22/2010 …® Updates –MAG --- 4th revision.
01/26/2010… ® Updates ……………5th revision
1. If you do not receive a latest copy of EMC STD-BCV pair document, please contact Nuri, Tamara(NGIT) or Vera, Gil (AITC) to request for the ${DATE}_vbaperf_refresh_lun.xls file that should include the latest EMC new STD- for you to complete a refresh.
2. Verify STD-BCV device pair configuration file on vbaperf, which is the /vbaperf/refresh/EMC/FULL_RESTORE_CFG. If there is a new one received, compare it with the existing one on the system, and resolve the difference with EMC admins. See the step one. If you have any concerns or questions about the new STD-BCV pair information, you should contact either Nuri, Tamara(NGIT) or Vera, Gil (AITC). Tamara’s number is 512-438-9210.
3. When Tamara’s file is received sort both the FULL_RESTORE_CFG and the file Tamara provides. In Tamara’s file make sure the fields between the BCV’s and STD’s are separated with one space. After sorting both files do a diff on the two files. Any pair that appears is Tamara’s file should be added to the FULL_RESTORE_CFG file at the bottom.
4. Login as root, type #df –k /, /var and /vbaperf on the command line, then save the outputs in a file for later use. Using Wordpad on your desktop is a good method.
5. Set up environment variables.
(Note that new scripts are put under /vbaperf/refresh/new)
# . /vbaperf/refresh/new/refresh.env
#-- set up the script env
6. Archive files.
a. First of all, visually verify the archived restore list.
#more $VPR_CONF/tar_restore.lst.
#-- contains files and directories that will be restored after bcv-std device restore
#more $VPR_CONF/tar_restore_parms.lst.
#-- contains files and their variables definitions for vbaperf environment.
b. Then run:
#cd $VPR_SCRIPTS
#tar_them.ksh restore
1). #-- archive/tar files in $VPR_CONF/tar_restore.lst.
2). #Updated tar_them.ksh to put /usr/ucb/whoami.
#collect_cfg.bash
#-- collection cfg files of vbaperf. Please note: this step will take about 15 to 20
minutes. So be patient.
The following are the retired steps:
(
to be retired steps:
verify lists:
/vbaperf/refresh/config/ tar_include
run:
# cd /
# /vbaperf/refresh/refresh_collect.ksh
)
7. Archive crontab files under /var/spool/cron/crontabs into a tar ball and store it under the current archived directory. The archive directory is /vbaperf/home/s101dxg/haha/re/work/
8. Send email messages to inform current login users, and request them to log off the system/vbaperf. Only execute this step if there are active users on the system. There shouldn’t be anyone logged on at this time.
9. Execute /vbaperf/refresh/EMC/stop_rc.ksh to bring Oracle, Tuxedo and Patrol application processes down before you bring the system down to {OK} prompt.
10. Login onto the vbaperf console via 10.204.7.30>SC0:
11. On SC0> Type console –d m on the command line. Log in as root directly. If you see the root password is expired, then the system will force you to change it. Once you give root a new password, you will be able to login. But please change the root password back to the one people know.
11. Once you log in as root, bring the systems down to {OK} prompt, and then boot the
system from a local device. This is accomplished by issuing an #init 0 from an SSH
window and monitoring the console output. This step usually takes about 6 – 8 minutes.
12. {OK} devalias
#to verify all the device aliases including the local boot device.
13. {OK}boot svm-boot
#this will boot the vbaperf from the local device. Please note: this step will take
about 15 to 20 minutes. So be patient.
14. FYI - Once the system comes back, you can perform the BCV full restore.
15. FYI - Since the massive messages display on the console, it is better to logon to vbaperf from ssh session through vbadev which can also avoid any network disruptions between your desktops and vbaperf server.
16. Commented out a line of #bcv_status >& in the check_procs.sh script, which can
save some time for completion of a FULL restore.
17. Perform the following steps in this order.
a. # cd /vbaperf/refresh/EMC.
b. # cp -p run_rc.ksh run_rc.ksh.bkup
c. # > run_rc.ksh
#-- disable rc3 scripts called in the next script.
d. # ./bcv_mgmt.pl -o restore -f –g FULL_RESTORE_CFG
#Make sure that the FULL_RESTORE_CFG is the latest and
updated. This step will perform a full restore bcv to std devices. Alternately use restore.ksh after modifying the file to do a FULL restore. This will generate a log file and can be monitored by executing #tail –f (LOG FILE)
Note: The –f flag is mandatory!
18. Issue one of the following command to check the status of FULL restore.
#symmir –f FULL_RESTORE_CFG –sid 0411 query or
#symmir –f FULL_RESTORE_CFG –sid 0411 query -bg - or –
#symmir -sid 0411 -f FULL_RESTORE_CFG query -i 5 | more
19. When the output from the command line above shows the FULL restore completed,
you can proceed with the next steps. The full restore for 5 TB might take 4 to 5
hours. The hours and data bytes to be copied will show with the
# symmir -sid 0411 -f FULL_RESTORE_CFG query -i 5|more
20. Mount the new EMC root device (note that Live Upgrade may change emc root
device). Before you do step a and b, verify the root / and /var device information
from a file made on the step 3, which is the output of “df –k” command.
a. # mount /dev/dsk/c4t5006048ACCAB32F2d2s0 /mnt
#-- mount new root from emc
b. # mount /dev/dsk/c4t5006048ACCAB32F2d2s3 /mnt/var
#-- mount new /var from emc
Note: Pay attention to the root and var devices. Refer back to the df –k command that was run and saved earlier in this document.
c. /vbaperf, /, /var should be mounted at this point.
21. Inspect production vfstab and vbaperf vfstab and resolve the differences.
a. # cd /vbaperf/refresh/new
b. # . ./refresh.env
#-- set up script env
c. # diff /mnt/etc/vfstab (PROD) $VPR_WORK/vfstab (PERF Original)
#-- compare prod vfstab and perf vfstab
At this point, the /mnt/etc/vfstab is the new production one, and the $VPR_WORK/vfstab is the original perf one. But if the $VPR_WORK/vfstab does not exist, then you need to make a copy of the /etc/vfstab file on the vbaperf to $VPR_WORK/vfstab-vbaperf.
Do a #diff vbaprod vbaperf manually and make a file called vfstab-perf.minimal. This file contans differences between vbaperf-vfstab.
d. #Once you visually checked the differences between vbaprod and vbaperf, manually make a file called vfstab-perf.minimal.
e. The $VPR_WORK/vfstab-perf.minimal file contains only minimal mount points which are root /, /var and /vbaperf. Please note: The /vbaperf is also important for you to access directories, files or scripts for the late tasks.
f. Restore necessary files as follows:
1). #cp -p /mnt/etc/vfstab /mnt/etc/vfstab.orig_prod
#make a copy of the production one.
2). #cp -p $VPR_WORK/vfstab-perf.minimal /mnt/etc/vfstab
#After this step, only “/”, “/var” and “/vbaperf” will be mounted when
the server gets rebooted.
3). #cd $VPR_SCRIPTS
4). #./post_bcv_restore.pl. Note: In this script the devices for /, and /var may need to be modified to the correct device.
Please note: This step will restore the files archived in step 5), they
are only OS configuration files this time.
5). If this step has to be happened in the next day after the archived files
were completed in previous steps, then you have to make a directory
with a name of current date. After that, please copy the restore.tar.gz
into the new directory that was just made.
6). #./runsvc.ksh move.
#disable rc scripts by cut and past cmds: “mv * __*”
g. Reboot vbaperf from emc boot device. But first of all, please verify boot-device and devalias, then reboot the systems.
h. # eeprom
# verify boot devices including local, EMC, etc if any more.
i. # reboot “dmx2-boot” –r.
j. #At this point the system will be booted from the EMC boot
device. The boot device alias is “dmx2-boot”.
22. At this point, there were errors regarding “Disk device not found” for db1,
etc. These disk errors or issues have to be resolved before proceeding with next
steps. Refer to Appendix A & B.
23. Mount the rest of file systems, and restore other files.
1). Uncomment the entries in /etc/vfstab file to mount the rest of
filesystems, then restore the other vbaperf OS directories, files and
other critical aspects that were archived from original vbaperf.
2). Copy all the entries under the line of #VBA MOUNT POINTS from
vfstab.orig_prod file which is the vfstab file from the current
production system. Then pasted them onto vfstab file under /etc on
vbaperf.
3). Make sure all other necessary entries above the #VBA MOUNT
POINTS line should be there in vfstab file under /etc directory.
These entries are for vbaperf system only, which are different from
the ones on vbaprod. Make sure that all the mount points for
database on the local disks on VBAPERF are valid in vfstab file before
it gets rebooted.
4). # . /vbaperf/refresh/new/refresh.env.
5). #cd $VPR_SCRIPTS.
6). # ./post_mountall_restore.pl
This step restore files archived in step 5), they are
non OS files including Tuxedo, Oracle and Patrol file,
etc. Check the source and make sure / and /var have the correct device files saved earlier from the df command.
24. Post refresh checks.
1). #chk_tar_restore.ksh.
#-- compare all files that are restore in steps 21) and 23) including
Timestamps and contents.
2). #chk_tar_restore_parms.pl
#-- check variable definition of the restored files against
$VPR_CONF/tar_restore_parms.lst. this script exit with an “ERROR
COUNT = 0”.
3). # ipfstat –nio.
#-- inspect IP filter config. Compare the output with
$VPR_WORK/ipfstat-nio.out. You need to do the following three steps.
a). #svcs –a | grep –i ipfilter. This shows that
ipfilter is disabled.
b). #svcadm enable ipfilter.
c). #ipfstat –nio to enable necessary ipfilter rules
on firewall.
4). # ./runsvc.ksh move
#enable rc scripts by cut and past “mv __* *”
5). #cd /vbaperf/refresh/EMC
6). #cp run_rc.ksh.bkup run_rc.ksh
#-- enable RC scripts in bcv_mgmt.pl script.
i. Verify the sudoers file is correct
ii. Copy cron tar ball to /var/spool/cron/crontabs and untar the file. #tar xvf (Tar file)
25. After the refresh is completed without any issues, please perform a FULL SYNC from STD to BCV using FULL_RESTRORE_CFG file on VBAPERF. You can skip the –f option to avoid full DB sync. This will give you a full backup for everything on the BCV. (See Appendix C for more details)
26. Updated /vbaperf/refresh/EMC/filesystems.txt file based on the new mount
points in vfstab on vbaperf if there were new VBA MOUNT POINTS added on
vbaprod before the refresh. Or else, to run bcv_mgmt.pl script will fail
somewhere in the middle of the process.
27. Remove all the crontabs entries under /var/spool/cron/crontabs. Then untar the tar ball that was made from step 6 into /var/spool/cron/crontabs directory.
28. Recreate two file systems for /export/home/datastage and /devl/vnetbatch on VBAPERF after each refresh. These two file systems are for data staging project, and have to be residing on EMC. Once this is completed, update both FULL_RESTROE_CFG and ALL_EXCEPT_OS_CFG files without the information of /export/home/datastage and /devl/vnetbatch for datastage. This will prevent from overwriting the data of datastage project when a DB restore is performed.
29. After the refresh, you need to perform additional tasks.
a. umount the /vbadata/cnpltrs which is from vbaprod. #umount /vbadata/cnpltrs
b. #mount /dev/dsk/c1t2d0s3 /vbadata/cnpltrs that is on local devices on vbaperf.
c. Vi /etc/vfstab file on vbaperf with the following line enabled.
/dev/dsk/c1t2d0s3 /dev/rdsk/c1t2d0s3 /vbadata/cnpltrs ufs 1 yes -
d. Disable the following line in /etc/vfstab file:
#/dev/vx/dsk/sys2/vvcnpltrs /dev/vx/rdsk/sys2/vvcnpltrs /vbadata/cnpltrs ufs 3 yes largefiles,suid.
e. Inform webLogic team to umount and mount the /AdobeDoc on three webLogic application servers.
f. Verify the accesses to the /AdobeDoc directory on 3 webLogic servers.
g. Verify NFS client and server are running after reboot.
#svcs –a |grep –I nfs
h. Remove STD-BCV pair from step #3 from FULL_RESTORE_CFG & ALL_EXCEPT_OS_CFG
i. Get the sizes of /export/home/datastage and /devl/vnetbatch from John Prescott.
j. Using VEA or command line Veritas do the following:
a. Find free disks as large or larger than requirements in above step (i). #vxdisk list
b. If necessary remove the disk device from the disk group
# /usr/lib/vxvm/bin/vxevac -g
c. Create a new Disk Group with the disk devices allocated in above step. #vxdiskadm – or – # vxassist -g
see Appendix A for further help.
d. Create a Volume of designated size, add to vfstab and have the file system automount and FSCK automatically with 1 pass
# vxassist -g
e. Edit Vfstab to enable FSCK, and automount
30. On the VBAPERF, shut down all the print services by typing #lpshut.
31. Release vbaperf for more manual post refresh checks (David & Vikas).
Appendix A
During the process of booting the system, if a warning similar to the following is displayed, ignore the warning.
WARNING: The following files in / differ from the boot archive: changed file-name
The system will enter system maintenance mode.
Clear the boot-archive service by typing the following command:
# svcadm clear system/boot-archive
After this command is run, the bootadm update-archive command runs silently. If the boot archive is updated successfully, the system is rebooted.
Verify the service is running.
# svcs boot-archiveSTATE STIME FMRIonline 9:02:38 svc:/system/boot-archive:default
fsck [ -F fstype] [-V] [-yY] [-o options] special
-F fstype type of file system to be repaired ( ufs , vxfs etc)
-V verify the command line syntax but do not run the command
-y or -Y Run the command in non interactive mode – repair all errors encountered without waiting for user response.
-o options Three options can be specified with -o flag
b=n where n is the number of next super block if primary super block is corrupted in a file system .
p option used to make safe repair options during the booting process.
f force the file system check regardless of its clean flag.
special – Block or character device name of the file system to be checked/repaired – for example /dev/rdsk/cXtXdX0sX .Character device should be used for consistencies check & repair
fsck phases
fsck checks the file system in a series of 5 pages and checks a specific functionality of file system in each phase.
** phase 1 – Check Blocks and Sizes
** phase 2 – Check Pathnames
** phase 3 – Check Connectivity
** phase 4 – Check Reference Counts
** phase 5 – Check Cylinder Groups
fsck error messages & Corrective action :
1. Corrupted superblock – fsck fails to run
If the superblock is corrupted the file system still can be repaired using alternate superblock which are formed while making new file system .
the first alternate superblock number is 32 and others superblock numbers can be found using the following command :
newfs -N
for example to run fsck using first alternate superblock following command is used
fsck -F ufs -o b=32
PowerPath
HBA Info /etc/powermt display
Disk Info /etc/powermt display dev=all
Rebuild /kernel/drv/emcp.conf /etc/powercf -q
Reconfigure powerpath using emcp.conf /etc/powermt config
Save the configuration /etc/powermt save
Enable and Disable HBA cards used for testing /etc/powermt display (get card ID) /etc/powermt disable hba=3072/etc/powermt enable hba=3072
symmir -sid 0411 -f FULL_RESTORE_CFG que -i 5|more
symmir -sid 0411 -f FULL_RESTORE_CFG split -nop
Veritas Volume Manager Command Line Examples
Volume Manager CLI examples:
---------------------------
display disk listings:
# vxdisk list
# /usr/lib/vxvm/bin/vxevac -g testdg testdg01 testdg02
#/etc/vx/bin/vxdisksetup -i
display volume manager object listings
# vxprint -ht
display free space in a disk group
# vxdg -g
list all volume manager tasks currently running on the system
# vxtask list
add a disk to Volume Manager (devicename = cXtXdX) (prompt driven)
# vxdiskadd
designate/remove a disk as a hot-relocation spare
# vxedit set spare=on
# vxedit set spare=off
rename a disk
# vxedit rename
reserve/unreserve a disk (space won't be allocated unless specifically mentioned in vxassist)
# vxedit set reserve=on
# vxedit set reserve=off
take a disk offline (first remove the disk from its disk group) (devicename=cXtXdXs2)
#vxdisk offline
remove a disk
(first, stop any applications associated with the volume,
unmount the volume, stop the volume,
if you need the data on the disk, move the volumes to another disk or back up volume)
--removing the disk from its disk group:
# vxdg -g
--remove the disk from volume manager control (devicename=cXtXdXs2)
# vxdisk rm
display multipath information
# vxdisk list
create a disk group
(see 'add a disk to volume manager - if the group does not exist, you will be prompted)
upgrade a disk group
--list disk group version
# vxdg list
--upgrade disk group to highest version currently running
# vxdg upgrade
destroy a disk group
# vxdg destroy
display disk group information
# vxdg list
# vxdg list
move a disk group
(stop any applications associated with all volumes in the disk group,
unmount and stop all volumes in the disk group:
--deport (disable local access) the disk group to be moved on first system
# vxdg deport
--import (enable local access) the disk group and its disks from the second system
# vxdg import
--start all volumes in the disk group on the second system
# vxrecover -g
or # vxvol -g
estimating maximum volume size (any_valid_type = raid5, stripe, mirror)
# vxassist -g
create a concatenated volume (length examples = 15g, 15m)
# vxassist -g
create a striped volume (length examples = 15g, 15m)
# vxassist -g
create a raid5 volume (without logging) (length examples = 15g, 15m)
# vxassist -g
create a raid5 volume (with logging) (length examples = 15g, 15m)
# vxassist -g
create a mirrored volume (without DRL) (length examples = 15g, 15m)
# vxassist -g
create a mirrored volume (with DRL) (length examples = 15g, 15m)
# vxassist -g
mirror an existing volume
# vxassist mirror
mirror all volumes within a disk group
# vxmirror -g
mirror the root (boot) disk
EEPROM variable "use-nvramrc?" must be set to true
# vxrootmir -v
remove a mirror (use 'vxprint -g
# vxplex -o rm dis
add a log to an existing volume
# vxassist addlog
remove a log from an existing volume
# vxassist remove log
create a raid 0+1 volume (without DRL)
# vxassist -g
nmirror=# nstripe=#
create a raid 1+0 volume (without DRL)
# vxassist -g
nmirror=# nstripe=#
resize a volume
# vxassist -g
# vxassist -g
# vxassist -g
# vxassist -g
estimate how much a volume can grow
# vxassist -g
remove a volume
(stop all applications associated with the volume,
unmount volume (remove /etc/vfstab entry))
--stop volume
# vxvol stop
-- remove volume (for raid 1+0, use 'rf-rm')
# vxedit -r rm
or # vxassist remove volume
change the volume read policy
# vxvol rdpol
# vxvol rdpol prefer
change volume attributes
# vxedit set
resize a filesystem (ufs cannot be shrunk, only grown)
# vxresize -g
# vxresize -g
join subdisks (must be in the order of offset on disks)
# vxsd join
»
create a striped volume (length examples = 15g, 15m)
The syntax is incorrect for:
create a striped volume (length examples = 15g, 15m)
# vxassist -g make layout=stripe \
It should be something like this:
# vxassist -g testdg make testdgvol1 285503488 layout=stripe alloc=disk1,disk2,disk3,disk4
· This brings the credibility of the rest of this document into question; You have not specified
Appendix B
Troubleshooting
Disk Device Not Found
a. Mount the root partition on the local drive. #mount /dev/dsk/c0t8d0s0 /mnt.
b. Make a copy of /etc/powermt.custom and /kernel/drv/emcp.conf files.
c. Copy /mnt/etc/powermt.custom file into /etc and copy /mnt/ kernel/drv/emcp.conf into /kernel/drv/emcp.conf.
d. Umount /mnt.
e. Perform a #reboot “dmx2-boot” –r. When the system comes back into a single user mode, the errors went away. You can bring the system up to muti-user mode without the errors. But when you do a reboot vbaperf with an init 6, the same errors will display again.
f. Go to /dev/dsk and /dev/rdsk to remove all the emcp* links.
g. Then you can perform a #reboot “dmx2-boot” with a –r. You will not see the same errors.
h. Perform another reboot with an init 6 to bring VBAPERF cleanly without the same errors, such as “Disk device not found” for db1, etc.
i. Make sure all the basic services are started up successfully.
Power Path / Boot Issues
b. To rebuild the powerpath configuration file boot to single user mode #boot –s
c. #/etc/powercf –q
d. #powermt config
e. #powermt save
f. The commands above will rebuild /kernel/drv/emcp.conf
g. You also need to clear the boot archive with #svcadm clear system/boot-archive
Rebuild missing device files
a. If the device files do not rebuild or id / (root) needs to be fsck’d perform the following steps.
b. Mount vbaperf / to /mnt
c. #devfsadm –Cv –r /mnt
d. #umount /mnt
e. #fsck –F ufs –o f –y (block root device path saved earlier)
f. #fsck –F ufs –o f –y (character root device path saved earlier)
g. #sync;sync;sync
h. #init 0
i. Reboot as follows: boot dmx2-boot –r from ok> prompt
Appendix C
Useful tips for this refresh process.
a. #./bcv_mgmt.pl -o vximport if all the Veritas disk groups are deported, but the bcv_mgmt.pl script exits with some issues. You can run this to re-import disk groups and use mountall command to mount all the file systems specified in /etc/vfstab file.
b. These are the TUX important steps you need to check before you bring
/tuxtpa/prod/tptocics.env
CICSHOSTNAME=SYP
###CICSIPADDR=10.224.131.33
CICSSENDNAME=cicssendtt TT is for vbaperf. Make sure they are not PP, or else it will go directly to Tux production DB.
AUSTIN_DB_IND=C C is for vbaperf.
CICS_TIMEOUT=60
CICS_DEBUG_IND=N
EMAIL_ERROR_LIMIT=2
c. #./bcv_mgmt.pl -g FULL_RESTORE_CFG -o sync_split
#./bcv_mgmt.pl –f -g FULL_RESTORE_CFG -o sync_split to
sync and then split them(SDT/BCV).
d. After the sym commands get executed, there a re logs that
you can check under /var/symapi/log for troubleshooting any
sym issues.
File Name:
Refresh Procedures Revision 5-01222010.doc
Last Updated By:
MAG
0 comments:
Post a Comment