I had a nice Oracle DB crash recently on one of my environments with RHEL 7.1.  Do you wonder why?
Some OS updates had been installed by the OS admin and my DB got crazy.

See what had happened

ORA-27157: OS post/wait facility removed
ORA-27300: OS system dependent operation:semop failed with status: 43
ORA-27301: OS failure message: Identifier removed
ORA-27302: failure occurred at: sskgpwwait1 USER (ospid: 103559): terminating the instance due to error 27157
System state dump requested by (instance=2, osid=103559 (GTX0)),summary=[abnormal instance termination].

After some investigating I have found that note: https://docs.oracle.com/cd/E52668_01/E67200/html/section-t51_kcn_f5.html

If is configured for , interprocess communication (IPC) is terminated for a non-system user’s processes when that user logs out. This setting, which is intended for laptops, can cause software problems on server systems. For example, if the user is a database software owner such as for Oracle Database, this configuration can cause database installation to fail or database services to crash.
By default, Oracle Linux 7 Update 2 configures in to prevent systemd from terminating IPC. However, if you have touched this file before updating your system to Oracle Linux 7 Update 2, the update installs the new version of the file as and does not set in . To avoid database crashes, set in and run systemctl reboot to reboot the system. (Bug ID 22224874)
Oracle Support confirms:
”The default value for RemoveIPC in RHEL7.2 is yes”
Btw: further investigation by the OS Admininstrator led to conclusion that the security patch update for rpcbind (Rpcbind-0.2.0-33.el7_2.x86_64) installed some dependencies like Systemd-libs-219-19.el7.x86_64 and  the latter caused the problems.

 

Found some beautifully named notes on MOS:
  • ALERT: Setting RemoveIPC=yes on Redhat 7.2 Crashes ASM and Database Instances as Well as Any Application That Uses a Shared Memory Segment (SHM) or Semaphores (SEM) (Doc ID 2081410.1)
  • RHEL 7.2: The default RemoveIPC setting crashes asm, databases, and any application that uses a Shared Memory Segment (SHM) (Doc ID 2082270.1)
Conclusions:
  1. If updating RHEL or OEL 7.1 or configuring database on 7.2 onward be aware that you need to set RemoveIPC=no explicitly
  2. I wonder when Oracle is going to update their DB and GI installation guides,
  3. Remember the old truth: Test or Die

 

Any comments?

LinkedIn Auto Publish Powered By : XYZScripts.com