I had a nice Oracle DB crash recently on one of my environments with RHEL 7.1. Do you wonder why?
Some OS updates had been installed by the OS admin and my DB got crazy.
See what had happened
ORA-27157: OS post/wait facility removed
ORA-27300: OS system dependent operation:semop failed with status: 43
ORA-27301: OS failure message: Identifier removed
ORA-27302: failure occurred at: sskgpwwait1 USER (ospid: 103559): terminating the instance due to error 27157
System state dump requested by (instance=2, osid=103559 (GTX0)),summary=[abnormal instance termination].
After some investigating I have found that note: https://docs.oracle.com/cd/E52668_01/E67200/html/section-t51_kcn_f5.html
If is configured for , interprocess communication (IPC) is terminated for a non-system user’s processes when that user logs out. This setting, which is intended for laptops, can cause software problems on server systems. For example, if the user is a database software owner such as for Oracle Database, this configuration can cause database installation to fail or database services to crash.By default, Oracle Linux 7 Update 2 configures in to prevent systemd from terminating IPC. However, if you have touched this file before updating your system to Oracle Linux 7 Update 2, the update installs the new version of the file as and does not set in . To avoid database crashes, set in and run systemctl reboot to reboot the system. (Bug ID 22224874)
”The default value for RemoveIPC in RHEL7.2 is yes”
- ALERT: Setting RemoveIPC=yes on Redhat 7.2 Crashes ASM and Database Instances as Well as Any Application That Uses a Shared Memory Segment (SHM) or Semaphores (SEM) (Doc ID 2081410.1)
- RHEL 7.2: The default RemoveIPC setting crashes asm, databases, and any application that uses a Shared Memory Segment (SHM) (Doc ID 2082270.1)
- If updating RHEL or OEL 7.1 or configuring database on 7.2 onward be aware that you need to set RemoveIPC=no explicitly
- I wonder when Oracle is going to update their DB and GI installation guides,
- Remember the old truth: Test or Die