ORA-27123: Unable to Attach Shared Memory - Fix Permissions

ORA-27123: Unable to Attach Shared Memory Segment

Error Overview

Error Text: ORA-27123: unable to attach to shared memory segment

ORA-27123 is a startup-phase error that occurs after Oracle has successfully created a shared memory segment (unlike ORA-27125, which occurs at creation time) but cannot attach it to the process address space. The OS system call shmat() has returned an error. This is almost always a permissions, resource limit, or address space exhaustion problem rather than a size problem.

ORA-27123 is commonly accompanied by a secondary OS error code. Capturing that secondary error (from the alert log or trace file) is the single most important diagnostic step.

Common Causes

1. IPC Permission Mismatch

• The shared memory segment was created by a different OS user (e.g., root or a different oracle account)
• The oracle OS user is not a member of the dba or oinstall group
• File system permissions on /dev/shm restrict access to the oracle user
• SELinux or AppArmor policy denies the shmat() call without logging a visible error

2. Oracle User Group Membership

• oracle not in the oinstall (OSDBA install group) or dba group
• Group membership change requires a new login session to take effect
• /etc/group and LDAP/NIS group databases are out of sync
• The vm.hugetlb_shm_group sysctl does not match the oracle user’s primary group

3. /dev/shm Sizing for AMM

• When Automatic Memory Management (AMM) is in use, Oracle maps SGA through /dev/shm
• If /dev/shm is smaller than MEMORY_TARGET, the attach fails after creation
• This is a common cause on systems where the default /dev/shm size is 50% of RAM and the DBA set MEMORY_TARGET above that threshold
• Other processes (shared memory databases, Java apps) may have filled /dev/shm

4. Process Virtual Address Space Limits

• The oracle background process virtual memory limit (ulimit -v) is too small
• On 32-bit systems, the 4GB virtual address space can be exhausted
• HugePages segments require contiguous virtual address space
• mmap() failures appear as ORA-27123 in some kernel versions

5. Semaphore Resource Exhaustion

• Although ORA-27140 is the primary semaphore error, a semaphore failure during the attach sequence can surface as ORA-27123
• If semmni (max semaphore sets) is exhausted, the IPC attach sequence fails mid-way

6. Orphaned Shared Memory from a Previous Crash

• After a crash or kill -9 on the instance, the shared memory segment persists
• On restart, Oracle finds an existing segment that it cannot attach to (wrong key or permissions)
• Orphaned segments must be removed manually before the instance can restart cleanly

Diagnostic Queries

Check Oracle Instance Memory Configuration

-- Verify AMM vs manual memory management
SELECT name, value
FROM v$parameter
WHERE name IN ('memory_target', 'memory_max_target', 'sga_target', 'use_large_pages')
ORDER BY name;

-- Verify actual SGA components
SELECT name,
       ROUND(bytes/1024/1024, 2) AS mb,
       resizeable
FROM v$sgainfo
ORDER BY bytes DESC;

-- Check background process memory
SELECT
  p.pid,
  p.spid,
  p.program,
  ROUND(p.pga_used_mem/1024/1024, 2) AS pga_used_mb
FROM v$process p
WHERE p.background = 1
ORDER BY p.pga_used_mem DESC
FETCH FIRST 10 ROWS ONLY;

Inspect IPC Shared Memory Segments

# List all shared memory segments with ownership and permissions
ipcs -m

# Verbose listing with size, permissions, key, and attached processes
ipcs -m -p

# Show totals and limits
ipcs -lm

# Find segments belonging to oracle OS user
ipcs -m | awk 'NR>2 { print }' | while read line; do
  shmid=$(echo $line | awk '{print $2}')
  owner=$(ipcs -m -i $shmid 2>/dev/null | grep owner | awk '{print $2}')
  echo "SHMID=$shmid OWNER=$owner LINE=$line"
done

# Check /dev/shm contents and sizes
ls -lah /dev/shm/
df -h /dev/shm

Diagnose Permissions and Group Membership

# Check oracle OS user identity and groups
id oracle

# Verify oracle is in dba and oinstall groups
getent group dba
getent group oinstall

# Check /dev/shm permissions
ls -la /dev/
stat /dev/shm

# Check SELinux status and denials
getenforce
ausearch -m avc -ts recent 2>/dev/null | grep shm | tail -20

# Check AppArmor (Ubuntu/Debian)
aa-status 2>/dev/null

Examine Alert Log and Trace Files

# Find the alert log
ALERT_LOG="$ORACLE_BASE/diag/rdbms/$ORACLE_SID/$ORACLE_SID/trace/alert_$ORACLE_SID.log"
tail -100 $ALERT_LOG

# Search for the secondary OS error code (errno) — this is critical
grep -A3 "ORA-27123" $ALERT_LOG | tail -30

# Find the most recent startup trace
ls -lt $ORACLE_BASE/diag/rdbms/$ORACLE_SID/$ORACLE_SID/trace/*.trc | head -5

# Common secondary errors:
#   Linux-X86_64 Error: 13: Permission denied  -> permissions problem
#   Linux-X86_64 Error: 12: Cannot allocate memory -> /dev/shm full
#   Linux-X86_64 Error: 22: Invalid argument  -> segment key mismatch (orphaned segment)

Step-by-Step Resolution

Step 1: Read the Secondary Error Code

# The secondary error is printed immediately after ORA-27123 in the alert log
# Example:
#   ORA-27123: unable to attach to shared memory segment
#   Linux-X86_64 Error: 13: Permission denied
#
# Error 13 (EACCES)  -> Go to Step 2: Fix Permissions
# Error 12 (ENOMEM)  -> Go to Step 3: Fix /dev/shm size
# Error 22 (EINVAL)  -> Go to Step 4: Clean orphaned segments
# Error  1 (EPERM)   -> Go to Step 2: Fix Permissions (likely SELinux)

grep -A5 "ORA-27123" $ORACLE_BASE/diag/rdbms/$ORACLE_SID/$ORACLE_SID/trace/alert_$ORACLE_SID.log | tail -20

Step 2: Fix IPC Permissions and Group Membership

# Verify oracle's groups
id oracle

# Add oracle to required groups if missing
sudo usermod -aG dba oracle
sudo usermod -aG oinstall oracle

# The oracle user must log out and back in (or start a new session) for group
# changes to take effect for new processes. Existing processes are not affected.

# Check /dev/shm permissions — should be 1777 (world-writable with sticky bit)
stat /dev/shm
# Fix if wrong:
sudo chmod 1777 /dev/shm

# If SELinux is enforcing, add the boolean or set permissive temporarily to test
sudo setenforce 0   # temporary — permissive mode for testing only
# If startup succeeds, work with your security team to create a proper SELinux policy

Step 3: Fix /dev/shm Size for AMM

# Check MEMORY_TARGET from SPFILE
strings $ORACLE_HOME/dbs/spfile${ORACLE_SID}.ora | grep -i memory_target

# Check current /dev/shm size
df -h /dev/shm

# Expand /dev/shm immediately (root required)
# Rule: /dev/shm must be >= MEMORY_TARGET + 10%
sudo mount -o remount,size=20G /dev/shm
df -h /dev/shm

# Make permanent in /etc/fstab
sudo cp /etc/fstab /etc/fstab.bak
# Edit /etc/fstab and set the tmpfs line:
# tmpfs   /dev/shm   tmpfs   defaults,size=20G   0 0
sudo vi /etc/fstab
sudo mount -o remount /dev/shm

Step 4: Remove Orphaned Shared Memory Segments

# Identify Oracle-owned orphaned segments
# An orphaned segment has no attached processes (nattch = 0)
ipcs -m | grep oracle

# For each orphaned SHMID, remove it
for shmid in $(ipcs -m | grep oracle | awk '{print $2}'); do
  nattch=$(ipcs -m -i $shmid | grep "nattch" | awk '{print $NF}')
  if [ "$nattch" = "0" ]; then
    echo "Removing orphaned segment SHMID=$shmid"
    ipcrm -m $shmid
  else
    echo "Segment SHMID=$shmid is still attached (nattch=$nattch) — skipping"
  fi
done

# Verify no oracle segments remain
ipcs -m | grep oracle

Step 5: Configure ulimits for the Oracle Process

# Check current limits for the oracle user
su - oracle -c "ulimit -a"

# Key limits for shared memory attachment:
# virtual memory (ulimit -v): unlimited or large enough for SGA + PGA + stack
# max locked memory (ulimit -l): unlimited (required for HugePages mlock)

# Update /etc/security/limits.conf
sudo tee -a /etc/security/limits.conf > /dev/null << 'EOF'
oracle   soft   nofile     1024
oracle   hard   nofile     65536
oracle   soft   nproc      2047
oracle   hard   nproc      16384
oracle   soft   stack      10240
oracle   hard   stack      32768
oracle   soft   memlock    unlimited
oracle   hard   memlock    unlimited
EOF

# Start a new oracle session and verify
su - oracle -c "ulimit -l && ulimit -v"

Step 6: Restart Oracle

-- After OS-level fixes, start the instance
STARTUP;

-- Confirm SGA is attached
SELECT name, ROUND(bytes/1024/1024/1024, 2) AS gb
FROM v$sgainfo
WHERE name IN ('Total SGA Size', 'Free SGA Memory Available');

-- Confirm all background processes are up
SELECT name, description, status
FROM v$bgprocess
WHERE paddr != '00'
ORDER BY name;

Prevention Strategies

Automated IPC Health Check Script

#!/bin/bash
# oracle_ipc_check.sh — Verify IPC readiness before Oracle startup

ORACLE_USER=${1:-oracle}
ORACLE_GROUP=${2:-oinstall}
SGA_GB=${3:-8}
SGA_BYTES=$(( SGA_GB * 1024 * 1024 * 1024 ))
ERRORS=0

echo "=== Oracle IPC Readiness Check ==="
echo "User: $ORACLE_USER | Group: $ORACLE_GROUP | SGA: ${SGA_GB}GB"
echo ""

# 1. Group membership
if id "$ORACLE_USER" | grep -q "$ORACLE_GROUP"; then
  echo "[OK]   $ORACLE_USER is member of $ORACLE_GROUP"
else
  echo "[FAIL] $ORACLE_USER not in $ORACLE_GROUP"; ERRORS=$((ERRORS+1))
fi

# 2. /dev/shm size
SHM_AVAIL=$(df --output=avail /dev/shm | tail -1)
SHM_AVAIL_BYTES=$(( SHM_AVAIL * 1024 ))
if [ $SHM_AVAIL_BYTES -ge $SGA_BYTES ]; then
  echo "[OK]   /dev/shm has $(( SHM_AVAIL / 1024 / 1024 ))GB available"
else
  echo "[FAIL] /dev/shm too small: $(( SHM_AVAIL / 1024 / 1024 ))GB < ${SGA_GB}GB"; ERRORS=$((ERRORS+1))
fi

# 3. /dev/shm permissions
SHM_PERMS=$(stat -c %a /dev/shm)
if [ "$SHM_PERMS" = "1777" ]; then
  echo "[OK]   /dev/shm permissions = 1777"
else
  echo "[FAIL] /dev/shm permissions = $SHM_PERMS (expected 1777)"; ERRORS=$((ERRORS+1))
fi

# 4. Orphaned oracle segments
ORPHANS=$(ipcs -m | grep "$ORACLE_USER" | awk '{print $2}' | while read id; do
  n=$(ipcs -m -i $id 2>/dev/null | grep nattch | awk '{print $NF}')
  [ "$n" = "0" ] && echo $id
done)
if [ -z "$ORPHANS" ]; then
  echo "[OK]   No orphaned Oracle shared memory segments"
else
  echo "[FAIL] Orphaned segments found: $ORPHANS"; ERRORS=$((ERRORS+1))
fi

# 5. SELinux
if command -v getenforce &>/dev/null; then
  SELINUX=$(getenforce)
  echo "[INFO] SELinux mode: $SELINUX"
  [ "$SELINUX" = "Enforcing" ] && echo "       Review avc denials if ORA-27123 persists"
fi

echo ""
echo "Check complete: $ERRORS issue(s) found"
[ $ERRORS -eq 0 ] && exit 0 || exit 1

Monitoring for /dev/shm Saturation

-- Alert when /dev/shm-backed AMM memory is nearly full
-- (Requires external OS monitoring; this query shows Oracle-side pressure)
SELECT
  name,
  ROUND(bytes/1024/1024/1024, 2) AS gb,
  CASE
    WHEN name = 'Free SGA Memory Available'
         AND bytes < 100*1024*1024 THEN 'WARNING: Low free SGA'
    ELSE 'OK'
  END AS status
FROM v$sgainfo
WHERE name IN ('Maximum SGA Size', 'Total SGA Size', 'Free SGA Memory Available');

Post-Crash Cleanup Procedure

#!/bin/bash
# oracle_ipc_cleanup.sh — Safe IPC cleanup after instance crash
# Run as oracle OS user

ORACLE_SID=${1:?Usage: $0 ORACLE_SID}

echo "Cleaning up IPC resources for SID: $ORACLE_SID"

# Remove shared memory segments owned by oracle with no attached processes
ipcs -m | grep oracle | awk '{print $2}' | while read shmid; do
  nattch=$(ipcs -m -i "$shmid" 2>/dev/null | grep nattch | awk '{print $NF}')
  if [ "$nattch" = "0" ]; then
    echo "  Removing SHM segment $shmid (unattached)"
    ipcrm -m "$shmid" && echo "  Removed." || echo "  Failed — may need root"
  else
    echo "  Skipping SHM segment $shmid (attached, nattch=$nattch)"
  fi
done

# Remove semaphore sets owned by oracle
ipcs -s | grep oracle | awk '{print $2}' | while read semid; do
  echo "  Removing semaphore set $semid"
  ipcrm -s "$semid"
done

echo "IPC cleanup complete"
ipcs -m | grep oracle
ipcs -s | grep oracle

Related Errors

• ORA-27125 - Unable to create shared memory segment (precedes ORA-27123 in startup)
• ORA-27102 - Out of memory (kernel refused allocation entirely)
• ORA-27140 - Attach to post/wait facility failed (semaphore attach failure)
• ORA-27300 - OS system dependent operation failed (generic OS error wrapper)
• ORA-00845 - MEMORY_TARGET not supported (/dev/shm too small for AMM)
• ORA-04031 - Unable to allocate shared memory within the SGA

Emergency Response

Quick Fixes

1. Check the secondary error first

   tail -50 $ORACLE_BASE/diag/rdbms/$ORACLE_SID/$ORACLE_SID/trace/alert_$ORACLE_SID.log \
     | grep -A3 "ORA-27123"

2. Fix permissions (Error 13)

   sudo chmod 1777 /dev/shm
   sudo usermod -aG oinstall oracle
   # Log out and back in as oracle, then retry startup

3. Fix /dev/shm size (Error 12)

   sudo mount -o remount,size=20G /dev/shm
   sqlplus / as sysdba
   STARTUP;

4. Remove orphaned segments (Error 22)

   ipcs -m | grep oracle | awk '{print $2}' | xargs -I{} ipcrm -m {}
   sqlplus / as sysdba
   STARTUP;

5. Disable AMM as a last resort

   strings $ORACLE_HOME/dbs/spfile${ORACLE_SID}.ora \
     | grep -v memory_target > /tmp/init_noamm.ora
   # Add: sga_target=8G and pga_aggregate_target=2G to /tmp/init_noamm.ora
   sqlplus / as sysdba
   STARTUP PFILE='/tmp/init_noamm.ora';
   CREATE SPFILE FROM PFILE='/tmp/init_noamm.ora';