ORA-15017 Diskgroup Cannot Be Mounted - ASM Mount Failure Resolution

ORA-15017: Diskgroup Cannot Be Mounted

Error Overview

Error Text: ORA-15017: diskgroup "diskgroup_name" cannot be mounted

This error occurs when ASM (Automatic Storage Management) cannot mount a diskgroup due to various issues such as missing disks, corrupted metadata, insufficient disk resources, or configuration problems. It indicates a more serious issue than ORA-15001, as the diskgroup exists but has problems preventing it from being mounted.

Understanding ASM Mount Process

Mount Operation Steps

ASM Mount Process
├── Disk Discovery
│   ├── Scan ASM_DISKSTRING paths
│   ├── Read disk headers
│   └── Validate disk membership
├── Metadata Validation
│   ├── Check disk group metadata
│   ├── Verify redundancy
│   └── Validate allocation units
├── Resource Allocation
│   ├── Allocate memory structures
│   ├── Start background processes
│   └── Initialize extent maps
└── Mount Completion
    ├── Update v$asm_diskgroup
    └── Make diskgroup available

Common Mount Failures

• Insufficient disks for redundancy
• Corrupted ASM metadata
• Disk path changes
• Permission issues
• Resource limitations

Common Causes

Disk Availability Issues

-- Missing disks from diskgroup
-- Disk failures exceeding redundancy
-- Disk path changes
-- Multipath configuration issues

Metadata Corruption

• PST (Partnership Status Table) corruption
• ACD (Active Change Directory) issues
• File directory corruption
• Allocation table problems

Configuration Problems

• Incorrect ASM_DISKSTRING
• Wrong disk permissions
• ASM parameter issues
• Memory constraints

Environmental Issues

• Storage array problems
• Operating system issues
• Cluster synchronization problems
• Network storage disconnections

Diagnostic Steps

Detailed Mount Error Analysis

-- Connect to ASM instance
sqlplus / as sysasm

-- Attempt mount with detailed error
ALTER DISKGROUP data MOUNT;

-- Check alert log for specific errors
-- Location: $ORACLE_BASE/diag/asm/+asm/+ASM/trace/alert_+ASM.log

-- Query diskgroup information
SELECT group_number, name, state, type, total_mb, required_mirror_free_mb
FROM v$asm_diskgroup
WHERE name = 'DATA';

Disk Status Investigation

-- Check all disks for the diskgroup
SELECT disk_number, path, mount_status, header_status, state, failgroup
FROM v$asm_disk
WHERE group_number = (SELECT group_number FROM v$asm_diskgroup WHERE name = 'DATA')
ORDER BY disk_number;

-- Find missing or problematic disks
SELECT path, header_status, state, total_mb, free_mb
FROM v$asm_disk
WHERE header_status IN ('CANDIDATE', 'FORMER', 'PROVISIONED', 'INCOMPATIBLE')
   OR state != 'NORMAL';

-- Check disk visibility across nodes (RAC)
SELECT inst_id, path, header_status
FROM gv$asm_disk
WHERE path LIKE '%DISK%'
ORDER BY inst_id, path;

Metadata Validation

# Check disk headers with kfed
kfed read /dev/oracleasm/disks/DISK1

# Verify ASM disk header
kfed read /dev/oracleasm/disks/DISK1 | grep -E "kfdhdb.grptyp|kfdhdb.dskname|kfdhdb.grpname"

# Check for metadata corruption
amdu -diskstring '/dev/oracleasm/disks/*' -dump 'DATA' -noimage

System Resource Check

-- Check ASM memory allocation
SHOW PARAMETER asm_power_limit;
SHOW PARAMETER memory_target;
SHOW PARAMETER sga_target;

-- Check ASM processes
SELECT name, description FROM v$bgprocess
WHERE name LIKE 'AR%' OR name IN ('RBAL', 'GMON');

-- Verify system resources
!free -m
!df -h

Resolution Steps

1. Fix Disk Discovery Issues

Update Disk Discovery String

-- Check current discovery string
SHOW PARAMETER asm_diskstring;

-- Update to include all disk paths
ALTER SYSTEM SET asm_diskstring='/dev/oracleasm/disks/*','/dev/mapper/asm*','/dev/sd*' SCOPE=BOTH;

-- Force disk discovery
ALTER SYSTEM SET asm_diskstring='/dev/oracleasm/disks/DISK*' SCOPE=MEMORY;
ALTER SYSTEM SET asm_diskstring='/dev/oracleasm/disks/*' SCOPE=MEMORY;

Verify Disk Permissions

# Check and fix disk ownership
ls -la /dev/oracleasm/disks/
chown grid:asmadmin /dev/oracleasm/disks/*
chmod 660 /dev/oracleasm/disks/*

# For multipath devices
multipath -ll
chown grid:asmadmin /dev/mapper/asm*
chmod 660 /dev/mapper/asm*

# Verify udev rules (Linux)
cat /etc/udev/rules.d/99-oracle-asmdevices.rules

2. Force Mount Operations

Mount with Limited Disks

-- Force mount ignoring missing disks
ALTER DISKGROUP data MOUNT FORCE;

-- Check mount status
SELECT name, state, type, total_mb, free_mb
FROM v$asm_diskgroup WHERE name = 'DATA';

-- If successful, check missing disks
SELECT * FROM v$asm_disk
WHERE group_number = (SELECT group_number FROM v$asm_diskgroup WHERE name = 'DATA')
AND header_status = 'FORMER';

Mount in Restricted Mode

-- Mount for maintenance only
ALTER DISKGROUP data MOUNT RESTRICTED;

-- Perform maintenance operations
ALTER DISKGROUP data CHECK ALL REPAIR;

-- Dismount and remount normally
ALTER DISKGROUP data DISMOUNT;
ALTER DISKGROUP data MOUNT;

3. Metadata Repair

Check and Repair Diskgroup

-- Run consistency check
ALTER DISKGROUP data CHECK ALL NOREPAIR;

-- Review results in alert log
-- Then repair if needed
ALTER DISKGROUP data CHECK ALL REPAIR;

-- Check specific disk
ALTER DISKGROUP data CHECK DISK data_0001 REPAIR;

-- Check specific file
ALTER DISKGROUP data CHECK FILE '+DATA/orcl/datafile/system.256.854737291' REPAIR;

Manual Metadata Recovery

# Backup disk headers before repair
for disk in /dev/oracleasm/disks/DISK*; do
    dd if=$disk of=/backup/$(basename $disk)_header.bak bs=1M count=10
done

# Use AMDU to extract metadata
amdu -diskstring '/dev/oracleasm/disks/*' -extract 'DATA.255'

4. Diskgroup Recreation (Last Resort)

Export Diskgroup Metadata

-- Generate diskgroup creation script
SET LONG 20000
SELECT dbms_metadata.get_ddl('DISKGROUP', 'DATA') FROM dual;

-- Backup important file list
SELECT file_number, file_incarnation, file_type, file_name
FROM v$asm_file
WHERE group_number = (SELECT group_number FROM v$asm_diskgroup WHERE name = 'DATA');

Recreate Diskgroup

-- Drop and recreate (WARNING: Data loss)
DROP DISKGROUP data FORCE INCLUDING CONTENTS;

-- Recreate with same configuration
CREATE DISKGROUP data NORMAL REDUNDANCY
FAILGROUP fg1 DISK
    '/dev/oracleasm/disks/DISK1',
    '/dev/oracleasm/disks/DISK2'
FAILGROUP fg2 DISK
    '/dev/oracleasm/disks/DISK3',
    '/dev/oracleasm/disks/DISK4'
ATTRIBUTE
    'compatible.asm' = '19.0.0.0',
    'compatible.rdbms' = '19.0.0.0',
    'au_size' = '4M';

Advanced Troubleshooting

Disk Header Analysis

#!/bin/bash
# Analyze all disk headers

DISKSTRING="/dev/oracleasm/disks/*"

for disk in $DISKSTRING; do
    echo "=== Checking $disk ==="
    kfed read $disk | grep -E "kfdhdb.grptyp|kfdhdb.dskname|kfdhdb.grpname|kfdhdb.dsknum|kfdhdb.driver"
done > /tmp/disk_header_analysis.txt

# Check for inconsistencies
grep -E "KFDGTP_INVALID|KFDGTP_UNKNOWN" /tmp/disk_header_analysis.txt

RAC-Specific Diagnostics

# Check disk visibility on all nodes
for node in node1 node2 node3; do
    echo "=== Node: $node ==="
    ssh $node "ls -la /dev/oracleasm/disks/ | wc -l"
    ssh $node "oracleasm listdisks | wc -l"
done

# Compare v$asm_disk across instances
crsctl stat res ora.DATA.dg -t
srvctl status diskgroup -g DATA -detail

Force Online Operations

-- Force disk online
ALTER DISKGROUP data ONLINE DISK data_0001 FORCE;

-- Force all disks online
ALTER DISKGROUP data ONLINE ALL FORCE;

-- Drop problematic disk
ALTER DISKGROUP data DROP DISK data_0001 FORCE;

-- Add disk back
ALTER DISKGROUP data ADD DISK '/dev/oracleasm/disks/DISK1' FORCE;

Monitoring and Prevention

Proactive Mount Monitoring

-- Create mount status tracking
CREATE TABLE asm_mount_history (
    check_time TIMESTAMP,
    diskgroup_name VARCHAR2(30),
    mount_attempt VARCHAR2(10),
    error_message VARCHAR2(4000)
);

-- Mount monitoring procedure
CREATE OR REPLACE PROCEDURE monitor_diskgroup_mounts AS
    v_state VARCHAR2(20);
BEGIN
    FOR dg IN (SELECT name FROM v$asm_diskgroup WHERE state != 'MOUNTED') LOOP
        BEGIN
            -- Attempt mount
            EXECUTE IMMEDIATE 'ALTER DISKGROUP ' || dg.name || ' MOUNT';

            INSERT INTO asm_mount_history VALUES (
                SYSTIMESTAMP, dg.name, 'SUCCESS', NULL
            );
        EXCEPTION
            WHEN OTHERS THEN
                INSERT INTO asm_mount_history VALUES (
                    SYSTIMESTAMP, dg.name, 'FAILED', SQLERRM
                );

                -- Alert on failure
                DBMS_OUTPUT.PUT_LINE('Mount failed for ' || dg.name || ': ' || SQLERRM);
        END;
    END LOOP;
    COMMIT;
END;
/

Disk Health Monitoring

#!/bin/bash
# Monitor disk health and availability

LOG_FILE="/var/log/asm_disk_health.log"
ALERT_EMAIL="[email protected]"

check_disk_health() {
    local issues=0

    # Check disk accessibility
    for disk in /dev/oracleasm/disks/*; do
        if [ ! -r "$disk" ]; then
            echo "$(date): ERROR - Cannot read disk $disk" >> $LOG_FILE
            ((issues++))
        fi
    done

    # Check multipath status
    multipath -ll | grep -i failed > /tmp/multipath_issues.txt
    if [ -s /tmp/multipath_issues.txt ]; then
        echo "$(date): ERROR - Multipath failures detected" >> $LOG_FILE
        cat /tmp/multipath_issues.txt >> $LOG_FILE
        ((issues++))
    fi

    # Check ASM disk status
    su - grid -c "
        sqlplus -s / as sysasm << EOF > /tmp/asm_disk_status.txt
        SET PAGESIZE 0 FEEDBACK OFF HEADING OFF
        SELECT path || ':' || header_status || ':' || state
        FROM v\\\$asm_disk
        WHERE header_status != 'MEMBER'
           OR state != 'NORMAL';
EOF
    "

    if [ -s /tmp/asm_disk_status.txt ]; then
        echo "$(date): WARNING - ASM disk issues detected" >> $LOG_FILE
        cat /tmp/asm_disk_status.txt >> $LOG_FILE
        ((issues++))
    fi

    # Send alert if issues found
    if [ $issues -gt 0 ]; then
        echo "ASM disk health check found $issues issues" | \
        mail -s "ASM Disk Health Alert" $ALERT_EMAIL
    fi
}

# Run check
check_disk_health

Emergency Procedures

Emergency Recovery Script

#!/bin/bash
# Emergency diskgroup recovery

export ORACLE_HOME=/u01/app/19.0.0/grid
export ORACLE_SID=+ASM
DISKGROUP=$1

if [ -z "$DISKGROUP" ]; then
    echo "Usage: $0 <diskgroup_name>"
    exit 1
fi

echo "Starting emergency recovery for diskgroup $DISKGROUP..."

# Phase 1: Try normal mount
echo "Phase 1: Attempting normal mount..."
$ORACLE_HOME/bin/sqlplus -s / as sysasm << EOF
ALTER DISKGROUP $DISKGROUP MOUNT;
EOF

# Check if mounted
MOUNTED=$($ORACLE_HOME/bin/sqlplus -s / as sysasm << EOF
SET PAGESIZE 0 FEEDBACK OFF HEADING OFF
SELECT COUNT(*) FROM v\$asm_diskgroup
WHERE name = '$DISKGROUP' AND state = 'MOUNTED';
EOF
)

if [ "$MOUNTED" -eq 1 ]; then
    echo "Diskgroup mounted successfully"
    exit 0
fi

# Phase 2: Force mount
echo "Phase 2: Attempting force mount..."
$ORACLE_HOME/bin/sqlplus -s / as sysasm << EOF
ALTER DISKGROUP $DISKGROUP MOUNT FORCE;
EOF

# Phase 3: Restricted mount
echo "Phase 3: Attempting restricted mount..."
$ORACLE_HOME/bin/sqlplus -s / as sysasm << EOF
ALTER DISKGROUP $DISKGROUP MOUNT RESTRICTED;
EOF

# Phase 4: Check and repair
echo "Phase 4: Running check and repair..."
$ORACLE_HOME/bin/sqlplus -s / as sysasm << EOF
ALTER DISKGROUP $DISKGROUP CHECK ALL REPAIR;
EOF

echo "Emergency recovery completed. Check alert log for details."

Critical Data Extraction

# Extract critical files before diskgroup failure
amdu -diskstring '/dev/oracleasm/disks/*' \
     -extract 'DATA.256' \
     -output /backup/extracted_files/

# Extract all database files
amdu -diskstring '/dev/oracleasm/disks/*' \
     -extract 'DATA.*' \
     -output /backup/extracted_files/ \
     -report /backup/extraction_report.txt

Best Practices

1. Regular metadata backups using md_backup
2. Monitor disk health and availability continuously
3. Maintain proper redundancy levels
4. Test mount/dismount procedures regularly
5. Keep ASM and Grid Infrastructure patched
6. Document disk-to-diskgroup mappings
7. Implement automated recovery procedures

Related Errors

• ORA-15001: Diskgroup does not exist or is not mounted
• ORA-15032: Not all alterations performed
• ORA-15040: Diskgroup is incomplete
• ORA-15042: ASM disk is missing

Troubleshooting Checklist

• Check alert log for specific mount errors
• Verify all disks are accessible
• Confirm disk permissions are correct
• Check disk header status
• Validate ASM discovery string
• Test force mount operation
• Run diskgroup check and repair
• Verify sufficient resources (memory/processes)