public

Expert-Level GDB Mastery Guide

linux/advanced_gdb.md

Concept Flow

Expert GDB Core Dump Analysis Without Symbols

1Foundation and Compilation
2Core Debugging Workflow
3Breakpoint Mastery
4Examining Program State
5Navigation and Execution Control
6Advanced Stack Frame Manipulation
7Memory and Data Structure Analysis
8Multi-threaded Debugging

I'll guide you through mastering GDB, building from core concepts to advanced techniques that distinguish expert-level debugging.

Foundation and Compilation

To debug effectively with GDB, compile with debug symbols using -g and consider disabling optimization during development with -O0. For production debugging, -Og provides a reasonable compromise. You can also use -ggdb for GDB-specific enhancements or -g3 to include macro definitions in debug info.

Core Debugging Workflow

Launch GDB with gdb ./program or attach to running processes with gdb -p <pid>. The fundamental cycle involves setting breakpoints, running the program, examining state when execution stops, and continuing or stepping through code.

Breakpoint Mastery

Beyond basic breakpoints with break function or break file.cpp:42, expert usage involves conditional breakpoints: break main if argc > 2 stops only when the condition is true. Temporary breakpoints with tbreak automatically delete after being hit once.

Watchpoints monitor memory changes. Use watch variable to break when a variable changes, rwatch for read access, and awatch for any access. For watching specific memory addresses: watch *(int*)0x7fffffffe000. This is invaluable for tracking down corruption bugs.

Catchpoints trap events like catch throw for C++ exceptions, catch syscall write for system calls, or catch fork for process creation. You can disable, enable, and delete breakpoints by number, and use save breakpoints file.gdb to persist complex breakpoint configurations.

Examining Program State

The print command evaluates expressions: print variable, print *pointer, print array[5]. Use format specifiers like print/x for hex, print/t for binary, print/c for characters. The convenience variable $ holds the last printed value, and $$ the one before that.

The x command examines raw memory with format and size specifiers: x/10x $rsp displays 10 hex words at the stack pointer, x/s 0x400500 shows a string, x/10i $rip disassembles 10 instructions at the program counter.

Use display to automatically show expressions after each stop: display/i $pc shows the next instruction, display argc keeps argc visible. Remove with undisplay.

The info command family is essential: info locals shows local variables, info args displays function arguments, info registers dumps all registers, info threads lists threads, info frame shows the current stack frame details.

Step through code with step (into functions), next (over functions), stepi and nexti for instruction-level stepping. The finish command runs until the current function returns, while until continues to a specific line.

The continue command resumes execution until the next breakpoint. You can continue and skip a certain number of hits: continue 5 ignores the next 5 breakpoint hits.

Advanced Stack Frame Manipulation

Use backtrace or bt to see the call stack, bt full to include local variables. Navigate frames with frame N to select a specific frame, up and down to move through the stack. Once in a frame, examine its locals and arguments as if you were at that point in execution.

The return command can force a premature return from a function, optionally with a specified value: return 42. This is powerful for testing error paths or skipping problematic code sections.

Memory and Data Structure Analysis

For complex structures, enable pretty-printing with set print pretty on. Cast and dereference to navigate data structures: print *(struct my_struct*)0x7ffff000. Call functions directly: call malloc(100) or call my_debug_function().

Examine types with ptype variable to see the full type definition including struct members. Use whatis for a quick type summary.

Multi-threaded Debugging

List threads with info threads, switch contexts with thread N. The thread apply all command runs a command on all threads, like thread apply all bt for all backtraces. Control whether other threads run during stepping with set scheduler-locking on (only current thread), off (all threads), or step (only during stepping).

Core Dump Analysis

Generate core dumps with ulimit -c unlimited then reproduce the crash. Load with gdb ./program core. Examine the state at crash time using all the standard inspection commands. This is post-mortem debugging and works identically to live debugging for inspection purposes.

For generating cores programmatically, call gcore <pid> while the process runs, or use generate-core-file from within GDB.

Reverse Execution and Record/Replay

Enable with record or record full. Once recording, use reverse-continue, reverse-step, reverse-next, and reverse-finish to execute backwards. This lets you wind back to see what led to a bug. Use record stop to end recording. Note that recording has performance overhead and memory limits.

Scripting and Automation

GDB uses a Python API for powerful automation. Access with python for single commands or python ... end for blocks:

python
class MyBreakpoint(gdb.Breakpoint):
    def stop(self):
        frame = gdb.selected_frame()
        if frame.read_var('x') > 100:
            print("x exceeded threshold!")
            return True
        return False
MyBreakpoint("function_name")
end

Write GDB command scripts in files with sequences of GDB commands, then load with source script.gdb. Use command-line arguments: gdb -x script.gdb ./program.

Define custom commands with define:

define print_locals_hex
  info locals
  print/x $arg0
end

Advanced Techniques

Convenience Variables: Create with set $myvar = expression. These persist across stops and can store complex values.

Checkpoints: Use checkpoint to save process state, then restart N to return to that checkpoint. This creates a fork and is useful for exploring different execution paths.

Debugging Optimized Code: Use set substitute-path to map source paths. Debug info may be incomplete, so use disassembly with disassemble and instruction-level stepping.

Remote Debugging: Use target remote host:port to debug processes on remote machines or embedded systems via gdbserver.

TUI Mode: Launch with gdb -tui or toggle with Ctrl-x Ctrl-a for split-screen source and assembly views. Navigate with layout src, layout asm, layout split, layout regs.

Signal Handling: Control with handle SIGNAL action where action is stop, nostop, print, noprint, pass, or nopass. Example: handle SIGPIPE nostop noprint pass.

Expert Workflow Tips

Create a .gdbinit file in your home directory or project directory for automatic initialization. Enable history with set history save on and set history filename ~/.gdb_history. Add set print pretty on, set pagination off for scripting, and custom commands.

Use command abbreviations: b for break, r for run, c for continue, n for next, s for step, p for print, bt for backtrace.

Combine with other tools: valgrind --vgdb=yes --vgdb-error=0 launches your program under Valgrind with GDB control. This combines GDB's inspection with Valgrind's memory checking.

For debugging without source, become proficient with disassembly, register examination (info registers), and understanding calling conventions. Know your architecture's ABI—where arguments go, return value locations, caller vs callee-saved registers.

Common Expert Patterns

Finding segfaults: Run until crash, bt to see the call stack, examine the faulting address and surrounding pointers.

Memory corruption: Set watchpoints on the corrupted memory, run until the watchpoint triggers to catch the culprit.

Race conditions: Use set scheduler-locking step to control thread execution, set thread-specific breakpoints with break file.cpp:42 thread 3.

Heisenbug investigation: Use reverse execution to replay exactly what happened, or use core dumps for deterministic analysis.

This should establish a comprehensive foundation for expert GDB usage. Would you like me to dive deeper into any specific area, such as Python scripting, debugging techniques for particular bug classes, or integration with specific development workflows?

Expert GDB Core Dump Analysis Without Symbols

Debugging stripped binaries and core dumps without symbols is a critical skill. Here's the complete toolkit.

Loading and Initial Analysis

# Load core dump
gdb /path/to/binary core

# Or if binary path is unknown
gdb -c core
# Then: file /path/to/binary

# First commands to run
info registers      # See register state at crash
info frame          # Current frame (may be garbage without symbols)
info threads        # List all threads
x/20i $pc           # Disassemble around program counter

Understanding the Crash

Examine Registers

Registers hold critical information:

info registers
# or
info all-registers  # Includes floating point/SIMD

# Key registers (x86-64):
# rip/pc  - instruction pointer (where crash occurred)
# rsp     - stack pointer
# rbp     - base pointer (frame pointer)
# rdi, rsi, rdx, rcx, r8, r9 - first 6 function arguments
# rax     - return value

Look for anomalies:

Identify the Faulting Instruction

x/i $pc                    # Current instruction
x/10i $pc-20               # Instructions before crash
x/20i $pc                  # Instructions around crash

# Check what memory was being accessed
info registers $rsi        # If instruction was using rsi
x/x $rsi                   # Try to read that address

Common crash patterns:

Manual Stack Walking

Without symbols, you must manually walk the stack:

# Examine stack memory
x/40gx $rsp                # 40 8-byte values in hex
x/100gx $rsp               # More context

# Look for return addresses (typically in text segment)
info proc mappings         # See memory layout
# or
info files                 # Show loaded sections

Return addresses on the stack point to locations just after call instructions. They typically fall within the executable's text segment (often 0x400000-0x600000 range for static, 0x555555554000+ for PIE).

Identifying Stack Frames

# Pattern for x86-64 stack frame:
# saved rbp
# return address
# local variables
# ...

# Walk manually
x/2gx $rbp                 # Saved rbp and return address
x/2gx 0x<saved_rbp_value>  # Next frame
# Continue until you reach main or garbage

# Automated attempt (may fail without symbols)
backtrace
# or
bt 50                      # Try deeper

Create a manual backtrace:

# At each potential return address
x/5i <return_addr - 10>    # Disassemble before return
# Look for 'call' instruction - that's the function that was called

Disassembly Analysis

Understanding Assembly Without Symbols

# Disassemble around PC
disassemble $pc,$pc+100

# Find function boundaries (look for function prologue/epilogue)
x/50i $pc-100
# Prologue typically: push %rbp; mov %rsp,%rbp; sub $X,%rsp
# Epilogue typically: leave; ret  or  pop %rbp; ret

# Disassemble potential function starts
disassemble <address>

Recognizing Function Signatures

Without symbols, recognize patterns:

Function call setup (x86-64 System V ABI):

mov $0x..., %rdi           # 1st argument
mov $0x..., %rsi           # 2nd argument  
mov $0x..., %rdx           # 3rd argument
mov $0x..., %rcx           # 4th argument
mov $0x..., %r8            # 5th argument
mov $0x..., %r9            # 6th argument
# Stack arguments for 7+
call <address>

Common library functions have characteristic patterns:

Finding Function Boundaries

# Scan backwards from crash point for function start
x/i $pc-4
x/i $pc-8
# ... continue until you see function prologue

# Or use disassemble with estimated range
disassemble $pc-200,$pc+50

# Look for:
push   %rbp              # Common function start
mov    %rsp,%rbp
sub    $0x..,%rsp       # Allocate stack space

Memory Examination Techniques

Finding Strings and Data

# Search for ASCII strings around crash
x/s <address>              # Try to interpret as string

# Search memory for patterns
find <start>, <end>, "string"
find 0x400000, 0x500000, "error"

# Examine data around pointers
x/20wx $rdi                # 20 words in hex
x/20s $rsi                 # Try as strings

# Look for C++ vtables (start with pointer to type_info)
x/10gx <object_pointer>    # First value is vtable pointer
x/10gx <vtable_pointer>    # Vtable contains function pointers

Interpreting Memory Layout

# Get memory map
info proc mappings

# Typical layout (ASLR enabled):
# 0x555555554000-0x555555556000  .text (code)
# 0x555555756000-0x555555757000  .data (initialized data)
# 0x555555757000-0x555555758000  .bss  (uninitialized data)
# 0x7ffff7a0e000-0x7ffff7bce000  libc.so
# 0x7ffffffde000-0x7ffffffff000  [stack]

# Identify regions
x/10gx 0x555555554000      # Code (will show instructions)
x/10gx 0x7fffffffe000      # Stack (will show addresses/data)

Heap Analysis

# Find heap allocations
info proc mappings         # Look for [heap]

# Examine heap chunks (glibc malloc)
# Chunk header: prev_size, size (with flags in low 3 bits)
x/4gx <heap_address-16>    # Show chunk metadata

# Look for heap corruption patterns
x/100gx <heap_start>       # Scan heap

Reconstructing the Call Chain

Manual Stack Trace Construction

# Python script to help extract call chain
gdb -batch -ex "set pagination off" \
    -ex "thread apply all bt" \
    -ex "quit" \
    /path/to/binary core > bt.txt

From within GDB:

# Semi-automated approach
python
import gdb

sp = int(gdb.parse_and_eval("$rsp"))
# Read stack looking for code pointers
for offset in range(0, 400, 8):
    addr = sp + offset
    try:
        val = gdb.parse_and_eval(f"*(unsigned long*){addr}")
        val_int = int(val)
        # Check if value looks like a code address
        if 0x400000 < val_int < 0x800000 or \
           0x555555554000 < val_int < 0x555555600000:
            # Try to see if it's preceded by a call
            gdb.execute(f"x/2i {val_int-7}")
    except:
        pass
end

Using External Tools

# Get additional info from core
eu-unstrip -n --core=core           # Find which libraries were loaded

# Extract executable from core if needed  
eu-readelf -n core | grep "NT_FILE"

# Examine with readelf/objdump
readelf -a /path/to/binary
objdump -d /path/to/binary          # Full disassembly
objdump -d --start-address=0x<addr> --stop-address=0x<addr> /path/to/binary

# Try to get symbols from separate debug info
apt-get install <package>-dbgsym    # Debian/Ubuntu
debuginfo-install <package>         # RedHat/Fedora

# Search for debug symbols
find /usr/lib/debug -name "*.debug"

Advanced Pattern Recognition

Identify C++ Crashes

# C++ often crashes in:
# - Virtual function calls (look for indirect calls through vtable)
# - Exception handling (look for _Unwind_* or __cxa_throw)
# - Destructor calls (often called in loops or during unwind)

# Check for C++ runtime functions
x/i $pc
# If you see calls to addresses in libstdc++, check:
info proc mappings | grep libstdc

# C++ object layout
x/20gx <this_pointer>      # First is usually vtable pointer
x/10gx *(void**)0x<obj>    # Examine vtable

Recognize Common Crash Causes

Null pointer dereference:

# Look for register containing 0 being dereferenced
info registers
x/i $pc
# If instruction is "mov (%rax),%rdx" and rax=0, that's the bug

Use after free:

# Freed memory often filled with patterns
x/20gx <accessed_address>
# Look for: 0xfefefefe (Windows), 0x5a5a5a5a, repeating patterns

Stack overflow:

info proc mappings         # Check stack size
# Calculate stack usage
print $rsp - <stack_start>
# If near stack limit, likely overflow

# Look for recursive calls
x/100gx $rsp              # Repeating patterns of return addresses

Buffer overflow:

# Look for corrupted return address on stack
x/20gx $rbp
# Return address should be valid code address
# If it's garbage or pattern (0x4141414141...), that's overflow

Signal Information

info signal               # What signal caused crash
info program              # Why program stopped

# Common signals:
# SIGSEGV (11) - Segmentation fault (bad memory access)
# SIGABRT (6)  - Abort (often from assert/abort())
# SIGBUS (7)   - Bus error (alignment issue)
# SIGFPE (8)   - Floating point exception
# SIGILL (4)   - Illegal instruction

Practical Debugging Workflow

Complete Analysis Checklist

# 1. Load and get orientation
file /path/to/binary
core core
info program
info signal

# 2. Examine crash context
info registers
x/20i $pc-20
x/40gx $rsp

# 3. Identify faulting instruction
x/i $pc
# Determine what memory was accessed

# 4. Check register values
info registers
# Are pointers valid?
x/x $rdi
x/x $rsi
x/x $rdx

# 5. Examine memory map
info proc mappings
# Is PC in valid code region?
# Is SP in stack region?

# 6. Attempt backtrace
bt
thread apply all bt

# 7. Manual stack walk if needed
x/100gx $rsp
# Pick out return addresses

# 8. Examine each potential caller
x/10i <return_addr-20>
# Find the call instruction

# 9. Look for data/strings
find 0x400000, 0x800000, "error"
find 0x400000, 0x800000, "assert"

# 10. Check for common patterns
# - Null pointers
# - Stack corruption  
# - Heap corruption
# - Use after free

Creating Helper Scripts

Save this as ~/.gdbinit:

# Helper commands for stripped debugging

define xxd
  dump binary memory dump.bin $arg0 $arg0+$arg1
  shell xxd dump.bin
  shell rm dump.bin
end

define find_calls
  python
import gdb
frame = gdb.selected_frame()
pc = int(frame.pc())
for i in range(-200, 50, 1):
    try:
        ins = gdb.execute(f"x/i {pc+i}", to_string=True)
        if 'call' in ins:
            print(ins)
    except:
        pass
  end
end

define search_stack_returns
  python
import gdb
sp = int(gdb.parse_and_eval("$rsp"))
print("Potential return addresses on stack:")
for offset in range(0, 800, 8):
    try:
        addr = sp + offset
        val = int(gdb.parse_and_eval(f"*(unsigned long*){addr}"))
        # Check if looks like code address
        if (0x400000 < val < 0x800000) or (0x555555554000 < val < 0x600000000):
            print(f"[{hex(addr)}] = {hex(val)}")
            gdb.execute(f"x/i {val}")
    except:
        pass
  end
end

set pagination off
set print pretty on

Multi-threaded Core Analysis

info threads              # List all threads
thread apply all bt       # Backtrace of all threads

# Switch to each thread
thread 1
info registers
x/20i $pc

thread 2  
info registers
x/20i $pc

# Find thread that crashed
thread apply all info registers
# Look for unusual PC values

Reverse Engineering Techniques

Finding Function Arguments

Since you know the calling convention:

# For x86-64, before function call:
# Arg 1: rdi
# Arg 2: rsi  
# Arg 3: rdx
# Arg 4: rcx
# Arg 5: r8
# Arg 6: r9
# Arg 7+: on stack

# Inside function, arguments may be in:
# - Original registers (if not clobbered)
# - Stack at known offsets  
# - Local stack slots

# Check function prologue to see where args are saved
x/20i <function_start>

Identifying Variables

# Local variables are at [rbp-offset]
# Find function prologue
x/10i <function_start>
# Note: sub $0x40,%rsp  <- 0x40 bytes of locals

# Examine local space
x/16gx $rbp-0x40

This toolkit should enable you to debug virtually any core dump, even completely stripped binaries. The key is understanding assembly, calling conventions, and memory layout. Practice on your own programs first - compile with and without symbols to verify your analysis techniques.

Expert GDB Core Dump Analysis Without Symbols - AArch64

Debugging stripped AArch64 binaries requires understanding ARM64 architecture specifics. Here's the complete guide.

Loading and Initial Analysis

# Load core dump
gdb /path/to/binary core

# First commands for AArch64
info registers      # See register state at crash
info all-registers  # Include NEON/FP registers
info frame          
info threads        
x/20i $pc           # Disassemble around program counter

AArch64 Register Architecture

Understanding AArch64 Registers

info registers

# General Purpose Registers (64-bit):
# x0-x7    - Function arguments and return values
# x8       - Indirect result location (structs)
# x9-x15   - Temporary registers (caller-saved)
# x16-x17  - Intra-procedure call scratch (IP0, IP1)
# x18      - Platform register (varies by OS)
# x19-x28  - Callee-saved registers
# x29      - Frame pointer (FP)
# x30      - Link register (LR) - stores return address
# sp       - Stack pointer
# pc       - Program counter

# Also have 32-bit versions: w0-w30 (lower 32 bits of x0-x30)

# NEON/FP registers:
# v0-v31   - 128-bit SIMD/FP registers
# Can be accessed as:
#   - q0-q31 (128-bit)
#   - d0-d31 (64-bit double)
#   - s0-s31 (32-bit float)
#   - h0-h31 (16-bit half)
#   - b0-b31 (8-bit byte)

Key differences from x86-64:

Examining Crash State

info registers
# Check critical registers:
# pc  - where crash occurred
# sp  - stack pointer (must be 16-byte aligned)
# x29 - frame pointer
# x30 - return address (where we came from)
# x0-x7 - function arguments

# Look for anomalies:
print/x $pc          # Is this valid code address?
print/x $sp          # Is stack aligned? (low 4 bits should be 0)
print/x $x30         # Valid return address?
x/i $pc              # What instruction faulted?
x/i $x30             # Where would we return to?

AArch64 Calling Convention (AAPCS64)

Function Arguments

# Arguments 1-8: x0-x7
# Argument 9+: on stack
# Return value: x0 (or x0+x1 for 128-bit)
# Floating point args: v0-v7
# FP return: v0

# Before a function call, examine:
info registers x0 x1 x2 x3 x4 x5 x6 x7

# Common function patterns:
# strlen(char *s)           - s in x0
# memcpy(void *d, void *s, size_t n) - d in x0, s in x1, n in x2
# printf(const char *fmt, ...) - fmt in x0, args in x1-x7, then stack

Register Preservation

# Caller-saved (may be clobbered by function):
# x0-x18 (except x16-x17 are special)

# Callee-saved (must be preserved):
# x19-x28, x29 (FP), x30 (LR), sp

# If debugging in middle of function, callee-saved registers
# from caller should still have original values

Understanding the Crash

Identify the Faulting Instruction

x/i $pc                    # Current instruction
x/10i $pc-40               # Instructions before crash (ARM is 4-bytes/instruction)
x/20i $pc                  # Instructions around crash

# Common crash patterns on AArch64:

# 1. Null pointer dereference
ldr x1, [x0]               # Load from x0, if x0=0 → crash
str x2, [x1, #16]          # Store to x1+16, if x1=0 → crash

# 2. Invalid memory access
ldr x0, [x1, #0x1000]      # If x1+0x1000 is unmapped
ldrb w0, [x2]              # Byte load from invalid x2

# 3. Function pointer call
blr x3                     # Branch to address in x3, if invalid → crash
br x4                      # Branch without link

# 4. Alignment fault (if strict alignment enabled)
ldr x0, [x1]               # If x1 not 8-byte aligned
ldp x0, x1, [x2]           # Load pair, needs 8-byte alignment

# Check what memory was being accessed
info registers x0 x1 x2 x3
x/x $x0                    # Try to read addresses
x/x $x1

AArch64 Instruction Patterns

# Memory access patterns:
ldr x0, [x1]               # Load 64-bit: x0 = *x1
ldr w0, [x1]               # Load 32-bit: w0 = *(uint32_t*)x1
ldrb w0, [x1]              # Load byte: w0 = *(uint8_t*)x1
ldrh w0, [x1]              # Load half: w0 = *(uint16_t*)x1
ldp x0, x1, [x2]           # Load pair: x0=*x2, x1=*(x2+8)
ldr x0, [x1, #16]          # Load with offset: x0 = *(x1+16)
ldr x0, [x1, x2]           # Load with register offset: x0 = *(x1+x2)

str x0, [x1]               # Store 64-bit: *x1 = x0
str w0, [x1]               # Store 32-bit
strb w0, [x1]              # Store byte
strh w0, [x1]              # Store half
stp x0, x1, [x2]           # Store pair

# Branching:
bl <offset>                # Branch with link (call), LR=PC+4
blr x0                     # Branch with link to register
b <offset>                 # Unconditional branch
br x0                      # Branch to register
ret                        # Return (br x30)
ret x5                     # Return to x5 instead of x30

# Conditional branches:
cbz x0, <offset>           # Compare and branch if zero
cbnz x0, <offset>          # Compare and branch if non-zero
b.eq <offset>              # Branch if equal (after cmp)
b.ne, b.gt, b.lt, etc.

Manual Stack Walking

AArch64 Stack Frame Layout

# Typical stack frame (with frame pointer):
#
# High addresses
# +------------------+
# | Previous FP      | <- x29 points here in previous frame
# | Return Addr (LR) | <- x30 saved here
# +------------------+
# | Saved x19-x28    | <- Callee-saved registers
# | (if used)        |
# +------------------+
# | Local variables  |
# | and spills       |
# +------------------+
# | Outgoing args    | <- For calls to other functions
# | (if needed)      |
# +------------------+ <- sp (16-byte aligned)
# Low addresses

# Walk the stack using frame pointer
x/2gx $x29                 # Show saved FP and LR
# [x29]     = previous FP
# [x29+8]   = saved LR (return address)

x/2gx *(unsigned long*)$x29   # Next frame
# Continue until you hit garbage or known address

Manual Stack Trace

# Method 1: Using saved frame pointers
set $fp = $x29
while ($fp != 0)
  x/2gx $fp
  set $fp = *(unsigned long*)$fp
end

# Method 2: Using saved link registers
x/40gx $sp                 # Look for return addresses

# Return addresses should:
# 1. Be in code segment (check with info proc mappings)
# 2. Point just after a 'bl' or 'blr' instruction

info proc mappings         # Find code regions
# Look for addresses in executable regions

Identifying Stack Frames Without Frame Pointer

If compiled with -fomit-frame-pointer:

# Look for saved LR on stack
x/100gx $sp

# Return addresses are typically:
# - In executable memory ranges
# - 4-byte aligned
# - Preceded by 'bl' instruction when you examine them

# For each potential return address:
x/3i <addr-12>             # Look for 'bl' before it

Function Recognition

AArch64 Function Prologue/Epilogue

# Standard function prologue (with frame pointer):
stp x29, x30, [sp, #-16]!  # Save FP and LR, pre-decrement sp
mov x29, sp                # Set up frame pointer
sub sp, sp, #<size>        # Allocate stack space
# May also see:
stp x19, x20, [sp, #-16]!  # Save callee-saved registers
stp x21, x22, [sp, #-16]!

# Without frame pointer (leaf functions often):
sub sp, sp, #<size>        # Just allocate stack
str x30, [sp, #<offset>]   # Save LR if needed

# Function epilogue:
ldp x29, x30, [sp], #16    # Restore FP and LR, post-increment sp
ret                        # Return (br x30)

# Or:
add sp, sp, #<size>        # Deallocate stack
ldp x29, x30, [sp], #16
ret

Finding Function Boundaries

# Scan backwards from crash point
x/i $pc-4
x/i $pc-8
# Continue until you see function prologue

# Or disassemble a range
disassemble $pc-200,$pc+50

# Look for prologue pattern:
# stp x29, x30, [sp, #-XX]!
# This is almost always function start

# Find function extent
x/100i <function_start>
# Look for 'ret' instruction for function end

Memory Examination Techniques

Finding Strings and Data

# AArch64 string loading patterns:
adrp x0, <page>            # Load page address
add x0, x0, #<offset>      # Add page offset
# Result: x0 points to string/data

# Or:
ldr x0, [pc, #<offset>]    # PC-relative load

# Examine potential string pointers
x/s $x0
x/s $x1
x/s $x2

# Search for strings
find 0x400000, 0x500000, "error"
find <start>, <end>, "assert"

# Check all argument registers for strings
x/s $x0
x/s $x1
x/s $x2
x/s $x3
x/s $x4
x/s $x5
x/s $x6
x/s $x7

Memory Layout (AArch64 Linux)

info proc mappings

# Typical layout (with ASLR):
# 0x0000005500000000-0x0000005500001000  .text (code) - PIE
# 0x0000005500010000-0x0000005500011000  .data
# 0x0000005500011000-0x0000005500012000  .bss
# 0x0000007f00000000-0x0000007f00200000  libc.so
# 0x0000007ffffff000-0x0000008000000000  [stack]

# Stack grows downward from high addresses
# On AArch64, stack is typically at very high addresses

# Check if address is valid
x/x <address>
# or
find /b <address>, <address>+1, 0xff

Examining Data Structures

# C++ objects (vtable pointer is first)
x/10gx <this_pointer>      # First is vtable
x/10gx *(void**)$x0        # If x0 is 'this'

# Arrays
x/20gx $x1                 # If x1 points to array

# Structures - manually decode offsets
# struct example {
#   uint64_t field1;  // offset 0
#   uint32_t field2;  // offset 8
#   char *field3;     // offset 16
# };
x/3gx $x0                  # Show all fields
print *(uint64_t*)($x0)
print *(uint32_t*)($x0+8)
x/s *(char**)($x0+16)

Disassembly Analysis

Common AArch64 Patterns

# Function calls
bl <function>              # Direct call
blr x0                     # Indirect call (function pointer)

# Check what was called
x/i $pc
# If: blr x3
info registers x3          # What function?
x/10i $x3                  # Disassemble target

# Parameter setup before call
# mov x0, x1               # Set arg1
# mov x1, #0               # Set arg2 = 0
# bl <function>

# Null checks
cbz x0, <error_label>      # if (x0 == 0) goto error
cbnz x0, <ok_label>        # if (x0 != 0) goto ok

# Comparisons
cmp x0, #0                 # Compare x0 with 0
b.eq <label>               # Branch if equal
cmp x1, x2                 # Compare registers
b.ne <label>               # Branch if not equal

# Loop patterns
.loop:
  ldr x1, [x0], #8         # Load and post-increment
  cbz x1, .done
  bl <process>
  b .loop
.done:

Virtual Function Calls (C++)

# Pattern for virtual call:
ldr x0, [x19]              # Load 'this' pointer
ldr x8, [x0]               # Load vtable pointer
ldr x8, [x8, #24]          # Load function pointer from vtable
blr x8                     # Call virtual function

# Examine vtable
x/10gx *(void**)$x19       # Show vtable entries
x/10i <vtable_entry>       # Disassemble virtual function

Advanced Debugging Techniques

Reconstructing the Call Chain

# Method 1: Follow saved LR values
set $current_lr = $x30
x/10i $current_lr-4        # Should show 'bl' instruction

# Walk frames
set $fp = $x29
printf "Frame 0: LR=%p\n", $x30
set $i = 1
while ($fp != 0 && $i < 20)
  set $saved_lr = *(unsigned long*)($fp + 8)
  printf "Frame %d: FP=%p LR=%p\n", $i, $fp, $saved_lr
  x/2i $saved_lr-8
  set $fp = *(unsigned long*)$fp
  set $i = $i + 1
end

# Method 2: Search stack for return addresses
python
import gdb

sp = int(gdb.parse_and_eval("$sp"))
print("Potential return addresses on stack:")

for offset in range(0, 800, 8):
    try:
        addr = sp + offset
        val = int(gdb.parse_and_eval(f"*(unsigned long*){addr}"))
        
        # Check if looks like code address (adjust ranges for your binary)
        if (0x5500000000 < val < 0x5600000000) or \
           (0x7f00000000 < val < 0x8000000000):
            # Check if preceded by bl/blr
            try:
                insn = gdb.execute(f"x/i {val-4}", to_string=True)
                if 'bl' in insn:
                    print(f"[{hex(addr)}] = {hex(val)}")
                    print(insn)
            except:
                pass
    except:
        pass
end

Signal Information

info signal
info program

# Common signals on AArch64:
# SIGSEGV (11) - Invalid memory reference
#   - si_addr shows the faulting address
# SIGBUS (7)   - Bus error (alignment on strict-align systems)
# SIGILL (4)   - Illegal instruction
# SIGFPE (8)   - Arithmetic exception
# SIGTRAP (5)  - Trace/breakpoint trap

# Get more signal details
$_siginfo
# or
print $_siginfo

Crash Pattern Recognition

Null Pointer Dereference:

# Look for x0 (or other register) being 0
info registers
x/i $pc

# Example:
# ldr x1, [x0]
# If x0 = 0x0000000000000000 → crash here

# Check all registers that might be dereferenced
info registers | grep "0x0"

Stack Overflow:

info proc mappings         # Find stack region
# Stack is typically: 0x007ffffff000-0x008000000000

# Calculate stack usage
print/x $sp
# If very close to stack start → overflow

# Look for deep recursion
x/200gx $sp
# Repeating patterns of same return addresses indicate recursion

Use After Free:

# Check for poison patterns in freed memory
x/20gx <accessed_address>

# Common patterns:
# 0xfefefefefefefefe (debug malloc)
# 0xdeadbeefdeadbeef
# 0x5a5a5a5a5a5a5a5a

# Also check if address is in heap region
info proc mappings | grep heap

Buffer Overflow:

# Check saved LR for corruption
x/2gx $x29
# Saved LR at [x29+8] should be valid code address

# If it's garbage or pattern (0x41414141...) → overflow

# Check stack for patterns
x/100gx $sp

Alignment Fault:

# AArch64 usually allows unaligned access, but:
# - Can be disabled by kernel
# - Atomic operations require alignment
# - LDP/STP require 8-byte alignment in some cases

info registers
x/i $pc

# Check if address is misaligned
# ldp x0, x1, [x2]
# If x2 = 0x12345 (not 8-byte aligned) → may crash
print/x $x2 & 7
# Should be 0 for 8-byte operations

Practical Debugging Workflow

Complete Analysis Checklist

# 1. Load core dump
file /path/to/aarch64-binary
core-file core

# 2. Basic orientation
info program
info signal
info threads

# 3. Examine crash state
info registers
info all-registers

# 4. Analyze the crash
x/i $pc                    # Faulting instruction
x/10i $pc-40              # Context before
x/10i $pc                 # Context after

# 5. Check register contents
# Especially argument registers and pointers
x/x $x0
x/x $x1
x/x $x2
# Check for null, invalid addresses

# 6. Examine return address
x/i $x30                   # Where we came from
x/5i $x30-20              # Show the call site

# 7. Stack examination
x/2gx $x29                 # Saved FP and LR
x/40gx $sp                 # Stack contents

# 8. Memory mapping
info proc mappings
# Verify PC, SP, LR are in valid regions

# 9. Attempt backtrace
bt
bt full
thread apply all bt

# 10. Manual stack walk if needed
# (use scripts from earlier section)

# 11. Look for strings/data
x/s $x0
x/s $x1
find 0x5500000000, 0x5600000000, "error"

# 12. Multi-threaded analysis
info threads
thread apply all bt
thread apply all info registers

Helper GDB Script for AArch64

Save as ~/.gdbinit-aarch64:

# AArch64 debugging helpers

define arm64-regs
  printf "x0:  %016lx  x1:  %016lx\n", $x0, $x1
  printf "x2:  %016lx  x3:  %016lx\n", $x2, $x3
  printf "x4:  %016lx  x5:  %016lx\n", $x4, $x5
  printf "x6:  %016lx  x7:  %016lx\n", $x6, $x7
  printf "x8:  %016lx  x29: %016lx (FP)\n", $x8, $x29
  printf "x30: %016lx (LR) sp:  %016lx\n", $x30, $sp
  printf "pc:  %016lx\n", $pc
end

define arm64-args
  printf "arg0 (x0): "
  x/gx $x0
  printf "arg1 (x1): "
  x/gx $x1
  printf "arg2 (x2): "
  x/gx $x2
  printf "arg3 (x3): "
  x/gx $x3
  printf "arg4 (x4): "
  x/gx $x4
  printf "arg5 (x5): "
  x/gx $x5
  printf "arg6 (x6): "
  x/gx $x6
  printf "arg7 (x7): "
  x/gx $x7
end

define arm64-frame-walk
  python
import gdb

fp = int(gdb.parse_and_eval("$x29"))
frame_num = 0

print("Manual stack frame walk:")
print("=" * 60)

while fp != 0 and frame_num < 30:
    try:
        saved_fp = int(gdb.parse_and_eval(f"*(unsigned long*){fp}"))
        saved_lr = int(gdb.parse_and_eval(f"*(unsigned long*)({fp}+8)"))
        
        print(f"\nFrame {frame_num}:")
        print(f"  FP: {hex(fp)}")
        print(f"  Saved LR: {hex(saved_lr)}")
        
        # Try to show the call site
        try:
            insn = gdb.execute(f"x/2i {saved_lr-8}", to_string=True)
            print(f"  Call site:\n{insn}", end='')
        except:
            pass
        
        # Sanity check
        if saved_fp < fp or saved_fp > fp + 0x100000:
            print("  (Frame pointer looks invalid, stopping)")
            break
            
        fp = saved_fp
        frame_num += 1
    except:
        print(f"  Error reading frame {frame_num}")
        break

print("\n" + "=" * 60)
  end
end

define arm64-stack-scan
  python
import gdb

sp = int(gdb.parse_and_eval("$sp"))
print("Scanning stack for potential return addresses:")
print("=" * 60)

for offset in range(0, 1024, 8):
    try:
        addr = sp + offset
        val = int(gdb.parse_and_eval(f"*(unsigned long*){addr}"))
        
        # Check if looks like code (adjust for your binary)
        # Typical ranges: 0x5500000000-0x5600000000 for PIE
        #                 0x7f00000000-0x8000000000 for libs
        if ((0x5500000000 < val < 0x5600000000) or 
            (0x7f00000000 < val < 0x8000000000)):
            
            # Check if 4-byte aligned (all ARM64 instructions are)
            if val & 3 == 0:
                try:
                    # Look for 'bl' before this address
                    insn = gdb.execute(f"x/2i {val-8}", to_string=True)
                    if 'bl' in insn:
                        print(f"\n[sp+{hex(offset)}] = {hex(val)}")
                        print(insn, end='')
                except:
                    pass
    except:
        pass

print("\n" + "=" * 60)
  end
end

define arm64-check-null-deref
  python
import gdb

pc = int(gdb.parse_and_eval("$pc"))

# Get the instruction
insn = gdb.execute(f"x/i {pc}", to_string=True)

print("Checking for null pointer dereference:")
print("=" * 60)
print(f"Faulting instruction:\n{insn}")

# Check if it's a memory access instruction
mem_ops = ['ldr', 'str', 'ldp', 'stp', 'ldrb', 'strb', 'ldrh', 'strh']

for op in mem_ops:
    if op in insn.lower():
        print(f"\nFound memory operation: {op}")
        
        # Check common pointer registers
        for reg in ['x0', 'x1', 'x2', 'x3', 'x4', 'x5']:
            try:
                val = int(gdb.parse_and_eval(f"${reg}"))
                status = "NULL" if val == 0 else "INVALID" if val < 0x1000 else "OK"
                print(f"  {reg}: {hex(val)} [{status}]")
            except:
                pass
        break

print("=" * 60)
  end
end

define arm64-crash-summary
  printf "\n"
  printf "=== AArch64 Crash Summary ===\n"
  printf "\n"
  
  printf "Signal: "
  info signal
  
  printf "\nCrash Location:\n"
  x/i $pc
  
  printf "\nRegister State:\n"
  arm64-regs
  
  printf "\nStack Frame:\n"
  x/2gx $x29
  
  printf "\nPotential Return Path:\n"
  x/3i $x30-8
  
  printf "\nStack Top:\n"
  x/8gx $sp
  
  printf "\n"
end

set pagination off
set print pretty on

printf "AArch64 debugging helpers loaded.\n"
printf "Commands: arm64-regs, arm64-args, arm64-frame-walk,\n"
printf "          arm64-stack-scan, arm64-check-null-deref,\n"
printf "          arm64-crash-summary\n"

Load with:

gdb -x ~/.gdbinit-aarch64 /path/to/binary core

Quick Reference

First Steps:

arm64-crash-summary        # Quick overview
info threads               # Check all threads
thread apply all bt        # All backtraces

Manual Analysis:

arm64-check-null-deref     # Check for null pointer
arm64-frame-walk           # Walk call stack
arm64-stack-scan           # Find return addresses

Memory Investigation:

x/20i $pc-40              # Context around crash
x/s $x0                    # Check string args
x/40gx $sp                 # Stack dump
info proc mappings         # Memory layout

AArch64-Specific Considerations

Position Independent Executable (PIE)

# AArch64 heavily uses ADRP for PIC
adrp x0, <page>            # Get page address
add x0, x0, #<pageoff>     # Add page offset
ldr x1, [x0]               # Load from computed address

# To find what data/string this references:
# 1. Execute the adrp+add mentally or in calculator
# 2. Or step through and examine x0 after

NEON/SIMD Crashes

# If crash is in SIMD code
info all-registers         # Show v0-v31
info vector                # SIMD register state

# SIMD load/store
ld1 {v0.16b}, [x0]         # Load 16 bytes to v0
st1 {v1.8h}, [x1]          # Store 8 halfwords from v1

# Check alignment for SIMD operations
print $x0 & 15             # Should be 0 for 128-bit ops

Thread Local Storage (TLS)

# AArch64 uses special register for TLS
# tpidr_el0 on Linux (not directly accessible in GDB)

# TLS access pattern:
mrs x0, tpidr_el0          # Read thread pointer
ldr x1, [x0, #<offset>]    # Access TLS variable

# In core dump, TLS data may be visible in thread-specific stack

This guide should equip you to debug any AArch64 core dump without symbols. The key differences from x86-64 are the calling convention, register layout, and instruction encoding, but the fundamental principles remain the same: understand the architecture, read assembly, and manually reconstruct program state.

Expert Guide to GDB Scripting: Python & Internal Commands

Part 1: GDB Internal Scripting

Basic Command Scripts

GDB has its own command language for creating custom commands and automating tasks.

Defining Custom Commands

# Basic command definition
define mycommand
  # Commands go here
  info registers
  backtrace
end

# Command with documentation
define mycommand
  document mycommand
    Custom command that shows registers and backtrace.
    Usage: mycommand
  end
  info registers
  backtrace
end

# Command with arguments
define print_array
  set $i = 0
  while $i < $arg0
    print array[$i]
    set $i = $i + 1
  end
end

# Usage: print_array 10

Arguments in Commands

# $arg0, $arg1, etc. are the arguments
define dump_memory
  dump binary memory $arg0 $arg1 $arg2
end
# Usage: dump_memory output.bin 0x400000 0x401000

# With documentation
define dump_memory
  document dump_memory
    Dump memory region to file.
    Usage: dump_memory <filename> <start_addr> <end_addr>
  end
  if $argc != 3
    printf "Usage: dump_memory <filename> <start_addr> <end_addr>\n"
  else
    dump binary memory $arg0 $arg1 $arg2
    printf "Dumped memory from %p to %p into %s\n", $arg1, $arg2, $arg0
  end
end

Control Flow

# Conditional execution
define check_null
  if $arg0 == 0
    printf "Pointer is NULL!\n"
  else
    printf "Pointer: %p\n", $arg0
    x/x $arg0
  end
end

# While loops
define print_list
  set $node = $arg0
  while $node != 0
    printf "Node: %p, data: %d\n", $node, $node->data
    set $node = $node->next
  end
end

# Loop with counter
define hexdump
  set $addr = $arg0
  set $count = $arg1
  set $i = 0
  while $i < $count
    printf "%p: ", $addr + $i
    x/x $addr + $i
    set $i = $i + 1
  end
end

Convenience Variables

# User-defined convenience variables start with $
define save_context
  set $saved_rip = $rip
  set $saved_rsp = $rsp
  set $saved_rbp = $rbp
  printf "Context saved\n"
end

define restore_context
  set $rip = $saved_rip
  set $rsp = $saved_rsp
  set $rbp = $saved_rbp
  printf "Context restored\n"
end

# Persistent variables across debugging sessions
set $myvar = 42
show convenience  # Show all convenience variables

Hooks

# Hooks run before/after commands
define hook-stop
  # Runs every time execution stops
  printf "\n=== Stopped ===\n"
  x/i $pc
  printf "===============\n"
end

define hook-next
  # Runs before 'next' command
  printf "About to execute next...\n"
end

define hookpost-next
  # Runs after 'next' command
  printf "Completed next\n"
  info registers rax rbx rcx
end

# Hook for specific breakpoints
define hook-break
  # Runs before setting breakpoint
  printf "Setting breakpoint...\n"
end

Output and Formatting

define show_state
  printf "PC: 0x%016lx\n", $pc
  printf "SP: 0x%016lx\n", $sp
  printf "Frame: "
  info frame
  
  # Output redirection
  set logging file debug.log
  set logging on
  backtrace full
  set logging off
end

# Formatted output
define print_struct
  printf "struct {\n"
  printf "  field1: %d\n", $arg0->field1
  printf "  field2: 0x%x\n", $arg0->field2
  printf "  field3: %s\n", $arg0->field3
  printf "}\n"
end

# Conditional output
define verbose_print
  if $verbose_mode
    printf "Debug: value = %d\n", $arg0
  end
end

Advanced Internal Scripting

# Error handling
define safe_deref
  # GDB doesn't have try-catch, but we can check
  set $valid = 1
  
  # Check if address looks valid
  if $arg0 < 0x1000
    printf "Invalid pointer: %p\n", $arg0
    set $valid = 0
  end
  
  if $valid
    x/x $arg0
  end
end

# Function-like behavior
define calculate
  set $result = $arg0 + $arg1 * $arg2
  # Return via convenience variable
  set $retval = $result
end

# Call it
calculate 10 20 30
printf "Result: %d\n", $retval

# Complex data structure traversal
define walk_binary_tree
  set $node = $arg0
  
  if $node != 0
    # Pre-order traversal
    printf "Value: %d\n", $node->value
    walk_binary_tree $node->left
    walk_binary_tree $node->right
  end
end

# Breakpoint management
define set_conditional_break
  break $arg0 if $arg1 > $arg2
  printf "Conditional breakpoint set at %s\n", $arg0
end

Command Files

# Save to file: startup.gdb
set pagination off
set print pretty on
set history save on
set history filename ~/.gdb_history

define init
  printf "Custom GDB environment loaded\n"
  source ~/project/.gdbinit
end

# Auto-run on startup
init

# Load with: gdb -x startup.gdb

Part 2: Python Scripting in GDB

Basic Python Interface

# Single-line Python
(gdb) python print("Hello from Python")

# Multi-line Python
(gdb) python
>import sys
>print(f"Python version: {sys.version}")
>for i in range(5):
>    print(f"Count: {i}")
>end

# Or in script files
python
import gdb

class MyCommand(gdb.Command):
    """My custom command"""
    
    def __init__(self):
        super().__init__("mypy", gdb.COMMAND_USER)
    
    def invoke(self, args, from_tty):
        print("Hello from Python command!")

MyCommand()
end

GDB Python API - Core Concepts

Accessing Inferior (Debugged Program)

python

import gdb

# Get the current inferior
inferior = gdb.selected_inferior()

# Read memory
address = 0x400000
size = 100
memory = inferior.read_memory(address, size)
# Returns memoryview object
data = bytes(memory)
print(f"First byte: {data[0]:02x}")

# Write memory
new_data = b'\x90\x90\x90\x90'  # NOP instructions
inferior.write_memory(0x400500, new_data)

# Check if program is running
if inferior.is_valid():
    print("Inferior is valid")

# Get all inferiors (for multi-process debugging)
for inf in gdb.inferiors():
    print(f"Inferior {inf.num}: PID={inf.pid}")

end

Working with Values

python

import gdb

# Parse and evaluate expressions
value = gdb.parse_and_eval("my_variable")
ptr = gdb.parse_and_eval("my_pointer")
expr = gdb.parse_and_eval("x + y * 2")

# Get value as Python types
int_val = int(value)
str_val = str(value)
float_val = float(value)

# Dereference pointers
dereferenced = ptr.dereference()

# Access struct members
struct_val = gdb.parse_and_eval("my_struct")
field = struct_val['field_name']

# Cast values
casted = value.cast(gdb.lookup_type("int *"))

# Array access
array = gdb.parse_and_eval("my_array")
element = array[5]

# Value attributes
print(f"Type: {value.type}")
print(f"Address: {value.address}")
print(f"Is optimized out: {value.is_optimized_out}")

# Arithmetic operations
result = value + 10
result = ptr + 4  # Pointer arithmetic

end

Type System

python

import gdb

# Lookup types
int_type = gdb.lookup_type("int")
struct_type = gdb.lookup_type("struct my_struct")
ptr_type = gdb.lookup_type("char").pointer()

# Type properties
print(f"Type name: {struct_type.name}")
print(f"Type code: {struct_type.code}")  # TYPE_CODE_STRUCT, etc.
print(f"Type size: {struct_type.sizeof}")

# Type codes
# gdb.TYPE_CODE_PTR - pointer
# gdb.TYPE_CODE_ARRAY - array
# gdb.TYPE_CODE_STRUCT - struct
# gdb.TYPE_CODE_UNION - union
# gdb.TYPE_CODE_ENUM - enum
# gdb.TYPE_CODE_FUNC - function
# gdb.TYPE_CODE_INT - integer
# gdb.TYPE_CODE_FLT - float

# Navigate type properties
if struct_type.code == gdb.TYPE_CODE_STRUCT:
    for field in struct_type.fields():
        print(f"Field: {field.name}, offset: {field.bitpos//8}")

# Pointer type operations
base_type = ptr_type.target()  # Get pointed-to type
array_type = int_type.array(10)  # Create array type

# Type comparison
if value.type == int_type:
    print("Value is int")

# Type casting
value_as_ptr = gdb.parse_and_eval("&my_var")
as_long_ptr = value_as_ptr.cast(gdb.lookup_type("long").pointer())

end

Frames and Backtraces

python

import gdb

# Get current frame
frame = gdb.selected_frame()

# Frame information
print(f"Frame name: {frame.name()}")
print(f"PC: {hex(frame.pc())}")
print(f"Frame level: {frame.level()}")

# Get older/newer frames
older = frame.older()
newer = frame.newer()

# Navigate all frames
frame = gdb.newest_frame()
while frame:
    print(f"Level {frame.level()}: {frame.name()} at {hex(frame.pc())}")
    frame = frame.older()

# Read variables from frame
try:
    var = frame.read_var("local_variable")
    print(f"Variable value: {var}")
except ValueError as e:
    print(f"Variable not found: {e}")

# Read registers in frame
rip = frame.read_register("rip")
rsp = frame.read_register("rsp")

# Get all registers
arch = frame.architecture()
for reg in arch.registers():
    try:
        value = frame.read_register(reg)
        print(f"{reg.name}: {value}")
    except:
        pass

# Select frame
gdb.execute("frame 2")  # Switch to frame 2

# Block and symbol information
block = frame.block()
if block:
    print(f"Block start: {hex(block.start)}")
    print(f"Block end: {hex(block.end)}")
    
    # Iterate symbols in block
    for symbol in block:
        print(f"Symbol: {symbol.name}")

end

Breakpoints

python

import gdb

# Create simple breakpoint
bp = gdb.Breakpoint("main")
bp = gdb.Breakpoint("file.cpp:42")
bp = gdb.Breakpoint("*0x400500")  # Address breakpoint

# Breakpoint properties
bp.enabled = True
bp.silent = True  # Don't print when hit
bp.ignore_count = 5  # Ignore first 5 hits
bp.condition = "x > 100"  # Conditional breakpoint

# Watchpoints
wp = gdb.Breakpoint("my_variable", gdb.BP_WATCHPOINT)
rwp = gdb.Breakpoint("my_variable", gdb.BP_READ_WATCHPOINT)
awp = gdb.Breakpoint("my_variable", gdb.BP_ACCESS_WATCHPOINT)

# Custom breakpoint with callback
class MyBreakpoint(gdb.Breakpoint):
    def __init__(self, spec):
        super().__init__(spec)
        self.hit_count = 0
    
    def stop(self):
        self.hit_count += 1
        frame = gdb.selected_frame()
        
        # Access variables
        try:
            x = frame.read_var("x")
            print(f"Hit #{self.hit_count}: x = {x}")
            
            # Conditional stop
            if int(x) > 100:
                print("x exceeded 100, stopping")
                return True  # Stop execution
            else:
                return False  # Continue execution
        except:
            return True

# Create custom breakpoint
mybp = MyBreakpoint("function_name")

# Breakpoint with commands
class CommandBreakpoint(gdb.Breakpoint):
    def stop(self):
        gdb.execute("info locals")
        gdb.execute("backtrace 3")
        return True

# Temporary breakpoint
temp_bp = gdb.Breakpoint("init", temporary=True)

# Get all breakpoints
for bp in gdb.breakpoints():
    print(f"Breakpoint {bp.number}: {bp.location}")
    bp.delete()  # Delete breakpoint

end

Threads

python

import gdb

# Get all threads
for thread in gdb.selected_inferior().threads():
    print(f"Thread {thread.num}: {thread.name}")
    print(f"  PTID: {thread.ptid}")
    print(f"  Is stopped: {thread.is_stopped()}")
    print(f"  Is running: {thread.is_running()}")

# Switch thread
thread = gdb.selected_inferior().threads()[0]
thread.switch()

# Or use thread number
gdb.execute("thread 2")

# Get selected thread
current = gdb.selected_thread()
if current:
    print(f"Current thread: {current.num}")
    
    # Get thread-specific inferior
    inferior = current.inferior()

# Iterate threads and get backtraces
for thread in gdb.selected_inferior().threads():
    thread.switch()
    print(f"\n=== Thread {thread.num} ===")
    
    frame = gdb.newest_frame()
    while frame:
        print(f"  {frame.name()}")
        frame = frame.older()

end

Advanced Python Scripting

Custom Commands

python

import gdb

class DumpStructCommand(gdb.Command):
    """Dump a C structure with formatting.
    Usage: dump-struct <expression>
    """
    
    def __init__(self):
        super().__init__("dump-struct", gdb.COMMAND_DATA)
    
    def invoke(self, args, from_tty):
        try:
            value = gdb.parse_and_eval(args)
            self.dump_value(value, 0)
        except Exception as e:
            print(f"Error: {e}")
    
    def dump_value(self, value, indent):
        indent_str = "  " * indent
        vtype = value.type
        
        # Strip typedefs and references
        while vtype.code == gdb.TYPE_CODE_TYPEDEF or \
              vtype.code == gdb.TYPE_CODE_REF:
            vtype = vtype.target()
        
        if vtype.code == gdb.TYPE_CODE_STRUCT:
            print(f"{indent_str}{{")
            for field in vtype.fields():
                if field.is_base_class:
                    continue
                try:
                    field_val = value[field.name]
                    print(f"{indent_str}  {field.name}: ", end="")
                    self.dump_value(field_val, indent + 1)
                except:
                    print(f"{indent_str}  {field.name}: <error>")
            print(f"{indent_str}}}")
        
        elif vtype.code == gdb.TYPE_CODE_PTR:
            try:
                addr = int(value)
                print(f"0x{addr:x}", end="")
                if addr != 0:
                    deref = value.dereference()
                    print(" -> ", end="")
                    self.dump_value(deref, indent)
                else:
                    print(" (NULL)")
            except:
                print("<invalid pointer>")
        
        elif vtype.code == gdb.TYPE_CODE_ARRAY:
            length = vtype.range()[1] - vtype.range()[0] + 1
            print(f"[{length}] {{")
            for i in range(min(length, 10)):  # Limit to 10 elements
                print(f"{indent_str}  [{i}]: ", end="")
                self.dump_value(value[i], indent + 1)
            if length > 10:
                print(f"{indent_str}  ... ({length - 10} more)")
            print(f"{indent_str}}}")
        
        else:
            print(value)
    
    def complete(self, text, word):
        # Tab completion
        return gdb.COMPLETE_SYMBOL

DumpStructCommand()

end

Pretty Printers

python

import gdb
import gdb.printing

class VectorPrinter:
    """Pretty printer for std::vector"""
    
    def __init__(self, val):
        self.val = val
    
    def to_string(self):
        start = self.val['_M_impl']['_M_start']
        finish = self.val['_M_impl']['_M_finish']
        end_storage = self.val['_M_impl']['_M_end_of_storage']
        
        size = int(finish - start)
        capacity = int(end_storage - start)
        
        return f"std::vector of length {size}, capacity {capacity}"
    
    def children(self):
        start = self.val['_M_impl']['_M_start']
        finish = self.val['_M_impl']['_M_finish']
        
        size = int(finish - start)
        for i in range(size):
            element = (start + i).dereference()
            yield (f"[{i}]", element)
    
    def display_hint(self):
        return 'array'

class LinkedListPrinter:
    """Pretty printer for custom linked list"""
    
    def __init__(self, val):
        self.val = val
    
    def to_string(self):
        count = 0
        node = self.val['head']
        while node != 0:
            count += 1
            node = node['next']
            if count > 1000:  # Prevent infinite loops
                break
        return f"LinkedList with {count} nodes"
    
    def children(self):
        node = self.val['head']
        index = 0
        while node != 0:
            yield (f"[{index}]", node['data'])
            node = node['next']
            index += 1
            if index > 100:  # Limit display
                break

# Register pretty printers
def build_pretty_printer():
    pp = gdb.printing.RegexpCollectionPrettyPrinter("my_printers")
    
    # Register by type name (regex)
    pp.add_printer('vector', '^std::vector<.*>$', VectorPrinter)
    pp.add_printer('list', '^LinkedList$', LinkedListPrinter)
    
    return pp

gdb.printing.register_pretty_printer(
    gdb.current_objfile(),
    build_pretty_printer(),
    replace=True
)

# Or register globally
# gdb.printing.register_pretty_printer(None, build_pretty_printer())

end

Memory Analysis

python

import gdb
import struct

class MemoryAnalyzer(gdb.Command):
    """Analyze memory regions.
    Usage: memanalyze <start> <size>
    """
    
    def __init__(self):
        super().__init__("memanalyze", gdb.COMMAND_DATA)
    
    def invoke(self, args, from_tty):
        args_list = gdb.string_to_argv(args)
        if len(args_list) != 2:
            print("Usage: memanalyze <start_address> <size>")
            return
        
        start = int(gdb.parse_and_eval(args_list[0]))
        size = int(args_list[1])
        
        # Read memory
        inferior = gdb.selected_inferior()
        try:
            memory = inferior.read_memory(start, size)
            data = bytes(memory)
        except gdb.MemoryError as e:
            print(f"Memory read error: {e}")
            return
        
        self.analyze(data, start)
    
    def analyze(self, data, start_addr):
        print(f"Memory analysis from 0x{start_addr:x}:")
        print(f"  Size: {len(data)} bytes")
        
        # Check for patterns
        null_bytes = data.count(0)
        print(f"  NULL bytes: {null_bytes} ({null_bytes*100//len(data)}%)")
        
        # Look for ASCII strings
        strings = self.find_strings(data, start_addr)
        if strings:
            print(f"  Found {len(strings)} strings:")
            for addr, s in strings[:5]:  # Show first 5
                print(f"    0x{addr:x}: {s}")
        
        # Look for pointers (assuming 64-bit)
        pointers = self.find_pointers(data, start_addr)
        if pointers:
            print(f"  Found {len(pointers)} potential pointers")
        
        # Entropy check (simple)
        entropy = self.calculate_entropy(data)
        print(f"  Entropy: {entropy:.2f}")
    
    def find_strings(self, data, base_addr, min_length=4):
        strings = []
        current = []
        current_start = 0
        
        for i, byte in enumerate(data):
            if 32 <= byte < 127:  # Printable ASCII
                if not current:
                    current_start = i
                current.append(chr(byte))
            else:
                if len(current) >= min_length:
                    s = ''.join(current)
                    strings.append((base_addr + current_start, s))
                current = []
        
        if len(current) >= min_length:
            s = ''.join(current)
            strings.append((base_addr + current_start, s))
        
        return strings
    
    def find_pointers(self, data, base_addr):
        pointers = []
        
        # Unpack as 64-bit values
        for i in range(0, len(data) - 7, 8):
            value = struct.unpack('<Q', data[i:i+8])[0]
            
            # Check if looks like a pointer (heuristic)
            # Typical userspace: 0x00005500_00000000 - 0x00007fff_ffffffff
            if 0x0000550000000000 <= value <= 0x00007fffffffffff:
                pointers.append((base_addr + i, value))
        
        return pointers
    
    def calculate_entropy(self, data):
        if not data:
            return 0
        
        # Simple byte frequency entropy
        freq = [0] * 256
        for byte in data:
            freq[byte] += 1
        
        import math
        entropy = 0
        for count in freq:
            if count > 0:
                p = count / len(data)
                entropy -= p * math.log2(p)
        
        return entropy

MemoryAnalyzer()

end

Heap Analysis

python

import gdb
import struct

class HeapWalker(gdb.Command):
    """Walk glibc heap and show allocations.
    Usage: heapwalk [arena_address]
    """
    
    def __init__(self):
        super().__init__("heapwalk", gdb.COMMAND_DATA)
    
    def invoke(self, args, from_tty):
        # This is a simplified version for glibc malloc
        # Real implementation would need to handle all chunk types
        
        # Find heap region
        inferior = gdb.selected_inferior()
        
        # Get process mappings
        try:
            mappings = gdb.execute("info proc mappings", to_string=True)
        except:
            print("Cannot get process mappings")
            return
        
        # Parse mappings to find [heap]
        heap_start = None
        heap_end = None
        for line in mappings.split('\n'):
            if '[heap]' in line:
                parts = line.split()
                if len(parts) >= 2:
                    addrs = parts[0].split('-')
                    if len(addrs) == 2:
                        heap_start = int(addrs[0], 16)
                        heap_end = int(addrs[1], 16)
                        break
        
        if not heap_start:
            print("Heap not found")
            return
        
        print(f"Heap: 0x{heap_start:x} - 0x{heap_end:x}")
        self.walk_heap(heap_start, heap_end)
    
    def walk_heap(self, start, end):
        # Simplified glibc chunk header:
        # struct malloc_chunk {
        #   size_t prev_size;  // Only if previous chunk is free
        #   size_t size;       // Size with flags in low 3 bits
        #   // For free chunks:
        #   struct malloc_chunk *fd;
        #   struct malloc_chunk *bk;
        # };
        
        PREV_INUSE = 1
        IS_MMAPPED = 2
        NON_MAIN_ARENA = 4
        
        inferior = gdb.selected_inferior()
        ptr = start
        chunk_num = 0
        
        print("\nChunk  Address           Size       Flags  Status")
        print("=" * 70)
        
        while ptr < end and chunk_num < 100:  # Limit iterations
            try:
                # Read chunk header (16 bytes)
                header = inferior.read_memory(ptr, 16)
                prev_size, size = struct.unpack('<QQ', bytes(header))
                
                # Extract flags
                flags = size & 7
                actual_size = size & ~7
                
                # Determine status
                status = "ALLOCATED" if (flags & PREV_INUSE) else "FREE"
                
                flags_str = ""
                if flags & PREV_INUSE:
                    flags_str += "P"
                if flags & IS_MMAPPED:
                    flags_str += "M"
                if flags & NON_MAIN_ARENA:
                    flags_str += "N"
                
                print(f"{chunk_num:5d}  0x{ptr:016x}  {actual_size:8d}   "
                      f"{flags_str:4s}  {status}")
                
                # Try to show some data for allocated chunks
                if status == "ALLOCATED" and actual_size >= 16:
                    try:
                        data = inferior.read_memory(ptr + 16, min(32, actual_size - 16))
                        data_bytes = bytes(data)
                        
                        # Check if looks like a string
                        if all(32 <= b < 127 or b == 0 for b in data_bytes[:16]):
                            try:
                                s = data_bytes.decode('utf-8', errors='ignore')
                                s = s.split('\0')[0]
                                if s:
                                    print(f"       Data: {s[:50]}")
                            except:
                                pass
                    except:
                        pass
                
                # Move to next chunk
                if actual_size == 0:
                    break
                ptr += actual_size
                chunk_num += 1
                
            except gdb.MemoryError:
                print(f"       <memory read error at 0x{ptr:x}>")
                break
            except Exception as e:
                print(f"       <error: {e}>")
                break
        
        print("=" * 70)
        print(f"Total chunks analyzed: {chunk_num}")

HeapWalker()

end

Call Stack Analysis

python

import gdb
import re

class SmartBacktrace(gdb.Command):
    """Enhanced backtrace with argument detection.
    Usage: sbt
    """
    
    def __init__(self):
        super().__init__("sbt", gdb.COMMAND_STACK)
    
    def invoke(self, args, from_tty):
        frame = gdb.newest_frame()
        level = 0
        
        print("Level  Function                               Arguments")
        print("=" * 80)
        
        while frame:
            try:
                name = frame.name() or "??"
                pc = frame.pc()
                
                # Try to get function arguments
                args_str = self.get_arguments(frame)
                
                print(f"{level:5d}  {name:40s} {args_str}")
                
                level += 1
                frame = frame.older()
                
            except Exception as e:
                print(f"{level:5d}  <error: {e}>")
                break
        
        print("=" * 80)
    
    def get_arguments(self, frame):
        """Try to determine function arguments"""
        arch = frame.architecture().name()
        
        if 'x86-64' in arch or 'i386:x86-64' in arch:
            return self.get_x86_64_args(frame)
        elif 'aarch64' in arch:
            return self.get_aarch64_args(frame)
        else:
            return ""
    
    def get_x86_64_args(self, frame):
        """Get arguments for x86-64 calling convention"""
        arg_regs = ['rdi', 'rsi', 'rdx', 'rcx', 'r8', 'r9']
        args = []
        
        for i, reg in enumerate(arg_regs):
            try:
                value = frame.read_register(reg)
                val_int = int(value)
                
                # Try to determine type
                arg_str = self.format_value(val_int, frame)
                args.append(f"arg{i}={arg_str}")
                
                if len(args) >= 4:  # Limit to 4 args for display
                    break
            except:
                break
        
        return ", ".join(args)
    
    def get_aarch64_args(self, frame):
        """Get arguments for AArch64 calling convention"""
        arg_regs = ['x0', 'x1', 'x2', 'x3', 'x4', 'x5', 'x6', 'x7']
        args = []
        
        for i, reg in enumerate(arg_regs):
            try:
                value = frame.read_register(reg)
                val_int = int(value)
                
                arg_str = self.format_value(val_int, frame)
                args.append(f"arg{i}={arg_str}")
                
                if len(args) >= 4:
                    break
            except:
                break
        
        return ", ".join(args)
    
    def format_value(self, value, frame):
        """Try to format a value intelligently"""
        # Check if NULL
        if value == 0:
            return "NULL"
        
        # Check if looks like a pointer
        if value > 0x1000:
            try:
                # Try to read as string
                inferior = gdb.selected_inferior()
                data = inferior.read_memory(value, 32)
                
                # Check if printable ASCII
                is_string = True
                for i, byte in enumerate(bytes(data)):
                    if byte == 0:
                        break
                    if not (32 <= byte < 127):
                        is_string = False
                        break
                
                if is_string:
                    s = bytes(data).split(b'\0')[0].decode('utf-8', errors='ignore')
                    if len(s) > 20:
                        s = s[:20] + "..."
                    return f'"{s}"'
            except:
                pass
        
        # Just return as hex
        return f"0x{value:x}"

SmartBacktrace()

end

Watchpoint with History

python

import gdb

class WatchHistory:
    """Track watchpoint hits and show history"""
    
    def __init__(self):
        self.history = []
        self.max_history = 100
    
    def record(self, variable, old_val, new_val, location):
        import time
        entry = {
            'time': time.time(),
            'variable': variable,
            'old_value': old_val,
            'new_value': new_val,
            'location': location,
            'backtrace': self.get_backtrace()
        }
        
        self.history.append(entry)
        if len(self.history) > self.max_history:
            self.history.pop(0)
    
    def get_backtrace(self):
        bt = []
        frame = gdb.newest_frame()
        while frame and len(bt) < 5:
            name = frame.name() or "??"
            bt.append(name)
            frame = frame.older()
        return bt
    
    def show_history(self, count=10):
        print(f"\nWatch History (last {count}):")
        print("=" * 80)
        
        for entry in self.history[-count:]:
            print(f"\nVariable: {entry['variable']}")
            print(f"  Old value: {entry['old_value']}")
            print(f"  New value: {entry['new_value']}")
            print(f"  Location: {entry['location']}")
            print(f"  Backtrace: {' -> '.join(entry['backtrace'])}")

# Global history object
watch_history = WatchHistory()

class TrackedWatchpoint(gdb.Breakpoint):
    """Watchpoint that tracks history of changes"""
    
    def __init__(self, variable):
        super().__init__(variable, gdb.BP_WATCHPOINT)
        self.variable = variable
        self.last_value = None
    
    def stop(self):
        global watch_history
        
        try:
            frame = gdb.selected_frame()
            current_value = gdb.parse_and_eval(self.variable)
            
            if self.last_value is not None:
                location = f"{frame.name()}:{frame.pc():#x}"
                watch_history.record(
                    self.variable,
                    str(self.last_value),
                    str(current_value),
                    location
                )
            
            self.last_value = current_value
            
            print(f"\nWatchpoint: {self.variable} changed")
            print(f"  New value: {current_value}")
            watch_history.show_history(5)
            
            return True
        except Exception as e:
            print(f"Error in watchpoint: {e}")
            return True

class WatchHistoryCommand(gdb.Command):
    """Show watchpoint history.
    Usage: watch-history [count]
    """
    
    def __init__(self):
        super().__init__("watch-history", gdb.COMMAND_DATA)
    
    def invoke(self, args, from_tty):
        global watch_history
        
        count = 10
        if args:
            try:
                count = int(args)
            except:
                print("Usage: watch-history [count]")
                return
        
        watch_history.show_history(count)

WatchHistoryCommand()

# Example usage:
# python TrackedWatchpoint("my_variable")

end

Event Handling

python

import gdb

# Stop event handlers
def on_stop(event):
    """Called whenever execution stops"""
    if isinstance(event, gdb.SignalEvent):
        print(f"Stopped by signal: {event.stop_signal}")
    elif isinstance(event, gdb.BreakpointEvent):
        print(f"Stopped at breakpoint")
        for bp in event.breakpoints:
            print(f"  Breakpoint {bp.number} at {bp.location}")
    
    # Show current location
    frame = gdb.selected_frame()
    if frame:
        print(f"Stopped at: {frame.name()}")

gdb.events.stop.connect(on_stop)

# Continue event
def on_continue(event):
    print("Execution continuing...")

gdb.events.cont.connect(on_continue)

# Exited event
def on_exit(event):
    print(f"Program exited with code: {event.exit_code}")

gdb.events.exited.connect(on_exit)

# New thread event
def on_new_thread(event):
    thread = event.inferior_thread
    print(f"New thread created: {thread.num}")

gdb.events.new_thread.connect(on_new_thread)

# Breakpoint created
def on_breakpoint_created(bp):
    print(f"Breakpoint created: #{bp.number} at {bp.location}")

gdb.events.breakpoint_created.connect(on_breakpoint_created)

# Breakpoint deleted
def on_breakpoint_deleted(bp):
    print(f"Breakpoint deleted: #{bp.number}")

gdb.events.breakpoint_deleted.connect(on_breakpoint_deleted)

# New objfile (library) loaded
def on_new_objfile(event):
    objfile = event.new_objfile
    print(f"New object file loaded: {objfile.filename}")

gdb.events.new_objfile.connect(on_new_objfile)

# Clear inferior (when debugging ends)
def on_clear_objfiles(event):
    print("Object files cleared")

gdb.events.clear_objfiles.connect(on_clear_objfiles)

# Disconnect event handler
# gdb.events.stop.disconnect(on_stop)

end

Complex Example: Automatic Leak Detection

python

import gdb

class LeakDetector:
    """Track malloc/free calls to detect memory leaks"""
    
    def __init__(self):
        self.allocations = {}  # address -> (size, backtrace)
        self.malloc_bp = None
        self.free_bp = None
        self.enabled = False
    
    def start(self):
        """Start tracking allocations"""
        if self.enabled:
            print("Already tracking")
            return
        
        # Set breakpoints on malloc and free
        try:
            self.malloc_bp = MallocBreakpoint(self)
            self.free_bp = FreeBreakpoint(self)
            self.enabled = True
            print("Leak detection started")
        except Exception as e:
            print(f"Failed to start tracking: {e}")
    
    def stop(self):
        """Stop tracking and report leaks"""
        if not self.enabled:
            print("Not tracking")
            return
        
        if self.malloc_bp:
            self.malloc_bp.delete()
        if self.free_bp:
            self.free_bp.delete()
        
        self.enabled = False
        self.report_leaks()
    
    def record_malloc(self, address, size):
        """Record a malloc"""
        bt = self.get_backtrace()
        self.allocations[address] = (size, bt)
        print(f"Malloc: 0x{address:x} size={size}")
    
    def record_free(self, address):
        """Record a free"""
        if address in self.allocations:
            size, bt = self.allocations[address]
            del self.allocations[address]
            print(f"Free: 0x{address:x} size={size}")
        else:
            if address != 0:  # free(NULL) is valid
                print(f"WARNING: Free of untracked address: 0x{address:x}")
    
    def get_backtrace(self):
        """Get current backtrace"""
        bt = []
        frame = gdb.newest_frame()
        while frame:
            name = frame.name() or "??"
            bt.append(f"{name}:0x{frame.pc():x}")
            frame = frame.older()
            if len(bt) >= 10:  # Limit depth
                break
        return bt
    
    def report_leaks(self):
        """Report any leaked memory"""
        if not self.allocations:
            print("\nNo leaks detected!")
            return
        
        print(f"\n{'='*80}")
        print(f"MEMORY LEAKS DETECTED: {len(self.allocations)} allocations")
        print(f"{'='*80}")
        
        total_leaked = 0
        for addr, (size, bt) in self.allocations.items():
            total_leaked += size
            print(f"\nLeak: 0x{addr:x}, size={size} bytes")
            print("  Allocated at:")
            for frame in bt:
                print(f"    {frame}")
        
        print(f"\n{'='*80}")
        print(f"Total leaked: {total_leaked} bytes")
        print(f"{'='*80}")

class MallocBreakpoint(gdb.FinishBreakpoint):
    """Breakpoint on malloc that captures return value"""
    
    def __init__(self, detector):
        # Find malloc
        try:
            super().__init__(gdb.newest_frame(), internal=True)
            self.detector = detector
            self.silent = True
        except Exception as e:
            # Fallback: set regular breakpoint
            super().__init__()
            self.spec = "malloc"
            self.detector = detector
    
    def stop(self):
        try:
            # Get return value (allocated address)
            addr = int(self.return_value)
            
            # Get size argument (first argument, in rdi on x86-64)
            frame = gdb.selected_frame()
            size = int(frame.read_register("rdi"))
            
            if addr != 0:
                self.detector.record_malloc(addr, size)
        except Exception as e:
            print(f"Error in malloc breakpoint: {e}")
        
        return False  # Continue execution

class FreeBreakpoint(gdb.Breakpoint):
    """Breakpoint on free"""
    
    def __init__(self, detector):
        super().__init__("free", internal=True)
        self.detector = detector
        self.silent = True
    
    def stop(self):
        try:
            # Get address argument (first argument)
            frame = gdb.selected_frame()
            addr = int(frame.read_register("rdi"))
            
            self.detector.record_free(addr)
        except Exception as e:
            print(f"Error in free breakpoint: {e}")
        
        return False  # Continue execution

# Global leak detector
leak_detector = LeakDetector()

class LeakDetectCommand(gdb.Command):
    """Control leak detection.
    Usage: leak-detect [start|stop]
    """
    
    def __init__(self):
        super().__init__("leak-detect", gdb.COMMAND_DATA)
    
    def invoke(self, args, from_tty):
        global leak_detector
        
        if args == "start":
            leak_detector.start()
        elif args == "stop":
            leak_detector.stop()
        else:
            print("Usage: leak-detect [start|stop]")

LeakDetectCommand()

end

Useful Utility Functions

python

import gdb

def read_string(addr, max_len=1024):
    """Read a null-terminated string from memory"""
    inferior = gdb.selected_inferior()
    try:
        data = inferior.read_memory(addr, max_len)
        # Find null terminator
        for i, byte in enumerate(bytes(data)):
            if byte == 0:
                return bytes(data[:i]).decode('utf-8', errors='ignore')
        return bytes(data).decode('utf-8', errors='ignore')
    except:
        return None

def is_valid_address(addr):
    """Check if address is valid/mapped"""
    try:
        inferior = gdb.selected_inferior()
        inferior.read_memory(addr, 1)
        return True
    except gdb.MemoryError:
        return False

def get_register_size():
    """Get register size (4 for 32-bit, 8 for 64-bit)"""
    return gdb.parse_and_eval("sizeof(void*)").type.sizeof

def follow_pointer(ptr, depth=1):
    """Follow pointer chain"""
    current = ptr
    for i in range(depth):
        try:
            val = gdb.parse_and_eval(f"*(void**){current}")
            current = int(val)
            print(f"Level {i}: 0x{current:x}")
        except:
            print(f"Level {i}: invalid")
            break

def find_containing_function(addr):
    """Find which function contains an address"""
    try:
        block = gdb.block_for_pc(addr)
        while block and not block.function:
            block = block.superblock
        if block and block.function:
            return block.function.name
    except:
        pass
    return None

# Helper to convert between types
def cast_to(value, type_name):
    """Cast a value to a different type"""
    target_type = gdb.lookup_type(type_name)
    return value.cast(target_type)

# Example usage in GDB:
# python print(read_string(0x400500))
# python follow_pointer(0x12345678, 3)
# python print(find_containing_function(0x400123))

end

This comprehensive guide covers both GDB's internal scripting language and the powerful Python API. The Python API is generally more flexible and powerful for complex tasks, while internal scripts are lighter weight for simple automation. In practice, many advanced GDB users combine both approaches as needed.