Cannot get kdump to dump в vmcore using crashkernel
гm trying to get kernel crashdump to work properly in Ubuntu 15.10 with the запас 4.2.0-22-generic kernel. I have followed the methods described here and here precisely. But, when I manually trigger в крах путь:
echo c | sudo tee /proc/sysrq-trigger
the system крах, and reboots, but не крах output is saved in /var/crash.
Эксперт this is EC2, I don't have в read/write я утешил - I хан only get read-only я утешил output, and гm not seeing much useful output:
[ 473.666303] sysrq: SysRq : Trigger a crash
[ 473.668278] BUG: unable to handle kernel NULL pointer dereference at (null)
[ 473.671624] IP: [<ffffffff814c79e6>] sysrq_handle_crash+0x16/0x20
[ 473.672244] PGD 3e235c067 PUD 3e2351067 PMD 0
[ 473.672244] Oops: 0002 [#1] SMP
[ 473.672244] Modules linked in: isofs xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack xt_tcpudp bridge stp llc iptable_filter ip_tables x_tables ppdev xen_fbfront intel_rapl fb_sys_fops iosf_mbi input_leds serio_raw parport_pc 8250_fintek i2c_piix4 parport mac_hid autofs4 crct10dif_pclmul crc32_pclmul cirrus syscopyarea aesni_intel aes_x86_64 sysfillrect lrw sysimgblt gf128mul ttm glue_helper ablk_helper drm_kms_helper cryptd psmouse drm ixgbevf pata_acpi floppy
[ 473.672244] CPU: 3 PID: 2814 Comm: bash Not tainted 4.2.0-22-generic #27-Ubuntu
[ 473.672244] Hardware name: Xen HVM domU, BIOS 4.2.amazon 12/07/2015
[ 473.672244] task: ffff8803d1d86e00 ti: ffff8803dc414000 task.ti: ffff8803dc414000
[ 473.672244] RIP: 0010:[<ffffffff814c79e6>] [<ffffffff814c79e6>] sysrq_handle_crash+0x16/0x20
[ 473.672244] RSP: 0018:ffff8803dc417e28 EFLAGS: 00010246
[ 473.672244] RAX: 000000000000000f RBX: 0000000000000063 RCX: 0000000000000000
[ 473.672244] RDX: 0000000000000000 RSI: ffff8803ff2ce938 RDI: 0000000000000063
[ 473.672244] RBP: ffff8803dc417e28 R08: 0000000000000002 R09: 000000000000024d
[ 473.672244] R10: 000000000000a614 R11: 000000000000024d R12: 0000000000000004
[ 473.672244] R13: 0000000000000000 R14: ffffffff81cb48e0 R15: 0000000000000000
[ 473.672244] FS: 00007fecdb3ca700(0000) GS:ffff8803ff2c0000(0000) knlGS:0000000000000000
[ 473.672244] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 473.672244] CR2: 0000000000000000 CR3: 00000003e234b000 CR4: 00000000001406e0
[ 473.672244] Stack:
[ 473.672244] ffff8803dc417e58 ffffffff814c821a 0000000000000002 fffffffffffffffb
[ 473.672244] ffff8803dc417f18 0000000000000002 ffff8803dc417e78 ffffffff814c86a3
[ 473.672244] 0000000000000002 ffff8803f816c900 ffff8803dc417e98 ffffffff81266aa2
[ 473.672244] Call Trace:
[ 473.672244] [<ffffffff814c821a>] __handle_sysrq+0xea/0x140
[ 473.672244] [<ffffffff814c86a3>] write_sysrq_trigger+0x33/0x40
[ 473.672244] [<ffffffff81266aa2>] proc_reg_write+0x42/0x70
[ 473.672244] [<ffffffff811fca68>] __vfs_write+0x18/0x40
[ 473.672244] [<ffffffff811fd3f6>] vfs_write+0xa6/0x1a0
[ 473.672244] [<ffffffff810c3e21>] ? __raw_callee_save___pv_queued_spin_unlock+0x11/0x20
[ 473.672244] [<ffffffff811fe0e5>] SyS_write+0x55/0xc0
[ 473.672244] [<ffffffff8121b31f>] ? __close_fd+0x8f/0xb0
[ 473.672244] [<ffffffff817f02b2>] entry_SYSCALL_64_fastpath+0x16/0x75
[ 473.672244] Code: 45 3b 7d 34 75 e5 4c 89 ef e8 f7 f7 ff ff eb db 0f 1f 44 00 00 0f 1f 44 00 00 55 c7 05 a8 74 a2 00 01 00 00 00 48 89 e5 0f ae f8 <c6> 04 25 00 00 00 00 01 5d c3 0f 1f 44 00 00 55 48 89 e5 53 8d
[ 473.672244] RIP [<ffffffff814c79e6>] sysrq_handle_crash+0x16/0x20
[ 473.672244] RSP <ffff8803dc417e28>
[ 473.672244] CR2: 0000000000000000
[ 0.000000] Initializing cgroup subsys cpuset
[ 0.000000] Initializing cgroup subsys cpu
[ 0.000000] Initializing cgroup subsys cpuacct
[ 0.000000] Linux version 4.2.0-22-generic (buildd@lcy01-22) (gcc version 5.2.1 20151010 (Ubuntu 5.2.1-22ubuntu2) ) #27-Ubuntu SMP Thu Dec 17 22:57:08 UTC 2015 (Ubuntu 4.2.0-22.27-generic 4.2.6)
[ 0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-4.2.0-22-generic root=UUID=9bd55602-81dd-4868-8cfc-b7d63f8f8d7e ro console=tty1 console=ttyS0 crashkernel=384M
...
[ 3.021894] piix4_smbus 0000:00:01.3: SMBus base address uninitialized - upgrade BIOS or use force_addr=0xaddr
...
[ OK ] Started memcached daemon.
Starting LSB: Execute the kexec -e command to reboot system...
...
[ OK ] Started LSB: Record successful boot for GRUB.
[ OK ] Started LSB: automatic crash report generation.
[ OK ] Started LSB: Execute the kexec -e command to reboot system.
...
[ OK ] Started LSB: Load kernel image with kexec.
ondemand.service
rc-local.service
grub-common.service
Stopping LSB: Start NTP daemon...
apport.service
kexec.service
...
lxc.service
[ OK ] Started LXC Container Initialization and Autoboot Code.
Starting Container hypervisor based on LXC - boot time check...
[ 34.181647] kdump-tools[773]: Starting kdump-tools: * loaded kdump kernel
kdump-tools.service
[ OK ] Started Kernel crash dump capture service.
[ OK ] Started Container hypervisor based on LXC - boot time check.
Онлайн And then the system is fully back, with nothing in /var/crash except .lock and kexec_cmd.
гve tried crashkernel=128M, crashkernel=256M, crashkernel=384M, 512M, 256@0, 256@16M, и т.д.
I even tried enabling SSH in /etc/default/grub.d/kexec-tools.cfg, with в machine гve verified I хан access from this machine, with в учебник SSH_KEY configured that exists, works, and существуешь proper permissions set, but the переизречение machine doesn't шоу в connection attempt at all.
The output of kdump-config show looks proper:
DUMP_MODE: kdump
USE_KDUMP: 1
KDUMP_SYSCTL: kernel.panic_on_oops=1
KDUMP_COREDIR: /var/crash
crashkernel addr: 0x2c000000
SSH: ubuntu@10.xxx.xxx.xxx
SSH_KEY: /root/.ssh/id_rsa
HOSTTAG: ip
current state: ready to kdump
kexec command:
/sbin/kexec -p --command-line="BOOT_IMAGE=/boot/vmlinuz-4.2.0-22-generic root=UUID=9bd55602-81dd-4868-8cfc-b7d63f8f8d7e ro console=tty1 console=ttyS0 irqpoll maxcpus=1 nousb systemd.unit=kdump-tools.service" --initrd=/var/lib/kdump/initrd.img /var/lib/kdump/vmlinuz
However, when I manually trigger в крах путь:
echo 1 > /proc/sys/kernel/sysrq
echo c > /proc/sysrq-trigger
The system reboots and не vmcore or .crash is ever written to /var/crash or to my переизречение SSH host. The SSH host never sees в login attempt. I see the наметил output путь ec2-get-console-output -r <instance>, and the system immediately reboots эксперт shown above.
гm very very stuck trying to debug it - everything seems correct, but не крах доклад.
Now, гm not sure if this could possibly be related, but, ifquery is also crashing on startup, and therВs не .crash доклад ever, and apport isn't aware that it crashed. гve yet to see apport ever actually поверьте в .crash here. Could that possibly be what's going wrong with my крах dumps? Could anyone provide any insight into debugging this?
2 ответа
Это может быть связано с этой проблемой: https://bugs.launchpad.net/ubuntu / + источник/Linux / + ошибка/1421391
Просто идея - попробуйте отключить некоторые из модулей, наиболее интенсивно использующих память, которые не связаны с коллекцией kdump. Я видел много высокопроизводительных сетевых драйверов, вызывающих OOM на работе, в том числе и у меня дома машина имеет высококачественную графику, оба примера приводят к тому, что в kdump загружается много памяти, что приводит к нехватке памяти, в конце концов, зарезервированная память kdump составляет крошечную часть установленной оперативной памяти на хосте, поскольку она резервируется при загрузке а потом будет недоступен.
Итак, чтобы определить, какие модули потребляют много памяти:
$ lsmod | sort -nk2 -r | head
amdgpu 4116480 16
btrfs 1228800 2
kvm 655360 0
nfsv4 638976 2
drm 487424 8 gpu_sched,drm_kms_helper,amdgpu,ttm
sunrpc 380928 9 nfsv4,auth_rpcgss,lockd,rpcsec_gss_krb5,nfs
aesni_intel 372736 0
fscache 368640 2 nfsv4,nfs
nfs 299008 2 nfsv4
igb 221184 0
В моем случае amdgpu находится наверху, но у вас могут быть все модули, на которые я наткнулся на работе, например ixgbe , i40e , mlx5_core и т. Д.
Чтобы отключить их только для ядра kdump, отредактируйте / etc / default / kdump-tools и , раскомментируйте (возможно, скопируйте, а затем раскомментируйте) KDUMP_CMDLINE_APPEND , затем добавьте драйверы в черный список. Некоторые могут быть в ядре, некоторые в initrd, поэтому для уверенности добавьте каждый как $ driver_name.blacklist = 1 и rd.driver.blacklist = $ driver_name в этой строке. , это пример с amdgpu ниже:
[snip]
#KDUMP_CMDLINE_APPEND="reset_devices systemd.unit=kdump-tools-dump.service nr_cpus=1 irqpoll nousb ata_piix.prefer_ms_hyperv=0"
KDUMP_CMDLINE_APPEND="reset_devices systemd.unit=kdump-tools-dump.service nr_cpus=1 irqpoll nousb ata_piix.prefer_ms_hyperv=0 amdgpu.blacklist=1 rd.driver.blacklist=amdgpu"
[snip]
Затем просто перезагрузите kdump-tools и убедитесь, что загружена новая конфигурация:
$ sudo systemctl restart kdump-tools
$ kdump-config show
DUMP_MODE: kdump
USE_KDUMP: 1
KDUMP_SYSCTL: kernel.panic_on_oops=1
KDUMP_COREDIR: /var/crash
crashkernel addr: 0x
/var/lib/kdump/vmlinuz: symbolic link to /boot/vmlinuz-5.3.0-40-lowlatency
kdump initrd:
/var/lib/kdump/initrd.img: symbolic link to /var/lib/kdump/initrd.img-5.3.0-40-lowlatency
current state: ready to kdump
kexec command:
/sbin/kexec -p --command-line="BOOT_IMAGE=/@/boot/vmlinuz-5.3.0-40-lowlatency root=UUID=a745358b-a4e6-4a16-a347-5fa3d65e78a7 ro rootflags=subvol=@ quiet splash vt.handoff=1 reset_devices systemd.unit=kdump-tools-dump.service nr_cpus=1 irqpoll nousb ata_piix.prefer_ms_hyperv=0 amdgpu.blacklist=1 rd.driver.blacklist=amdgpu" --initrd=/var/lib/kdump/initrd.img /var/lib/kdump/vmlinuz
Затем повторите сбор.
Ура, Т.