Firefly开源社区

标题: firefly-RK3288非正常死机 [打印本页]

作者: lh2008xp 时间: 2018-8-2 14:00
标题: firefly-RK3288非正常死机
本帖最后由 lh2008xp 于 2018-8-2 14:44 编辑

运行系统Ubuntu 16.04 LTS，运行代码平台ROS中的Kinetic，工程启动后5分钟左右，死机！通过远程ssh访问的RK3288，ssh全部掉线且无法重连！
如果连接hdmi接显示器，显示器桌面卡死，表现为系统桌面时间不再刷新！
1.死机时RK3288串口打印结果如下：
firefly login: [  960.285805] INFO: task sd-resolve:597 blocked for more than 120 seconds.
[  960.292533]    Not tainted 4.4.55 #2
[  960.296406] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  960.305372] Kernel panic - not syncing: hung_task: blocked tasks
[  960.311375] CPU: 3 PID: 29 Comm: khungtaskd Not tainted 4.4.55 #2
[  960.317458] Hardware name: Rockchip (Device Tree)
[  960.322174] [<c01103a8>] (unwind_backtrace) from [<c010be58>] (show_stack+0x20/0x24)
[  960.329919] [<c010be58>] (show_stack) from [<c03ff91c>] (dump_stack+0x88/0xa8)
[  960.337138] [<c03ff91c>] (dump_stack) from [<c01e36cc>] (panic+0x98/0x204)
[  960.344013] [<c01e36cc>] (panic) from [<c01aa8fc>] (watchdog+0x32c/0x398)
[  960.350800] [<c01aa8fc>] (watchdog) from [<c013a3bc>] (kthread+0xf8/0x10c)
[  960.357671] [<c013a3bc>] (kthread) from [<c01074f8>] (ret_from_fork+0x14/0x3c)
[  960.364891] CPU1: stopping
[  960.367612] CPU: 1 PID: 2137 Comm: robot_pose_ekf Not tainted 4.4.55 #2
[  960.374213] Hardware name: Rockchip (Device Tree)
[  960.378923] [<c01103a8>] (unwind_backtrace) from [<c010be58>] (show_stack+0x20/0x24)
[  960.386659] [<c010be58>] (show_stack) from [<c03ff91c>] (dump_stack+0x88/0xa8)
[  960.393873] [<c03ff91c>] (dump_stack) from [<c010e900>] (handle_IPI+0x168/0x2c4)
[  960.401257] [<c010e900>] (handle_IPI) from [<c010147c>] (gic_handle_irq+0x88/0xa4)
[  960.408813] [<c010147c>] (gic_handle_irq) from [<c010c994>] (__irq_svc+0x54/0x70)
[  960.416282] Exception stack(0xee193d60 to 0xee193da8)
[  960.421325] 3d60: ed2e13f0 3beb3bea 00000001 dde42580 ed2e1380 00000140 ed2e13f0 00000000
[  960.429487] 3d80: ed2e17b8 00000000 00000144 ee193dbc ee193db0 ee193db0 c0b69500 c0b69500
[  960.437647] 3da0: 600f0013 ffffffff
[  960.441136] [<c010c994>] (__irq_svc) from [<c0b69500>] (_raw_spin_lock_bh+0x30/0x34)
[  960.448873] [<c0b69500>] (_raw_spin_lock_bh) from [<c090d160>] (release_sock+0x24/0x14c)
[  960.456955] [<c090d160>] (release_sock) from [<c0993db4>] (tcp_recvmsg+0x754/0xa28)
[  960.464601] [<c0993db4>] (tcp_recvmsg) from [<c09bd370>] (inet_recvmsg+0x9c/0xb8)
[  960.472071] [<c09bd370>] (inet_recvmsg) from [<c09094e4>] (sock_recvmsg+0x4c/0x50)
[  960.479628] [<c09094e4>] (sock_recvmsg) from [<c090a734>] (SyS_recvfrom+0xac/0xfc)
[  960.487183] [<c090a734>] (SyS_recvfrom) from [<c090a7ac>] (SyS_recv+0x28/0x30)
[  960.494393] [<c090a7ac>] (SyS_recv) from [<c0107440>] (ret_fast_syscall+0x0/0x3c)
[  960.501860] CPU2: stopping
[  960.504566] CPU: 2 PID: 0 Comm: swapper/2 Not tainted 4.4.55 #2
[  960.510473] Hardware name: Rockchip (Device Tree)
[  960.515173] [<c01103a8>] (unwind_backtrace) from [<c010be58>] (show_stack+0x20/0x24)
[  960.522905] [<c010be58>] (show_stack) from [<c03ff91c>] (dump_stack+0x88/0xa8)
[  960.530116] [<c03ff91c>] (dump_stack) from [<c010e900>] (handle_IPI+0x168/0x2c4)
[  960.537499] [<c010e900>] (handle_IPI) from [<c010147c>] (gic_handle_irq+0x88/0xa4)
[  960.545056] [<c010147c>] (gic_handle_irq) from [<c010c994>] (__irq_svc+0x54/0x70)
[  960.552523] Exception stack(0xee971f50 to 0xee971f98)
[  960.557565] 1f40:                                  00000001 00000000 ee971fb0 c0118fa0
[  960.565728] 1f60: ee970000 00000000 00000000 ee971fc0 c10b03ac c10b4858 c0c01a44 ee971fac
[  960.573890] 1f80: ee971fb0 ee971fa0 c010805c c0108060 60010013 ffffffff
[  960.580493] [<c010c994>] (__irq_svc) from [<c0108060>] (arch_cpu_idle+0x3c/0x48)
[  960.587881] [<c0108060>] (arch_cpu_idle) from [<c0160fe4>] (default_idle_call+0x38/0x3c)
[  960.595961] [<c0160fe4>] (default_idle_call) from [<c0161240>] (cpu_startup_entry+0x1fc/0x334)
[  960.604559] [<c0161240>] (cpu_startup_entry) from [<c010e524>] (secondary_start_kernel+0x12c/0x148)
[  960.613590] [<c010e524>] (secondary_start_kernel) from [<0010152c>] (0x10152c)
[  960.620799] CPU0: stopping
[  960.623503] CPU: 0 PID: 10 Comm: watchdog/0 Not tainted 4.4.55 #2
[  960.629583] Hardware name: Rockchip (Device Tree)
[  960.634283] [<c01103a8>] (unwind_backtrace) from [<c010be58>] (show_stack+0x20/0x24)
[  960.642014] [<c010be58>] (show_stack) from [<c03ff91c>] (dump_stack+0x88/0xa8)
[  960.649225] [<c03ff91c>] (dump_stack) from [<c010e900>] (handle_IPI+0x168/0x2c4)
[  960.656608] [<c010e900>] (handle_IPI) from [<c010147c>] (gic_handle_irq+0x88/0xa4)
[  960.664163] [<c010147c>] (gic_handle_irq) from [<c010c994>] (__irq_svc+0x54/0x70)
[  960.671631] Exception stack(0xee96dd00 to 0xee96dd48)
[  960.676673] dd00: ee39d940 89188917 00000000 00000000 ee39d93c ee39d940 ee39d950 ee96dd60
[  960.684836] dd20: 00000000 c11d59c0 ee96ddb8 ee96dd5c ee96dd50 ee96dd50 c0b6955c c0b69560
[  960.692996] dd40: 60010113 ffffffff
[  960.696480] [<c010c994>] (__irq_svc) from [<c0b69560>] (_raw_spin_unlock_irq+0x1c/0x20)
[  960.704475] [<c0b69560>] (_raw_spin_unlock_irq) from [<c07515b0>] (usb_giveback_urb_bh+0xc0/0xc8)
[  960.713335] [<c07515b0>] (usb_giveback_urb_bh) from [<c0122b48>] (tasklet_action+0x9c/0xfc)
[  960.721674] [<c0122b48>] (tasklet_action) from [<c0122058>] (__do_softirq+0x138/0x31c)
[  960.729577] [<c0122058>] (__do_softirq) from [<c01224f8>] (irq_exit+0x98/0x108)
[  960.736876] [<c01224f8>] (irq_exit) from [<c016ed98>] (__handle_domain_irq+0x9c/0xc0)
[  960.744695] [<c016ed98>] (__handle_domain_irq) from [<c0101450>] (gic_handle_irq+0x5c/0xa4)
[  960.753030] [<c0101450>] (gic_handle_irq) from [<c010c994>] (__irq_svc+0x54/0x70)
[  960.760497] Exception stack(0xee96de70 to 0xee96deb8)
[  960.765538] de60:                                  eef923c0 00000000 00000002 ed660e28
[  960.773702] de80: ed660e00 00000000 eef923c0 ee943840 ee940c80 c0b66040 c0c02218 ee96df04
[  960.781864] dea0: ee96de70 ee96dec0 c0b6955c c0b65cd4 80010013 ffffffff
[  960.788468] [<c010c994>] (__irq_svc) from [<c0b65cd4>] (__schedule+0x470/0x730)
[  960.795766] [<c0b65cd4>] (__schedule) from [<c0b66040>] (schedule+0xac/0xbc)
[  960.802805] [<c0b66040>] (schedule) from [<c013e378>] (smpboot_thread_fn+0x16c/0x19c)
[  960.810621] [<c013e378>] (smpboot_thread_fn) from [<c013a3bc>] (kthread+0xf8/0x10c)
[  960.818265] [<c013a3bc>] (kthread) from [<c01074f8>] (ret_from_fork+0x14/0x3c)
[  960.825483] rockchip-thermal ff280000.tsadc: channal 0: temperature(43 C)
[  960.832259] THERMAL REGS:
[  960.834878] 00000000: 00000200 00030061 00000602 00000000 00000000 00000000 00000000 00000000
[  960.843389] 00000020: 00000000 00000e2d 00000e37 00000000 00000000 00000deb 00000000 00000000
[  960.851898] 00000040: 00000000 00000db6 00000db6 00000000 00000000 00000000 00000000 00000000
[  960.860408] 00000060: 00000004 00000004 000000fa 00000032 00000000 00000000 00000000 00000000
[  960.868915] 00000080: 00000000 00000000
[  960.872748] ---[ end Kernel panic - not syncing: hung_task: blocked tasks
[  996.105661] ------------[ cut here ]------------
[  996.110280] WARNING: CPU: 3 PID: 29 at kernel/watchdog.c:352 watchdog_timer_fn+0xf8/0x344()
[  996.118613] Watchdog detected hard LOCKUP on cpu 0
[  996.123221] Modules linked in: ch341 cp210x
[  996.127617] CPU: 3 PID: 29 Comm: khungtaskd Not tainted 4.4.55 #2
[  996.133697] Hardware name: Rockchip (Device Tree)
[  996.138397] [<c01103a8>] (unwind_backtrace) from [<c010be58>] (show_stack+0x20/0x24)
[  996.146129] [<c010be58>] (show_stack) from [<c03ff91c>] (dump_stack+0x88/0xa8)
[  996.153342] [<c03ff91c>] (dump_stack) from [<c011de5c>] (warn_slowpath_common+0x98/0xc4)
[  996.161419] [<c011de5c>] (warn_slowpath_common) from [<c011dec8>] (warn_slowpath_fmt+0x40/0x48)
[  996.170103] [<c011dec8>] (warn_slowpath_fmt) from [<c01ab098>] (watchdog_timer_fn+0xf8/0x344)
[  996.178619] [<c01ab098>] (watchdog_timer_fn) from [<c01823fc>] (__hrtimer_run_queues+0x1b0/0x324)
[  996.187478] [<c01823fc>] (__hrtimer_run_queues) from [<c0182ccc>] (hrtimer_interrupt+0xac/0x1f8)
[  996.196253] [<c0182ccc>] (hrtimer_interrupt) from [<c086f978>] (arch_timer_handler_phys+0x38/0x40)
[  996.205200] [<c086f978>] (arch_timer_handler_phys) from [<c01730ac>] (handle_percpu_devid_irq+0xd8/0x1ac)
[  996.214753] [<c01730ac>] (handle_percpu_devid_irq) from [<c016ea54>] (generic_handle_irq+0x28/0x38)
[  996.223784] [<c016ea54>] (generic_handle_irq) from [<c016ed94>] (__handle_domain_irq+0x98/0xc0)
[  996.232467] [<c016ed94>] (__handle_domain_irq) from [<c0101450>] (gic_handle_irq+0x5c/0xa4)
[  996.240803] [<c0101450>] (gic_handle_irq) from [<c010c994>] (__irq_svc+0x54/0x70)
[  996.248271] Exception stack(0xeeb61e08 to 0xeeb61e50)
[  996.253313] 1e00:                eeb61e94 00000005 fd640800 c086f8d0 eeb61e94 00005dbf
[  996.261476] 1e20: 00008a48 c11d5568 0000004f c1154604 00000399 eeb61e64 eeb61e68 eeb61e58
[  996.269636] 1e40: c010f730 c086f8e0 a00e0013 ffffffff
[  996.274680] [<c010c994>] (__irq_svc) from [<c086f8e0>] (arch_counter_get_cntpct+0x10/0x18)
[  996.282932] [<c086f8e0>] (arch_counter_get_cntpct) from [<c010f730>] (arch_timer_read_counter_long+0x20/0x28)
[  996.292830] [<c010f730>] (arch_timer_read_counter_long) from [<c03fd7d0>] (read_current_timer+0x30/0x48)
[  996.302294] [<c03fd7d0>] (read_current_timer) from [<c03fd820>] (__timer_delay+0x38/0x5c)
[  996.310458] [<c03fd820>] (__timer_delay) from [<c03fd870>] (__timer_const_udelay+0x2c/0x34)
[  996.318798] [<c03fd870>] (__timer_const_udelay) from [<c01e37fc>] (panic+0x1c8/0x204)
[  996.326618] [<c01e37fc>] (panic) from [<c01aa8fc>] (watchdog+0x32c/0x398)
[  996.333396] [<c01aa8fc>] (watchdog) from [<c013a3bc>] (kthread+0xf8/0x10c)
[  996.340259] [<c013a3bc>] (kthread) from [<c01074f8>] (ret_from_fork+0x14/0x3c)
[  996.347466] ---[ end trace 7bec9f8eaf3d9ec7 ]---
注意到以上log中有提到[  996.118613] Watchdog detected hard LOCKUP on cpu 0，请问是否是锁死原因造成的原因？

同时提供其他信息以供参考：
2.死机时RK3288温度测试结果,死机后实测芯片温度73.9°C，温度是否过高？

3.死机前CPU使用情况，其中CPU0的软中断（Software IRQ）占用CPU的百分比100%，是否异常？
file:///C:/Users/pc/Documents/My%20Knowledge/temp/9adca0f7-7393-49d7-8451-35e0035edcbf/128/index_files/de077649-718d-410c-8553-469583a9130f.png

作者: lh2008xp 时间: 2018-8-2 16:08
从串口打印消息Kernel panic - not syncing: hung_task: blocked tasks知道，发生了内核错误！请问这个该如何处理了定位？我仅仅是在现有的RK3288上进行应用开发！

欢迎光临 Firefly开源社区 (https://dev.t-firefly.com/)