运行SOPHON-DEMO的Qwen出现内存报错

bing20138

运行SOPHON-DEMO的Qwen出现内存报错
步骤1、git clone https://github.com/sophgo/sophon-demo工程步骤2、cd Qwen目录
步骤3、先执行./scripts/download.sh下载模型
步骤4、pip3 install -r python/requirements.txt 和安装sophon_arm_pcie-3.7.0-py3-none-any.whl依赖

步骤5、streamlit run python/web_demo.py
错误显示-cmd终端显示：
[2024-06-17 10:28:10.560] [error] [tensor.cpp:403] bm_memset_device failed, return=4
[bmlib_memory][error] bm_alloc_gmem failed, dev_id = 0, size = 0x0
[BM_CHECK][error] BM_CHECK_RET fail /workspace/libsophon/bmlib/src/bmlib_memory.cpp: bm_malloc_device_byte_heap_mask: 705
[2024-06-17 10:28:10.597] [error] [tensor.cpp:396] bm_malloc_device_type() err=5, size=0
[bmlib_memory][error] bm_device_mem_range_valid saddr=0xfffffffff eaddr=0xfffffffff out of range
[bmlib_memory][error] memset wrong memory addr 0xfffffffff
[2024-06-17 10:28:10.597] [error] [tensor.cpp:403] bm_memset_device failed, return=4
[bmlib_memory][error] bm_alloc_gmem failed, dev_id = 0, size = 0x0
[BM_CHECK][error] BM_CHECK_RET fail /workspace/libsophon/bmlib/src/bmlib_memory.cpp: bm_malloc_device_byte_heap_mask: 705
[2024-06-17 10:28:10.635] [error] [tensor.cpp:396] bm_malloc_device_type() err=5, size=0
[bmlib_memory][error] bm_device_mem_range_valid saddr=0xfffffffff eaddr=0xfffffffff out of range
[bmlib_memory][error] memset wrong memory addr 0xfffffffff
[2024-06-17 10:28:10.635] [error] [tensor.cpp:403] bm_memset_device failed, return=4
[bmlib_memory][error] bm_alloc_gmem failed, dev_id = 0, size = 0x0
[BM_CHECK][error] BM_CHECK_RET fail /workspace/libsophon/bmlib/src/bmlib_memory.cpp: bm_malloc_device_byte_heap_mask: 705
[2024-06-17 10:28:10.672] [error] [tensor.cpp:396] bm_malloc_device_type() err=5, size=0
[bmlib_memory][error] bm_device_mem_range_valid saddr=0xfffffffff eaddr=0xfffffffff out of range
[bmlib_memory][error] memset wrong memory addr 0xfffffffff
[2024-06-17 10:28:10.672] [error] [tensor.cpp:403] bm_memset_device failed, return=4
[bmlib_memory][error] bm_alloc_gmem failed, dev_id = 0, size = 0x0
[BM_CHECK][error] BM_CHECK_RET fail /workspace/libsophon/bmlib/src/bmlib_memory.cpp: bm_malloc_device_byte_heap_mask: 705
[2024-06-17 10:28:10.710] [error] [tensor.cpp:396] bm_malloc_device_type() err=5, size=0
[bmlib_memory][error] bm_device_mem_range_valid saddr=0xfffffffff eaddr=0xfffffffff out of range
[bmlib_memory][error] memset wrong memory addr 0xfffffffff
[2024-06-17 10:28:10.710] [error] [tensor.cpp:403] bm_memset_device failed, return=4
[bmlib_memory][error] bm_alloc_gmem failed, dev_id = 0, size = 0x0
[BM_CHECK][error] BM_CHECK_RET fail /workspace/libsophon/bmlib/src/bmlib_memory.cpp: bm_malloc_device_byte_heap_mask: 705
[2024-06-17 10:28:10.747] [error] [tensor.cpp:396] bm_malloc_device_type() err=5, size=0
[bmlib_memory][error] bm_device_mem_range_valid saddr=0xfffffffff eaddr=0xfffffffff out of range
[bmlib_memory][error] memset wrong memory addr 0xfffffffff
[2024-06-17 10:28:10.747] [error] [tensor.cpp:403] bm_memset_device failed, return=4

错误显示-串口终端显示：
[ 3690.300620][4]  ion_alloc failed(-22): len 0 heap_id_mask 4 flags 0
[ 3690.306888][4]  ion_ioctl ion alloc failed, fd=-22, from streamlit
[ 3690.313185][4]  ion_alloc failed(-22): len 0 heap_id_mask 1 flags 0
[ 3690.319452][4]  ion_ioctl ion alloc failed, fd=-22, from streamlit
[ 3690.325633][4]  ion_alloc failed(-22): len 0 heap_id_mask 2 flags 0
[ 3690.331897][4]  ion_ioctl ion alloc failed, fd=-22, from streamlit
[ 3690.338074][4]  ion_alloc failed(-22): len 0 heap_id_mask 4 flags 0
[ 3690.344337][4]  ion_ioctl ion alloc failed, fd=-22, from streamlit
[ 3690.350833][4]  ion_alloc failed(-22): len 0 heap_id_mask 1 flags 0
[ 3690.357103][4]  ion_ioctl ion alloc failed, fd=-22, from streamlit
[ 3690.363282][4]  ion_alloc failed(-22): len 0 heap_id_mask 2 flags 0
[ 3690.369548][4]  ion_ioctl ion alloc failed, fd=-22, from streamlit
[ 3690.375725][4]  ion_alloc failed(-22): len 0 heap_id_mask 4 flags 0
[ 3690.381988][4]  ion_ioctl ion alloc failed, fd=-22, from streamlit
[ 3690.388292][4]  ion_alloc failed(-22): len 0 heap_id_mask 1 flags 0
[ 3690.394554][4]  ion_ioctl ion alloc failed, fd=-22, from streamlit
[ 3690.400743][4]  ion_alloc failed(-22): len 0 heap_id_mask 2 flags 0
[ 3690.407011][4]  ion_ioctl ion alloc failed, fd=-22, from streamlit
[ 3690.413193][4]  ion_alloc failed(-22): len 0 heap_id_mask 4 flags 0
[ 3690.419463][4]  ion_ioctl ion alloc failed, fd=-22, from streamlit
[ 3690.425752][4]  ion_alloc failed(-22): len 0 heap_id_mask 1 flags 0
[ 3690.432020][4]  ion_ioctl ion alloc failed, fd=-22, from streamlit
[ 3690.438205][4]  ion_alloc failed(-22): len 0 heap_id_mask 2 flags 0
[ 3690.444486][4]  ion_ioctl ion alloc failed, fd=-22, from streamlit
[ 3690.450671][4]  ion_alloc failed(-22): len 0 heap_id_mask 4 flags 0
[ 3690.456941][4]  ion_ioctl ion alloc failed, fd=-22, from streamlit

Multius

可能是 FireflyChat 占用了内存导致的，执行 `sudo systemctl stop FireflyChat` 命令关闭 FireflyChat 后再继续尝试运行 demo

bing20138

Multius 发表于 2024-6-17 15:11
可能是 FireflyChat 占用了内存导致的，执行 `sudo systemctl stop FireflyChat` 命令关闭 FireflyChat 后 ...

试过还是不行

Multius

bing20138 发表于 2024-6-17 15:33
试过还是不行

使用的具体是 Qwen 的哪一个模型？模型最终编译出来的 bmodel 文件有多大体积？

bing20138

也试过./memory_edit.sh -c -npu 7615 -vpu 3072 -vpp 3072修改TPU内存也是不行的

bing20138

Multius 发表于 2024-6-17 15:52
使用的具体是 Qwen 的哪一个模型？模型最终编译出来的 bmodel 文件有多大体积？

qwen-7b_int4_1dev.bmodel 4.78GB

bing20138

难道是因为这个原因？

Multius

bing20138 发表于 2024-6-17 16:06
难道是因为这个原因？

这里的说明是对于编译模型的机器的要求，与模型运行无关

Multius

bing20138 发表于 2024-6-17 16:02
qwen-7b_int4_1dev.bmodel 4.78GB

这个尺寸的 int4 的量化模型应该是足够在 1684x 上面运行的，但由于 Qwen 第一代模型的适配工作也有些时日了，具体原因难以排查。
建议使用 Qwen1.5 或者 Qwen2 进行部署验证。

Firefly | 让科技更简单，让生活更智能 (t-firefly.com)，我们在这里提供了 Qwen1.5 编译好的 bmodel 文件，可直接使用。

bing20138

换QWEN1.5会报AttributeError: module 'sophon.sail' has no attribute 'EngineLLM'