常见日志报错及解决方法(一)

EMQX 企业版技术支持发表于:2022年05月18日 11:09:30

1.关键字:ecpool_pool_sup

日志原文:

Supervisor: {<0.18803.107>,ecpool_pool_sup}. Context: start_error. Reason: {shutdown,{failed_to_start_child,{worker,1},{{badmatch,{error,einval}},[{eredis_client,connect_with_tcp,2,[{file,"eredis_client.erl"},{line,369}]},{eredis_client,init,1,[{file,"eredis_client.erl"},{line,98}]},{gen_server,init_it,2,[{file,"gen_server.erl"},{line,417}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,385}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]}}}. Offender: id=worker_sup,pid=undefined.

错误原因:eredis_client 里面的 badmatch,是个bug,redis驱动因为DNS解析失败等情况产生的问题

解决方案:已在4.4.3修复

2.关键字:Warning 1265

日志原文:

Warning 1265: Data truncated for column 'temp' at row 1, Warning 1265: Data truncated for column 'hum' at row 1, in insert into , temp_hum(up_timestamp, client_id, temp, hum)  VALUES  , (FROM_UNIXTIME(1647761018341/1000), 'diaocheguanli1', 'undefined', 'undefined')

错误原因:一般是 因为数据类型的不对应,或者字符串长度不够而造成的

解决方案:更改为正确的数据类型或者增加字符串长度

3.关键字: initial call: cowboy_clear

日志原文:

crasher:    initial call: cowboy_clear:connection_process/4    pid: <0.25409.710>    registered_name: []    exception error: no match of right hand side value {error,closed}      in function  cowboy_clear:connection_process/4 (cowboy_clear.erl, line 38)    ancestors: [<0.2302.0>,<0.2301.0>,ranch_sup,<0.2062.0>]    message_queue_len: 1    messages: [{handshake,'mqtt:ws:9083',ranch_tcp,#Port<0.9487546>,5000}]    links: [<0.2302.0>,#Port<0.9487546>]    dictionary: []    trap_exit: false    status: running    heap_size: 610    stack_size: 27    reductions: 263  neighbours:2022-03-13 13:27:35.816 [error] Ranch listener 'mqtt:ws:9083' had connection process started with cowboy_clear:start_link/4 at <0.25409.710> exit with reason: {{badmatch,{error,closed}},[{cowboy_clear,connection_process,4,[{file,"cowboy_clear.erl"},{line,38}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,249}]}]}

错误原因:ws的在握手的过程中被中断了

解决方案:检查下网络问题

4.关键字: Parse failed for frame_too_large

日志原文:

10.192.198.141:42568 [MQTT] , Parse failed for frame_too_large, [{emqx_frame,parse_remaining_len,5,[{file,"emqx_frame.erl"},{line,159}]},{emqx_connection,parse_incoming,3,[{file,"emqx_connection.erl"},{line,625}]},{emqx_connection,handle_msg,2,[{file,"emqx_connection.erl"},{line,618}]},{emqx_connection,process_msg,2,[{file,"emqx_connection.erl"},{line,364}]},{emqx_connection,handle_recv,3,[{file,"emqx_connection.erl"},{line,328}]},{proc_lib,wake_up,3,[{file,"proc_lib.erl"},{line,236}]}], Frame data:<<0,164,255,83,77,66,114,0,0,0,0,8,1,64,0,0,0,0,0,0,0,0,0,0,0,0,0,0,64,6,0,0,1,0,0,129,0,2,80,67,32,78,69,84,87,79,82,75,32,80,82,79,71,82,65,77,32,49,46,48,0,2,77,73,67,82,79,83,79,70,84,32,78,69,84,87,79,82,75,83,32,49,46,48,51,0,2,77,73,67,82,79,83,79,70,84,32,78,69,...>>

错误原因:数据报文太长了

解决方案:检查下发送的相关的报文

5.关键字: esockd_acceptor_sup

日志原文:

Supervisor: {<0.2393.0>,esockd_acceptor_sup}. Context: shutdown_error. Reason: noproc. Offender: id=acceptor,nb_children=2.

错误原因:进程挂掉了

解决方案:需要上下文详细定位

6.关键字: PoolWorker

日志原文:

[PoolWorker] supervisee <0.12604.2> is force killed

错误原因:ecpool 的功能,被底层库强制关闭了,这个是预期行为,如果子进程卡住是要强杀的

解决方案

7.关键字: resource

日志原文:

[Alarm Handler] Alarm resource/web_hook/resource:6777aed2/down is deactivated

错误原因:webhook资源down掉了

解决方案:检查webhook的url情况

8.关键字: More than one channel found

日志原文:

<<"zhtbhz_beidou_server_test">>@14.108.214.6:61987 [CM] More than one channel found: [<0.23782.682>,<0.23804.682>]

错误原因:同一个 ClientId 出现了多个会话

解决方案

9.关键字: Failed to discard

日志原文:

<<"android:7706416585007415566:7b6a3110779b89e3">>@112.97.83.197:37017 [CM] Failed to discard <59069.26551.1562>: {'EXIT',{{shutdown,tcp_closed},{gen_server,call,[<59069.26551.1562>,discard,infinity]}}}

错误原因:客户端网络断了,导致任务分发失败,有可能是网络导致的,强断也有可能

解决方案:检查网络问题

10.关键字: Warning 1292

日志原文:

Warning 1292: Truncated incorrect DOUBLE value: 'undefined', in INSERT INTO monitor_data_H621382_2022 (id, cod, nh3n, turb, ec, ph, dio, temp, vol, gps, monitor_dt, create_dt, state)  VALUES  ('H621382', ROUND(7.0591, 2), abs(ROUND(1.025, 2)), ROUND(10.93, 2), ROUND(0.71, 2), ROUND(8.22, 2), ROUND(4.2, 2), ROUND(13.67, 2),ROUND('undefined', 2), '0,0', FROM_UNIXTIME(1648004224), now(), '1')

错误原因:写的mysql查询语句中所提供的字段,与表中字段类型不匹配

解决方案:更改为对应的字段

11.关键字: failed_to_connect_all

日志原文:

failed_to_connect_all: [{{"127.0.0.1",9092},{{{kpro_req,#Ref<60403.1985112988.647757825.240140>,api_versions,0,false,[]},closed},[{kpro_lib,send_and_recv_raw,4,[{file,"kpro_lib.erl"},{line,70}]},{kpro_lib,send_and_recv,5,[{file,"kpro_lib.erl"},{line,81}]},{kpro_connection,query_api_versions,4,[{file,"kpro_connection.erl"},{line,251}]},{kpro_connection,init_connection,2,[{file,"kpro_connection.erl"},{line,238}]},{kpro_connection,init,4,[{file,"kpro_connection.erl"},{line,175}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]}}]

错误原因:kafka连不上

解决方案:检查kafka连接

12.关键字: http_connectivity failed

日志原文:

check http_connectivity failed: <<"http://119.3.223.224:7402">>

错误原因:http请求的url连不上

解决方案:检查http://119.3.223.224:7402 这个链接是否可以telnet通

13.关键字: Removing (timedout) connection

日志原文:

** Node 'emqx@10.13.67.20' not responding **, ** Removing (timedout) connection **

错误原因:EMQ未正常启动

解决方案:启动EMQ

14.关键字: awaiting_rel

日志原文:

TEST00-0000-000-C050@47.103.69.133:54588 [Channel] Dropped the qos2 packet 24 due to awaiting_rel is full.

错误原因:qos2的消息队列满了,等待的满了,订阅端接收消息的能力不足

解决方案:增加订阅端的消费能力

15.关键字: login failed

日志原文:

[27784,38451,22235,38498]@58.220.83.118:57540 [Channel] Client æ²•é•³å••é•¢ (Username: 'admin') login failed for password_error

错误原因:用户名admin密码错误

解决方案:更换为正确的密码

16.关键字: message=channel_closed

日志原文:

message=channel_closed driver=tcp socket="#Port<0.741>" action=stopping

错误原因:进程正在停掉的日志

解决方案

17.关键字: mysql_conn

日志原文:

crasher: initial call: mysql_conn:init/1, pid: <0.3807.0>, registered_name: [], exit: {{1044,<<"42000">>,<<"Access denied for user 'MQTTTopci'@'%' to database 'mqttdata'">>},[{gen_server,init_it,6,[{file,"gen_server.erl"},{line,401}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]}, ancestors: [<0.3806.0>,<0.3805.0>,<0.3803.0>,ecpool_sup,<0.2530.0>], message_queue_len: 0, messages: [], links: [<0.3806.0>], dictionary: [], trap_exit: false, status: running, heap_size: 376, stack_size: 28, reductions: 345; neighbours:

错误原因:emqx 访问被 MySQL 拒绝了

解决方案:检查mysql设置

18.关键字: Socket error

日志原文:

<<"dbce405abb00eee7">>@218.82.138.78:52242 [MQTT] Socket error: einval

错误原因: 可能域名解析失败,也可能网络出现故障

解决方案:4.4版本应该不会有这种报错,需要检查下客户端的域名和网络情况

19.关键字: license_quota

日志原文:

[Alarm Handler] Alarm license_quota is activated, License: the number of connections exceeds 80%

错误原因:连接数过高,超过了限制的80%

解决方案:减少连接数

20.关键字: HTTP request failed

日志原文:

NGROK_TEST00-0000-000-C007@47.103.69.133:33792 HTTP request failed path: <<"/iot/hub/v1alpha1/devices/mqtt-events">> error: {closed,"The connection was lost."}

错误原因:发送的http  认证请求被关掉了,HTTP 服务器端的认证连接失败

解决方案:检查http认证服务器相应配置

21.关键字: high_system_memory_usage

日志原文:

[Alarm Handler] Alarm high_system_memory_usage is activated, System memory usage is higher than 70.0%

错误原因:EMQ占os系统内存过高

解决方案:找到占内存的原因,4.4.1前是bug,已在4.4.2修复

22.关键字: emqx shutdown for join

日志原文:

[EMQ X] emqx shutdown for join

错误原因:加集群时暂时关闭emq

解决方案:加集群时暂时关闭emq

23.关键字: init_module_failure

日志原文:

cluster_call error found, ResL: [{{{init_module_failure,'emqx@10.13.67.122'}, {{emqx_module_auth_redis,on_module_create},badarg}},[{erlang,list_to_integer,["6379 "],[]},{emqx_module_auth_redis,format_server,1,[{file,"emqx_module_auth_redis.erl"},{line,303}]},{emqx_module_auth_redis,format_servers,2,[{file,"emqx_module_auth_redis.erl"},{line,295}]},{emqx_module_auth_redis,on_module_create,2,[{file,"emqx_module_auth_redis.erl"},{line,225}]},{emqx_modules,'-init_module/4-fun-0-',4,[{file,"emqx_modules.erl"},{line,253}]},{emqx_modules,init_module,4,[{file,"emqx_modules.erl"},{line,253}]},{erpc,execute_call,4,[{file,"erpc.erl"},{line,416}]}]},{{{init_module_failure,'emqx@10.13.67.20'},{{emqx_module_auth_redis,on_module_create},badarg}},[{erlang,list_to_integer,["6379 "],[]},{emqx_module_auth_redis,format_server,1,[{file,"emqx_module_auth_redis.erl"},{line,303}]},{emqx_module_auth_redis,format_servers,2,[{file,"emqx_module_auth_redis.erl"},{line,295}]},{emqx_module_auth_redis,on_module_create,2,[{file,"emqx_module_auth_redis.erl"},{line,225}]},{emqx_modules,'-init_module/4-fun-0-',4,[{file,"emqx_modules.erl"},{line,253}]},{emqx_modules,init_module,4,[{file,"emqx_modules.erl"},{line,253}]},{erpc,execute_call,4,[{file,"erpc.erl"},{line,416}]}]}]

错误原因:redis认证模块端口6379后加了空格

解决方案:删掉多出空格后重新启动该模块

24.关键字: emqx_license

日志原文:

[emqx_license] emqx_channel_conn size is undefined:{[undefined],['emqx@10.14.237.109']}

错误原因:这个是同时启动集群中的多个节点, 启动时会去别的节点拿信息,由于另的节点也没准备好,就返回了个undefined

解决方案:只要不是一直报,就没问题的

25.关键字: emqx_acl_mnesia_cli

日志原文:

[Ctl] CMD acl is overidden by {emqx_acl_mnesia_cli,cli}

错误原因:ACL配置被命令重写了

解决方案

26.关键字: JT808 Conn

日志原文:

[JT808 Conn] Parser failed for {invalid_message,                                
[{emqx_jt808_frame,extract_message,3,                                  
[{file,"emqx_jt808_frame.erl"},{line,111}]},                                 
{emqx_jt808_frame,parse_main,3,                                  
[{file,"emqx_jt808_frame.erl"},{line,67}]},                                 
{emqx_jt808_connection,received,2,                                  
[{file,"emqx_jt808_connection.erl"},                                   {line,243}]},                                 
{gen_server,try_dispatch,4,                                  [{file,"gen_server.erl"},
{line,637}]},                                 {gen_server,handle_msg,6,                                  
[{file,"gen_server.erl"},{line,711}]},                                 
{proc_lib,init_p_do_apply,3,                                  [{file,"proc_lib.erl"},
{line,249}]}]}2022-03-13 10:26:16.216 [error] [JT808 Conn] Error data: 
<<126,0,2,0,0,2,0,0,0,0,21,0,3,50,126>>

错误原因:发过来的报文不合法

解决方案:检查发送的报文

27.关键字: Cannot subscribe

日志原文:

<<"20220314192301">>@61.164.57.169:62445 [Channel] Cannot subscribe YD_MICROWAVE_DATA_PRETREAT due to Not authorized.

错误原因:没订阅的权限,所以订阅不了

解决方案:开放订阅的权限

28.关键字: TcpClosed producer

日志原文:

TcpClosed producer: <0.5964.0>

错误原因:日志太少,producer的连接断开了

解决方案

29.关键字: Redis

日志原文:

[Redis] Can't connect to Redis server: {{badmatch,{error,einval}},[{eredis_client,connect_with_tcp,2,[{file,"eredis_client.erl"},{line,369}]},{eredis_client,init,1,[{file,"eredis_client.erl"},{line,98}]},{gen_server,init_it,2,[{file,"gen_server.erl"},{line,417}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,385}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]}

错误原因:需要升级下redis库版本

解决方案

30.关键字: lock for too long

日志原文:

kill <0.1401.4063> as it has held the lock for too long, resource: <<"dbce405abb00eee7">>

错误原因 在 ekka 里面某个资源持有的锁一直没有释放,强制杀死持有锁时间较长的进程

解决方案

31.关键字: connect_to_remote_server

日志原文:

event=connect_to_remote_server peer="emqx@10.13.67.20" result=failure reason="econnrefused"

错误原因:连接被拒绝了

解决方案:检查连接

32.关键字: maximum heap size reached

日志原文:

Process:          <0.19563.3699> on node 'emqx@10.13.67.23', Context:          maximum heap size reached, 
Max Heap Size:    10485760, 
Total Heap Size:  23943461, 
Kill:             true, 
Error Logger:     true, GC Info:          [{old_heap_block_size,318187},{heap_block_size,12909534},
{mbuf_size,10715776},{recent_size,6388},{stack_size,33},{old_heap_size,22167},{heap_size,75077},{bin_vheap_size,77750},{bin_vheap_block_size,75110},{bin_old_vheap_size,54000},{bin_old_vheap_block_size,98314}]

错误原因:进程达到了最大的堆内存,一般原因就是消息积压太多了

解决方案

33.关键字: init_action_failure

日志原文:

cluster_call error found, ResL: [{{init_action_failure,'emqx@10.14.69.101'},{{emqx_bridge_kafka_actions,on_action_create_data_to_kafka},{error,{error,kafka_topic_not_found},[{emqx_bridge_kafka_actions,check_kafka_topic,2,[{file,"emqx_bridge_kafka_actions.erl"},{line,459}]},{emqx_bridge_kafka_actions,on_action_create_data_to_kafka,2,[{file,"emqx_bridge_kafka_actions.erl"},{line,386}]},{emqx_rule_engine,'-init_action/4-fun-0-',4,[{file,"emqx_rule_engine.erl"},{line,551}]},{emqx_rule_engine,init_action,4,[{file,"emqx_rule_engine.erl"},{line,551}]},{erpc,execute_call,4,[{file,"erpc.erl"},{line,416}]}]}}},{{init_action_failure,'emqx@10.14.69.7'},{{emqx_bridge_kafka_actions,on_action_create_data_to_kafka},{error,{error,kafka_topic_not_found},[{emqx_bridge_kafka_actions,check_kafka_topic,2,[{file,"emqx_bridge_kafka_actions.erl"},{line,459}]},{emqx_bridge_kafka_actions,on_action_create_data_to_kafka,2,[{file,"emqx_bridge_kafka_actions.erl"},{line,386}]},{emqx_rule_engine,'-init_action/4-fun-0-',4,[{file,"emqx_rule_engine.erl"},{line,551}]},{emqx_rule_engine,init_action,4,[{file,"emqx_rule_engine.erl"},{line,551}]},{erpc,execute_call,4,[{file,"erpc.erl"},{line,416}]}]}}}]

错误原因 写入到kafka失败,Kafka的topic没找到

解决方案 在kafka里建topic或者写入时换kafka里有的topic

34.关键字: mc_worker

日志原文:

Generic server <0.2926.84> terminating. Reason: tcp_closed. Last message: {tcp_closed,#Port<0.2554055>}. State: {state,#Port<0.2554055>,#{7 => #Fun<mc_worker_logic.0.117303875>},<<>>,{conn_state,unsafe,master,<<"admin">>,<<"admin">>},undefined,#Fun<mc_worker.0.76890317>,gen_tcp}.

错误原因:没有设置 set name,这是一个必填字段

解决方案:设置后写入mongo

35.关键字: metadata

日志原文:

Failed to get metadata, reason: unknown_topic_or_partition

错误原因 未知的topic或者分区

解决方案 检查kafka配置

36.关键字: Alarm resource

日志原文:

[Alarm Handler] Alarm resource/web_hook/resource:6777aed2/down is deactivated

错误原因 webhook资源不可用

解决方案 检查webhook资源

37.关键字: action: discard

日志原文:

mqttdoctor_d4e9afe1@10.0.71.227:30041 action: discard, file: emqx_cm.erl, line: 317, mfa: {emqx_cm,kick_or_kill,3}, msg: session_kick_exception, pid: <0.2083.446>, reason: normal, stacktrace: [{emqx_ws_connection,call,3,[{file,"emqx_ws_connection.erl"},{line,167}]},{emqx_cm,kick_or_kill,3,[{file,"emqx_cm.erl"},{line,299}]},{lists,foreach,2,[{file,"lists.erl"},{line,1342}]},{emqx_cm,'-open_session/3-fun-0-',5,[{file,"emqx_cm.erl"},{line,219}]},{emqx_cm_locker,trans,3,[{file,"emqx_cm_locker.erl"},{line,46}]},{emqx_channel,process_connect,2,[{file,"emqx_channel.erl"},{line,492}]},{emqx_ws_connection,with_channel,3,[{file,"emqx_ws_connection.erl"},{line,575}]},{cowboy_websocket,handler_call,6,[{file,"cowboy_websocket.erl"},{line,487}]},{proc_lib,wake_up,3,[{file,"proc_lib.erl"},{line,236}]}], stale_channel: undefined

错误原因 相同的clientID互踢或者手动踢出

解决方案 检查clientid重复登录问题

38.关键字: Dropped msg

日志原文:

rcsReceiver@ba5eb70e-8c03-4563-bd6a-b4737552bb15@101.132.135.166:53926 [Session] Dropped msg due to mqueue is full: Message(Id=•Ú•Z•okªLJ•sЕo, QoS=1, Topic=/maintenance/GS142-0120-T9M-0000/status, From=<<"GS142-0120-T9M-0000">>, Flags=[], Headers=#{peerhost => {101,132,135,166}, properties => #{},proto_ver => 4,protocol => mqtt, username => <<"gaussian">>})

错误原因:消息队列已经满了,订阅端消费能力不足导致

解决方案:增加订阅端消费能力

39.关键字: SYSMON

日志原文:

[SYSMON] large_heap warning: pid = <0.15399.1611>, info: [{old_heap_block_size, 10695351}, {heap_block_size,8912793}, {mbuf_size,0}, {stack_size,49}, {old_heap_size,4158305}, {heap_size,4043698}], [{initial_call,{proc_lib,init_p,5}},{current_function,{ets,select_trap,1}},{registered_name,[]},{status,running},{message_queue_len,0},{group_leader,<0.2147.0>},{priority,normal},{trap_exit,false},{reductions,336485},{last_calls,false},{catchlevel,5},{trace,0},{suspending,[]},{sequential_trace_token,[]},{error_handler,error_handler},{memory,156866100},{total_heap_size,19608144},{heap_size,8912793},{stack_size,51},{min_heap_size,233}]

错误原因:进程达到了最大的堆内存的告警,消息积压太多了

解决方案

40.关键字: ACL http

日志原文:

evocationClient@SIM00-0000-000-F049@47.103.69.133:32814 [ACL http] Request ACL path /iot/v1alpha1/devices/authorize, error: {closed,"The connection was lost."}

错误原因:http认证服务器连不上

解决方案:检查认证服务器连接

41.关键字: PUBREL

日志原文:

TEST00-0000-000-C001@47.103.69.133:56746 [Channel] The PUBREL PacketId 151 is not found.

错误原因:一种是客户端重复发送了 PUBREL 报文,导致相应的 PacketID 在 EMQ X 这里已经处理掉了;另外一种是客户端发送的 PUBREL 报文里面的 PacketID 是错误的,在这之前客户端没有发送过相应的 PUBLISH 报文。

解决方案:检查客户端

42.关键字: PUBREC

日志原文:

69A386BD-62F9-45B6-B582-5A389380061C@10.0.71.228:54410 [Channel] The PUBREC PacketId 1 is inuse.

错误原因:警告信息,EMQX 这边收到的 PUBREC 包使用的 PacketId 重复了,可能是一条 PUBREC 信息被重复发送了,也可能是用错了 Id, 大多数情况下都是重复发送的问题

解决方案

43.关键字: gen_rpc_client_sup

日志原文:

crasher: initial call: gen_rpc_client:init/1, pid: <0.3835.0>, registered_name: 'gen_rpc.client.emqx@10.13.67.20/2614250', error: {{badmatch,{error,einval}},[{gen_rpc_driver_tcp,set_send_timeout,2,[{file,"gen_rpc_driver_tcp.erl"},{line,191}]},{gen_rpc_client,send_cast,4,[{file,"gen_rpc_client.erl"},{line,418}]},{gen_server,try_dispatch,4,[{file,"gen_server.erl"},{line,689}]},{gen_server,handle_msg,6,[{file,"gen_server.erl"},{line,765}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]}, ancestors: [gen_rpc_client_sup,gen_rpc_sup,<0.2131.0>], message_queue_len: 23, messages: [{tcp_error,#Port<0.300>,econnreset},{tcp_closed,#Port<0.300>},{{cast,emqx_broker,dispatch,[<<"thing/product/mockdevicesn/osd">>,{delivery,<0.24144.1>,{message,<<0,5,218,42,74,192,37,28,39,137,1,0,94,80,0,124>>,0,<<"30bc90c0b567_bench_pub_677_2014658720">>,#{dup => false,retain => false},#{peerhost => {111,222,129,229},properties => #{},proto_ver => 5,protocol => mqtt,username => <<"mockuser">>},<<"thing/product/mockdevicesn/osd">>,<<"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa">>,1647250061141}}]},undefined},{{cast,emqx_broker,dispatch,[<<"thing/product/mockdevicesn/osd">>,{delivery,<0.24637.1>,{message,<<0,5,218,42,74,192,43,42,39,137,1,0,96,61,0,123>>,0,<<"30bc90c0b567_bench_pub_776_2348236373">>,#{dup => false,retain => false},#{peerhost => {111,222,129,229},properties => #{},proto_ver => 5,protocol => mqtt,username => <<"mockuser">>},<<"thing/product/mockdevicesn/osd">>

错误原因:与别一个节点的RPC连接不可用了

解决方案:要看另外个节点有什么报错, 是不是重启,还是升级了。还是其它网络原因

44.关键字: Cannot publish message

日志原文:

<<"6afc4fb6-23ee-4010-9971-05e3c2aa5d9a">>@39.144.11.81:34937 [Channel] Cannot publish message to device//events due to Not authorized.

错误原因:没有发布到这个topic的权限

解决方案:开放权限

45.关键字: JT808 Proto

日志原文:

[JT808 Proto] Unexpected frame #{<<"body">> => #{},                                 <<"header">> =>                                     #{<<"encrypt">> => 0,<<"len">> => 0,                                       <<"msg_id">> => 3,<<"msg_sn">> => 9,                                       <<"phone">> => <<"014161736453">>}}2022-03-14 01:18:05.107 [error] ** Generic server <0.646.655> terminating ** Last message in was {inet_async,#Port<0.6944860>,0,                           {ok,<<126,0,3,0,0,1,65,97,115,100,83,0,9,111,126>>}}** When Server state == {state,esockd_transport,#Port<0.6944860>,                            {{117,132,196,139},22319},                            {not_detect_heading_0x7e,8192},                            {pstate,                                #{conn_mod => emqx_jt808_connection,                                  peercert => nossl,                                  peername => {{117,132,196,139},22319},                                  sendfun =>                                      {fun emqx_jt808_connection:send/3,                                       [esockd_transport,#Port<0.6944860>]},                                  sockname => {{10,65,141,225},8090},                                  socktype => tcp},                                #{clientid => undefined,is_bridge => false,                                  is_superuser => false,                                  mountpoint => undefined,                                  peerhost => {117,132,196,139},                                  protocol => jt808,sockport => 8090,                                  username => undefined,zone => undefined},                                undefined,undefined,undefined,                                {fun emqx_jt808_connection:send/3,                                 [esockd_transport,#Port<0.6944860>]},                                undefined,0,<<"jt808/%c/dn">>,                                <<"jt808/%c/up">>,                                {auth,undefined,undefined,true},                                [0,                                 <<0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,                                   0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,                                   0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,                                   0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,                                   0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,                                   0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,                                   0,0>>],                                {inflight,128,{0,nil}},                                undefined,undefined},                            undefined,undefined,8192,true,30000}** Reason for termination ==** unexpected_frame2022-03-14 01:18:05.108 [error]   crasher:    initial call: emqx_jt808_connection:init/1    pid: <0.646.655>    registered_name: []    exception exit: unexpected_frame      in function  gen_server:handle_common_reply/8 (gen_server.erl, line 751)    ancestors: [<0.2670.0>,<0.2669.0>,esockd_sup,<0.2055.0>]    message_queue_len: 1    messages: [{inet_async,#Port<0.6944860>,1,{error,closed}}]    links: [<0.2670.0>]    dictionary: []    trap_exit: false    status: running    heap_size: 6772    stack_size: 27    reductions: 15706  neighbours

错误原因:解析失败了,设备上来的数据不对

解决方案  

46.关键字: rocketmq

日志原文:

State machine <0.5958.0> terminating. Reason: {error,econnrefused}. Stack: [{gen_statem,loop_state_callback_result,11,[{file,"gen_statem.erl"},{line,1360}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]. Last event: {info,connecting}. State: {idle,{state,<<"bridge_rocketmq:resource:d0de7c39_al_device_up">>,<<"al_device_up">>,<<"172.24.1.152:10911">>,undefined,5,17208,[{sndbuf,1048576}],{emqx_bridge_rocket_actions,rocket_callback,[<<"data_to_rocket_1646912587576490567">>]},100,#{},<<>>}}.

错误原因:连接被rocketmq服务器拒绝

解决方案:检查rocketmq的连接

47.关键字: esockd_connection_sup

日志原文:

supervisor: 'esockd_connection_sup - <0.2390.0>', errorContext: connection_shutdown, reason: {badmatch,<<>>}, offender: [{pid,<0.10594.41>},{name,connection},{mfargs,{emqx_connection,start_link,[[{deflate_options,[]},{max_conn_rate,1000},{active_n,100},{zone,external},{proxy_address_header,<<>>},{proxy_port_header,<<>>},{supported_subprotocols,[]}]]}}]

错误原因:连接还没有创建完成,就被断开了

解决方案:检查连接

48.关键字: get_status_of_pool_workers

日志原文:

get_status_of_pool_workers failed: {throw,{timeout,<0.8230.420>}}, stacktrace: []

错误原因:在创建测试资源的进程池时,测试的资源没有在1秒内回复正常.

举个例子:如果是mongo,创建测试资源时,去查执行一个最简单的mongo查询,他没有在一秒内收到回复

解决方案

49.关键字: Take action

日志原文:

<<"android:7706416585007415566:7b6a3110779b89e3">>@112.97.83.197:36545 Take action <<"data_to_kafka_1646315366030644983">> failed, continue next action, reason: {error,timeout,[{wolff_producer,send_sync,3,[{file,"wolff_producer.erl"},{line,139}]},{emqx_bridge_kafka_actions,produce,6,[{file,"emqx_bridge_kafka_actions.erl"},{line,408}]},{emqx_rule_runtime,take_action,5,[{file,"emqx_rule_runtime.erl"},{line,236}]},{emqx_rule_runtime,'-take_actions/4-lc$^0/1-0-',4,[{file,"emqx_rule_runtime.erl"},{line,227}]},{emqx_rule_runtime,'-take_actions/4-lc$^0/1-0-',4,[{file,"emqx_rule_runtime.erl"},{line,228}]},{emqx_rule_runtime,do_apply_rule,2,[{file,"emqx_rule_runtime.erl"},{line,110}]},{emqx_rule_runtime,apply_rules,2,[{file,"emqx_rule_runtime.erl"},{line,52}]},{emqx_hooks,safe_execute,2,[{file,"emqx_hooks.erl"},{line,164}]}]}

错误原因 Kafka连接timeout

解决方案 检查相关填写参数是否正确,以及Kafka服务是否正常。

50.关键字: epgsql

日志原文:

Generic server <0.27348.0> terminating. Reason: nxdomain. Last message: {command,epgsql_cmd_connect,#{database => "dwhdb",ecpool_worker_id => 1,host => "matrixdb.c4.srv",password => #Fun<epgsql_cmd_connect.0.29916615>,port => 80,username => "mxadmin"}}. State: {state,undefined,undefined,<<>>,undefined,on_message,undefined,{[],[]},undefined,undefined,undefined,undefined,[],information_redacted,[],undefined,undefined,undefined,undefined,undefined}. Client <0.27347.0> stacktrace: [{logger_config,allow,2,[{file,"logger_config.erl"},{line,64}]},{proc_lib,crash_report,4,[{file,"proc_lib.erl"},{line,525}]},{proc_lib,exit_p,3,[{file,"proc_lib.erl"},{line,246}]}].

错误原因:matrixdb需要进行设置pg_hba.conf才可以连接数据库

解决方案:进行设置

51.关键字: Quota exceeded

日志原文:

mqtt-benchmark-227@100.121.120.249:57499 [Channel] Cannot publish messages to thing/product/mockdevicesn/osd due to Quota exceeded.

错误原因 :超出配额限制

解决方案:避免超出配额限制。

52.关键字: WebHook Action

日志原文:

d3dea20d-f45f-4243-8bc0-ddd556472708@39.144.5.19:16876 [WebHook Action] HTTP request error: timeout

错误原因:请求地址超时

解决方案:检查URL地址是否能正常通信

53.关键字: cluster_call error found

日志原文:

cluster_call error found, ResL: [{{init_action_failure,'emqx@10.14.69.101'},{{emqx_bridge_kafka_actions,on_action_create_data_to_kafka},{error,{error,kafka_topic_not_found},[{emqx_bridge_kafka_actions,check_kafka_topic,2,[{file,"emqx_bridge_kafka_actions.erl"},{line,459}]},{emqx_bridge_kafka_actions,on_action_create_data_to_kafka,2,[{file,"emqx_bridge_kafka_actions.erl"},{line,386}]},{emqx_rule_engine,'-init_action/4-fun-0-',4,[{file,"emqx_rule_engine.erl"},{line,551}]},{emqx_rule_engine,init_action,4,[{file,"emqx_rule_engine.erl"},{line,551}]},{erpc,execute_call,4,[{file,"erpc.erl"},{line,416}]}]}}},{{init_action_failure,'emqx@10.14.69.7'},{{emqx_bridge_kafka_actions,on_action_create_data_to_kafka},{error,{error,kafka_topic_not_found},[{emqx_bridge_kafka_actions,check_kafka_topic,2,[{file,"emqx_bridge_kafka_actions.erl"},{line,459}]},{emqx_bridge_kafka_actions,on_action_create_data_to_kafka,2,[{file,"emqx_bridge_kafka_actions.erl"},{line,386}]},{emqx_rule_engine,'-init_action/4-fun-0-',4,[{file,"emqx_rule_engine.erl"},{line,551}]},{emqx_rule_engine,init_action,4,[{file,"emqx_rule_engine.erl"},{line,551}]},{erpc,execute_call,4,[{file,"erpc.erl"},{line,416}]}]}}}]

错误原因:Kafka topic没找到

解决方案:请创建相应topic

54.关键字: Auth http

日志原文:

CENSYS@167.94.146.58:45530 [Auth http] Deny connection from path: /emqx/mqtt/auth, response http code: 400

错误原因:HTTP认证服务拒绝连接

解决方案:检查HTTP服务

55.关键字: PUBACK

日志原文:

76EE2AF1220D@121.69.9.98:50634 [Channel] The PUBACK PacketId 6 is not found.

错误原因:PUBACK报文PacketId 6没找到

解决方案

56.关键字: Parse failed for {badmatch

日志原文:

172.105.87.91:35516 [MQTT] , Parse failed for {badmatch,<<>>}, [{emqx_frame,parse_packet,3,[{file,"emqx_frame.erl"},{line,237}]},{emqx_frame,parse_frame,4,[{file,"emqx_frame.erl"},{line,201}]},{emqx_connection,parse_incoming,3,[{file,"emqx_connection.erl"},{line,625}]},{emqx_connection,handle_msg,2,[{file,"emqx_connection.erl"},{line,618}]},{emqx_connection,process_msg,2,[{file,"emqx_connection.erl"},{line,364}]},{emqx_connection,handle_recv,3,[{file,"emqx_connection.erl"},{line,328}]},{proc_lib,wake_up,3,[{file,"proc_lib.erl"},{line,236}]}], Frame data:<<22,3,0,0,83,1,0,0,79,3,0,63,71,215,247,186,44,238,234,178,96,126,243,0,253,130,123,185,213,150,200,119,155,230,196,219,60,61,219,111,239,16,110,0,0,40,0,22,0,19,0,10,0,102,0,5,0,4,0,101,0,100,0,99,0,98,0,97,0,96,0,21,0,18,0,9,0,20,0,17,0,8,0,6,0,3,1,0>>

错误原因:报文解析失败

解决方案:检查发布报文

57.关键字: Connection rejected

日志原文:

47.92.85.45:47136 Connection rejected due to max clients limitation

错误原因:达到最大连接数限制

解决方案:请保持连接数在限制以内。

58.关键字: ehttpc

日志原文:

ehttpc: Received 'gun_data' message from unknown stream ref: #Ref<0.1986289746.2927886337.204390>

错误原因:内部把LOG打错了

解决方案:已修复

59.关键字: JT808 Frame

日志原文:

[JT808 Frame] unknow message id 256, <<0,44,1,47,55,48,49,49,49,77,71,45,49,49,                                       0,0,0,48,48,48,48,48,48,48,1,212,193,66,                                       56,56,56,56,56>>2022-03-14 01:15:30.139 [error] [JT808 Conn] Parser failed for {invalid_message,                                [{emqx_jt808_frame,parse_message_body,2,                                  [{file,"emqx_jt808_frame.erl"},{line,219}]},                                 {emqx_jt808_frame,parse_message,1,                                  [{file,"emqx_jt808_frame.erl"},{line,121}]},                                 {emqx_jt808_frame,parse_main,3,                                  [{file,"emqx_jt808_frame.erl"},{line,69}]},                                 {emqx_jt808_connection,received,2,                                  [{file,"emqx_jt808_connection.erl"},                                   {line,243}]},                                 {gen_server,try_dispatch,4,                                  [{file,"gen_server.erl"},{line,637}]},                                 {gen_server,handle_msg,6,                                  [{file,"gen_server.erl"},{line,711}]},                                 {proc_lib,init_p_do_apply,3,                                  [{file,"proc_lib.erl"},{line,249}]}]}2022-03-14 01:15:30.139 [error] [JT808 Conn] Error data: <<126,1,0,0,33,1,65,97,115,100,83,0,0,0,44,1,47,55,48,                           49,49,49,77,71,45,49,49,0,0,0,48,48,48,48,48,48,48,                           1,212,193,66,56,56,56,56,56,8,126>>

错误原因:发的消息不符合jt808 protocol规范

解决方案

60.关键字: Unexpected sock_closed

日志原文:

<<"emq_4b74fa73b8454fe2a7bcbe5b2b4e8066">>@183.129.130.2:55474 [Channel] Unexpected sock_closed: tcp_closed

错误原因:socket被close了

解决方案:查传输层面的影响 比如网络或os


    您需要登录后才可以回复