Mysql server reboots every 3 minutes

In /var/log/mysql/error.log repeated messages like

Kernel version: Linux version 5.4.0-125-generic ([email protected]) (gcc version 9.4.0 (Ubuntu 9.4.0-1ubuntu1~20.04.1)) #141-Ubuntu SMP Wed Aug 10 13:42:03 UTC 2022

2022-09-11 12:33:37 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2022-09-11 12:33:37 0 [Note] InnoDB: Number of pools: 1
2022-09-11 12:33:37 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
2022-09-11 12:33:37 0 [Note] InnoDB: Using Linux native AIO
2022-09-11 12:33:37 0 [Note] InnoDB: Initializing buffer pool, total size = 8589934592, chunk size = 134217728
2022-09-11 12:33:38 0 [Note] InnoDB: Completed initialization of buffer pool
2022-09-11 12:33:38 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=137214403179,137214403179
2022-09-11 12:33:41 0 [Note] InnoDB: 2 transaction(s) which must be rolled back or cleaned up in total 192 row operations to undo
2022-09-11 12:33:41 0 [Note] InnoDB: Trx id counter is 1433585
2022-09-11 12:33:41 0 [Note] InnoDB: Starting final batch to recover 29975 pages from redo log.
2022-09-11 12:33:41 0 [Note] InnoDB: 128 rollback segments are active.
2022-09-11 12:33:41 0 [Note] InnoDB: Starting in background the rollback of recovered transactions
2022-09-11 12:33:41 0 [Note] InnoDB: Removed temporary tablespace data file: "./ibtmp1"
2022-09-11 12:33:41 0 [Note] InnoDB: Creating shared tablespace for temporary tables
2022-09-11 12:33:41 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
2022-09-11 12:33:41 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
2022-09-11 12:33:41 0 [Note] InnoDB: 10.6.9 started; log sequence number 138501288720; transaction id 1433586
2022-09-11 12:33:41 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
2022-09-11 12:33:41 0 [Note] Plugin 'FEEDBACK' is disabled.
2022-09-11 12:33:41 0 [Warning] You need to use --log-bin to make --expire-logs-days or --binlog-expire-logs-seconds work.
2022-09-11 12:33:41 0 [Note] Server socket created on IP: '127.0.0.1'.
2022-09-11 12:33:41 0 [Note] InnoDB: Rolled back recovered transaction 1433575
2022-09-11 12:33:41 0 [Note] InnoDB: Rolled back recovered transaction 1433580
2022-09-11 12:33:41 0 [Note] InnoDB: Rollback of non-prepared transactions completed
2022-09-11 12:33:41 0 [Note] /usr/sbin/mariadbd: ready for connections.
Version: '10.6.9-MariaDB-1:10.6.9+maria~ubu2004-log'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  mariadb.org binary distribution
2022-09-11 12:33:44 0 [Note] InnoDB: Buffer pool(s) load completed at 220911 12:33:43
2022-09-11 12:39:21 1077 [ERROR] InnoDB: (Duplicate key) writing word node to FTS auxiliary index table `psyche_psyche`.`ibf_core_message_posts`
2022-09-11 12:39:21 0x7fca6c332700  InnoDB: Assertion failure in file /home/buildbot/buildbot/build/mariadb-10.6.9/storage/innobase/que/que0que.cc line 728
InnoDB: Failing assertion: trx->error_state == DB_SUCCESS
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mariadbd startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: https://mariadb.com/kb/en/library/innodb-recovery-modes/
InnoDB: about forcing recovery.
220911 12:39:21 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.

To report this bug, see https://mariadb.com/kb/en/reporting-bugs

We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.

Server version: 10.6.9-MariaDB-1:10.6.9+maria~ubu2004-log
key_buffer_size=268435456
read_buffer_size=1048576
max_used_connections=6
max_threads=302
thread_count=6
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 40162976 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x7fc7dc000c58
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x7fca6c331d98 thread_stack 0x49000
??:0(my_print_stacktrace)[0x559d5392ef62]
??:0(handle_fatal_signal)[0x559d533effb5]
??:0(__restore_rt)[0x7fca72852420]
??:0(gsignal)[0x7fca7235600b]
??:0(abort)[0x7fca72335859]
??:0(Wsrep_server_service::log_dummy_write_set(wsrep::client_state&, wsrep::ws_meta const&))[0x559d53075e5d]
??:0(Wsrep_server_service::log_dummy_write_set(wsrep::client_state&, wsrep::ws_meta const&))[0x559d53061a9a]
??:0(void std::vector<unsigned long, std::allocator<unsigned long> >::_M_realloc_insert<unsigned long>(__gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, unsigned long&&))[0x559d5389e82d]
??:0(void std::vector<unsigned long, std::allocator<unsigned long> >::_M_realloc_insert<unsigned long>(__gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, unsigned long&&))[0x559d5388bd53]
??:0(void std::vector<unsigned long, std::allocator<unsigned long> >::_M_realloc_insert<unsigned long>(__gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, unsigned long&&))[0x559d53891091]
??:0(void std::vector<unsigned long, std::allocator<unsigned long> >::_M_realloc_insert<unsigned long>(__gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, unsigned long&&))[0x559d53892915]
??:0(void std::vector<unsigned long, std::allocator<unsigned long> >::_M_realloc_insert<unsigned long>(__gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, unsigned long&&))[0x559d53892b39]
??:0(void std::vector<unsigned long, std::allocator<unsigned long> >::_M_realloc_insert<unsigned long>(__gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, unsigned long&&))[0x559d53892d0c]
??:0(main)[0x559d530d2356]
??:0(std::_Rb_tree<unsigned int, unsigned int, std::_Identity<unsigned int>, std::less<unsigned int>, std::allocator<unsigned int> >::_M_erase(std::_Rb_tree_node<unsigned int>*))[0x559d537e6ff9]
??:0(std::_Rb_tree<unsigned int, unsigned int, std::_Identity<unsigned int>, std::less<unsigned int>, std::allocator<unsigned int> >::_M_erase(std::_Rb_tree_node<unsigned int>*))[0x559d537e71f2]
??:0(wsrep_notify_status(wsrep::server_state::state, wsrep::view const*))[0x559d536dbc90]
??:0(ha_check_and_coalesce_trx_read_only(THD*, Ha_trx_info*, bool))[0x559d533f3cd9]
??:0(ha_commit_trans(THD*, bool))[0x559d533f4db6]
??:0(trans_commit_stmt(THD*))[0x559d532e0537]
??:0(mysql_execute_command(THD*, bool))[0x559d531c0219]
??:0(Prepared_statement::execute(String*, bool))[0x559d531d456a]
??:0(Prepared_statement::execute_loop(String*, bool, unsigned char*, unsigned char*))[0x559d531d4791]
??:0(Prepared_statement::execute_bulk_loop(String*, bool, unsigned char*, unsigned char*))[0x559d531d5675]
??:0(mysqld_stmt_execute(THD*, char*, unsigned int))[0x559d531d5850]
??:0(dispatch_command(enum_server_command, THD*, char*, unsigned int, bool))[0x559d531bd17b]
??:0(do_command(THD*, bool))[0x559d531be718]
??:0(do_handle_one_connection(CONNECT*, bool))[0x559d532d0137]
??:0(handle_one_connection)[0x559d532d048d]
??:0(MyCTX_nopad::finish(unsigned char*, unsigned int*))[0x559d53636916]
??:0(start_thread)[0x7fca72846609]
??:0(clone)[0x7fca72432133]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0x7fc7dc010af0): /*psyche_psyche::psyche_psycho::IPS\Content\_Comment::create:151*/ INSERT INTO `ibf_core_message_posts` ( `msg_topic_id`, `msg_date`, `msg_author_id`, `msg_post`, `msg_ip_address`, `msg_is_first_post` ) VALUES ( 250534, 1662889161, 188044, '<p>\n	<em>Электронная почта? Новый форум?</em>\n</p>\n\n<p>\n	Как Вам удобно. Мне без разницы.\n</p>\n\n<p>\n	 \n</p>\n\n<p>\n	Напишите мне по адресу: [email protected], и дальше разберемся. Я уже полтора часа бьюсь над тем, чтобы отправить гребанное сообщение.\n</p>\n', '89.43.20.129', 0 )

Connection ID (thread ID): 1077
Status: NOT_KILLED

Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on,not_null_range_scan=off

The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
information that should help you find out what is causing the crash.
Writing a core file...
Working directory at /var/lib/mysql
Resource Limits:
Limit                     Soft Limit           Hard Limit           Units     
Max cpu time              unlimited            unlimited            seconds   
Max file size             unlimited            unlimited            bytes     
Max data size             unlimited            unlimited            bytes     
Max stack size            8388608              unlimited            bytes     
Max core file size        0                    unlimited            bytes     
Max resident set          unlimited            unlimited            bytes     
Max processes             63803                63803                processes 
Max open files            32768                32768                files     
Max locked memory         65536                65536                bytes     
Max address space         unlimited            unlimited            bytes     
Max file locks            unlimited            unlimited            locks     
Max pending signals       63803                63803                signals   
Max msgqueue size         819200               819200               bytes     
Max nice priority         0                    0                    
Max realtime priority     0                    0                    
Max realtime timeout      unlimited            unlimited            us        
Core pattern: core

Kernel version: Linux version 5.4.0-125-generic ([email protected]) (gcc version 9.4.0 (Ubuntu 9.4.0-1ubuntu1~20.04.1)) #141-Ubuntu SMP Wed Aug 10 13:42:03 UTC 2022

2022-09-11 12:39:28 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2022-09-11 12:39:28 0 [Note] InnoDB: Number of pools: 1
2022-09-11 12:39:28 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
2022-09-11 12:39:28 0 [Note] InnoDB: Using Linux native AIO
2022-09-11 12:39:28 0 [Note] InnoDB: Initializing buffer pool, total size = 8589934592, chunk size = 134217728
2022-09-11 12:39:28 0 [Note] InnoDB: Completed initialization of buffer pool
2022-09-11 12:39:28 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=137214403179,137214403179
2022-09-11 12:39:32 0 [Note] InnoDB: 2 transaction(s) which must be rolled back or cleaned up in total 196 row operations to undo
2022-09-11 12:39:32 0 [Note] InnoDB: Trx id counter is 1435689
2022-09-11 12:39:32 0 [Note] InnoDB: Starting final batch to recover 30207 pages from redo log.
2022-09-11 12:39:33 0 [Note] InnoDB: 128 rollback segments are active.
2022-09-11 12:39:33 0 [Note] InnoDB: Starting in background the rollback of recovered transactions
2022-09-11 12:39:33 0 [Note] InnoDB: Removed temporary tablespace data file: "./ibtmp1"
2022-09-11 12:39:33 0 [Note] InnoDB: Creating shared tablespace for temporary tables
2022-09-11 12:39:33 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
2022-09-11 12:39:33 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
2022-09-11 12:39:33 0 [Note] InnoDB: 10.6.9 started; log sequence number 138591958786; transaction id 1435690
2022-09-11 12:39:33 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
2022-09-11 12:39:33 0 [Note] Plugin 'FEEDBACK' is disabled.
2022-09-11 12:39:33 0 [Note] InnoDB: Rolled back recovered transaction 1435677
2022-09-11 12:39:33 0 [Warning] You need to use --log-bin to make --expire-logs-days or --binlog-expire-logs-seconds work.
2022-09-11 12:39:33 0 [Note] Server socket created on IP: '127.0.0.1'.
2022-09-11 12:39:33 0 [Note] InnoDB: Rolled back recovered transaction 1435682
2022-09-11 12:39:33 0 [Note] InnoDB: Rollback of non-prepared transactions completed
2022-09-11 12:39:33 0 [Note] /usr/sbin/mariadbd: ready for connections.
Version: '10.6.9-MariaDB-1:10.6.9+maria~ubu2004-log'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  mariadb.org binary distribution
2022-09-11 12:39:34 55 [ERROR] mariadbd: Table './forumz_psycheua/ibf_core_members' is marked as crashed and should be repaired
2022-09-11 12:39:34 55 [Warning] Checking table:   './forumz_psycheua/ibf_core_members'
2022-09-11 12:39:36 0 [Note] InnoDB: Buffer pool(s) load completed at 220911 12:39:36

Looks some values are wrong did you make some changes

1 Like

Yes I did. Problem was with:

sort_buffer_size = 128M
thread_cache_size = 302

Thanks for help! :heart: :heart: :heart: