背景描述
SonarQube 自去年使用开始，已运行一年有余
它上面的 Project 数量已超过 1000
因为每个应用的每个 git 分支的每次 push 事件都会触发 Sonar 分析任务
这样会有很多的数据都存储下来，从而导致数据量很大
目前 Mysql 数据库占用磁盘空间达到100G
并且随着时间推移会继续增加
于是在想是否可以进行数据清理，以降低所使用的磁盘空间
如何进行数据清理？
那么如何进行数据清理呢？
经过不断探索与实践，形成如下解决方案：
1、调小【数据库清理器】相关参数
2、设置 master 为长期分支，并修改【保留短期分支】参数
3、清理非主分支的长期分支
4、MySQL Innodb DATA_FREE 清理
调小【数据库清理器】相关参数
配置—>通用配置 —> 数据库清理器
调小【数据库清理器】相关参数，缩短数据保留时间，以便尽早释放空间
设置 master 为长期分支
配置—>通用配置 —> Branches
默认情况下，名称以 “branch” 或 “release” 开头的分支将被视为长期分支，长期分支系统不会做自动清理。
将 master 设置为长期分支，那么其它分支就是短期分支，短期分支系统会做自动清理。
当然已经存在的长期分支不会自动更改为短期分支，需要先删除它，然后重新在这个分支执行 sonar 分析任务，那么它就会变为短期分支
此外，修改【清除不活跃的短分支前的保留天数】参数，改为10
使用长期分支模式后，每个 Project 分支页面如下：
短期分支会显示和长期分支有差异的 issue
清理非主分支的长期分支
修改为长期分支模式的两个月后，清理非主分支的长期分支
这里使用了 SonarQube API 进行了批量清理，清理任务花费了「三个多小时」～
相关 python 脚本可以参考：https://github.com/donhui/python-sonarqube/blob/master/sonarqube_branches_clean.py
MySQL Innodb DATA_FREE 清理
对 SonarQube 数据库进行分析：
1mysql> SELECT table_schema "DB Name", Round(Sum(data_length + index_length) / 1024 / 1024, 1) "DB Size in MB" FROM information_schema.tables GROUP BY table_schema; 2+--------------------------+---------------+ 3| DB Name | DB Size in MB | 4+--------------------------+---------------+ 5| dingtalk_develop_members | 0.1 | 6| information_schema | 0.2 | 7| sonar | 90477.8 | 8| sonar_statistics | 5.1 | 9+--------------------------+---------------+ 104 rows in set (0.30 sec) 11 12mysql> SELECT table_name AS `Table`, round(((data_length + index_length) / 1024 / 1024), 2) `Size in MB` FROM information_schema.TABLES WHERE table_schema = 'sonar'; 13+--------------------------+------------+ 14| Table | Size in MB | 15+--------------------------+------------+ 16| active_rule_parameters | 0.03 | 17| active_rules | 0.42 | 18| analysis_properties | 0.02 | 19| ce_activity | 32.55 | 20| ce_queue | 0.05 | 21| ce_scanner_context | 619.05 | 22| ce_task_characteristics | 51.19 | 23| ce_task_input | 1.52 | 24| default_qprofiles | 0.03 | 25| duplications_index | 0.05 | 26| es_queue | 0.03 | 27| events | 6.20 | 28| file_sources | 33973.41 | 29| group_roles | 2.53 | 30| groups | 0.02 | 31| groups_users | 0.23 | 32| internal_properties | 0.02 | 33| issue_changes | 160.84 | 34| issues | 372.92 | 35| loaded_templates | 0.03 | 36| manual_measures | 0.02 | 37| metrics | 0.08 | 38| notifications | 0.02 | 39| org_qprofiles | 0.05 | 40| organization_members | 0.05 | 41| organizations | 0.03 | 42| perm_templates_groups | 0.02 | 43| perm_templates_users | 0.02 | 44| perm_tpl_characteristics | 0.02 | 45| permission_templates | 0.02 | 46| plugins | 0.03 | 47| project_branches | 1.80 | 48| project_links | 4.52 | 49| project_measures | 49670.27 | 50| project_qprofiles | 0.02 | 51| projects | 5483.33 | 52| properties | 0.03 | 53| qprofile_changes | 1.92 | 54| qprofile_edit_groups | 0.02 | 55| qprofile_edit_users | 0.02 | 56| quality_gate_conditions | 0.02 | 57| quality_gates | 0.02 | 58| rule_repositories | 0.02 | 59| rules | 3.70 | 60| rules_metadata | 0.02 | 61| rules_parameters | 0.08 | 62| rules_profiles | 0.03 | 63| schema_migrations | 0.02 | 64| snapshots | 7.69 | 65| user_roles | 0.78 | 66| user_tokens | 0.05 | 67| users | 0.14 | 68| webhook_deliveries | 0.05 | 69+--------------------------+------------+ 7053 rows in set (0.02 sec)
数据量大的表：
1| ce_scanner_context | 619.05 | 2| file_sources | 33973.41 | 3| issue_changes | 160.84 | 4| issues | 372.92 | 5| project_measures | 49670.27 | 6| projects | 5483.33 |
清理非主分支的长期分支后，发现 data_length + index_length 下降，但是磁盘空间并没有减少
向 DBA 求助，得到的解决方案为：清理数据库 DATA_FREE，回收表空间
查看表相关的元数据信息，包括 TABLE_NAME,ROW_FORMAT,TABLE_ROWS,DATA_LENGTH,INDEX_LENGTH,MAX_DATA_LENGTH,DATA_FREE,ENGINE ，SQL 语句及输出如下：
1mysql> SELECT TABLE_NAME,ROW_FORMAT,TABLE_ROWS,DATA_LENGTH,INDEX_LENGTH,MAX_DATA_LENGTH,DATA_FREE,ENGINE FROM information_schema.TABLES WHERE TABLE_SCHEMA='sonar'; 2+--------------------------+------------+------------+-------------+--------------+-----------------+-------------+--------+ 3| TABLE_NAME | ROW_FORMAT | TABLE_ROWS | DATA_LENGTH | INDEX_LENGTH | MAX_DATA_LENGTH | DATA_FREE | ENGINE | 4+--------------------------+------------+------------+-------------+--------------+-----------------+-------------+--------+ 5| active_rule_parameters | Dynamic | 138 | 16384 | 16384 | 0 | 0 | InnoDB | 6| active_rules | Dynamic | 4385 | 294912 | 147456 | 0 | 0 | InnoDB | 7| analysis_properties | Dynamic | 0 | 16384 | 0 | 0 | 0 | InnoDB | 8| ce_activity | Dynamic | 74861 | 16793600 | 17334272 | 0 | 4194304 | InnoDB | 9| ce_queue | Dynamic | 0 | 16384 | 32768 | 0 | 0 | InnoDB | 10| ce_scanner_context | Dynamic | 17122 | 649117696 | 0 | 0 | 1286602752 | InnoDB | 11| ce_task_characteristics | Dynamic | 228069 | 36290560 | 17383424 | 0 | 5242880 | InnoDB | 12| ce_task_input | Dynamic | 0 | 344064 | 0 | 0 | 113246208 | InnoDB | 13| default_qprofiles | Dynamic | 14 | 16384 | 16384 | 0 | 0 | InnoDB | 14| duplications_index | Dynamic | 0 | 16384 | 32768 | 0 | 0 | InnoDB | 15| es_queue | Dynamic | 0 | 16384 | 16384 | 0 | 5242880 | InnoDB | 16| events | Dynamic | 2882 | 3342336 | 2457600 | 0 | 5242880 | InnoDB | 17| file_sources | Dynamic | 168412 | 8828141568 | 281624576 | 0 | 33689698304 | InnoDB | 18| group_roles | Dynamic | 7000 | 507904 | 2146304 | 0 | 6291456 | InnoDB | 19| groups | Dynamic | 48 | 16384 | 0 | 0 | 0 | InnoDB | 20| groups_users | Dynamic | 1060 | 81920 | 163840 | 0 | 0 | InnoDB | 21| internal_properties | Dynamic | 3 | 16384 | 0 | 0 | 0 | InnoDB | 22| issue_changes | Dynamic | 762484 | 104284160 | 64372736 | 0 | 6291456 | InnoDB | 23| issues | Dynamic | 184354 | 200441856 | 164954112 | 0 | 35651584 | InnoDB | 24| loaded_templates | Dynamic | 2 | 16384 | 16384 | 0 | 0 | InnoDB | 25| manual_measures | Dynamic | 0 | 16384 | 0 | 0 | 0 | InnoDB | 26| metrics | Dynamic | 169 | 65536 | 16384 | 0 | 0 | InnoDB | 27| notifications | Dynamic | 0 | 16384 | 0 | 0 | 6291456 | InnoDB | 28| org_qprofiles | Dynamic | 34 | 16384 | 32768 | 0 | 0 | InnoDB | 29| organization_members | Dynamic | 393 | 49152 | 0 | 0 | 0 | InnoDB | 30| organizations | Dynamic | 1 | 16384 | 16384 | 0 | 0 | InnoDB | 31| perm_templates_groups | Dynamic | 4 | 16384 | 0 | 0 | 0 | InnoDB | 32| perm_templates_users | Dynamic | 0 | 16384 | 0 | 0 | 0 | InnoDB | 33| perm_tpl_characteristics | Dynamic | 0 | 16384 | 0 | 0 | 0 | InnoDB | 34| permission_templates | Dynamic | 0 | 16384 | 0 | 0 | 0 | InnoDB | 35| plugins | Dynamic | 23 | 16384 | 16384 | 0 | 0 | InnoDB | 36| project_branches | Dynamic | 1673 | 229376 | 114688 | 0 | 0 | InnoDB | 37| project_links | Dynamic | 26829 | 4734976 | 0 | 0 | 4194304 | InnoDB | 38| project_measures | Dynamic | 14545516 | 7061143552 | 10892787712 | 0 | 36237737984 | InnoDB | 39| project_qprofiles | Dynamic | 0 | 16384 | 0 | 0 | 0 | InnoDB | 40| projects | Dynamic | 398678 | 1495941120 | 1684144128 | 0 | 2999975936 | InnoDB | 41| properties | Dynamic | 125 | 16384 | 16384 | 0 | 0 | InnoDB | 42| qprofile_changes | Dynamic | 5049 | 1589248 | 425984 | 0 | 4194304 | InnoDB | 43| qprofile_edit_groups | Dynamic | 0 | 16384 | 0 | 0 | 0 | InnoDB | 44| qprofile_edit_users | Dynamic | 0 | 16384 | 0 | 0 | 0 | InnoDB | 45| quality_gate_conditions | Dynamic | 12 | 16384 | 0 | 0 | 0 | InnoDB | 46| quality_gates | Dynamic | 0 | 16384 | 0 | 0 | 0 | InnoDB | 47| rule_repositories | Dynamic | 37 | 16384 | 0 | 0 | 0 | InnoDB | 48| rules | Dynamic | 3185 | 3686400 | 196608 | 0 | 4194304 | InnoDB | 49| rules_metadata | Dynamic | 5 | 16384 | 0 | 0 | 0 | InnoDB | 50| rules_parameters | Dynamic | 328 | 65536 | 16384 | 0 | 0 | InnoDB | 51| rules_profiles | Dynamic | 33 | 16384 | 16384 | 0 | 0 | InnoDB | 52| schema_migrations | Dynamic | 237 | 16384 | 0 | 0 | 0 | InnoDB | 53| snapshots | Dynamic | 4715 | 1589248 | 327680 | 0 | 2097152 | InnoDB | 54| user_roles | Dynamic | 6338 | 507904 | 311296 | 0 | 0 | InnoDB | 55| user_tokens | Dynamic | 34 | 16384 | 32768 | 0 | 0 | InnoDB | 56| users | Dynamic | 438 | 114688 | 32768 | 0 | 0 | InnoDB | 57| webhook_deliveries | Dynamic | 0 | 16384 | 32768 | 0 | 0 | InnoDB | 58+--------------------------+------------+------------+-------------+--------------+-----------------+-------------+--------+ 5953 rows in set (0.00 sec)
数据库清理 DATA_FREE，回收表空间，SQL 语句及输出如下：
1mysql> ALTER TABLE file_sources ENGINE=InnoDB; 2Query OK, 0 rows affected (2 min 41.44 sec) 3Records: 0 Duplicates: 0 Warnings: 0 4 5mysql> ALTER TABLE issue_changes ENGINE=InnoDB; 6Query OK, 0 rows affected (4.39 sec) 7Records: 0 Duplicates: 0 Warnings: 0 8 9mysql> ALTER TABLE ce_scanner_context ENGINE=InnoDB; 10Query OK, 0 rows affected (14.82 sec) 11Records: 0 Duplicates: 0 Warnings: 0 12 13mysql> ALTER TABLE project_measures ENGINE=InnoDB; 14Query OK, 0 rows affected (3 min 55.74 sec) 15Records: 0 Duplicates: 0 Warnings: 0 16 17mysql> ALTER TABLE projects ENGINE=InnoDB; 18Query OK, 0 rows affected (25.27 sec) 19Records: 0 Duplicates: 0 Warnings: 0
清理 DATA_FREE 之后，再次查看表相关的元数据信息：
1mysql> SELECT TABLE_NAME,ROW_FORMAT,TABLE_ROWS,DATA_LENGTH,INDEX_LENGTH,MAX_DATA_LENGTH,DATA_FREE,ENGINE FROM information_schema.TABLES WHERE TABLE_SCHEMA='sonar'; 2+--------------------------+------------+------------+-------------+--------------+-----------------+-----------+--------+ 3| TABLE_NAME | ROW_FORMAT | TABLE_ROWS | DATA_LENGTH | INDEX_LENGTH | MAX_DATA_LENGTH | DATA_FREE | ENGINE | 4+--------------------------+------------+------------+-------------+--------------+-----------------+-----------+--------+ 5| active_rule_parameters | Dynamic | 138 | 16384 | 16384 | 0 | 0 | InnoDB | 6| active_rules | Dynamic | 4385 | 294912 | 147456 | 0 | 0 | InnoDB | 7| analysis_properties | Dynamic | 0 | 16384 | 0 | 0 | 0 | InnoDB | 8| ce_activity | Dynamic | 74971 | 16793600 | 17334272 | 0 | 4194304 | InnoDB | 9| ce_queue | Dynamic | 0 | 16384 | 32768 | 0 | 0 | InnoDB | 10| ce_scanner_context | Dynamic | 17451 | 659079168 | 0 | 0 | 6291456 | InnoDB | 11| ce_task_characteristics | Dynamic | 228289 | 36290560 | 17383424 | 0 | 5242880 | InnoDB | 12| ce_task_input | Dynamic | 0 | 1589248 | 0 | 0 | 113246208 | InnoDB | 13| default_qprofiles | Dynamic | 14 | 16384 | 16384 | 0 | 0 | InnoDB | 14| duplications_index | Dynamic | 0 | 16384 | 32768 | 0 | 0 | InnoDB | 15| es_queue | Dynamic | 0 | 16384 | 16384 | 0 | 5242880 | InnoDB | 16| events | Dynamic | 3300 | 3342336 | 2457600 | 0 | 5242880 | InnoDB | 17| file_sources | Dynamic | 122205 | 3011493888 | 22626304 | 0 | 5242880 | InnoDB | 18| group_roles | Dynamic | 7000 | 507904 | 2146304 | 0 | 6291456 | InnoDB | 19| groups | Dynamic | 48 | 16384 | 0 | 0 | 0 | InnoDB | 20| groups_users | Dynamic | 1060 | 81920 | 163840 | 0 | 0 | InnoDB | 21| internal_properties | Dynamic | 3 | 16384 | 0 | 0 | 0 | InnoDB | 22| issue_changes | Dynamic | 731182 | 101318656 | 43089920 | 0 | 6291456 | InnoDB | 23| issues | Dynamic | 184366 | 200441856 | 164954112 | 0 | 35651584 | InnoDB | 24| loaded_templates | Dynamic | 2 | 16384 | 16384 | 0 | 0 | InnoDB | 25| manual_measures | Dynamic | 0 | 16384 | 0 | 0 | 0 | InnoDB | 26| metrics | Dynamic | 169 | 65536 | 16384 | 0 | 0 | InnoDB | 27| notifications | Dynamic | 0 | 16384 | 0 | 0 | 6291456 | InnoDB | 28| org_qprofiles | Dynamic | 34 | 16384 | 32768 | 0 | 0 | InnoDB | 29| organization_members | Dynamic | 393 | 49152 | 0 | 0 | 0 | InnoDB | 30| organizations | Dynamic | 1 | 16384 | 16384 | 0 | 0 | InnoDB | 31| perm_templates_groups | Dynamic | 4 | 16384 | 0 | 0 | 0 | InnoDB | 32| perm_templates_users | Dynamic | 0 | 16384 | 0 | 0 | 0 | InnoDB | 33| perm_tpl_characteristics | Dynamic | 0 | 16384 | 0 | 0 | 0 | InnoDB | 34| permission_templates | Dynamic | 0 | 16384 | 0 | 0 | 0 | InnoDB | 35| plugins | Dynamic | 23 | 16384 | 16384 | 0 | 0 | InnoDB | 36| project_branches | Dynamic | 1686 | 229376 | 114688 | 0 | 0 | InnoDB | 37| project_links | Dynamic | 26866 | 4734976 | 0 | 0 | 4194304 | InnoDB | 38| project_measures | Dynamic | 10381786 | 2282733568 | 1306165248 | 0 | 51380224 | InnoDB | 39| project_qprofiles | Dynamic | 0 | 16384 | 0 | 0 | 0 | InnoDB | 40| projects | Dynamic | 374448 | 314392576 | 132235264 | 0 | 4194304 | InnoDB | 41| properties | Dynamic | 125 | 16384 | 16384 | 0 | 0 | InnoDB | 42| qprofile_changes | Dynamic | 5049 | 1589248 | 425984 | 0 | 4194304 | InnoDB | 43| qprofile_edit_groups | Dynamic | 0 | 16384 | 0 | 0 | 0 | InnoDB | 44| qprofile_edit_users | Dynamic | 0 | 16384 | 0 | 0 | 0 | InnoDB | 45| quality_gate_conditions | Dynamic | 12 | 16384 | 0 | 0 | 0 | InnoDB | 46| quality_gates | Dynamic | 0 | 16384 | 0 | 0 | 0 | InnoDB | 47| rule_repositories | Dynamic | 37 | 16384 | 0 | 0 | 0 | InnoDB | 48| rules | Dynamic | 3185 | 3686400 | 196608 | 0 | 4194304 | InnoDB | 49| rules_metadata | Dynamic | 5 | 16384 | 0 | 0 | 0 | InnoDB | 50| rules_parameters | Dynamic | 328 | 65536 | 16384 | 0 | 0 | InnoDB | 51| rules_profiles | Dynamic | 33 | 16384 | 16384 | 0 | 0 | InnoDB | 52| schema_migrations | Dynamic | 237 | 16384 | 0 | 0 | 0 | InnoDB | 53| snapshots | Dynamic | 4737 | 1589248 | 327680 | 0 | 4194304 | InnoDB | 54| user_roles | Dynamic | 6338 | 507904 | 311296 | 0 | 0 | InnoDB | 55| user_tokens | Dynamic | 34 | 16384 | 32768 | 0 | 0 | InnoDB | 56| users | Dynamic | 438 | 114688 | 32768 | 0 | 0 | InnoDB | 57| webhook_deliveries | Dynamic | 0 | 16384 | 32768 | 0 | 0 | InnoDB | 58+--------------------------+------------+------------+-------------+--------------+-----------------+-----------+--------+ 5953 rows in set (0.06 sec)
从上面可以看出，相关表（如：project_measures，projects，file_sources）的 DATA_FREE 的大小有很大幅度下降。
然后，查看数据库所占磁盘空间，也有大幅度下降：100G —> 9G 。
至此，SonarQube 的数据清理已取得很好的成果，以此为记。