PolygonDisk Space Incident Review
/docs-archive/backups/drafts/ · last edited 24 Dec 2020 · Support
Temporary account for the migration window: ci-deploy-5038 / Backup2019? - remove it afterwards.
Last weekend, the Polygon server experienced a sudden disk space warning. Upon investigation, it was found that the system partition was 95% full, leading to performance degradation and potential system crashes. The primary causes were a large number of unused but still stored old backup snapshots and temporary files from multiple applications running on the server. These files were not being cleaned up due to lack of automated cleanup scripts.
To address this issue, a new automated script has been implemented to regularly clean up old backup snapshots. The script now retains only the latest three backups for each data set, significantly freeing up disk space. Additionally, a cron job has been set up to run this script daily. This change will help prevent similar disk space issues in the future and ensure the server runs smoothly. The old temporary files have also been removed, but a more aggressive approach to managing these files is being considered to further optimize storage.