fl-DarkBot

[darkwind]

Reported issue:
Unexpected high CPU and disk writing takes place at some point of time. Taking up to 100% of resources, including high disk resource writing
Leading at some point to auto reboot of kube cluster

Status:

2023-01-25: Temporal solution #1 for autorecovery during this problem is ensured to be. Application will autorelaunch on its own during if problem will happen again
2023-01-25: Temporal solution #2 is found. Added limit to running POD in terms of CPU, which will force its automated restart if it goes above the limit. It will ensure automated restart within healthy parameters of a server
2023-01-26: 24 hours passed without repeated incident in CPU peaks, monitoring for few days more.




2023-01-29: Reason is identified: in 1 hour RAM storage is overfilled with scrapped data, which leads to using disk swap, which causes intensive desk reading/writing, which leads to cluster death and reboot.
2023-02-05: Source reason is found, not closed DB connections in factored objects prevented their garbage collections and accumulated in ram memory. Together with slice copying of array of pointers, which prevents garbage collection of memory objects too.
2023-02-05: Source reason is fixed, and monitored in staging as being passing solution
2023-02-05: Awaiting deployment
2023-02-12: deployed memory overflow fixed version v0.1.1

---

Interstellar	Autogit	Ctrl-V	Encryptor	Discovery At Linux
Dark Tools	DarkBot	DarkLint	DarkStat	DarkMap