While a few customers run Discovery appliances on physical hardware, the vast majority run on VMs, and those being on VMware ESX. An important consideration is the ability to use VMware Snapshots, which we mention in the docs, here. So what exactly is a snashot?
When triggered by an ESX administrator, the VM's entire memory, disk and configuration state is captured at that point in time. The original virtual disk (.vmdk file) s "frozen" and refered to as the parent. A new child .vmdk file is created, which contains differences to the snapshot state. Over time, the number of changes may grow, up to a limit of the entire orignal virtual disk size (plus small overhead). If more snapshots are taken subsequently, a hierarchy of child deltas is created, looking something like this:
As you can imagine, the more snapshots the system has to manage, and the more changes the system makes, the more overhead in IO and storage use. Discovery's datastore makes intensive use of the disk.
See here for VMware's KB article "Best practices for using snapshots in the vSphere environment (1025279)". Of note are the headlines:
Do not use snapshots as backups
For better performance, use only 2 to 3 snapshots
Do not use a single snapshot for more than 72 hours
When using 3rd party backup software, ensure the snapshots are deleted after sucessful backup.
It is typical that Discovery Administrators are not also ESX Administrators, but I would recommended checking what the current snapshot policoes and procedures are for your Discovery virtual appliances. I have logged docs defect DRUD1-29882 to update the page with background and links to VMware's KB article.
Do let me know if you use snapshots, how often, and for what purpose. What processes do you have around them? Do you use any other backup and management software with them?