Did more googling and this time found the keywords that helped - you can create a DWORD registry value called NoRecycleFiles and set it to 1. This will permanently delete files. Here's an article relating to this setting:
It recommends placing the entry in HKEY_CURRENT_USERS, but I'm planning on placing it in HKLM (same subpath exists). This would require a reboot for any setting changes to take effect.