While these brought the server back online and at a higher level of performance then previously it is still performing sub-optimally. It was found that we needed to rebuild part of the storage array. This process is currently running. It is running in the background, but while this is running it takes a lot of I/O resources and slows the server down, unfortunately it is impossible to operate the server at full capacity until this is done.
Currently the server is up and running but due to the high I/O usage of the rebuild process there can be some negative consequences during the rebuild period:
- First and foremost the server is able to accept less mail connections. The server is processing incoming and outgoing mail, but if there are too many users sending and receiving mail during this rebuild process some users may find their mail not going through.
- Mail that is being received is being queued. While most mail is being accepted the server does not have enough resources to deliver it instantly as normal. Mail that is queued will be delivered as the queu is processed.
- Slow page loads. While pages are loading faster then they were, some database heavy sites may feel slow and could even time out.
We expect the rebuild to finish on Sunday. Unfortunately the rebuild process does not provide an ETA on the rebuild, but we do expect it to finish during the weekend and services to resume to normal.
Friday, August 25, 2017