Retrospective: On Saturday, September 21st, a disruption at one of our two cloud services partners caused slow uploads and image rendering delays. There should be no ambiguity--we own this problem irrespective of the root cause. In the interest of transparency, we do want to share the facts as they occurred. The cause of the issue was related to additional storage and throughput capacity being added by our primary cloud services provider at their facility. That planned improvement did not go smoothly, impacting Zenfolio and our customers. We began working with them immediately on a solution to restore service to expected levels.
Our service provider performed various hardware and software changes to mitigate the issue and restore full services to us. At the same time, our own in-house Engineering teams accelerated the deployment of a previously planned ”failover” solution which was planned to go into service in mid-October. The solution -- in the event of a problem at one of the cloud service vendors -- would enable a seamless switch to an alternate cloud provider avoiding any disruption to our customers. We rushed this planned enhancement into testing on Tuesday evening and it was successful. Simultaneous with the implementation of our new software, our vendor made progress in resolving their issue resulting in improved speeds for our customers.
As of Wednesday morning, service was fully restored by the original cloud provider. We have unambiguously communicated our disappointment with (and consequences of) their failure. Our new “failover solution” software has now been deployed, tested, and monitored, which allows for instantaneous switching to alternate vendors in the event of a cloud issue at primary providers. This will prevent an issue such as the one experienced this week from disrupting your business. Again, we apologize and have implemented steps to prevent this from occurring in the future.
Posted Sep 27, 2019 - 09:35 PDT
Update
We have been closely monitoring and testing both our primary and secondary storage services and will continue to do so in the near-term. Customers should see images loading at prior expected speeds, so if there are any specific concerns please reach out to our support team at support@zenfolio.com.
Posted Sep 26, 2019 - 09:54 PDT
Update
Image loading continues to operate at expected speeds. We will continue to monitor things very closely and update again if there is anything of note. Otherwise, we plan to provide the next update on Thursday, September 26.
Posted Sep 25, 2019 - 11:14 PDT
Monitoring
Our provider addressed hardware and software issues that were causing the image generation delays and were able to process the bulk of images overnight. Tests also indicate that newly uploaded images are generating at rates much closer to what is expected. We will continue to monitor this very closely throughout the day. If you continue to experience delays, please reach out to support@zenfolio.com so we can look into your specific question. Unless needed, the next update will be at 6 pm UTC (2 pm EDT, 11 am PDT). Thank you to all of our customers for your patience.
Posted Sep 25, 2019 - 04:13 PDT
Update
Our service provider continues to be focused on improving image generation speed, and your uploaded images will be available for viewing, but please continue to expect a delay. Based on progress today, we do expect speed to gradually improve. No need to try re-uploading if the images don't display immediately. Thank you again for your patience. We will provide another update no later than Wednesday, Sept 25 at 11 am UTC (7 am EDT, 4 am PDT).
Posted Sep 24, 2019 - 16:17 PDT
Update
The work by our service provider continues, and your uploaded images will be available for viewing, but please continue to expect a delay. The speed at which new images display will vary on several factors and larger galleries will generally take more time. No need to try re-uploading if the images don't display immediately. Thank you again for your patience. We will provide another update no later than Tuesday, Sept 24 at 11 pm UTC (7 pm EDT, 4 pm PDT).
Posted Sep 24, 2019 - 12:57 PDT
Update
Our service provider has continued to work on increasing capacity so your image uploads are ready for your clients to view and purchase. The work continues, and your uploaded images will be available for viewing, but please expect a delay. No need to try re-uploading if the images don't display immediately. We will continue to assess this and try to give some rough estimates on the amount of time to expect for images to fully display. Thank you again for your patience. We will provide another update no later than Tuesday, Sept 24 at 8 pm UTC (4 pm EDT, 1 pm PDT).
Posted Sep 24, 2019 - 09:24 PDT
Update
Our service provider worked overnight to increase capacity so your image uploads are ready for your clients to view and purchase quickly after upload. The work continues, and your uploaded images will be available for viewing after a delay. No need to try re-uploading if the images don't display immediately. Thank you again for your patience. We will provide another update no later than Tuesday, Sept 24 at 4 pm UTC (12 pm EDT, 9 am PDT).
Posted Sep 24, 2019 - 03:54 PDT
Update
Our service provider is continuing to work on restoring full capacity so your image uploads are ready for your clients to view and purchase quickly after upload. Please note that while the teams continue their work during the overnight hours, your uploaded images will be available for viewing after a delay. There are no problems with the upload, so there is no need to try re-uploading if the images don't display immediately. Thank you again for your patience. We will provide another update no later than Tuesday, Sept 24 at 11 am UTC (7 am EDT, 4 am PDT).
Posted Sep 23, 2019 - 13:47 PDT
Update
Our service provider is conducting hardware maintenance to address the issue with displaying images in a timely manner. While this is happening, we expect to see continued delays and thank you for your patience. We do not have an ETA from our provider yet regarding estimated resolution. We will provide another update at 9 pm UTC (5 pm EDT, 2 pm PDT)
Posted Sep 23, 2019 - 08:26 PDT
Update
We are still waiting on an update from our provider. We will post an update as soon as it is available, or by 3 pm UTC (11 am EDT, 8 am PDT). We sincerely apologize for the inconvenience this is causing.
Posted Sep 23, 2019 - 05:20 PDT
Update
The issue at our provider has continued to impact our users overnight. We will be getting an update from their engineers by 12pm UTC (8am EDT, 5am PDT) and will post any new information here. We sincerely apologize for the inconvenience this is causing.
Posted Sep 23, 2019 - 03:18 PDT
Update
Our service provider is continuing to work on restoring full service. Our sincere apologies for the inconvenience. We will provide an update Monday, Sept 23 at 10am UTC, (3am PDT, 6am EDT).
Posted Sep 22, 2019 - 16:58 PDT
Identified
After a period of stability, we are starting to see system degradation again. Our engineers are working closely with our service provider to resolve this as quickly as possible. Next update will be at midnight UTC, 8pm Eastern US, 5pm Pacific US). Thank you for your patience
Posted Sep 22, 2019 - 14:59 PDT
Update
Our engineers are continuing to monitor performance while our service provider finishes its work. At this time images are generally loading normally, but there may be some delays as the system catches up with the backlog. We will provide another update Monday Sept 23 at 10:00am UTC (6am Eastern US, 3am Pacific US)
Posted Sep 22, 2019 - 14:05 PDT
Monitoring
Our service provider has continued restoration of service. We should expect some delays in image display on Zenfolio sites while they work on fully restoring service. Our engineers are continuing to monitor the situation and will provide an update at 8pm UTC (4pm East Coast US, 1pm West Coast US). Thank you for your patience!
Posted Sep 22, 2019 - 09:21 PDT
Update
Our service provider has begun restoration of service and is at limited capacity at this time. We should expect delays in image display on Zenfolio sites while they work on fully restoring service. Our engineers are continuing to monitor the situation and will provide an update at 4pm UTC (12pm East Coast US, 9am West Coast US)
Posted Sep 22, 2019 - 07:17 PDT
Update
Our engineers are continuing to work with our provider to resolve this issue. We will provide another update at 2:30pm UTC (10:30am US Eastern, 7:30am US Pacific). Thank you.
Posted Sep 22, 2019 - 05:40 PDT
Identified
Our engineers have identified the issue with one of our providers and are working with them to resolve this. We will provide another update at 12:30pm UTC (8:30am US Eastern, 5:30 am US Pacific). Thank you for your patience
Posted Sep 22, 2019 - 04:37 PDT
Investigating
We are currently experiencing an issue with image rendering delays. Our engineers are investigating the cause of this issue and will work to resolve it. Our apologies for this inconvenience. Thank you for your patience!
Posted Sep 22, 2019 - 03:51 PDT
This incident affected: Zenfolio Classic (Content Rendering).