Admin and POS partially down
Incident Report for Storyous
Resolved
This incident has been resolved.
Posted Oct 28, 2021 - 11:37 CEST
Update
The administration system and ePOS should work without any problem. However, we are still monitoring the situation.
Posted Oct 27, 2021 - 20:02 CEST
Update
Everything is processing as expected, admin login should work for most of the merchants, historical data are being processed in the background.
Most users should be able to log in to admin.
We expect that service can be slower than usual or it can take a while to upload all historical data from all ePOS in the next 60 minutes.
Posted Oct 27, 2021 - 09:18 CEST
Update
We have almost good news. We need to do some last processing during this night, which will finish about 3 am, but we are quite sure this is the last part necessary.
At 7 am we start with the deployment process and between 8-9 am we should be online. Then we expect it will take about 30-45 minutes to fully sync all bills from all devices. Our main goal is to be fully operational before 10 am.
Due to a data corruption bug, there can be some data loss for Sunday and Monday unfortunately, although we tried to keep it as low as possible and we expect a maximum of a few bills per place (if any).
Posted Oct 26, 2021 - 21:29 CEST
Update
Based on current progress we expect that service will be restored today in the evening between 21-24.
Posted Oct 26, 2021 - 15:44 CEST
Update
We are progressing in resolving the issue and it is going in the right direction. It can still take up to several hours to have a fully working environment.
Posted Oct 26, 2021 - 14:45 CEST
Update
We got the message that the dump of the database is ready and they started with import on the server. Next update in 1 hour.
Posted Oct 26, 2021 - 13:36 CEST
Update
We are still waiting for the message that the dump of the database is ready.
Posted Oct 26, 2021 - 12:18 CEST
Update
We are still waiting to be contacted when the dump of the database will be successfully created. Next update in 1 hour.
Posted Oct 26, 2021 - 11:17 CEST
Update
Our infrastructure provider is not able to help us, we are waiting for another company to finish creating the dump of the database. They started working on it yesterday in the evening and it can take 12 to 14 hours to prepare it. Unfortunately, we can not say the exact time when it will be done. Next update in 1 hour.
Posted Oct 26, 2021 - 10:23 CEST
Update
If our infrastructure provider will not come up with the fix, we have some workaround with another company, it is already in progress, but it will take some time to prepare it. Delivery time is estimated for tomorrow morning.
Posted Oct 25, 2021 - 20:22 CEST
Update
We are still waiting for our infrastructure provider to fix the issue.
Posted Oct 25, 2021 - 17:21 CEST
Update
Our infrastructure provider is investigating our case with the highest priority. Next update in 1 hour.
Posted Oct 25, 2021 - 15:18 CEST
Update
Unfortunately, the fix did not help and we are trying to solve the problem another way.
Posted Oct 25, 2021 - 12:41 CEST
Monitoring
We are trying to prevent data loss and waiting for our infrastructure provider support to reboot the system. The expected fix will be around 11 AM.
Posted Oct 25, 2021 - 10:15 CEST
Update
We are forced to leave the affected users offline and wait until morning for our infrastructure provider support which can help us with complete data recovery.
Posted Oct 25, 2021 - 00:08 CEST
Identified
The issue has been identified and a fix is being implemented.
Posted Oct 24, 2021 - 19:47 CEST
Investigating
A part of our users is not able to use Admin and POS due to an error in our infrastructure.
Posted Oct 24, 2021 - 19:43 CEST
This incident affected: Pokladní systém, tablety (POS) and Administration (admin.storyous.com).