Increased latency for several APIs

Resolved | May 06, 2025 | 11:00 GMT+02:00

Due to an error in a deployment we saw heavily increased network load leading to increased latencies.

+Show history

• Monitoring | May 05, 2025 | 18:42 GMT+02:00

  All systems are back to normal

• Investigating | May 05, 2025 | 17:51 GMT+02:00

  We are seeing increased latencies for several APIs at the moment, investigation is ongoing.

Service outages and delays

Resolved | Mar 29, 2025 | 22:26 GMT+01:00

After more than 24 hours intensified monitoring we are going back to our standard protocols.

All systems are working as usual and expected.

+Show history

• Monitoring | Mar 28, 2025 | 19:27 GMT+01:00

  All write operations that were queued are written to the systems now

• Monitoring | Mar 28, 2025 | 19:29 GMT+01:00

  Since about 19:03 o'clock the system answers without increased error rate again. While the system is recovering and emptying the queues response times can be a bit higher than usual. Customer systems using WebHooks will see increased WebHook volume due to emptying the queues.
  
  In response to the recent outage, we will intensify our monitoring of the systems beyond our standard protocols for the next several hours.

• Resolution in progress | Mar 28, 2025 | 18:21 GMT+01:00

  The root cause has been identified and is being resolved.
  
  We see 92% - 94% of all requests successfully completed. Read operations such as fetching bookings, coupons or transactions usually succeed, whereas write operations have a higher failure ratio.
  
  We expect the problem to be solved in the coming hours.

• Investigating | Mar 28, 2025 | 16:47 GMT+01:00

  Responses are still very slow leading to http 500 errors in many cases. Investigation ongoing.

• Investigating | Mar 28, 2025 | 16:14 GMT+01:00

  There are still problems with our managed database. Our service provider and we are still investigating. The performance is therefore reduced.
  
  Customers can see different effects as a result like bookings do not appear in the apps, activated coupons are deactivated again, cash register receipts do not arrive. No data is lost, all events are queued and are processed with a delay.

• Investigating | Mar 28, 2025 | 15:25 GMT+01:00

  Services are still responding with degraded performance, investigation ongoing.

• Investigating | Mar 28, 2025 | 14:50 GMT+01:00

  Due to a restart of a cluster node a resync of all nodes is running leading to degraded performance.
  
  Investigation is ongoing.

• Open | Mar 28, 2025 | 14:30 GMT+01:00

  Since about 14:26 o'clock we see problems with our main databases leading to service outages and slow responses across all services.
  
  Investigation of the problem is ongoing.

Several services respond slowly or time out

Resolved | Mar 20, 2025 | 12:53 GMT+01:00

Due to a switch of cluster nodes at about 12:51 some services responded slowly or timed out.
The reason for this switch has been identified and will be prevented for the future

Affected components

• Production

+Show history

• Identified | Mar 20, 2025 | 12:52 GMT+01:00

  Due to a switch of cluster nodes at about 12:51 some services responded slowly or timed out.
  The reason for this switch has been identified and will be prevented for the future
  

Service outages and delays

Resolved | Mar 19, 2025 | 11:25 GMT+01:00

The issue was identified, fixed and deployed to production

Affected components

• Production

+Show history

• Monitoring | Mar 19, 2025 | 10:49 GMT+01:00

  At about 10:49h a spontaneous cache service restart led to delays and failed calls for several services. After less than a minute the service recovered.
  
  The reason for the restart is being investigated.

Services Coupon, Account, Search resond slowly or do not respond at all

Resolved | Oct 21, 2024 | 13:51 GMT+02:00

Since 12:35 o'clock all queues including low priority queues have been processed

Affected components

• Production: Account, Coupon, Search

+Show history

• Monitoring | Oct 21, 2024 | 11:35 GMT+02:00

  Starting at around 22:35 on 20 October 2024, communication problems occurred in the central infrastructure that exchanges information between the services. As a result, many requests piled up and were answered very slowly or not at all.
  
  Communication was fully restored at around 9.03 a.m. on 21 October 2024 and the services are currently processing the backlogged requests. For this reason, responses may still be delayed.
  
  
  Beginnend gegen 22:35 Uhr am 20.10.2024 traten Kommunikationsstörungen in der zentralen Infrastruktur die Informationen zwischen den Services austauscht auf. In der Folge haben sich viele Anfragen aufgestaut und wurden sehr verzögert oder auch gar nicht mehr beantwortet.
  
  Gegen 09.03 Uhr am 21.10.2024 war die Kommunikation vollständig wiederhergestellt, die Services arbeiten momentan die aufgestauten Anfragen ab. Aus diesem Grund können Antworten noch verzögert erfolgen.

Search service returns no results

Resolved | Jan 25, 2024 | 12:18 GMT+01:00

Service fully operational again

Affected components

• Production: Search

+Show history

• Resolution in progress | Jan 25, 2024 | 12:10 GMT+01:00

  Due to misconfiguration during an upgrade of the service it returns no results

• Identified | Jan 25, 2024 | 11:45 GMT+01:00

  The search service responds normal but returns no results