Incident history

We are upgrading the software on our backup cluster. Snapshots and backups will be unavailable for about 1 hour between 17.30 and 18.30 CEST.

Progress

3 public update(s)

We detected high packet loss towards Webdock.io. The NOC team are investigating this. Sorry for the inconvenience.

Progress

1 public update(s)

We are currently investigating this incident.

Progress

2 public update(s)

We will be performing important security enhancements to our network on Tuesday the 28th betwen 10.00 and 14.00 CEST. The upgrade procedure includes a full kernel upgrade of all host systems. The procedure WILL result in brief network outage for ALL customers in Denmark. Typically we are looking at 5 seconds of interruption as systems are reloaded, but for s…

Progress

3 public update(s)

In order to balance load and stability we'll migrating Pico4 instances to other physical systems. Each server migration will be fairly quick (not more than 10 minutes, within 5 minutes in most cases). You will receive an email when the migration starts and also when the migration finishes. The migrations happen between a 4hr maintenace window, starting at 8…

Progress

4 public update(s)

We are currently investigating this incident.

Progress

2 public update(s)

We are currently investigating this incident. We are having issues with routing via IPV6\. The team is aware and looking into it. If you have any issues, you may turn off IPV6 via our dashboard.

Progress

2 public update(s)

We are upgrading the software on our backup cluster. Snapshots and backups will be unavailable for about 1 hour between 16.30 and 17.30 CEST.

Progress

3 public update(s)

We are currently investigating this incident.

Progress

2 public update(s)

We are upgrading the capabilities of our AI chat.

Progress

3 public update(s)

Rebooting the host. The downtime will be 5-10 minutes. Apologies for inconvenience.

Progress

2 public update(s)

We are currently investigating this incident.

Progress

3 public update(s)

We noticed an issue with our AI chat and are working to fix it.

Progress

2 public update(s)

The host needs a reboot. There will be downtime for few minutes. Apologies for the trouble caused.

Progress

2 public update(s)

We will be performing important security enhancements to our network on Wednesday the 20th betwen 10.00 and 14.00 CEST. The upgrade procedure includes a full kernel upgrade of all host systems. The procedure WILL result in brief network outage for ALL customers in Denmark. Typically we are looking at 5 seconds of interruption as systems are reloaded, but for…

Progress

3 public update(s)

After a major software upgrade of our backup cluster yesterday we are seeing frequent timeouts while doing backups. This is due to some regression in Incus, our hypervisor. We have the Incus team investigating this and we hope to have this solved within a day or two at most. In the interim, if you get an error trying to perform a snapshot you can retry and h…

Progress

3 public update(s)

We are expanding our backup storage with another 200TB of storage. The storage backend will be unavailable for 1-2 hours.

Progress

3 public update(s)

We are currently investigating this incident.

Progress

2 public update(s)

A single Ryzen system in our DK DC has a temperature anomaly. We believe one or more fans may have failed in the system and this needs to be investigated. We are reserving 2 hours for this maintenance on thursday morning at 06.00 CEST, just in case, but we expect this to take 45 minutes or less. Your server will go down during this time, and you have been no…

Progress

3 public update(s)

We are currently investigating this incident.

Progress

4 public update(s)

Monday April 27th at 20.00 CEST we will be performing major upgrades to the Webdock Dashboard at [app.webdock.io](http://app.webdock.io) \- the maintenance will take at most 2 hours, but we hope to complete it in 30 minutes or less. The upgrade requires us to take down the dashboard completely and in this time period you will be unable to use our dashboard t…

Progress

6 public update(s)

We are currently investigating this incident.

Progress

3 public update(s)

We are currently investigating this incident.

Progress

2 public update(s)

We are currently investigating this incident.

Progress

2 public update(s)

We are performing some needed maintenance in storage cluster DK DC1 now and for up to 2 hours. In this period, snapshot functionality will not be available.

Progress

2 public update(s)

One of the hosts (lxd based legacy host) seems unresponsive or crashed. This one needs a reboot.

Progress

2 public update(s)

One of the hosts (lxd based legacy host) seems unresponsive or crashed. This one needs a reboot.

Progress

2 public update(s)

We are having issues communicating with one of our Epyc hosts - all customers there are running fine, we just have problems interacting with and sending commands to the host system.

Progress

3 public update(s)

The host needs a reboot for proper functioning. Sorry for the inconvenience

Progress

2 public update(s)

We are currently investigating this incident.

Progress

2 public update(s)

The host requires a reboot. Instances will face downtime for 2-5 minutes. Sorry for the inconvenience.

Progress

2 public update(s)

We are currently investigating this incident.

Progress

3 public update(s)

The host will need a reboot for about 5-10 minutes. We apologize for any inconvenience.

Progress

2 public update(s)

Our DC guys looking into the issue. Apparently it looks like one of the CPU has failed (dual CPU setup). The guys are working on to bring up the host. Apologies for the inconvenience.

Progress

8 public update(s)

The server will be down for 5-10 minutes. Sorry for the inconvenience.

Progress

2 public update(s)

The physical host needs a reboot. Instances running there will see a 15-20 minutes of downtime.

Progress

4 public update(s)

We are seeing intermittent packet loss on certain prefixes (IP ranges) in DK DC-1\. This may be another small scale DOS which is not being properly mitigated. We are currently investigating this incident.

Progress

2 public update(s)

Something is happening with our network at this time. We are currently investigating this incident.

Progress

6 public update(s)

The host needs a reboot, there will be 5-10 minutes of downtime. Sorry for the inconvenience.

Progress

2 public update(s)

The host will be rebooted now. There will be 5-10 minutes of downtime.

Progress

2 public update(s)

The host needs a reboot. Server will see a 5-10 minutes of downtime. Apologies for the inconvenience.

Progress

2 public update(s)

One of our lxd-based host needs a reboot as it has become unresponsive.

Progress

2 public update(s)

The host needs reboot. The container VPSes will see a 5-10 minutes of downtime.

Progress

2 public update(s)

One of our EPYC system is down and hence VPSes will be unreachable. We are investigating this.

Progress

2 public update(s)

The server needs a reboot. Expect 5-10 minutes of downtime. Apologies for the inconvenience.

Progress

2 public update(s)

The host needs a reboot. There will be 5-10 minutes fo downtime. Sorry for any inconvenience caused

Progress

2 public update(s)

One of our legacy container-based host needs a reboot (system is frozen) for proper functioning. There will be dowtime for few minutes. Apologies for the inconvenience.

Progress

2 public update(s)

Our NOC team has identified and issue with a flapping link and high CRC errors rate for the past 2 days. We will attempt to fix the issue by changing optics.

Progress

3 public update(s)

The server needs a reboot. Expect 5-10 minutes of downtime. Apologies for the inconvenience.

Progress

2 public update(s)

One of our KVM host looks to be unresponsive. Our NOC guys are currently investiagating this. We'll do our best to bring things back to normal as soon as we can. Apologies for the inconvenience.

Progress

2 public update(s)

This thursday morning some electric works will be happening at the DK DC1\. We are getting new power supply added increasing our allowed power draw from the grid. The procedure means that mains power will need to be cut for about 5-10 minutes. This should be handled by our UPS and Generator systems - but just in case something unexpected happens we are annou…

Progress

3 public update(s)

There will be downtime for 5-10 minutes

Progress

2 public update(s)

There appears to be a problem with a Ryzen host which is giving trouble on boot up. The NOC team is working on it. Affected customers see an alert in the dashboard.

Progress

3 public update(s)

There will be downtime for 5-10 minutes. Apologies for the inconvenience.

Progress

2 public update(s)

We have a Xeon host down in DK DC1 - NOC team is working on bringing the system back up. We hope to have customer instances up and running in about 10-15 minutes.

Progress

2 public update(s)

The host needs a reboot. There will be 15 to 20 minutes of downtime.

Progress

2 public update(s)

The system is a bit frozen. The snapshot and start operations seem to hang. A reboot is required, hence a 5-10 minutes of downtime

Progress

2 public update(s)

The host needs a reboot and will be up in 5-10 minutes.

Progress

2 public update(s)

The host needs a reboot for proper functioning. Instances will see 5-10 minutes of downtime.

Progress

2 public update(s)

We had a small outage again affecting Ryzen and Epyc workloads as well as our dashboard and related services. The event lasted about 10 minutes. All customers should be operational again. We apologize for the inconvenience.

Progress

1 public update(s)

We are currently investigating this incident. A network-level event seems to have affected a single rack in our DC, taking down our dashboard and some related services, as well as a number of our Epyc/Ryzen customer instances.

Progress

3 public update(s)

One of our container-based hosts needs a reboot. There will be downtime for few minutes

Progress

2 public update(s)

The host needs a reboot. Few minutes of downtime is to be expected.

Progress

2 public update(s)

The host requires a reboot. There will be few minutes of downtime.

Progress

2 public update(s)

The host will require a reboot. Instances will see few minutes of downtime.

Progress

2 public update(s)

The host is unresponsive and requires a reboot. The will be downtime for 5-10 minutes.

Progress

2 public update(s)

One of our lxd-based hosts needs a reboot. Instances here will see some downtime.

Progress

2 public update(s)

One of our Ryzen host is down/inaccessible. We're looking into this. Sorry for the inconvenience.

Progress

2 public update(s)

Maintenance is now in progress.

Progress

3 public update(s)

One of our host needs a reboot after a kernel upgrade. All VPSes will experience a downtime. At max 15 minutes. Sorry for the inconvenience.

Progress

2 public update(s)

One of our host needs a reboot after a kernel upgrade. All VPSes will experience a downtime. At max 15 minutes. Sorry for the inconvenience.

Progress

2 public update(s)

We have a host machine which needs a kernel upgrade and reboot. Your VPS will go down for at most 5-10 minutes, usually a lot less. You can check if your server(s) are affected by logging in to Webdock as this will then be shown at the top of the page in a red alert window.

Progress

2 public update(s)

We have been working on expanding our storage cluster. Something has gone wrong with configuration meaning some snapshot operations (especially for older VPS servers) will not work properly. We are currently investigating this incident and hope to get this resolved sometime today. For some subset of our customers, no automated backups were run today due to t…

Progress

2 public update(s)

We had a small network outage in Denmark, for about 3 minutes certain services could not be reached. We are investigating this incident currently.

Progress

2 public update(s)

We noticed the network is intermittent in DK, some customer sites going down and coming up again. We are currently investigating this incident. Sorry for the inconvenience.

Progress

2 public update(s)