מצב הרשת חדשות, עדכונים ומידע

4 may 4:31 We have been informed of an outage at BHS5 IPv6

4 may 4:35 Outage has been resolved. We are monitoring the status of all VM's.

Update: Outage was due to a network loss from our datacenter provider. This has been solved.

We had an electricity outage today. We are unable to process support requests and are awaiting the arrival of electrical teams.

Valued Customers,

Due to spring break and "Semana Santa" in Mexico we will be taking some days off as a team from 31 March -10 April. In this time our response and activation times will be a lot slower than usual.

We apologize sincerely for this.

Best Regards,

Hosterlabs Team.

12:00 User reports that SolusVM control panel is unavailable.

00:30 Investigation begins.

00:45 Server Reboot of BHS1

00:55 Server Reboot completed.

1:00 Problem is related to failed backup that used all usable space of server, resulting in mysql errors on the control panel.

1:07 Problem Solved

Report:

5:02 AM Datacenter notifies us there is a power failure and hence downtime. We contact and monitor the situation closely.

8:27 AM We get notified it was a defective power connector.

We detected an issue with servers after an automatic upgrade from Solus.io

This update fried all compute resources! We are working on the outage of 100+ VPS

0:00 AM Update is started on Solus.IO without previous authorization. No reports are sent as nodes are online and working for most people

11:56 AM Outage is reported

12:26 AM Outage is solved

12 hours of downtime is a SLA of 98.8%, it is an issue that went unnoticed because automatic updates are disabled and must be monitored by our staff at all moments. Although it took us 30 minutes to fix, it went unnoticed. As such any customer wishing to claim 15 days of free credit is free to do so. Please open a ticket referencing this if you were affected. We are not contractualy forced to do so as we do not claim any SLA for virtual servers. Yet we will do so and update our ToS to include an SLA guarantee.

Best regards

There has been lately a certain amount of abuse on one of our nodes by many of our customers. This will be dealt with by blocking for SPAM the port 25th for many IP ranges. This only applies for VM's that have been abusing our services.

Also, a high amount of I/O has been detected on this precise node. We will be separating the accounts that generate a lot of I/O and might be upgrading the nodes to NVMe in the foreseeable future. This will not affect pricing. Now, if your node requires a lot of storage, we need to move it to a storage node.

The purpose of OPENVZ servers is NOT FOR INTENSIVE USE APPLICATIONS LIKE MINECRAFT OR ANY HIGH I/O USAGES. Loads of several VM's being more than 30 is alarming. Several restrictions have been applied on all nodes with loads over 15. This affects the user experience for all of our clients. This is NOT ACCEPTABLE.

You see slowness in this particular node because of the high amount of abuse regarding I/O. 

We will:

1. Generate a new node separating HIGH I/O usage and low I/o. 
2. Change the server for our control panel to solve other issues.

This will be dealt within the next few days. please be patient.

If you experience SLOW RESPONSIVENESS IN YOUR OPENVZ NODE, YOU ARE ELEGIBLE FOR A FREE UPGRADE TO A KVM SERVER WITH NVME Storage.

We have seen a lot of slowness on our control panel for virtual servers. This is not acceptable and it will be dealt with in the next few days.

If you experienced downtime the 27 seven of October here is why:

This issues happened on CA03KVM

1. Amplification DDoS attacks in the morning ( 9 AM ) were received. This means, due to lax security on many of our client's firewalls DDoS attacks were amplified using our infrastructure with IP Spoofing. We received a report of at least 8+ IP's on the same range being blocked. This caused downtime on all servers, as abuse blocks are automatic. 
2. Additional measures were put on the firewalls blocking several hundreds of thousands of IP's where the abuse came from. Mainly china and Russia. Now, an error on very stringent firewall rules caused some servers to have little accessibility as well as problems with the control panel. This caused around 15 minutes of downtime on many servers. Later on IP's were unblocked for the affected machines.

22:00 Hrs We are currently migrating ca00vz due to high loads. 

23:20 Server Deployment

00:15 Testing of environment for a Smoot transfer

1:00 IP Failover transfer

1:15 Transfer initialized

3:16 Most machines have been migrated.

6:12 Migration completed.

We have upgraded the system from 7.1 to 8.0.

Errors have been detected on the theme that does not allow customers to access their accounts. Theme has been set to default until we can troubleshoot the issue further.

Theme Updated

We have detected attacks on windows servers using port 389 UDP which is used for LADP. Unusual traffic is still being received, basic actions to mitigate the attack have been taken.

Further action pending: Implementation of a stronger firewall.

Firewall upgrades have been installed.

We have detected outages from ca-03-kvm at 23:30 Hrs the last two days. Actions have been taken to prevent further downtime.

We are waiting for today to test if actions were sufficient.

System wide improvements of automation will be performed.  As well as security improvements.

[Solved] Non-Bootable device issues reported on windows basic kvm for windows 2019 server. Issue has been pinged back to little space on the basic plan. Minimum disk space has been set up to 24GB to prevent further issues with this.

[Solved] Auto-setup configured a wrong NIC card that did not enable proper functioning of internet. NIC Virtio was setup instead of INTEL PRO 100/1000 or RealTek.

[Solved] Addition of articles related to common issues. Like Noon-bootable device and wrong passwords. We have provided fixes in the kb, but we plan to avoid this issues happening.

[Solved] Update of OS's to remove EOL's OS's that might not work properly with the system.

[Solved]  Updating EOL OS Removal on WHMCS Control Panel

[Solved] Some issues have been reported with the login password at setup. Login credentials are not working properly.

[Solved]  Auto-setup configures minimum speed at 200Mbps which will be changed further to 1Gbps in the upcoming update.

[Solved] There are some conflicts with private ip's being setup on nodes. This sometimes causes websites to be unresponsive/non - accesible by installed browsers.

Solution: Set up proper routing tables

[Planned] We plan to offer installed google chrome on windows server machines, as we know internet explorer might be hard to use due to security restrictions

[Solved]  Node Security Upgrades are planned on all nodes.

[Cancelled] Introduction of server-wide and system-wide firewall's with advanced tuning and security for extra mitigation of DDoS attacks.

Reason: This might cause conflicts with some users that need open firewalls and a lot of traffic.

[Solved] Improvement of Automation.

[Solved]  Implementation of anti IP-Stealing and other related issues on nodes.

[Solved] Change of OS Names to something more friendly.

[Pending] Implementation of friendlier OS Names on WHMCS

[Solved] Addition of Windows 10 OS

[Solved] Auto-FTP Backups are not being created properly . This issue will be mitigated by creating backups once a week instead of daily. This also has affected performance on some nodes while backups are being made. Some clients have reported slow internet connections when this is happening.

[Cancelled] Possibility of upgrading node ca01kvm to the same specifications as ca03kvm.

[Cancelled] Possibility of offering new locations in Europe and/or Singapore. 

[Solved] Upgrade ca00vz

We detected a DDOS attack the 14th of September at around 10-12 AM. We made an emergency migration to brand new servers with higher ddos rates.

15 September - Migration had been completed, but several issues were detected within the system. They were fixed.

16 September

15:00 Hours Early reports of unknown invoices on several customers. 

15:20 Tickets were found to have unknown mistakes related to whmcs billing system. WHMCS Was contacted for further assistance.

22:00 First response from WHMCS, Nevertheless proved unhelpful.

22:15 Upon further inspection, a corrupted database of some sort was found.

23:00 We luckily had a database from the day of 15 of September due to the migrations, we used the database and fixed all possible issues.

8:50. We have received notice that ca03kvm has been detected to be offline. We are taking all preventive measures and checking it as soon as possible.

8:54 It is not an issue relating to the particular server. It seems to be an issue with more than 100 servers on the same space and different racks. We can only suspect at this point that it is a kind of routing error left for our DC to fix. We will keep you informed.

10:06 Electrical issue detected and solved by DC. Server does not want to boot on normal mode, checking issues.

10:38 Server files have been detected to be corrupted. We are trying to fix it as soon as possible.

12:15 Connection has been gained through IPMI to host node. Corrupted partition is currently being repaired via fsck.

12:40 Lost  IPMI Access, waiting on DC.

13:01 IPMI being reset by DC.

14:24. Issue resolved.

After 6+ hours of our staff working nonstop, the issue has been at last successfully resolved.

Hello,

We have noticed backups are not being created for servers on node ca01kvm. We advise you should have your own backups, as the server has no free space for creation of Backups. We will not be able to provide you with any backup in case of data loss.

Kindly make sure you are keeping offsite backups.

Best regards

Hosterlabs Team

8:44 We have detected an issue on a node right now. Our master solusvm node appears to be offline.

9:53 Issue was detected and solved by DC. Switch port issue.

12 May - Possible report of auto-installation issues. All orders will be reviewed manually.

13May - Issue Resolved

4 July - Issue reappears. Investigation in progress. It extends to KVM Linux Templates.

Delays in activation of VPS are to be expected.