XEN/KVM OS Upgrade & security patch (Resolved)
  • Priority - High
  • Affecting Server - VPS Master
  • Dear Lowest Host/Empire Technology Customers

    In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.

    EVENT START DATE: 05/22/2022
    EVENT START TIME: 00:01 EDT

    EVENT END DATE: 05/22/2022 
    EVENT END TIME: 06:00 EST

    EVENT LOCATION: Midwest NOC

    EXPECTED DOWN TIME: 10 -45 minutes 

    EVENT  DETAILS: 
    Maintenance XEN Server OS Upgrade & security patch
    Site Affected: XEN Nodes 5

    Maintenance KVM  Server OS Upgrade & security patch
    Site Affected: KVM Nodes 1 

    Maintenance OPENVZ   Server OS Upgrade & security patch
    Site Affected: OVZ Node Nodes 3

     

     

     

    Update  /05/22/2022  1257 EDT  

    All Upgrades complete  No Issues 

  • Date - 05/21/2022 22:00 - 05/22/2022 18:49
  • Last Updated - 05/21/2022 22:58
Xen 6 Maintenance (Resolved)
  • Priority - High
  • Affecting Server - Las Vegas NOC
  • Dear Lowest Host/Empire Technology Customers

    In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.

    EVENT START DATE: 03/16/2022
    EVENT START TIME: 12:45 EST

    EVENT END DATE: 03/16/2022 
    EVENT END TIME: 14:00 EST

    EVENT LOCATION: Las Vegas NOC

    EXPECTED DOWN TIME: 10 -20 minutes 

    EVENT 3 DETAILS: 
    Maintenance XEN 6 Retirement

    Our XEN 6  Hardware Node is being replaced by newer hardware we will be migrating all VPS’s on XEN 6 In Las Vegas to XEN 2  Las Vegas, down time will be  approximately 10-20 minutes.   

     

     

    Upgrades complete no issues 

  • Date - 03/14/2022 21:50 - 03/23/2022 08:42
  • Last Updated - 03/23/2022 08:43
VPS Node OS upgrade (Resolved)
  • Priority - High
  • Affecting Server - VPS Master
  • Dear Lowest Host/Empire Technology Customers

    In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.

    EVENT START DATE: 04/17/2021
    EVENT START TIME: 00:01 EST

    EVENT END DATE: 04/21/2021 
    EVENT END TIME: 23:59 EST

    EVENT LOCATION: Midwest NOC

    EXPECTED DOWN TIME: 2 - 30 minutes 

    EVENT 3 DETAILS: 
    We will be upgrading our node OS to AlmaLinux 8 & Openvz 7.
    Site Affected: XEN Nodes All, KVM nodes ALL, OpenVZ nodes all.

    Las Vegas XEN nodes will be scheduled at a later date.

    We will be cutting all VPS servers over to Newer Node Operating systems expected downtime will be 2 to 30 Min depending on the size of your VPS.

    -----------------------------------------------

    Update  04/20   18:00 EST   All XEN VPS nodes have been completed 

    -------------------------------------------------

    Update 04/21/  The XEN nodes  took longer than expected extending  the maintenance  window to end of the day 4/21

    -----------------------------------

    Update 04/21/  All KVM Nodes have been completed  

    ---------------------------------------------------------

    Update 04/21/  All OPENVZ  Nodes have been completed  

    --------------------------------------------------------

    Update 04/22  one of the  Open VZ nodes is failing to boot we are repairing it 

    --------------------

    Update 4/23 all upgrades complete 

     

     

  • Date - 04/21/2021 04:30 - 04/28/2021 16:21
  • Last Updated - 04/28/2021 16:21
Server XEN 3 Failing Post on OS boot (Resolved)
  • Priority - Critical
  • Affecting Server - VPS Master
  • ·         Dear Lowest Host/Empire Technology Customers

    In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.

    EVENT START DATE: 09/04/2020
    EVENT START TIME: 10:51 PDT


    EVENT LOCATION: SB NOC (Midwest)


    Server XEN 3   Failing Post on OS boot


    EVENT 1 DETAILS:
    Server Xen 3 went into a kernel Panic & is failing to Post for Boot  
    The NOC tech is on scene & is investigating  

    Update 12:15 PDT 

    One of the  memory sticks  Froze on reboot  Error was cleared  & Ram/memory reseated  & server is on line  

     

    Update  2:20 PDT   

    We will be monitoring the system for errors and replacing the affected DIMM  if we get an Error  

  • Date - 10/04/2020 10:51 - 01/22/2021 23:08
  • Last Updated - 10/04/2020 15:34
Transit Provider Maintenance (Resolved)
  • Priority - Low
  • Affecting Server - Las Vegas NOC
  • In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.

     

    EVENT START DATE: 03/16/2020
    EVENT START TIME: 2300:00 PDT

    EVENT END DATE: 03/17/2020
    EVENT END TIME: 03:00 PDT

    EVENT LOCATION:  Hurricane Electric POP Las Vegas   

    EVENT  DETAILS:
    Lowest Host/Empire Technology LLC has been informed by Hurricane Electric will be performing Maintenance: Router software upgrade to enhance security and stability
    Associated circuit(s): Las Vegas 1 & 2
    Thank You,


    Datacenter Operations Team

  • Date - 03/16/2020 23:00 - 10/04/2020 11:29
  • Last Updated - 03/17/2020 01:04
Open VZ 1 Maint (Resolved)
  • Priority - High
  • Affecting Server - VPS Master
  • Dear Lowest Host/Empire Technology Customers

    In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.

    EVENT START DATE: 12/25/2019
    EVENT START TIME: 00:01 PST

    EVENT END DATE: 12/25/2019
    EVENT END TIME: 01:00 PST

    EVENT LOCATION: SB NOC Open VZ 1 (Midwest) 

    EVENT DETAILS: Reboot To sync up with backup server

    EXPECTED DOWN TIME: 5 MIN - 15 MIN
    EVENT DETAILS: Open VZ 1
    Maintenance: we will be reinstalling the OS Kernel as the current Kernel is not syncing up with our backup server. This will require 2 reboots

    ---

    Update 12/25/2019  06:09  PST

    Maintenance complete No issues

  • Date - 12/25/2019 00:01 - 01/06/2020 09:30
  • Last Updated - 12/25/2019 06:10
partial network Outage (Resolved)
  • Priority - Medium
  • In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.

    EVENT START DATE: 09/10/2019
    EVENT START TIME: 05:24 PST
    EVENT END TIME: 06:44 PDT

    EVENT LOCATION: South Bend NOC

    EVENT 1 DETAILS: 
    emergency Maint: Our Primary distribution switch in our South Bend Data center had a partial failure. We have dispatched a NOC technician to troubleshoot why specific ports remain dead even after resetting them. 

    ===============================================
    Update 06:20 PDT
    after Further Investigation Our Ups Power on FEED #2: Went into an overload condition. It was receiving Irregular unclean Power from the utility company (
    Indiana Michigan Power Company) 110.32 % of normal.  Which sent our UPS: into Bypass mode (Battery power) we remained on Battery power but the fault did not clear & our ups shut down as a safety precaution to not overload equipment. 


    ===============================================
    Update 06:44 PST

    We have reset the circuit and have requested an electrician be dispatched to take a closer look and adjust as necessary. All Network anomalies were resolved at this time (Xen 3’s network took a bit longer to come up but is on line again). All servers & network issues should be resolved if you are still having issues Please open a support ticket so we can take a closer look.

    ===============================================
    Update 07:20 PST

    we will be monitoring the circuit for the next 24 hours and send out further updates as needed

     

     

  • Date - 09/10/2019 05:24 - 09/13/2019 10:54
  • Last Updated - 09/10/2019 09:13
Router(s) Upgrade/Replacement (Resolved)
  • Priority - Medium
  • In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.

    EVENT START DATE: 06/24/2019
    EVENT START TIME: 00:10 EDT

    EVENT END DATE: 06/28/2019
    EVENT END TIME: 04:00 EDT

    EVENT LOCATION: SB NOC (Midwest)

    EVENT DETAILS: Router(s) Upgrade/Replacement


    EXPECTED DOWN TIME: 15-60 Seconds
    EVENT DETAILS:
    The week of Jun 23, 2019 we will be replacing our aging routers in our Midwest data center with new Cisco Routers to accommodate more capacity & resource usage.
    We are expecting 0 down time as we will be hot swapping each Router one at a time leaving at least one network link active. There will be a 15-60 second outage as the routes converge when your server/rack is cut over.  

    Note: BGP customers with Redundant Links you will get 2 separate emails for each Link (this includes private BGP sessions). Please verify that we are sending the correct routes. Single homed BGP customers please verify that your session is receiving the correct routes (default route for most of you).  

    ===========================================

    Update  06/28/2019 

    All upgrades are complete: routing adjusted to optimal, Redundant  Failover tested 

    ============================================

    Update 07/3/2019

    Adjusted IPV6 Routing for VPS servers only   (autoconfig disabled  it enabled itself during cut over)

    =============================================

    Update 07/15/2018

    Moved  IPV6  IBGP   temporary link off  old Primary  router & router decommissioned

     

     

     

     

     

     

     

  • Date - 06/24/2019 00:10 - 08/30/2019 05:22
  • Last Updated - 07/23/2019 06:52
CVE-2019-0211 Patch (Resolved)
  • Priority - High
  • Affecting System - Apache Web Server
  • Dear Lowest Host/Empire Technology Customers

     

    In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.

    EVENT START DATE: 4/4/20198
    EVENT START TIME: 06:10 PDT

    EVENT END DATE: NA
    EVENT END TIME: NA

    EVENT LOCATION: SB NOC (Midwest) Las Vegas NOC (West Coast)

     

    EVENT DETAILSCVE-2019-0211, According to the Apache team, less-privileged Apache child processes (such as cgi) can execute malicious code with the privileges of the parent process.

    Because on most Unix systems Apache httpd runs under the root user, any threat actor who has planted a malicious CGI script on an Apache server can use CVE-2019-0211 to take over the underlying system running the Apache httpd process, and inherently control the entire machine.

    WARNING: Apache HTTP 2.4.17 to 2.4.38 Local Root Exploit

    Apache HTTP 2.4.17 to 2.4.38 is vulnerable to a local root exploit when mod_prefork, mod_worker and mod_event are used:

    https://httpd.apache.org/security/vu...lities_24.html

    We are hearing reports of exploit(s) already being produced and strongly recommend that everyone update to Apache HTTP 2.4.39 as soon as possible - especially in shared hosting environments!

    https://www.zdnet.com/article/apache...-environments/

    CloudLinux

    https://cloudlinux.com/cloudlinux-os-blog/entry/beta-ea-apache24-2-4-39-1-cloudlinux-major-security-update

     

    Maintenance Apache: 4.39 patch

     

    EXPECTED DOWN TIME: NA
    EVENT DETAILS:
    Server OS Upgrade & security patch 
    Site Affected: All Managed Dedicated & VPS Servers Running the Apache web server.

     

    We will be verifying all managed cpanel servers running Apache have been updated.

    Directadmin managed servers we will be patching all servers.

    cPanel cloudlinux  managed servers we will be patching via the testing repo.

     

    Per usual this event can be tracked @ https://secure.empire-hosting.net/serverstatus.php

    --------------------------------------------------------

    For all unmanaged clients & some managed clients Please see your respective repo if this vulnerability affects you. See below for Control panel specific information.

    cPanel CentOS 6/7 if you are running easyapache 4  with automatic updates your server will be updated automatically.

    cPanel if you are running easyapache3 or an unsupported End OF Life Operating system  there is no patch available.

    cPanel Cloudlinux 6/7  Patch is available via the testing repo

    Directadmin Centos 6/7 patch is available via custombuild.

    Directadmin Centos an unsupported End OF Life Operating system  there is no patch available.

    Litespeed web server is unaffected.

    ================

     

    Update 4/4/2019  0730 PDT

    All Shared Hosting servers have been Patched/updated

    =======================================

     

    Update 4/4/2019 0800 PDT

    All Managed DirectAdmin servers have been Patched/updated 

    =============================================

     

    Update 4/4/2019 11:00 PDT

    All Managed cPanel servers verified & updated if needed 

    =============================================

     

    Update 4/4/219 11:00 PDT 

    All Managed Cpanel/CloudLinux servers Patched/updated

  • Date - 04/04/2019 06:00 - 04/09/2019 06:55
  • Last Updated - 04/04/2019 10:57
All Managed Dedicated VPS Servers CloudLinux (Resolved)
  • Priority - High
  • Affecting System - Systems running CloudLinux OS
  • Dear Lowest Host/Empire Technology Customers

     

    In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.

    EVENT START DATE: 11/24/2018
    EVENT START TIME: 00:10 EST

    EVENT END DATE: 11/24/2018
    EVENT END TIME: 04:00 EST

    EVENT LOCATION: SB NOC (Midwest) Las Vegas NOC (West Coast)

     

    EVENT DETAILS: CVE-2018-3615/3620/3646 [L1 terminal fault] aka 'Foreshadow & Foreshadow-NG'


    Maintenance OS Kernel: 'Foreshadow & Foreshadow-NG'Patch

     

    EXPECTED DOWN TIME: 5-10 Min 
    EVENT 1 DETAILS:
    Server OS Upgrade & security patch 
    Site Affected: All Managed Dedicated & VPS  Servers Running the CloudLinux operating system . 

    Note: if you are an unmanaged client Patch your OS at your leisure

    NOTE: Warning if you are running Kernelcare rebootless kernel (licensed Version) it will not patch for CVE-2018-3615/3620/3646 it requires an up2date kernel & reboot.

     

    ---------------------------------

    Update 07:00 EST

    Maintenance Complete No issues

     

     

     

  • Date - 11/23/2018 00:10 - 11/24/2018 20:00
  • Last Updated - 11/24/2018 05:56
OS Upgrade & security patch (Resolved)
  • Priority - High
  • Affecting Other - Shared Hosting Servers
  • Dear Lowest Host/Empire Technology Customers

     

    In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.

    EVENT START DATE: 11/24/2018
    EVENT START TIME: 00:10 EST

    EVENT END DATE: 11/24/2018
    EVENT END TIME: 03:00 EST

    EVENT LOCATION: SB NOC (Midwest) Las Vegas NOC (West Coast)

     

    EVENT DETAILS: Maintenance OS Kernel: OS Upgrade & security patch

     

     

    EXPECTED DOWN TIME: 5-10 Min 
    EVENT 1 DETAILS:
    Server OS Upgrade & security patch 
    Site Affected: All Shared Hosting servers

     

    ---------------------------------

    Update 07:00 EST

    Maintenance Complete No issues

     

     

  • Date - 11/24/2018 00:10 - 11/24/2018 20:00
  • Last Updated - 11/24/2018 05:56
Upstream IP Transit Edge Router Maintenance (Resolved)
  • Priority - Low
  • Dear Lowest Host/Empire Technology Customers

    In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.

    EVENT START DATE: 11/07/2018
    EVENT START TIME: 000:00 EST

    EVENT END DATE: 11/07/2018
    EVENT END TIME: 06:00 EST

    EVENT LOCATION: SB NOC (Midwest)

     


    This will be a no impact issue we will be turning down the circuit at 10:00 EST 11/06/2018 until completed. Below is a summery of  maintenance notice sent to us.

     ========

    As a valued customer, Cogent is committed to keeping you informed about any changes in the status of your service with us. This email is to alert you regarding maintenance we will be performing on our network: 

    Start time: 12:00 AM (local time) 11/7/2018
    End time: 06:00 AM (local time) 11/7/2018
    Work order number: NC840-80
    Order ID(s) impacted: XXXXXXX
    Expected Outage/Downtime: 45-60 minutes

    Cogent customers receiving service in the greater South Bend, IN metro area will be affected by this maintenance. Only the Cogent Order ID(s) listed above will be impacted. The purpose of this work is to apply and activate software module upgrades on code running on the edge routers per the manufacturer's recommendations.

     

     

    =============================================

    Maint complete NO issues 

  • Date - 11/06/2018 10:00 - 11/09/2018 09:10
  • Last Updated - 11/07/2018 17:44
Maintenance OS Kernel: (Resolved)
  • Priority - Low
  • Affecting System - All Managed Dedicated Servers
  • Dear Lowest Host/Empire Technology Customers

    In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.

    EVENT START DATE: 09/15/2018
    EVENT START TIME: 13:00 PDT

    EVENT END DATE: 9/16/2018
    EVENT END TIME: 04:00 PDT

    EVENT LOCATION: SB NOC (Midwest) Las Vegas NOC (West Coast)

     

    EVENT DETAILS: CVE-2018-3615/3620/3646 [L1 terminal fault] aka 'Foreshadow & Foreshadow-NG'


    Maintenance OS Kernel: 'Foreshadow & Foreshadow-NG'Patch

     

    EXPECTED DOWN TIME: 5-10 Min
    EVENT 1 DETAILS:
    Server OS Upgrade & security patch 
    Site Affected: All Managed Dedicated Servers. 

    Note: if you are an unmanaged client Patch your OS at your leisure

    Note2: all RedHat variants Have Patches available with the exception of CloudLinux OS

    Note3: Debian 8 No patch released

    NOTE4: Warning if you are running Kernelcare rebootless kernel (licensed Version) it will not patch for CVE-2018-3615/3620/3646 it requires an up2date kernel & reboot.

    NOTE5: If you are running CloudLinux OS, they have not pushed a patch yet all CloudLinux customers will be scheduled on a later date.


    Update 05:53 PDT

    All Servers Patched no issues 



  • Date - 09/15/2018 10:12 - 09/18/2018 16:07
  • Last Updated - 09/16/2018 10:28
Maintenance OS Kernel: (Resolved)
  • Priority - Low
  • Affecting System - All Managed VPS Servers
  • Dear Lowest Host/Empire Technology Customers

    In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.

    EVENT START DATE: 09/15/2018
    EVENT START TIME: 13:00 PDT

    EVENT END DATE: 9/16/2018
    EVENT END TIME: 01:00 PDT

    EVENT LOCATION: SB NOC (Midwest) Las Vegas NOC (West Coast)

     

    EVENT DETAILS: CVE-2018-3615/3620/3646 [L1 terminal fault] aka 'Foreshadow & Foreshadow-NG'


    Maintenance OS Kernel: 'Foreshadow & Foreshadow-NG'Patch

     

    EXPECTED DOWN TIME: 0 (new OS kernel & software patch will be loaded on Node reboot see below)
    EVENT 1 DETAILS:
    Server OS Upgrade & security patch 
    Site Affected: All Managed VPS XEN & KVM  Servers, ( OpenVZ NA). 

    Note: if you are an unmanaged client Patch your OS at your leisure

    Note2: all RedHat variants Have Patches available with the exception of CloudLinux OS

    Note3: Debian 8 No patch released

     

    EXPECTED DOWN TIME: 5MIN - 15 MIN
    EVENT 2 DETAILS:
    Our XEN/KVM/OpenVZ Maintenance Virtuozzo Server OS Upgrade & security patch 
    Site Affected: VPS Servers  All Nodes


    Update  O5:55 PDT


    Updates complete No issues 




  • Date - 09/15/2018 10:14 - 09/18/2018 16:07
  • Last Updated - 09/16/2018 10:28
OS Upgrade & security patch (Resolved)
  • Priority - Low
  • Affecting System - Shared Hosting Servers
  • Dear Lowest Host/Empire Technology Customers

    In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.

    EVENT START DATE: 09/15/2018
    EVENT START TIME: 13:00 PDT

    EVENT END DATE: 9/16/2018
    EVENT END TIME: 01:00 PDT

    EVENT LOCATION: SB NOC (Midwest) Las Vegas NOC (West Coast)


    Maintenance OS Kernel: OS Upgrade & security patch

     

    EXPECTED DOWN TIME: 5-10 min
    EVENT 1 DETAILS:
    Server OS Upgrade & security patch 
    Site Affected: All Shared Hosting servers  


    Update 05:56 PDT

    Updates complete No issues 

  • Date - 09/15/2018 10:09 - 09/18/2018 16:07
  • Last Updated - 09/16/2018 10:28
June 21 RTO and IPV6 Circuit Turn up (Resolved)
  • Priority - Low
  • Description of Outage:
    On Thursday , June 21, 2018 at approximately 06:59 PM PST, Lowest Host Empire Technology LLC was experiencing routing drops of our Level3 IPv6 Network in our Midwest Data Center. In an effort to Repair the disruption our Network Feed was moved to a New Router & port. Due to a failure of our uptream’s internal core routing configuration with our router, it caused our core2.SBII Router to drop large portions of our IP announcements entering our network. This caused degraded network performance, packet loss, and the inability to reach our network.

    Our core2.SBII Router handles our upstream BGP sessions. Which were ultimately unreachable due to the failed configuration on our primary feed, this condition lasted until our upstream tuned down their link.

    Resolution:
    The Faulty configuration was fixed by our upstream peer. Configuration was restored, and our routing protocols began operating normally at this time.

    Network Status IPv4 100% Primary and secondary Feeds 

    Network Status IPV6 Inbound 100% Outbound 50% degraded Level3 is not routing our outbound packets and they are still trying to find a resolution.

     
    -------------------------------------------------------------------------------------


    In progress: 

    Turnning up a new IPV6 Link
    Status Cable being  installed 


    ---------------------------------------------------------------------------------------

    Update June 23, 2018 
    Link has been established awaiting on seesion information 


    ----------------------------------------------------------------------------------------


    Update June 23, 2018
    Network Status IPV6 100% Primary and secondary Feeds
    Level3 IPV6 routing issue resolved at this time 




  • Date - 06/22/2018 10:56 - 08/11/2018 20:57
  • Last Updated - 06/23/2018 10:41
VPS Server Meltdown & Spectre Patch (Resolved)
  • Priority - High
  • Affecting Server - VPS Master
  • Dear Lowest Host/Empire Technology Customers

    In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.

    EVENT START DATE: 1/28/2018
    EVENT START TIME: 08:30 PST

    EVENT END DATE: TBA 
    EVENT END TIME: TBA

    EVENT LOCATION: ALL

    EXPECTED DOWN TIME: 5 -10 minutes Possible multiple patches.

    EVENT DETAILS: Meltdown


    Maintenance CPU Vulnerabilities: Meltdown  & Spectre Patch


    Site Affected: All VPS servers

    We will be patching all Managed dedicated VPS servers for the vulnerabilities above. Currently All managed dedicated clients can be patched with the exception of Cloud Linux & kernelcare (awaiting patch). If you are an unmanaged client please check with your OS vendor if they have a patch ready. Please refer to the security notice email dated 1-14 for the current status of the OS vendors we use. 

    Currently Redhat/Centos 6 & 7 is the only OS that has patched for CVE-2017-5753 & CVE-2017-5754 Spectre 1 & Meltdown 3, waiting on CPU manufactures to release patches for CVE-2017-5715 Spectre II.

    As of today

    Centos 6/7 Scientific Linux 6/7 Patched Spectre 1 & Meltdown 3

    Cloudlinux 6/7 waiting for Patches

     Kernelcare ALL waiting for Patches

    Ubuntu 14/16 Meltdown 3 only

    Debian 8/9 Meltdown 3 only

    Centos 5 EOL it won’t be patched upgrade to centos 7

    Debian 6 EOL it won’t be patched upgrade to Debian 9

    Gentoo 2011 EOL upgrade to 12.1 2018 patch

    ===============================================================

    WARNING WARNING WARNING !!!

    Do not patch your XEN VPS Currently the Latest kernels for XEN are unstable and may result in your server not booting. If you have upgraded your OS kernel on Xen please set the boot back to your previous OS kernel.

     

    At this time we may do a forced upgrade on all VPS server managed or unmanaged node(s) wide. For Centos, Scientific Linux, CloudLinux we will defiantly. For Ubuntu and Debian it is still undecided. We may push our own sources mirror with Patched Kernels if they do not patch their OS kernels in a timely manner.  

    For those running EOL OS’s centos 5, Debian 6, and Gentoo we are going to try building a custom Kernel and pushing the patch

    ===============================================================

    Virtualization types

    OpenVZ: Node only patch client OS does not require any Patch.

    KVM: Node and Client OS.

    XEN: Node and Client OS, still waiting on a stable Patch for XEN VPS servers see warning above.


    ====================================================================

    Update 1-28-2018  21:35:43 PST

    We will be patching All KVM servers Node software 
    We will be patching all OpenVZ Node software 
    for Meltdown & Spectre Patch

    ======================================================================


    Update 1-28-2018  11:38:43 PST 

    Upgrades to KVM OpenVZ  complete no issues 

    ======================================================================
    Update 2-4 2018  09:12:00 PST 

    Upgraded all Managed VPS Servers on KVM.
    Did a Forced upgrade on all unmanaged servers running CentOS 7 & CentOS 6 x86_64.


    ===========================================================
    Update 2-8 2018  04:22:00 PST 

    We will Be patching the all of the XEN nodes for the Meldown Spectre  Vulnerabilities Expected  Dowtime 10 MIN 


    ============================================================
     Upadate 2-14 2018 10:24 PST

    All Xen Nodes have Been Patched  No issues 

    ============================================================
     Upadate 3-4 2018 9:20 PST

    All Xen & all KVM servers have been forcfulliy patched for Meldown Spectre 

    ============================================================
     Upadate 3-4-2018 09:35 PST

    Just realized we did not Patch KVM 3 & Openvz 1  for spectre Meltdown
    we will be applying  the patch and rebooting the node
    expect 10 min of down time 

    ========================================================

    Closing all servers patched that can be 

  • Date - 01/28/2018 08:16 - 04/03/2018 02:41
  • Last Updated - 04/03/2018 02:41
shared Sever Meltdown Spectre (Resolved)
  • Priority - High
  • Affecting Server - rc1
  • Dear Lowest Host/Empire Technology Customers

    In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.

    EVENT START DATE: 02/14/2018
    EVENT START TIME: 08:59 PST

    EVENT END TIME: 16:00 PST

    EVENT DETAILS: Emergency Maintenance all Shared Hosting servers 

    We will be patching all of our shared hosting servers for the CPU Vulnerabilities: Meltdown & Spectre.

    For reference see security Notice email sent out On Jan 16 2018,

      

    We will be rebooting each shared hosting server expect 5-10 min  of down time while we apply the patch. 


    ====================================================================
    Update 02/14/2018 10:22 PST 

    All updates complete no Issues 

  • Date - 02/14/2018 09:02 - 02/17/2018 08:59
  • Last Updated - 02/14/2018 10:26
Dedi Server : Meltdown & Spectre Patch (Resolved)
  • Priority - Low
  • Dear Lowest Host/Empire Technology Customers

    In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.

    EVENT START DATE: 1/16/2018
    EVENT START TIME: 08:30 PST

    EVENT END DATE: TBA 
    EVENT END TIME: TBA

    EVENT LOCATION: ALL

    EXPECTED DOWN TIME: 5 -10 minutes 

     

    EVENT DETAILS: 
    Maintenance CPU Vulnerabilities: Meltdown & Spectre Patch


    Site Affected: All managed dedicated servers

    We will be patching all Managed dedicated server s for the vulnerabilities above. Currently All managed dedicated clients can be patched with the exception of Cloud Linux & kernelcare (awaiting patch). If you are an unmanaged client please check with your OS vendor if they have a patch ready. Please refer to the security notice email dated  1-14  for the current status of the OS vendors we use. 

    ========================================================
    Update Jan 17 2018 09:17 PST

    Update  All Centos  Managed dedicated server Patched with Partial support  ( awaiting Intel  patches)
    Update Still waiting on CloudLinux  patches 

    =========================================================
    Update Feb 2 2018 09:33 PST

    Update CloudLinux 7 Kernelcare 7 Patches have been released all managed servers have been patched
    Update Still waiting on CloudLinux 6 Kernelcare 6

    ===========================================================
    Update Feb 4 2018 08:00 PST

    Update All CloudLinux  6 all managed dedicated servers have been patched with the CloudLinux  6 Hybrid kernel.
    Update all managed dedicated have been patched  down grading this issue to low. 

  • Date - 01/16/2018 08:30 - 02/11/2018 21:05
  • Last Updated - 02/04/2018 09:17
Dedicated Server Clients: Bandwidth graphs (Resolved)
  • Priority - Low
  • Affecting Other - Dedicated Server Clients: Bandwidth graphs
  • In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.

    EVENT START DATE: 01/03/2017
    EVENT START TIME: 05:24 PST
    EVENT END TIME: TBA


    Due to the fact that our Edge router failed in Las Vegas our Data center Bandwidth Graphs have Become too out of sync. Restoring from backup did resolve the issue they would run for about 30 minutes and then would drop all logging from logging errors. We will be rebuilding all clients BW monitoring.


    =============================================================
    Update Jan 3 14:20 PST

    We have be rebuilt all client BW monitoring.


    =============================================================
    Update Jan 4 12:43 PST


    We have upgraded our monitoring software to the latest version unfortunately our entire Bandwidth display in our dedicated server management area is not compatible with the new graph & font size of the the most up to date software. We are currently rebuilding the dedicated server manager to accommodate the changes.

     

  • Date - 01/03/2018 05:24 - 01/11/2018 01:21
  • Last Updated - 01/04/2018 12:53
Emergency Network Maintenance Las Vegas (Resolved)
  • Priority - Low
  • In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.

    EVENT START DATE: 01/02/2017
    EVENT START TIME: 05:24 PST
    EVENT END TIME: 08:44 PST

    EVENT LOCATION: Las Vegas NOC

    EVENT 1 DETAILS: 
    emergency Maint: Our IP Network announcement in our Las Vegas Data center has failed  we have dispatched a NOC technician to the Datacenter to try and clear the Network fault  on or Edge router. 


    ===============================================
    Update 07:50
    After Further Investigation Our Edge Router is not rebooting we are currently replacing the router Please standby for further updates. 

     
    ===============================================
    Update 08:44 PST

    We have replaced Our Edge router  all server should be on line once again  if you are still having issues  Please open a support ticket so we can take a closer look. 

    ===============================================
    Update 16:20 PST

    we are downgrading this Maintenance as it was completed @ 08:44 PST. 
    We will leave issue open for 24 more hours. 

    ===============================================

    Update Jan 3 05:20 PST

    Our Data center Bandwidth Graphs have Become too out of sync. Restoring from backup did resolve the issue. We will be rebuilding all clients BW monitoring.  

  • Date - 01/02/2018 05:27 - 01/04/2018 12:54
  • Last Updated - 01/03/2018 05:33
Data center IP transit Maintenance: (Resolved)
  • Priority - Low
  • Dear Lowest Host/Empire Technology Customers

    In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.

    EVENT START DATE: 07/09/2017
    EVENT START TIME: 11:51 PST

    EVENT END TIME: TBA

    EVENT DETAILS: Emergency IP transit  Maintenance  

    On or about July 5th 2017 

    our Level3 Primary transit was having 10 sec to 5 Minute optical drops we opened a Ticket to have Level3 troubleshoot the issue

     
     ========

    After further investigation by Level3  they have determined that the one of Core Routers at the Point of presence (POP)  Is creating the issue.
    Level3 has dispatched a technician  to the data center to replace router or parts as necessary 

    We have  turned down on our primary Level3 Feed until the repairs are complete 

  • Date - 07/09/2017 11:49 - 07/14/2017 16:46
  • Last Updated - 07/09/2017 12:11
Moderate Packet Loss On Las Vegas Shared Serv (Resolved)
  • Priority - High
  • Affecting Server - rc1
  • Dear Lowest Host/Empire Technology Customers

    In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.

    EVENT START DATE: 06/30/2017
    EVENT START TIME: 09:45 PDT

    EVENT LOCATION: Las Vegs NOC

    EVENT DETAILS: 
    We are experiencing Moderate network packet Loss on Or RC1 shard Hosting server in Los Vegas  we are investigating the issue 

    ===========================

    We have resolved the issue  we will continue to monitor the issue for any further anomalies 

  • Date - 06/30/2017 08:45 - 07/09/2017 11:49
  • Last Updated - 06/30/2017 09:24
XEN 8 Raid Failure (Resolved)
  • Priority - Critical
  • Dear Lowest Host/Empire Technology Customers

    In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.

    EVENT START DATE: 06/28/2017
    EVENT START TIME: 05:44 PST

    EVENT LOCATION: Las Vegas NOC

    EVENT 1 DETAILS: 
    emergency Maint we have had a total Raid Failure on XEN 8 we are restoring servers from backup 


    ============================================================

    All vps servers restored from backup  

  • Date - 06/28/2017 05:47 - 06/29/2017 12:35
  • Last Updated - 06/28/2017 10:01
Maintenance Cisco Edge Switch (Resolved)
  • Priority - Medium
  • Dear Lowest Host/Empire Technology Customers

    In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.

    EVENT START DATE: 06/17/2017
    EVENT START TIME: 00:01 EST

    EVENT END DATE: 06/17/2017 
    EVENT END TIME: 00:06 EST

    EVENT LOCATION: Midwest NOC

    EXPECTED DOWN TIME: 5 minutes 

    EVENT DETAILS: 
    Maintenance Cisco Edge Switch IOS reload. We will be performing maintenance on one of our Edge switches to clear a memory fault.
    Site Affected: Midwest NOC 


    =======================================


    Maint Complete 

  • Date - 06/17/2017 22:01 - 06/28/2017 05:47
  • Last Updated - 06/16/2017 21:08
XEN Server OS Upgrade & security patch (Resolved)
  • Priority - High
  • Affecting Server - VPS Master
  • Dear Lowest Host/Empire Technology Customers

    In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.

    EVENT START DATE: 05/27/2017
    EVENT START TIME: 00:01 EST

    EVENT END DATE: 05/27/2017 
    EVENT END TIME: 06:00 EST

    EVENT LOCATION: Midwest NOC

    EXPECTED DOWN TIME: 10 -20 minutes 

    EVENT 3 DETAILS: 
    Maintenance XEN Server OS Upgrade & security patch
    Site Affected: XEN Nodes 3, 7, 5

    ===================

    Updates completed XEN 7 was cancleled will be resceduled at a later date 

  • Date - 05/27/2017 00:01 - 06/10/2017 17:33
  • Last Updated - 05/26/2017 22:44
XEN Server OS Upgrade & security patch (Resolved)
  • Priority - High
  • Affecting Server - VPS Master
  • Dear Lowest Host/Empire Technology Customers

    In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.

    EVENT START DATE: 05/6/2017
    EVENT START TIME: 00:01 EST

    EVENT END DATE: 05/6/2017 
    EVENT END TIME: 06:00 EST

    EVENT LOCATION: Las Vegas NOC (West Coast)

    EXPECTED DOWN TIME: 10 -30 minutes

    EVENT 1 DETAILS:
    Non emergency Drive(s) Replacement. On Xen 2 (retiring server)

    EVENT 2 DETAILS: Repair Of OS install on Xen 8, Our VPS control panel is not syncing with the OS.  

    EXPECTED DOWN TIME: 10 -30 minutes Note: Please do not reboot your server if you are on this node please open a ticket until maintenance is completed.

    EVENT 3 DETAILS:
    Maintenance XEN Server OS Upgrade & security patch
    Site Affected: XEN Nodes 2, 8, 6


    ------------------------------------------------

    Update: all node upgrades have been completed except XEN 2 
    we have extended the maintenance action to  Sunday May 7 for xen 2 

  • Date - 05/05/2017 10:01 - 05/15/2017 05:50
  • Last Updated - 05/07/2017 09:08
Xen Network Bridge crash (Resolved)
  • Priority - Low
  • Dear Lowest Host/Empire Technology Customers

    In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.

    EVENT START DATE: 4/19/2017
    EVENT START TIME: 12:00 PDT

    EVENT END DATE: 4/19/2017
    EVENT END TIME: 12:05 PDT

    EVENT LOCATION: Midwest NOC  (XEN3  ) 

    EVENT DETAILS:  emergency maintenance 
    Orur XEN 3 node Network crashed we took the time and patched the server and restasrted the network on all Virtual machines

    Thank You Server Operations Team.

     
    ==========================

  • Date - 04/19/2017 12:00 - 04/20/2017 07:37
  • Last Updated - 04/19/2017 12:09
Virtuozzo Server Maint (Resolved)
  • Priority - High
  • Affecting Other - SB NOC (Midwest)
  • Dear Lowest Host/Empire Technology Customers

    In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.

    EVENT START DATE: 4/2/2017
    EVENT START TIME: 12:00 PDT

    EVENT END DATE: 4/2/2017
    EVENT END TIME: 17:00 PDT

    EVENT LOCATION: Midwest NOC  (VPS 2 ) 

    EXPECTED DOWN TIME: 5-15 Min . 
    EVENT DETAILS:  emergency maintenance
    We will be retiring are our Openvz node 2 and moving data to new hardware  4/2/2017 12:00 PM PDT - 4/2/2017 1700:00 PM PDT

    Thank You Server Operations Team.

     
    ==========================

    Maint complete No issues 

  • Date - 04/02/2017 12:00 - 04/04/2017 08:09
  • Last Updated - 04/04/2017 08:09
Decommission Xen 1 (Resolved)
  • Priority - High
  • Affecting System - XEN 1
  • Dear Lowest Host/Empire Technology Customers

    In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.

    EVENT START DATE: 3/27/2017
    EVENT START TIME: 05:00 PDT

    EVENT END DATE: 4/2/2017
    EVENT END TIME: 17:00 PDT

    EVENT LOCATION: Midwest NOC  (XEN 1) 

    EXPECTED DOWN TIME: 5-20 Min . 
    EVENT DETAILS: 
    We will be retiring are our oldest XEN node and moving data to new hardware  3/27/2017 05:00 AM PDT - 4/2/2017 1700:00 PM PDT

    Thank You Server Operations Team


    ===============================

    update 
    This maintenance has been completed No issues 

     

  • Date - 03/27/2017 05:00 - 04/02/2017 10:55
  • Last Updated - 04/01/2017 09:43
DDOS Attack Las Vegas Shared hosting server. (Resolved)
  • Priority - Low
  • Affecting Server - rc1
  •  

    Dear Lowest Host/Empire Technology Customers

    In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.

    EVENT START DATE: 2/27/2015
    EVENT START TIME: 10:25 PST

    EVENT LOCATION:Las Vegas  NOC

    EVENT DETAILS:

    One of our shared hosting servers ins under a layer7 DDOS attack  RC1 attackers  are flooding the inbound network connection

    We are currently mitigating  the the attack at this time:
    Note: we will be making some modifications to the network on this server it will be unreachable until we stop the attack 
    Note: some of you  who are are receiving  the attack  your domains will be redirected and you will be notified   in  a separate Email 

    ============================

    Update 

    We have successfully mitigated the attack 
    For Now

    We are currently Running a File system check
    As the file system was damaged during the attack


    ========================================


    Update 

    We were able to mitigate the attack in fast fashion

    But do to the heavy volume of the attack  and the server crashing several times during the attack
    The server was Forced into a manual Maintenance cycle and the file system had to be manually repaired. 
    The server is back on line  we will continue to monitor  and mitigate the lingering attacks. 




  • Date - 02/27/2017 10:25 - 03/05/2017 09:09
  • Last Updated - 02/27/2017 18:42
Upgrade Easyapache 4 (Resolved)
  • Priority - Medium
  • Dear Lowest Host/Empire Technology Customers

    In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.

    EVENT START DATE: 1/29/2017
    EVENT START TIME: 05:00 PST

    EVENT END DATE: 2/28/2017
    EVENT END TIME: 17:00 PST

    EVENT LOCATION: Shared Hosting servers (ALL) 

    EXPECTED DOWN TIME: 0 there will be a cut over to the new Apache configuration which may cause a few sporadic restarts.
    EVENT DETAILS: 
    We will be performing server maintenance: on all of Shared Hosting Servers Reason: Easyapache 3 is going End of life and to keep our clients up to date with the latest security and software we must make this upgrade to Easyapache 4. Maintenance Window: 1/29/2017 05:00 AM - 2/28/2017 1700:00 PM PST

    Thank You Server Operations Team


    =============

    Update All Las Vegas shared hosting servers updated 


    =============

    Update all Midwest shared Hosting servers updated 

  • Date - 01/29/2017 05:04 - 02/02/2017 09:28
  • Last Updated - 01/31/2017 06:12
XEN Server OS Upgrade & security patch (Resolved)
  • Priority - High
  • Dear Lowest Host/Empire Technology Customers

    In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.

    EVENT START DATE: 12/23/2016
    EVENT START TIME: 00:01 EST

    EVENT END DATE: 12/23/2016
    EVENT END TIME: 22:00 EST


    EVENT LOCATION: Las Vegas NOC (West Coast)

    EXPECTED DOWN TIME: 10 -30 minutes
    EVENT 1 DETAILS:
    Non emergency Drive Replacement.

    EVENT 2 DETAILS: Start Date 12/23/2016 00:01 EST
    Maintenance XEN Server OS Upgrade & security patch
    Site Affected: XEN Nodes 2, 8, 1

    ===================================
    Updates complete 2, 1 

    Updates Complete with the exception of xen 8
    we are tweaking the drive sets for performace  

  • Date - 12/22/2016 10:01 - 01/12/2017 05:26
  • Last Updated - 12/27/2016 00:21
KVM Servers Hard Drive upgrade (Resolved)
  • Priority - High
  • Dear Lowest Host/Empire Technology Customers

    In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.

    EVENT START DATE: 10/15/2016
    EVENT START TIME: 00:01 EDT

    EVENT END DATE: 10/15/2016 
    EVENT END TIME: 06:00 EDT

    EVENT LOCATION: SB NOC (Midwest)

    EXPECTED DOWN TIME: 5-25 MIN 
    EVENT DETAILS: 
    We will be performing server maintenance: on all of KVM Nodes. Reason: we found that our KVM Nodes SSD drives have degraded in sustained write speed over time. We are going upgrade to a different manufacture & model that has proven to be much better in terms of performance & longevity with our current Raid card models.

    Thank You Server Operations Team

    ==========================================


    We are a bit behind schedule  but we are about 95% complete

  • Date - 10/14/2016 10:01 - 10/19/2016 16:01
  • Last Updated - 10/15/2016 06:58
Midwest Data center expansion (Resolved)
  • Priority - High
  • AUG 1 2016, we have expanded our Midwest Data center.

    • Yes we have done it again.
    • All Midwest data center clients we have expanded our Midwest Data center.
    • We have added additional Rack/floor Space and increased our network capacity by a Factor of 1.5.
    • During the first week Aug 2016 some servers will require relocation we will send out a notification if your server requires relocation.

     

    We will be starting our expansion on Aug 1 2016, and sending out notifications if your server requires a move. This will be short notice relocation/reboot. 


    ===============================================

    We are about 95% complete  had an Issue with one of our XEN nodes  it failed to Boot   operating system had to be reinstalled 

    ==================================================

    Upgrades complete 

  • Date - 08/01/2016 15:00 - 08/11/2016 08:04
  • Last Updated - 08/11/2016 08:04
Las Vegas shared hosting server Maint (Resolved)
  • Priority - Low
  • Affecting Server - rc1
  • Dear Lowest Host/Empire Technology Customers

    In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.

    EVENT START DATE: 06/06/2016
    EVENT START TIME: 07:59 PST

    EVENT END TIME: 16:00 PST

    EVENT DETAILS: Emergency Maintenance Shared Hosting server 

    At 07:59 PST We will be replacing a Failing Backup drive in our Rc1 Shard Hosting server. 

    This is a low impact event and we do not expect any Downtime if this should change we will update as required

     
     ========
     
    The failed Hard drive has been replaced we had to reboot the server as it did not recognize the new Drive 

  • Date - 06/06/2016 08:06 - 06/18/2016 13:46
  • Last Updated - 06/06/2016 11:26
VPS Servers Maint Patch (Resolved)
  • Priority - High
  • Dear Lowest Host/Empire Technology Customers

    In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.

    EVENT START DATE: 5/7/2016
    EVENT START TIME: 00:01 EDT

    EVENT END DATE: 5/8/2016 
    EVENT END TIME: 06:00 EDT

    EVENT LOCATION: SB NOC (Midwest)

    EXPECTED DOWN TIME: 5-15 MIN
    EVENT DETAILS: 
    We will be performing server maintenance: on all of KVM Nodes. Reason: server security & maintenance patch.                                                                   
    We will be performing server maintenance: on all of OPenVZ/Viruozzo Nodes. Reason: server security & maintenance patch.                                                                    
    Maintenance Window: 5/7/2016 00:01 - 5/8/2016 0600:00 PM EDT

    Thank You Server Operations Team


    =================================================


    Maint  complete no issues 

  • Date - 05/06/2016 10:01 - 05/17/2016 16:49
  • Last Updated - 05/08/2016 08:26
Degraded On Xen Node 6 (Resolved)
  • Priority - Low
  • Dear Lowest Host/Empire Technology Customers

    In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.

    EVENT START DATE: 04/24/2016
    EVENT START TIME: 07:59 PST

    EVENT END TIME: 09:20 PST

    EVENT DETAILS: Emergency Maintenance XEN 6

    At 07:59 PST a hard drive popped out of our Raid array and caused a lock up

    The Technician on Duty immediately noticed the error and rebooted the raid is currently rebuilding  

     
     ========

    We will monitor the rebuilding of the array and close this issue when it is complete 


    =============================================================

    Raid  rebuuild is complete 

  • Date - 04/24/2016 07:59 - 04/29/2016 10:46
  • Last Updated - 04/24/2016 13:29
VPS Servers Maint Patch (Resolved)
  • Priority - High
  • Dear Lowest Host/Empire Technology Customers

    In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.

    EVENT START DATE: 4/17/2015
    EVENT START TIME: 00:01 EDT

    EVENT END DATE: 4/18/2015 
    EVENT END TIME: 06:00 EDT

    EVENT LOCATION: SB NOC (Midwest)

    EXPECTED DOWN TIME: 5-15 MIN 2 Instances
    EVENT DETAILS: 
    We will be performing server maintenance: on all of KVM Nodes. Reason: server security & maintenance patch.                                                                   
    We will be performing server maintenance: on all of OPenVZ/Viruozzo Nodes. Reason: server security & maintenance patch.                                                                   
    Maintenance Window: 4/17/2015 00:01 - 4/18/2015 0600:00 PM EDT

    Thank You Server Operations Team


    ====================================

    This maint window has been canceled 

  • Date - 04/16/2016 21:01 - 04/19/2016 10:19
  • Last Updated - 04/17/2016 07:24
Upgrading our shared Hosting servers to the l (Resolved)
  • Priority - Low
  • Affecting Other - SB NOC (Midwest) Las Vegas NOC (West Coast)
  • Dear Lowest Host/Empire Technology Customers

    In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.

    EVENT START DATE: 3/05/2016
    EVENT START TIME: 13:00 PST

    EVENT END DATE: 03/19/2016
    EVENT END TIME: 06:00 PST

    EVENT LOCATION: SB NOC (Midwest) Las Vegas NOC (West Coast)

    EXPECTED DOWN TIME: 1-5 minutes when we cut the network over

    EVENT DETAILS:

    Maintenance Scheduled for Midwest & West Coast Data centers 
    Maintenance window: 5 March 2016 13:00 PST to 19 Mar 2016 06:00 PST


    Maintenance Window Duration: 2 weeks
    Maintenance Upgrading all shared Hosting servers to the latest OS Centos 7 
    Site Affected: Shared  Hosting servers ALL 
    Expected Down time 1-5 minutes

    Risk Classification: LOW
    Empire Hosting Master Ticket #: 20160305

    ===========================


    Upgrades complete No issues 

  • Date - 03/05/2016 11:04 - 03/15/2016 07:32
  • Last Updated - 03/15/2016 07:32
informational Notice XEN 2 (Resolved)
  • Priority - Low
  • Dear Lowest Host/Empire Technology Customers

    In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.

    EVENT START DATE: 02/23/2016
    EVENT START TIME: 21:12 PST

    EVENT END TIME: NA

    EVENT DETAILS: informational Notice XEN 2

    At 21:12 PST a client  on Xen Node 2 reported that their server was not coming up after a reboot

    After further investigation it was found that the Network Bridge lost connection (the server was running but no network).

    We have reconfigured the Bridge and restarted the network those servers that we noticed that lost connection we rebooted. If you cannot connect to your servers IP please do a reboot will reconnect your network.


    ============================================================

    No more issues  closing 

  • Date - 02/23/2016 22:34 - 03/05/2016 10:51
  • Last Updated - 03/05/2016 10:51
Network Event (Resolved)
  • Priority - Critical
  • Dear Lowest Host/Empire Technology Customers

    In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.

    EVENT START DATE: 02/09/2016
    EVENT START TIME: 09:00 PST

    EVENT END TIME: 09:20 PST

    EVENT DETAILS: Emergency Maintenance  

    At 09:00 PST we had a Network Event in our Mid West Data center

    We have investigated to or Level3 transit was null routing all of our IP routes to un-routable inbound and outbound

     
     ========

    After further investigation we had to power cycle the edge router and bring up our backup Peers manually.
    We are currently on our primary Level3 Feed monitoring the situation Network restored @ 09:20 PST

    ================================================================= 

     resolved

  • Date - 02/09/2016 09:00 - 02/10/2016 13:55
  • Last Updated - 02/10/2016 13:55
KVM Emergency Maintenance (Resolved)
  • Priority - High
  • Dear Lowest Host/Empire Technology Customers

    In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.

    EVENT START DATE: 12/26/2015
    EVENT START TIME: 4:04 PST

    EVENT DETAILS: Emergency Maintenance  

    KVM 1

    We have had reports of VM’s failing to boot

    We have investigated it to a software glitch on the we are currently migrating the affected VMS to another node

    ========

    After further investigation the affected VM's file systems went read only  and would not stay on line long enough to to run a FSCK.
    corrected the issue by moving  disk partitions to another Node  and ran manual FSCK's

    =================================================================
    resolved 

  • Date - 01/26/2016 16:05 - 01/27/2016 01:50
  • Last Updated - 01/27/2016 01:50
Emergency Maintenance XEN2 (Resolved)
  • Priority - High
  • Dear Lowest Host/Empire Technology Customers

    In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.

    EVENT START DATE: 1/22/2016
    EVENT START TIME: 8:17 PDT

    EVENT end TIME: 8:21 PDT

    EVENT DETAILS: Emergency Maintenance  

    Xen 2

    We have had reports of some client VPS servers hanging on reboot

    We have restarted the node to clear the error
    ========

     After further investigation it was found 2 Virtual servers were casing the issue 1 the first VPS that was hung required a file system check and another VPS was abusing the disk IO it has been suspended.

  • Date - 01/22/2016 20:17 - 01/25/2016 03:37
  • Last Updated - 01/22/2016 21:29
Shared hosting servers exploited (Resolved)
  • Priority - Low
  • Affecting System - Shared Hosting Server
  • Dear Lowest Host/Empire Technology Customers

    In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.

    EVENT START DATE: 12/25/2015
    EVENT START TIME: 2:00 PDT

    EVENT DETAILS: Emergency Maintenance  

    Hared Hosting Server(s)

    We have had reports of some client sites being redirected to a Hacked site page

    We are currently have investigated  the hacked sites and have determined that that 2 redirects have been place in the server  one in the public html and one in the Web server configuration  the apache configuration appears to be exploited. We are currently restoring from Backup.
    ========

    Update Server has been restored form Backup
    Old server is still intact removed from the network 

    Changing Status to Low as everything is  comming back on line

      
    ==========================================================

    All Data restored server running  without issue 



    ===================

    Note This issues is 100% resolved  if you were affected we have preparred a ROH please open a ticket 
    and request a copy 


  • Date - 12/25/2015 02:00 - 12/28/2015 20:52
  • Last Updated - 12/29/2015 09:55
Planned partial power outage: (Resolved)
  • Priority - High
  • Dear Lowest Host/Empire Technology Customers

    In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.

    EVENT START DATE: 11/3/2015
    EVENT START TIME: 22:00 EST

    EVENT END DATE: 11/4/2015 
    EVENT END TIME: 03:00 EST

    EVENT LOCATION: SB NOC (Midwest)

    EXPECTED DOWN TIME: 15-45 Minutes
    EVENT DETAILS:

    Planed Partial Power Outage: We will be replacing a defective Power distribution unit on circuit A1A. During this planed maintenance your server will be inaccessible. The extended window is to allow for the scheduling of the electrician.

    Please contact us at https://secure.empire-hosting.net/submitticket.php if you have any questions or concerns.

    Thank You Network Operations Team


    ====================================
    Maintenance is underway


    ================================


    Maintentence complelete 
    No Issues 








  • Date - 11/03/2015 18:00 - 11/22/2015 15:02
  • Last Updated - 11/03/2015 23:34
Planned power outage: (Resolved)
  • Priority - Low
  • Dear Lowest Host/Empire Technology Customers

    In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.

    EVENT START DATE: 10/31/2015
    EVENT START TIME: 08:00 EDT

    EVENT END DATE: 10/31/2015 
    EVENT END TIME: 12:00 EDT

    EVENT LOCATION: SB NOC (Midwest)

    EXPECTED DOWN TIME: 0
    EVENT DETAILS: 
    The power company had just informed us that there is a planned power outage from 8am to 12pm Saturday. Our UPSs and generators will be ready to take over the load. However some clients only have b power. There's a possibly some of these systems will bounce as the main and backup power switches.

    AEP did not give us any details on why there will be an outage other than a being a maintenance window.

    Please contact us at https://secure.empire-hosting.net/submitticket.php if you have any questions or concerns.

    Thank You Network Operations Team

    =======================================================

    No issues remained on Generator power until utility company restored power
    Closing



  • Date - 10/31/2015 05:00 - 11/01/2015 10:13
  • Last Updated - 11/01/2015 10:12
DDOS Attack Midwest Shared hosting server. (Resolved)
  • Priority - Low
  • Affecting Server - Columbia
  • Dear Lowest Host/Empire Technology Customers

    In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.

    EVENT START DATE: 10/21/2015
    EVENT START TIME: 15:06 PDT
    EVENT END TIME:      15:20 PDT

    EVENT LOCATION:Midwest  NOC

    EVENT DETAILS:

    One of our shared hosting servers ins under a layer7 DDOS attack  RC2/Columbia attackers  are flooding the inbound network connection

    We are currently mitigating  the the attack at this time:
    Note: we will be making some modifications to the network on this server it will be unreachable for about 5 minutes
    Note: some of you  who are are reviving the attack  your domains will be redirected and you will be notified   in aseparate Email 
    ============================

    the network has been adjusted 

    ========================

    The attack has been mitigated  if your web site is down you are one of the unlucky ones that was under a intense flood
    Please see the ticket we opened for you on your behalf

    ====================================

    Resolved 







  • Date - 10/21/2015 15:58 - 10/22/2015 22:42
  • Last Updated - 10/22/2015 22:42
KVM 3 Emergency Maintenance (Resolved)
  • Priority - Critical
  • Dear Lowest Host/Empire Technology Customers

    In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.

    EVENT START DATE: 8/26/2015
    EVENT START TIME: 09:20 PDT

    EVENT LOCATION: SB NOC (Midwest)

    EVENT DETAILS: Emergency Maintenance  

    VPS Server Node KVM 3

    Is currently having some issues the server is stuck at the boot menu

    We are currently investigating  

    ========


    Update  10:46 PDT

    This issue has been resolved

    The server was a total Loss (would not boot) we restored all Virtual machines to another server 0 data loss. Pulling old server down to see what failed.  

    =================================================================================




  • Date - 08/27/2015 09:20 - 09/03/2015 09:59
  • Last Updated - 08/28/2015 06:48
KVM Server OS Upgrade & security patch (Resolved)
  • Priority - High
  • Affecting Server - VPS Master
  • Dear Lowest Host/Empire Technology Customers

    In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.

    EVENT START DATE: 8/24/2015
    EVENT START TIME: 23:00 PDT

    EVENT END DATE: 08/27/2015
    EVENT END TIME: 06:00 PDT

    EVENT LOCATION: SB NOC (Midwest)

    EXPECTED DOWN TIME: 20 Minutes

    EVENT DETAILS:

    Maintenance Scheduled for Midwest Data centers 
    Maintenance window: 24 Aug 2015 23:00 PDT to 27 Aug 2015 06:00 EDT


    Maintenance Window Duration: 3 Day
    Maintenance KVM Server OS Upgrade & security patch 
    Site Affected: KVM All Nodes 
    Expected Down time per Node 10 - 25 Minutes

    Risk Classification: high 
    Empire Hosting Master Ticket #: 20150824


    =========================

    All updates complete no issues 




  • Date - 08/24/2015 23:00 - 08/27/2015 13:09
  • Last Updated - 08/27/2015 13:09
Virtuozzo 4.7 End of Life (Resolved)
  • Priority - Critical
  • Dear Lowest Host/Empire Technology Customers

    In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.

    EVENT START DATE: 8/23/2015
    EVENT START TIME: 0:15 PDT

    EVENT END DATE: 9/31/2015 
    EVENT END TIME: TBA

    EVENT LOCATION: Midwest NOC

    EVENT DETAILS:

    All Virtuozzo nodes:

    Virtuozzo 4.7 End of Life is September 2015.
    There is no replacement Parallels/Odin has discontinued the Virtuozzo line. We will be migrating our existing Virtuozzo infrastructure to Virtuozzo OpenVZ.

    There will be no changes to your plans or resources the only changes will be new control panel access (more features) and the latest Virtualization file system which you may see a performance increase. 

    ================================================


    Upgrades complete no issues 

  • Date - 08/23/2015 06:56 - 08/26/2015 10:16
  • Last Updated - 08/26/2015 10:15
DDOS Attack Midwest Shared hosting server. (Resolved)
  • Priority - High
  • Affecting Server - Columbia
  • Dear Lowest Host/Empire Technology Customers

    In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.

    EVENT START DATE: 8/12/2015
    EVENT START TIME: 8:15 PST

    EVENT LOCATION:Midwest  NOC

    EVENT DETAILS:

    One of our shared hosting servers ins under a layer7 DDOS attack  RC2/Columbia attackers  are flooding the inbound network connection

    We are currently investigating and mitigating the the attack at this time
    Thank You network Operations Team


    ============================
    Update   

    The DDos accttack has been mitigated we will leave this issue open and monitor for 12 more hours  



    ======================================================



    Resolved 

  • Date - 08/12/2015 20:22 - 08/13/2015 10:27
  • Last Updated - 08/13/2015 10:27
Circuit 1A1 Power Fault (Resolved)
  • Priority - High
  • Dear Lowest Host/Empire Technology Customers

    In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.

    EVENT START DATE: 7/31/2015
    EVENT START TIME: 02:45 PDT

    EVENT END DATE: 7/31/2015 
    EVENT END TIME: TBA

    EVENT LOCATION: SB NOC (Midwest)


    EVENT DETAILS: 

    Sorry for the blanket email this only affects select servers
    At approximately 2:45 PDT we lost power to one of circuits in our mid west facility some servers were power cycling until we could bring the whole system down and bring each server up 1 at a time currently power is stable and all affected servers are on line we will continue to montor the system and update if we need to perform any additional maintenance.

    Thank You NOC  Operations Team 

  • Date - 07/31/2015 02:45 - 08/02/2015 11:55
  • Last Updated - 07/31/2015 03:13
XEN 9 Hard Drive replacemnt (Resolved)
  • Priority - Critical
  • Dear Lowest Host/Empire Technology Customers

    In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.

    EVENT START DATE: 7/6/2015
    EVENT START TIME: 10:00 PDT

    EVENT END DATE: 7/6/2015
    EVENT END TIME: 18:00 PDT

    EVENT LOCATION: SB NOC (Midwest)

    EXPECTED DOWN TIME: 5 MIN
    EVENT DETAILS:
    We will be performing Emergency server maintenance: XEN 9 VPS server: During a routine Inspection on this server it was found that a hard drive is failing. Being this is the 4th hard drive that has failed on this particular machine in less than a year we are going pull this server down for extended maintenance and replace the entire array. We will be pulling a live snapshot of your server and moving it to a new hardware node.  
    Maintenance Window: 7/6/2015 10:00 - 7/6/2015 18:00 PDT

    Thank You Server Operations Team
    Last Updated - 07/03/2015 16:56


    =============================
    Completed No issues

  • Date - 07/06/2015 10:00 - 07/11/2015 09:46
  • Last Updated - 07/11/2015 09:46
Shared Hosting servers Maint Patch (Resolved)
  • Priority - Low
  • Affecting Server - rc1
  • Dear Lowest Host/Empire Technology Customers

    In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.

    EVENT START DATE: 7/5/2015
    EVENT START TIME: 00:01 EDT

    EVENT END DATE: 7/5/2015
    EVENT END TIME: 03:00 EDT

    EVENT LOCATION: SB NOC (Midwest) Las Vegas NOC (West Coast)

    EXPECTED DOWN TIME: 5 MIN
    EVENT DETAILS:
    We will be performing server maintenance: on all of shared hosting servers. Reason: server security & maintenance patch.
    Maintenance Window: 7/5/2015 00:01 - 7/5/2015 0300:00 PM EDT

    Thank You Server Operations Team


    ===========================================================
    Maint complete No issues



  • Date - 07/05/2015 02:00 - 07/11/2015 09:46
  • Last Updated - 07/06/2015 09:28
XEN/KVM OS Upgrade (Resolved)
  • Priority - High
  • Affecting Other - XEN/KVM OS Upgrade
  • Dear Lowest Host/Empire Technology Customers

    In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.

    EVENT START DATE: 5/10/2015
    EVENT START TIME: 06:00 PDT

    EVENT END DATE: 5/14/2015
    EVENT END TIME: 08:00 PDT

    EVENT LOCATION: SB NOC (Midwest) Las Vegas NOC (West Coast)

    EXPECTED DOWN TIME: 3MIN - 25 MIN
    EVENT 1 DETAILS:
    Our XEN/KVM Software vender has released some New storage features. To allow for use we will be converting all of our XEN & KVM machines to a new configuration that will allow for the use of Shared & Multi storage arrays for future expansion. Some of the vm Configurations are on an old configuration and cannot take advantage of this and the new configuration breaks our Backup configuration & NRT Migration. We will be doing most of the maintenance between 12:00 EDT - 3:00 EDT each Day.




    EVENT 2 DETAILS: Start Date as posted above  patch as needed per conversion
    Maintenance XEN Server OS Upgrade & security patch
    Site Affected: All XEN & KVM Nodes.


    ========================================
    Upgrade completed





     

  • Date - 05/10/2015 05:46 - 05/18/2015 07:38
  • Last Updated - 05/18/2015 07:38
Backup Server(s) Maint. (Resolved)
  • Priority - Low
  • Affecting Other - Backup Server(s) Maint.
  • Dear Lowest Host/Empire Technology Customers

    In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.

    EVENT START DATE: 5/10/2015
    EVENT START TIME: 06:00 PDT

    EVENT END DATE: 5/14/2015
    EVENT END TIME: 08:00 PDT

    EVENT LOCATION: SB NOC (Midwest) Las Vegas NOC (West Coast)

    EXPECTED DOWN TIME: NA
    EVENT  DETAILS:
    Per the release of Our XEN/KVM Software with new storage features. Our Backup servers will be off line or in a limited capacity to accommodate an update of our backup procedures. Please make a note of it

    ==============================
    Las Vegas NOC has been completed

    ==============================

    100% complete








  • Date - 05/10/2015 06:24 - 05/15/2015 04:34
  • Last Updated - 05/15/2015 04:34
Shared Hosting servers Maint (Resolved)
  • Priority - Low
  • Affecting Server - Columbia
  • Dear Lowest Host/Empire Technology Customers

    In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.

    EVENT START DATE: 5/10/2015
    EVENT START TIME: 06:00 PDT

    EVENT END DATE: 5/14/2015
    EVENT END TIME: 08:00 PDT

    EVENT LOCATION: SB NOC (Midwest) Las Vegas NOC (West Coast)

    EXPECTED DOWN TIME: 30 seconds -25 Minutes
    EVENT DETAILS: We will be making some changes to our Backup system for all of our shared hosting servers. During this time our R1 soft interface will be down: Normally we would not send out an email for only 30 seconds of network interruption but there is a chance we may need to shut one or more servers down longer to accommodate the change.

    ============
    Maint Complete no issues



  • Date - 05/10/2015 06:37 - 05/13/2015 01:27
  • Last Updated - 05/13/2015 01:27
Virtuozzo VPS server Maint Patch (Resolved)
  • Priority - Critical
  • Dear Lowest Host/Empire Technology Customers

    In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.

    EVENT START DATE: 5/12/2015
    EVENT START TIME: 10:00 PDT

    EVENT END DATE: 05/14/2014
    EVENT END TIME: 06:00 PDT

    EVENT LOCATION: SB NOC (Midwest)

    EXPECTED DOWN TIME: 20 Minutes

    EVENT DETAILS:

    Maintenance Scheduled for Midwest Data centers
    Maintenance window: 12 May 2015 00:01 EDT to  14 May 2015 06:00 EDT
    Maintenance Window Duration: 2 Day
    Maintenance Virtuozzo Server OS Upgrade & security patch
    Site Affected: Virtouzzo All Nodes
    Expected Down time per Node 10 - 25 Minutes

    Risk Classification: high
    Empire Hosting Master Ticket #: 20150510

     

    Thank You Server Operations Team

    ========================================

    Maintence complete  No issues




  • Date - 05/12/2015 06:51 - 05/13/2015 01:27
  • Last Updated - 05/13/2015 01:27
Shared Hosting servers Maint Patch (Resolved)
  • Priority - Medium
  • Affecting System - All Shared Hosting Servers
  • Dear Lowest Host/Empire Technology Customers

    In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.

    EVENT START DATE: 4/28/2015
    EVENT START TIME: 23:00 PST

    EVENT END DATE: 4/29/2015
    EVENT END TIME: 08:00 PST

    EVENT LOCATION: SB NOC (Midwest) Las Vegas NOC (West Coast)

    EXPECTED DOWN TIME: 5 MIN
    EVENT DETAILS:
    We will be performing server maintenance: on all of shared hosting servers. Reason: server security & maintenance patch.
    Maintenance Window: 4/28/2015 23:00 - 4/29/2015 08:00 PM PST

    Thank You Server Operations Team


    ========                

    Maint complete no issus

  • Date - 04/28/2015 23:00 - 04/29/2015 06:47
  • Last Updated - 04/29/2015 06:47
XEN 1 Network Maint: (Resolved)
  • Priority - Critical
  • Dear Lowest Host/Empire Technology Customers

    In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our
    data-centers nationwide. This does not necessarily mean that your facility or server will be affected.
    All pertinent information can be found in the description for each maintenance action to be performed.

    EVENT START DATE: 04/10/2015
    EVENT START TIME: 11:12 PST
    EVENT END DATE:04/10/2015
    EVENT END TIME: 12:10 PST
    EVENT LOCATION: Midwest NOC
    EXPECTED DOWN TIME: 20 Min
    EVENT DETAILS:


    Emergency Maintenance Scheduled for Midwest Data center
    Maintenance window: FRi 10 April 2014 11:20 PST
    Maintenance Window Duration: 15 Min
    Maintenance Network Failed On Node
    Site Affected: XEN Nodes 1
    Expected Down time 20 Min
    Risk Classification: High
    Empire Hosting Master Ticket #: 20150410

    The Main Network On this server has failed.
    We will be patching the server and rebooting to resolve the issue
    Thank You Server Operations Team


    ====================
    Update The network has been Restored

    ============================
    this has been 100% resoved

  • Date - 04/10/2015 11:15 - 04/10/2015 17:27
  • Last Updated - 04/10/2015 14:23
Maintenance Failed Hard drive replacement (Resolved)
  • Priority - Critical
  • Dear Lowest Host/Empire Technology Customers

    In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.

    EVENT START DATE: 3/13/2015
    EVENT START TIME: 12:00 PDT

    EVENT END DATE: 03/13/215
    EVENT END TIME: 16:00PDT

    EVENT LOCATION: Las Vegas NOC

    EVENT DETAILS:

    During a normal Maintenance inspection it was revealed that one of the Hard drive in the raid array has failed.
    We will be replacing the defective hard drive which will require a reboot of the Hardware node 10-20 minutes of down time is expected
    Thank You Server Operations Team

    ================================================================================================

    Upadte Defective drive has been replaced and the Raid Aray is rebuilding

    This issue will remain open until the raid rebuild is complete

    ==============================================

    Rebuild complete closing

  • Date - 03/13/2015 09:00 - 03/14/2015 09:18
  • Last Updated - 03/22/2015 14:21
Network Issues Las Vegas NOC (Resolved)
  • Priority - Low
  • EVENT START DATE: 3/8/2015
    EVENT START TIME: 10:45 PDT

    EVENT END DATE: 3/8/2015
    EVENT END TIME: 10:54 PDT

    EVENT LOCATION: Las Vegas NOC

    EXPECTED DOWN TIME: NA

    EVENT DETAILS:

    Risk Classification: Low
    Empire Hosting Master Ticket #: 20150308
    Our Primary Level3/TW Telecom Primary NTT Primary Route is has Gone down which is causing it to Flap with our Backup Feed we have routed all traffic away to our Backup until Our upstream transit providers can resolve their their issues. This will be a No impact event.


    ======================================================

    Update Our primary transit routes have come back on line
    We are going to remain on our Backup Feed Until we get and update from our upstream


    ================================================================

    issue resolved

  • Date - 03/08/2015 10:45 - 03/08/2015 12:13
  • Last Updated - 03/08/2015 11:35
Drive Maint VPS 4 (Resolved)
  • Priority - Critical
  • Dear Lowest Host/Empire Technology Customers

    In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.

    EVENT START DATE: 2/14/2015
    EVENT START TIME: 13:00 PST

    EVENT END DATE: 2/14/2015
    EVENT END TIME: 15:30 PST

    EVENT LOCATION: SB NOC (Midwest)
    EXPECTED DOWN TIME: 20 Min
    EVENT DETAILS:

    We will be doing Emergency Drive Maintenance on Virtuozzo server VPS4
    The Partitions are having drive errors and require a repair

    Thank You Server Operations Team
    ============================

    Update we are moving the time up on this action to 13:00 PST

    =======================================================

    Hard drive partitions have been repaired No issues



  • Date - 02/14/2015 13:00 - 02/14/2015 12:52
  • Last Updated - 02/14/2015 12:52
DDOS attack Las Vegas Data center (Resolved)
  • Priority - Critical
  • Dear Lowest Host/Empire Technology Customers

    In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.

    EVENT START DATE: 2/7/2015
    EVENT START TIME: 7:25 PST

    EVENT LOCATION: LV NOC (West Cost)

    EVENT DETAILS:

    XEN node 6 is currently Under a DDOS attack flooding the inbound network connection
    We are currently investigating and mitigating the the attack at this time
    Thank You network Operations Team

    ============================

    the Attack has been Mitigated and the attacking IP's have benn Null Routed

    ==============================================

  • Date - 02/07/2015 07:23 - 02/07/2015 21:44
  • Last Updated - 02/07/2015 07:36
SSD Raid Maint on VPS nodes (Resolved)
  • Priority - Low
  • Affecting Other - SSD VPS Nodes
  • Dear Lowest Host/Empire Technology Customers

    In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.




    EVENT START DATE: 1/31/2015

    EVENT START TIME: 15:00 PST

    EVENT END DATE: 2/02/2015
    EVENT END TIME: 06:00 PST

    EVENT LOCATION: SB NOC (Midwest)
    EXPECTED DOWN TIME: 30 Seconds
    EVENT DETAILS:

    We will be doing Raid Maintenance on all Hardware Raid SSD machines in our Midwest NOC
    this event will a a low impact event 30 seconds of down time 2 instances while we rearrange our raid configuration to increase performance.

    Thank You Server Operations Team

    ============================


    Maint completed No issues



  • Date - 01/31/2015 15:00 - 02/02/2015 17:55
  • Last Updated - 02/02/2015 17:54
XEN Maint Patch & Security Update (Resolved)
  • Priority - High
  • Dear Lowest Host/Empire Technology Customers

    In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.

    EVENT START DATE: 1/27/2015
    EVENT START TIME: 17:39 PST

    EVENT END DATE: 1/28/2015 (time pushed up 2 days)
    EVENT END TIME: 08:00 PST


    EVENT LOCATION: SB NOC (Midwest) Las Vegas NOC (West Coast)

    EXPECTED DOWN TIME: 10 MIN
    EVENT 1 DETAILS:
    A forced update from one of has software venders made our XEND instance shut down
    We will be rebooting XEN 1 XEN 6 XEN 4 Investigating the rest of the XEN Nodes.

    EVENT 2 DETAILS: Start Date 1/31/2015 00:00 PST
    Maintenance XEN Server OS Upgrade & security patch
    Site Affected: XEN Nodes 2, 7, 8, 10, 12
    ==============================================================

    Affected Nodes 1, 6, 4 patched No issues


    ===================================================================


    the Tme line on this has been pushed up forced update to all of our XEN nodes were pushed by our software vender

    ===============================================================================



    XEN Nodes 2, 7, 8, 10, 12 patched No issues

  • Date - 01/27/2015 18:37 - 01/31/2015 04:18
  • Last Updated - 01/28/2015 07:20
Mid west Data Center On Generator Power (Resolved)
  • Priority - Medium
  • Dear Lowest Host/Empire Technology Customers

    In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.

    Mid west Data Center On Generator Power

    EVENT START DATE: 1/25/2014
    EVENT START TIME: 18:36 PST

    EVENT LOCATION: Midwest NOC

    EXPECTED DOWN TIME: NA

    EVENT DETAILS:
    major weather related event in our Mid West data center has force us onto Generator power
    a few servers in on one of our power feeds power cycled them selves everything has been powered back on
    if you are having issues please open a support ticket.

    This issue will remain open until we go back on utility power


    ==============================================

    utility power has been restored

    Closing Issue

  • Date - 01/25/2015 19:41 - 01/26/2015 11:22
  • Last Updated - 01/25/2015 20:54
XEN 7 Network Maint: (Resolved)
  • Priority - Low
  • Dear Lowest Host/Empire Technology Customers

    In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.

    EVENT START DATE: 1/7/2014
    EVENT START TIME: 16:00 PST

    EVENT END DATE: 1/8/2014
    EVENT END TIME: 15:00 PST

    EVENT LOCATION: Midwest NOC

    EXPECTED DOWN TIME: 30 Seconds - 1 minute

    EVENT DETAILS:

    Emergency Maintenance Scheduled for Midwest Data center
    Maintenance window: Wed 7 Jan 2014 16:00 PST to Thurs 8 Jan 15:00 PST
    Maintenance Window Duration: 12 hour
    Maintenance Private Network Bridge failure
    Site Affected: XENNodes 7
    Expected Down time 30 Secs

    Risk Classification: Low
    Empire Hosting Master Ticket #: 20141229

    Our Private VPN that controls our Midwest Private network has failed on this server .
    We will be pulling this server down for extended Maintenance rather than having clients down during this period.
    We will be Live Migrating all VPS's to different Nodes

    Thank You Server Operations Team


    ==================================================================================

    This maintenance action was delayed because of a hard drive failure on another server took priority
    90% complete End time has been updated



    this

  • Date - 01/07/2015 16:00 - 01/17/2015 10:33
  • Last Updated - 01/08/2015 04:23
XEN 9 Hard Drive replacemnt (Resolved)
  • Priority - High
  • Dear Lowest Host/Empire Technology Customers

    In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.


    EVENT START DATE: 1/7/2014
    EVENT START TIME: 16:30 PST

    EVENT END DATE: 1/7/2014
    EVENT END TIME: 17:30 EDT

    EVENT LOCATION: Mid West NOC

    EXPECTED DOWN TIME: 10 Minutes

    EVENT DETAILS:

    Emergency Maintenance Scheduled for Midwest Nevada Data center
    Maintenance window: Wed 7 Jan 2014 16:30 PST to 17:30 PST
    Maintenance Window Duration: 2 hour
    Maintenance Failed Hard drive replacement
    Site Affected: XENNodes 9
    Expected Down time 10 15 Minutes

    Risk Classification: high
    Empire Hosting Master Ticket #: 2016107

    During a normal Maintenance inspection it was revealed that one of the Hard drive in the raid array has failed.
    We will be replacing the defective hard drive which will require a reboot of the Hardware node 10 minutes of down time is expected

    Thank You Server Operations Team

    ===============================

    UPDATE:

    The Raid array is failing to build

    ========================
    We will be migrating All VPS off of this server and pulling the server down and rebuilding the array from scratch
    Expect another reboot of your servers

    ===========================================

    Raid has been rebuilt and is functioning normally
    Downgrading to Low until new backups sync

    =========================================

    Backups syncing closing issue




  • Date - 01/07/2015 16:30 - 01/08/2015 04:17
  • Last Updated - 01/08/2015 04:17
XEN 6 Hard Drive replacemnt (Resolved)
  • Priority - Critical
  • Dear Lowest Host/Empire Technology Customers

    In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.

    EVENT START DATE: 12/29/2014
    EVENT START TIME: 06:01 PST

    EVENT END DATE: 12/29/2014
    EVENT END TIME: 06:00 EDT

    EVENT LOCATION: Las Vegas NOC (West Coast)

    EXPECTED DOWN TIME: 10 Minutes

    EVENT DETAILS:

    Emergency Maintenance Scheduled for Las Vegas Nevada Data centers
    Maintenance window: Monday 29 Dec 2014 06:00 PST to 12:00 PST
    Maintenance Window Duration: 6 hour
    Maintenance Failed Hard drive replacement
    Site Affected: XENNodes 6
    Expected Down time 10 15 Minutes

    Risk Classification: high
    Empire Hosting Master Ticket #: 20141229

    During a normal Maintenance inspection it was revealed that one of the Hard drive in the raid array has failed.
    We will be replacing the defective hard drive which will require a reboot of the Hardware node 5 minutes of down time is expected

    Thank You Server Operations Team

    =============================

    closed no issues

  • Date - 12/29/2014 06:20 - 01/02/2015 10:49
  • Last Updated - 01/02/2015 10:49
CVE-2014-9322 Patch (Resolved)
  • Priority - High
  • Dear Lowest Host/Empire Technology Customers

    In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.

    EVENT START DATE: 12/19/2014
    EVENT START TIME: 00:01 EDT

    EVENT END DATE: 12/19/2014
    EVENT END TIME: 06:00 EDT

    EVENT LOCATION: SB NOC (Midwest) Las Vegas NOC (West Coast)

    EXPECTED DOWN TIME: 10 Minutes

    EVENT DETAILS:

    Maintenance Scheduled for Midwest & West Coast Data centers
    Maintenance window: Friday 19 Dec 2014 00:01 EDTto 06:00 EDT
    Maintenance Window Duration: 6 hour
    Maintenance Shared Hosting Servers OS Upgrade & CVE-2014-9322 patch
    Site Affected: Shared Hosting Servers ALL
    Expected Down time 10 15 Minutes

    Maintenance Scheduled for Midwest & West Coast Data centers
    Maintenance window: Friday 19 Dec 2014 00:01 EDTto 23:59 EDT
    Maintenance Window Duration: 24 hour
    Maintenance Managed XEN Servers OS Upgrade & CVE-2014-9322 patch
    Site Affected: Shared Hosting Servers ALL
    Expected Down time 10 15 Minutes

    Maintenance Scheduled for Midwest Data centers
    Maintenance window: Friday 19 Dec 2014 00:01 EDTto 06:00 EDT
    Maintenance Window Duration: 6 hour
    Maintenance Managed Virtuozzo Servers OS Upgrade & CVE-2014-9322 patch
    Site Affected: Shared Hosting Servers ALL
    Expected Down time 10 15 Minutes

    Maintenance Scheduled for Midwest & West Coast Data centers
    Maintenance window: Friday 19 Dec 2014 00:01 EDTto 23.59 EDT
    Maintenance Window Duration: 24 hour
    Maintenance Managed Dedicated Servers OS Upgrade & CVE-2014-9322 patch
    Site Affected: Shared Hosting Servers ALL
    Expected Down time 10 15 Minutes

    Risk Classification: high
    Empire Hosting Master Ticket #: 20141218

    Thank You Server Operations Team

    ====================================

    update Dec 19 02:18 EST

    All shared Hosting Servers Completed
    All Virtuozzo servers completed

    Client Dedicated servers 10% done in progress
    Client Xen VPS servers 1% done in progress



  • Date - 12/19/2014 22:01 - 12/26/2014 05:24
  • Last Updated - 12/18/2014 23:22
XEN Maint Patch & Security Update (Resolved)
  • Priority - High
  • Dear Lowest Host/Empire Technology Customers

    In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.

    EVENT START DATE: 11/09/2014
    EVENT START TIME: 00:01 EDT

    EVENT END DATE: 11/09/2014
    EVENT END TIME: 06:00 EDT

    EVENT LOCATION: Las Vegas NOC

    EXPECTED DOWN TIME: 15 Minutes

    EVENT DETAILS:

    Maintenance Scheduled for Las Vegas Nevada Data centers
    Maintenance window: Monday 9 Nov 2014 00:01 EDTto 06:00 EDT
    Maintenance Window Duration: 6 hour
    Maintenance Xen Server OS Upgrade & security patch
    Site Affected: XENNodes 2, 4, 6, 8, 10
    Expected Down time per Node 10 15 Minutes

    Risk Classification: high
    Empire Hosting Master Ticket #: 2014081

    Thank You Server Operations Team

    ======================================
    Updates completed XEN 4,8,10

    Will reschedule xen 2 & 6 as it was not completed within the maintenance window
    Those effected will Be emailed for the new time


    ====================================================

    All Updates complete No issues





  • Date - 11/09/2014 00:01 - 11/15/2014 03:10
  • Last Updated - 11/11/2014 05:31
Virtuozzo Maint Patch & Security Update (Resolved)
  • Priority - High
  • Dear Lowest Host/Empire Technology Customers

    In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.

    EVENT START DATE: 11/09/2014
    EVENT START TIME: 00:01 EDT

    EVENT END DATE: 11/09/2014
    EVENT END TIME: 06:00 EDT

    EVENT LOCATION: SB NOC (Midwest)

    EXPECTED DOWN TIME: 20 Minutes

    EVENT DETAILS:

    Maintenance Scheduled for Midwest Data centers
    Maintenance window: Sunday 9 Nov 2014 00:01 EDTto 06:00 EDT
    Maintenance Window Duration: 6 hour
    Maintenance Virtuozzo Server OS Upgrade & security patch
    Site Affected: Virtouzzo Nodes 4, 7
    Expected Down time per Node 10 25 Minutes

    Risk Classification: high
    Empire Hosting Master Ticket #: 2014082

    Thank You Server Operations Team


    =====================================

    Update completed No issues

  • Date - 11/09/2014 00:01 - 11/09/2014 05:14
  • Last Updated - 11/09/2014 05:13
Level3 Network Issues (Resolved)
  • Priority - Low
  • EVENT START DATE: 9/9/2014
    EVENT START TIME: 10:00 PDT

    EVENT END DATE: UNK
    EVENT END TIME: UNK

    EVENT LOCATION:  Midwest NOC

    EXPECTED DOWN TIME:  NA

    EVENT DETAILS:

    Risk Classification: Low
    Empire Hosting Master Ticket #: 201409

    Our Primary Level3 Route is has some congestion issues which is causing it to Flap with our Primary Internap Feed we have routed all traffic away to our Backup until Level3 can resolve the issues. This will be a No impact event.  



    Thank You Network Operations Team

    ================================

    Update Level3 has resolved their Issue

  • Date - 09/09/2014 10:06 - 09/09/2014 11:35
  • Last Updated - 09/09/2014 11:35
EMERGENCY SCHEDULED MAINTENANCE (Resolved)
  • Priority - High
  • Dear Lowest Host/Empire Technology Customers

    In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.

     

    EVENT START DATE: 7/15/2014
    EVENT START TIME: 07:00 PDT

    EVENT END DATE: 07/18/2014
    EVENT END TIME: 23:00 PDT

    EVENT LOCATION:  Midwest NOC

    EXPECTED DOWN TIME:  10 Minutes

    EVENT DETAILS:

    Maintenance Scheduled for Midwest Data center
    Maintenance window: Tues 15 July 2014 07:00 PDT to Friday 18 July 23:00 PDT
    Maintenance Window Duration: 4 days
    Maintenance Failing pair of Hard Drives
    Site Affected: XEN Nodes 9
    Expected Down time 1 Reboot to new Server

    Risk Classification: high
    Empire Hosting Master Ticket #: 201408

    During a routine Drive inspection it was found that this server has 2 drives about to fail. There is currently there is no issues the raid array is continuing to function as expected,  But due to the nature that 2 drives are failing simultaneously that are in the same mirror we are electing to move all data off as a precaution just in case of data corruption.



    Thank You Server Operations Team


    =======================================

    No issues closing

  • Date - 07/15/2014 07:07 - 07/19/2014 13:30
  • Last Updated - 07/19/2014 13:26
Network Upgrades to 1G (Resolved)
  • Priority - Critical
  • EVENT START DATE: 6/04/2014
    EVENT START TIME: 09:58 PDT

    EVENT END DATE: 06/18/2014
    EVENT END TIME: 10:04PDT

    EVENT LOCATION:  Midwest NOC

    EVENT DETAILS:

    We will Be Expanding our network capacity at our Mid West Data Center June 4, 2014 – June 18, 2014. All Network connections will be 1G Minimum form a 10G backbone.

    This is follow up to our Network Upgrades “May 2014 Announcements Network Upgrades to 1G”

    We will Be Expanding our network capacity at our Mid West Data Center June 1, 2014 – June 18, 2014. All Network connections will be 1G Minimum form a 10G backbone.

    Dedicated Server clients only:  Beginning June 4, 2014 we will begin migrating existing clients tour All 1 G Network.
    Dedicated Server Clients you now have the Option of 100Mbs Metered, 100Mbs unmetered, and 1Gbs Metered.  Per Usual 100M & 1G dedicated line are still available as always. You may notice Some Slight latency during the week of Jun 8 while we cut over to the new Network.

    VPS Server clients only:  Beginning June 10, 2014 we will begin migrating existing clients to All 1 G Network.
    For XEN VPS clients you now have the option of 10Mbs unmetered, 100Mbs metered, 100Mbs unmetered, and 1Gbs metered. Virtouzzo you will have the option of 100Mbs & 100Mbs unmetered. You may notice Some Slight latency during the week of Jun 8 while we cut over to the new Network.   XEN additional: we will kill 2 birds with one stone and upgrading our few legacy XEN nodes that are still XEN 3 to XEN 4.

    Shared Hosting & Shared Hosting Resellers only: Shared Hosting clients will remain on 100Mbs& burstable to 1G. You may notice Some Slight latency during the week of Jun 8 while we cut over to the new Network.

    =====================================================================================
    Update 90% complete
    =====================================================================================
    This is 100% complete  leaving issue open because of some Lingering issues with Level3 Transit

  • Date - 06/04/2014 08:01 - 07/19/2014 13:26
  • Last Updated - 07/15/2014 07:26
EMERGENCY SCHEDULED MAINTENANCE (Resolved)
  • Priority - High
  • Dear Lowest Host/Empire Technology Customers

    In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.

     

    EVENT START DATE: 5/19/2014
    EVENT START TIME: 09:58 PDT

    EVENT END DATE: 05/19/2014
    EVENT END TIME: 10:04PDT

    EVENT LOCATION:  Las Vegas NOC

    EVENT  DETAILS:

    XEN Node 6 Locked up & required a emergency reboot
    Server has been rebooted and all VPS server are on line 

    Risk Classification: high 

    We will continue to monitor the Node for any other anomalies



    Thank You Server Operations Team



  • Date - 05/19/2014 10:58 - 05/21/2014 05:47
  • Last Updated - 05/19/2014 10:09
XEN Maint Patch & Security Update (Resolved)
  • Priority - High
  • Dear Lowest Host/Empire Technology Customers

    In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.

     

    EVENT START DATE: 5/17/2014
    EVENT START TIME: 00:01 PDT

    EVENT END DATE: 05/17/2014
    EVENT END TIME: 23:00 PDT

    EVENT LOCATION:  Las Vegas NOC

    EXPECTED DOWN TIME:  10 Minutes

    EVENT  DETAILS:

    Maintenance Scheduled for Las Vegas Nevada Data centers
    Maintenance window: Saturday 17 May 2014 00:01 PDT  to 23:00 PDT
    Maintenance Window Duration: 11 hour
    Maintenance Xen Server OS Upgrade  upgrade to XEN 4
    Site Affected: XEN  Nodes 10,8
    Expected Down time  1 Reboot to new Xen 4 image

    Risk Classification: high  
    Empire Hosting Master Ticket #: 201406



    Thank You Server Operations Team

    =================================

    Update 17 May 21:48
    Upgrade complete



  • Date - 05/17/2014 00:01 - 05/18/2014 10:08
  • Last Updated - 05/17/2014 21:48
EMERGENCY SCHEDULED POWER MAINTENANCE (Resolved)
  • Priority - Medium
  • EVENT START DATE: 05/15/2014
    EVENT START TIME: 08:00 PDT

    EVENT END DATE: 05/15/2014
    EVENT END TIME: 12:00 PDT

    EVENT LOCATION: Las Vegas NOC

    EVENT DETAILS:
    Maintenance Summary: Primary ATS Replacement

    Maintenance Details: Electrical contractors have discovered a critical defect in the existing ATS (Automatic Transfer Switch), which is responsible for transferring power from the utility grid to the backup generator in the event of a utility failure. The defect discovered could potentially cause the unit to malfunction in the event of a power outage, thereby causing power to be interrupted to customer equipment. Due to the potential sever of this problem, we've instructed the electrical contractor to proceed with the replacement on an emergency basis. Our electricians are not able to swap out this piece of equipment while it is still energized, so we will be transferring power load for our primary "A" power feed over to our backup generator for approximately 1-2 hours while the equipment is replaced. We expect this change to be invisible to our customers, however as with any type of power maintenance, unexpected issues can arise which could have an impact on power delivery. As such, we have elected to perform this maintenance during business hours in order to have full availability of all vendors involved, thereby minimizing the impact of any potential problems that could arise.

    ======================
    Maintenance completed no issues




  • Date - 05/15/2014 08:00 - 05/16/2014 19:15
  • Last Updated - 05/16/2014 19:15
EMERGENCY SCHEDULED MAINTENANCE (Resolved)
  • Priority - Low
  • EVENT START DATE: 04/30/2014
    EVENT START TIME: 00:00 CDT

    EVENT END DATE: 04/30/2014
    EVENT END TIME: 06:00 CDT

    EVENT LOCATION: Cedar Falls NOC

    EVENT DETAILS:
    Qwest/Century Link  Engineering will be reloading a core router in the Cedar Falls  datacenter on advice from vendor to resolve issue with packetloss on router.  This will be a service-impacting maintenance — customers in our Cedar Falls, IA datacenter location will be unavailable for approximately 10 minutes while the router reload is completed.  This reload is required to restore complete service to the datacenter.



    ==========================================
    Update 
    The maintenance has been completed  No issues noted.
    we will be downgrading this issue to low  until the end of the maintenance window.

    ============================================

    Closed



  • Date - 04/30/2014 00:00 - 04/30/2014 08:48
  • Last Updated - 04/30/2014 08:48
XEN Maint Patch & Security Update (Resolved)
  • Priority - High
  • Affecting Other - Las Vegas XEN Nodes
  • Dear Lowest Host/Empire Technology Customers

    In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.

     

    EVENT START DATE: 4/2/2014
    EVENT START TIME: 00:01 EDT

    EVENT END DATE: 4/2/2014
    EVENT END TIME: 06:00 EDT

    EVENT LOCATION:  Las Vegas NOC

    EXPECTED DOWN TIME:  15 Minutes

    EVENT  DETAILS:

    Maintenance Scheduled for Las Vegas Nevada Data centers
    Maintenance window: Wednesday 2 April 2014 00:01 EDT  to 06:00 EDT
    Maintenance Window Duration: 6 hour
    Maintenance Xen Server OS Upgrade & security patch  
    Site Affected: XEN  Nodes 2,4,6 8,10
    Expected Down time per Node  10 – 15 Minutes

    Risk Classification: high  
    Empire Hosting Master Ticket #: 201405

     

     Thank You Server Operations Team

    =============================================
    Update  Updated time window 10 11:30 EDT

    EVENT START DATE: 4/6/2014
    EVENT START TIME: 00:01 EDT

    EVENT END DATE: 4/6/2014
    EVENT END TIME: 06:00 EDT

    EVENT LOCATION:  Las Vegas NOC
    EVENT RESCHEDULED:  From 4/2/2014

    EXPECTED DOWN TIME:  15 Minutes

    EVENT  DETAILS:

    Original Maintenance Scheduled for Las Vegas Nevada Data centers  4/2/14
    Has been rescheduled: Sunday 6 April 2014 00:01 EDT  to 09:00 EDT.
    Maintenance Window Duration: 9 hour
    Maintenance Xen Server OS Upgrade & security patch  
    Site Affected: XEN  Nodes 6 & 8  All other Xen Nodes have been completed
    Expected Down time per Node  10 – 15 Minutes Risk Classification: high
    Empire Hosting Master Ticket #: 201405

     

     Thank You Server Operations Team



    ==============================
    Updated completed

  • Date - 04/02/2014 00:01 - 04/07/2014 06:59
  • Last Updated - 04/07/2014 06:59
Network Outage Cedar Falls Data Center (Resolved)
  • Priority - Low
  • Dear Lowest Host/Empire Technology Customers

    In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.

     

    EVENT START DATE: 12/13/2013
    EVENT START TIME: 04:59 PST

     

    EVENT  DETAILS:


    At 04:59 PST
    Our Cedar Falls Datacenter is currently experiencing a critical loss of connectivity. We are working with our upstream providers to restore this connectivity as soon as possible.
    At this time we do not have any more information than this we will update further as soon as we have more information 

    Thank You, 

    Datacenter Operations Team

    =============================
    Update 10:53 PST 
    Network  has Been Fully resoterd 
    ===================================

    =============================
    Update 13:00 PST 
    Issue has been downgraded to Low 
    ===================================

    =======================================
    Lowest Host/Empire Technology LLC Reason for Outage (RFO)

    Description of Outage:
    On Friday, December 13, 2013 at approximately 04:59 AM PST, Lowest Host Empire Technology LLC experienced a Network disruption do to a failure of our uptream’s internal core routing configuration. Which Caused our core2.CFII Router to drop all IP announcements leaving our network without any type of external routing protocols to enable routing between the core1.CFII Router and other devices within our network. This caused degraded network performance, packet loss, and the inability to reach the outside network.

    Our core1.CFII Router handles our upstream BGP sessions as well as all of our peering BGP sessions on the Mid West Exchange. That caused those networks to attempt to send traffic destined for our network down our peering BGP sessions, which were ultimately unreachable due to the failed routing protocols on both the primary and secondary. This condition lasted until the routing equipment was replaced by our upstream providers.

    Resolution:
    The Faulty Equipment was replaced by our upstream peer. Configuration was restored, and our external routing protocols began operating normally at this time. We have not had a total network failure in our Cedar Falls facility in over 7 years. It seems we have put too much faith in the ability of our upstream to provide uninterrupted connectivity. So we be looking into making some changes to our BGP configuration over the next few weeks in order to make our external routing more fault tolerant in the case of an internal routing failure on the part of our upstream Network providers. We sincerely apologize for any inconvenience that this matter has caused and thank you for your patience as we work to resolve these issues and improve our facility to serve you better.

  • Date - 12/13/2013 04:59 - 12/25/2013 20:23
  • Last Updated - 12/15/2013 04:10
Hard Drive failure on VPS Nodes (Resolved)
  • Priority - High
  • ear Lowest Host/Empire Technology Customers

    In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.

     

    EVENT START DATE: 10/16/2013
    EVENT START TIME: 09:00 CDT

    EVENT END DATE: 10/16/2013
    EVENT END TIME: 10:00 CDT

    EVENT LOCATION:  Cedar Falls NOC

    EXPECTED DOWN TIME:  15 Minutes

    EVENT  DETAILS:

    EMERGENCY MAINT: We have detected a failed hard drive on our xen 1 & virtuozzo 4  server  we will be replacing them & rebooting the servers  be

    Thank You, 

    Data center Operations 

     



    ========================================================

    Update 10/16 2013 1000 CDT   both servers hard drives have been replaced 
    VPS 4   raid  has rebuilt action closed

    XEN 1  Is being rebuilt
    ====================================================================

    Update 10/19/ 2013
    Xen 1 Has  file corruption in some of the VPS's and the raid will not rebuild 
    We will be cleaning  corruption out and rebuilding the array  expect a reboot of individual VPS servers No impact     


  • Date - 10/16/2013 09:00 - 10/25/2013 21:27
  • Last Updated - 10/19/2013 20:12
Data Center Maint: (Resolved)
  • Priority - High
  • Dear Lowest Host/Empire Technology Customers

    In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.

     

    EVENT START DATE: 10/14/2013
    EVENT START TIME: 09:00 CDT

    EVENT END DATE: 10/14/2013
    EVENT END TIME: 13:00 CDT

    EVENT LOCATION:  Cedar Falls NOC

    EXPECTED DOWN TIME:  15 Minutes

    EVENT  DETAILS:
    We apologize for the blanket notification . EGERNCY MAINT: We have detected a Power Fault in one our power Distribution switches in our cedar falls Data center. We will be pulling the defective unit and replacing it with a new unit during the replacement all Servers connected to this unit will be rebooted.

    Thank You, 

    Datacenter Operations Team

    ================================
    No issues   closing 

  • Date - 10/14/2013 06:02 - 10/15/2013 20:01
  • Last Updated - 10/15/2013 20:01
Network Transit Provider MAINT. (Resolved)
  • Priority - Medium
  • Dear Lowest Host/Empire Technology Customers

    In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.

     

    EVENT START DATE: 09/19/2013
    EVENT START TIME: 00:00 CDT

    EVENT END DATE: 09/19/2013
    EVENT END TIME: 06:00 CDT

    EVENT LOCATION:  Level 3  connection

    EVENT  DETAILS:
    Lowest Host/Empire Technology LLC has been informed by a Level 3 maintenance on a KC metro circuit.  This circuit is one of a pair of redundant circuits — customer impact is expected to be limited to brief  connectivity and latency between KC  and Cedar Falls.

    Thank You, 

    Datacenter Operations Team

    ============================

    No issues closing 

  • Date - 09/17/2013 23:00 - 09/18/2013 09:56
  • Last Updated - 09/18/2013 09:56
Network Maintenance (Resolved)
  • Priority - Critical
  • Affecting Other - Nework wide Routing Maintenance:
  • Dear Lowest Host/Empire Technology Customers

    In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.

     

    Network Maintenance Scheduled for all network links (excluding Cedar Falls)
    Maintenance window: Sunday  13 May 2013 00:01 AM   to Friday May 17  06:00 AM Pacific Time
    Maintenance Window Duration: 5 Days
    Maintenance Network routing modifications 
    Expected Down time: none                                                                                                                                                                                                                                                                                                                                  additional Notes: this will only affect the 198.102.29.XX net block  

     

    Network Engineers will be performing a network link swap as we will be turning down a old circuit (NC –NOC-1) and bringing up a new redundant (LV-NOC-1 & 2) to allow for  network expansion. Some of you may experience some light Latency for 6-48 hours until the routes converge (this is a conservative estimate it will likely be a matter of hours or minutes depending on your IP range ). 

    For XEN server customers only during this time we are going take the time during cut over we are going to retire some of our older hardware nodes and upgrade you to new Hardware. (Live migration).

     

     Thank You Network Operations Team

  • Date - 05/13/2013 00:01 - 06/28/2013 13:52
  • Last Updated - 05/11/2013 13:07
Transit Provider infrastructure maintenance (Resolved)
  • Priority - Medium
  • Dear Lowest Host/Empire Technology Customers

    In order to serve you better, Empire Hosting is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.

    Maintenance Scheduled for  North Carolina, Data centers
    Maintenance window: Saturday 6 April 2013 12:00 AM   to 0600 AM eastern  Time
    Maintenance Window Duration: 6 hours
    Maintenance Transit provider Maintenance
    Site Affected: North Carolina NOC
    Expected Down time   none

    Risk Classification: Med 
    Empire Hosting Master Ticket #: 550451




    From time to time Lowest Host Empire Technology LLC's  transit provider partners must perform maintenance on their infrastructure.  Charter Communications has notified us of a service impacting maintenance occurring on 4/6/2012 during the hours of 12:00 AM EDT to 6:00 AM EDT. 

     

    During the maintenance activities the following components will be impacted:

    • Charter Communications 10Gbps Transport Infrastructure –  in our North Carolina Facility       
      • During the maintenance window we will mark down Internet traffic to / from our Atlanta, GA peering points. we  will still have multiple 10G peers Via our other carriers Level 3, telia. Tw telcoem & our direct peer to Las Vegas    during this maintenance window.  Our customers may see brief periods of increased latency and packet loss associated with the shifting of Internet traffic.

     

     

    If you have any questions or concerns, please contact  Support

    Sincerely,

    Lowest Host/Empire Technology LLC  Support

     

  • Date - 04/06/2013 12:56 - 05/11/2013 12:14
  • Last Updated - 04/05/2013 09:15
Emergency hard Drive Maint (Resolved)
  • Priority - High
  • Affecting Server - rc1
  • Dear Lowest Host/Empire Technology Customers

    In order to serve you better, Empire Hosting is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.

    Maintenance Scheduled for  North Carolina, s Nevada Data centers
    Maintenance window: Saturday 9 Mar 2013 10:00 AM   to 10:30 AM Pacific Time
    Maintenance Window Duration: 30 Min
    Maintenance Hard Drive optimization
    Site Affected: RC1 Shared hosting server
    Expected Down time   10 15 Minutes

    Risk Classification: high  
    Empire Hosting Master Ticket #: 550450

    March 9, 2013 Lowest Host/Empire Technology Technicians notices that our RC1 shard hosting server was exhibiting slow drive performance issues.  We are attempted to clean up any corruption the drives may have.

     

     Thank You Server Operations Team

    ---------------------------------------
    Update 13 March 2013

    we have upgraded the server to New hardware and a new raid 10 Disk array
    we are closing this issue at this time 


  • Date - 03/09/2013 08:59 - 03/13/2013 09:49
  • Last Updated - 03/13/2013 09:49
XEN OS & security patch (Resolved)
  • Priority - High
  • Affecting Other - XEN Nodes 1-10 17,18
  • Dear Lowest Host/Empire Technology Customers

    In order to serve you better, Empire Hosting is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.

    Maintenance Scheduled for Asheville North Carolina, Cedar Falls Iowa Data, Las Vegas Nevada Data centers
    Maintenance window: Saturday 23 Feb 2013 00:01 AM   to 06:00 AM Pacific Time
    Maintenance Window Duration: 6 hour
    Maintenance Xen Server OS Upgrade & security patch  
    Site Affected: XEN  Nodes 1-10   17,18
    Expected Down time per Node  10 15 Minutes with the exception of XEN  5 which will require additional software installation 30-50 min (estimates)  

    Risk Classification: high  
    Empire Hosting Master Ticket #: 550449

    Lowest Host/Empire Technology LLC Data center operations will be performing OS Maint on  XEN  Nodes   1-10   17,18.  We will be performing upgrade between 00:01 AM and 06:00 AM Pacific on Saturday 23 Feb 2013 is will be a complete shutdown of all affected nodes the

     

     Thank You Server Operations Team

    ----------------------------------------


    All Updates Complete NO issues 






  • Date - 02/23/2013 00:01 - 03/09/2013 08:59
  • Last Updated - 02/23/2013 04:58
Routine server Maintenance (Resolved)
  • Priority - High
  • EVENT START DATE: 11-11-2012

    EVENT START TIME:11:00 PM PST

    EVENT END DATE:  11-11-2012

    EVENT END TIME: : 02:00 PM PST

    EVENT IMPACT: Routine server Maintenance

    EVENT DETAILS: 

    Server Maintenance Notice:

    We will be performing server maintenance : on our (XEN 2 & XEN 6 ) ,VPS servers.
    Reason: VPS NODE security & maintenance patch.
    Expected Down time: 15 Min
    Maintenance Window: 11:00 AM - 02:00 PM PST (07:00 PM - 10:00 PM GMT)

    =========================================
    Maintenance complete No issues



     

  • Date - 11/11/2012 11:00 - 11/11/2012 14:24
  • Last Updated - 11/11/2012 14:24
Routine server Maintenance (Resolved)
  • Priority - High
  • Affecting Server - rc1
  • EVENT START DATE: 11-11-2012

    EVENT START TIME:11:00 PM PST

    EVENT END DATE:  11-11-2012

    EVENT END TIME: : 02:00 PM PST

    EVENT IMPACT: Routine server Maintenance

    EVENT DETAILS:

    Server Maintenance Notice:
    We will be performing server maintenance : on our (RC1) ,shared hosting server.
    Reason: server security & maintenance patch.
    Expected Down time: 10Min
    Maintenance Window: 11:00 AM - 02:00 PM PST (07:00 PM - 10:00 PM GMT)

    ==================
    Maintenance complete No issues



     

  • Date - 11/11/2012 11:00 - 11/11/2012 14:24
  • Last Updated - 11/11/2012 14:22
Network Outage (Resolved)
  • Priority - Critical
  • EVENT START DATE: 07-20-2012

    EVENT START TIME: 02:15 PM PDT

    EVENT END DATE:  7-20-2012

    EVENT END TIME: : 04:10 PM PDT

    EVENT IMPACT: Connectivity stability in the ALO-North Carolina data center impacting multiple customers.

    EVENT DETAILS: At 0215  PDT 07-20-2012  NOC Techs observed connectivity issues impacting the North Carolina  data center. Network Services is working to identify the source of the issue and a resolution. Updates will be provided regularly throughout this event and all efforts are being made to minimize the impact of this issue. Our apologies for the inconvenience.


    Update  7-20-2012 04:19   connectivity has been restored

    network  Engineers have performed the line card swap and have Cisco support engaged. Disruptions in Internet connectivity are continuing to occur, however our engineers hope to restore connectivity and stability, shortly. Thank you very much for your patience and understanding.


    Update  7-20-2012 5:50 PM

    The Internet Connectivity issues our North Carolina data center was experiencing early today have subsided. Connectivity has been up and stable and we do not anticipate any further disruptions to service this evening. Our engineers are currently working with Cisco support in order to determine a root cause.

  • Date - 07/20/2012 02:15 - 07/22/2012 08:36
  • Last Updated - 07/20/2012 17:52
Ibound DDOS Flood (Resolved)
  • Priority - Critical
  • Affecting Other - Ibound DDOS Flood
  • EVENT START DATE: 07-12-2012

    EVENT START TIME: 11:05 PDT

    EVENT END DATE: 07-12-2012

    EVENT END TIME: : 11:50 PDT

    EVENT DETAILS: At 11:05 PDT 07-12-2012 one of our clients was under a direct inbound DDOS Attack saturating our network equipment

    The net work engineers we able to identify the attack and mitigate the attack quickly

  • Date - 07/13/2012 03:05 - 07/13/2012 03:06
  • Last Updated - 07/13/2012 03:06
Failing Cisco Switch replacement (Resolved)
  • Priority - Low
  • Affecting Other - Network infrastructure
  • Dear Lowest host Empire Technology LLC Customers,

    In order to serve you better, lowest host Empire Technology LLC is listing all upgrades and changes to our datacenters nationwide. Maintenance in other lowest host Empire Technology LLC Data Centers does not necessarily mean that your facility will be affected. All pertinent information can be found in the description for each maintenance action to be performed.

    Emergency Maintenance Scheduled for NOC (North Carolina )
    Maintenance Window: 10:00 AM to 01:00 PM PDT GMT -7 WednesdayJuly 11, 2012
    Maintenance Window Duration: 3 hour
    Maintenance Action: Cisco switch replacement
    Site(s) Affected: North Carolina NOC
    Risk Classification: High
    Lowest host Empire Technology LLC Master Ticket #: 60001

    Lowest host Empire Technology LLC operations will be performing maintenance on one of our Cisco switches in NOC (North Carolina ) this WednesdayJuly 11, 2012. During this window, engineers will Be performing a replacement of a failing Cisco switch customers experience 10 15 Minutes of down time while perform this Maintenance. We will perform this work between 10:00 AM to 01:00 PM PDT Time.

    Update July 11, 2012 11:50 PDT

    Switch replacement is complete no issues

    will down grade this to low as we monitor the switch performance

  • Date - 07/11/2012 10:00 - 07/13/2012 03:06
  • Last Updated - 07/11/2012 11:51
Node reboot (Resolved)
  • Priority - High
  • EVENT START DATE: 05-20-2012

    EVENT START TIME: 07:25 PDT

    EVENT END DATE: 07:30 PDT

    EVENT END TIME: : 05-20-2012

    EVENT DETAILS: At 06:38:25 PDT 05-20-2012 Client reported that his VPS was unresponsive and would not reboot

    the only way to get his server restarted was to reboot the node very little impact 5 minute outage there will be a minor shutdown while quotas are recalculated

  • Date - 05/20/2012 07:25 - 05/22/2012 06:32
  • Last Updated - 05/20/2012 07:53
Short Notice Power SYS MAINT: (Resolved)
  • Priority - High
  • Affecting System - Anual Power & UPS Sys Maint.
  • Dear Lowest host Empire Technology LLC Customers,

    In order to serve you better, lowest host Empire Technology LLC is listing all upgrades and changes to our datacenters nationwide. Maintenance in other lowest host Empire Technology LLC Data Centers does not necessarily mean that your facility will be affected. All pertinent information can be found in the description for each maintenance action to be performed.

    Maintenance Scheduled for NOC (North Carolina )
    Maintenance Window: 00:01 AM to 01:00 PM MDT GMT -6 Saturday March 24, 2012
    Maintenance Window Duration: 1 hour
    Maintenance Action: Annual power/UPS Maintenance
    Site(s) Affected: North Carolina NOC
    Risk Classification: High
    Lowest host Empire Technology LLC Master Ticket #: 5877215

    Lowest host Empire Technology LLC operations will be performing maintenance on our Power infrastructure in NOC (North Carolina ) this Friday, Saturday March 24, 2012. During this window, engineers will Be performing Annul Scheduled Maintenance on our Power circuit and UPS . We Will be power Cycling all affected servers during this maintenance window, customers may experience 10 15 Minutes of down time while perform this Maintenance. We will perform this work between 00:01 AM to 01:00 PM PDT Time.


    Thank You,

    Update 01:26 AM 24 March 2012

    No issues closing event

  • Date - 03/24/2012 00:01 - 03/24/2012 00:24
  • Last Updated - 03/24/2012 00:24
Moderate Packet Loss North Carolina Data cent (Resolved)
  • Priority - Low
  • Affecting Other - NOC/NC
  • EVENT START DATE: 02-25-2012

    EVENT START TIME: 10:52 MST

    EVENT END DATE: 02-25-2012

    EVENT END TIME: :

    EVENT IMPACT: Connectivity stability in the ALO-North Carolina data center impacting multiple customers.

    EVENT DETAILS: At 10:52 MST 02-25-2012 NOC Techs observed connectivity issues impacting the North Carolina data center. Network Services is working to identify the source of the issue and a resolution. Updates will be provided regularly throughout this event and all efforts are being made to minimize the impact of this issue. Our apologies for the inconvenience.

    UPATE : 02-25-2012 12:39 MST

    At approximately 10:52 MST on 2/25/2012 North Carolina experienced an
    unexpected event on one of its edge routers (not our equipment). At this time the Network
    Engineering team is working to resolve the issue. We will give you further
    updates as they become available.

    The Packet Loss has been reduced to Low at this time and all services are up

    UPATE : 02-25-2012 01:47 MST

    This network event has been resolved we are downgrading this event to LOW as a packet distribution has returned to normal.
    We will continue to monitor until 100% resolution

  • Date - 02/25/2012 21:49 - 02/26/2012 10:55
  • Last Updated - 02/26/2012 10:55
Server: XEN 2 OS/Hardware Upgrade (Resolved)
  • Priority - Medium
  • Dear Lowest Host/Empire Technology Customers

    In order to serve you better, Empire Hosting is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.

    Maintenance Scheduled for North Carolina Data Center 20 Dec 2011
    Maintenance Time:
    Tuesday 20 Dec 2011 00:90 AM to Wednesday December 27 09:00 AM Pacific Time
    Maintenance Window Duration: 1 week
    Maintenance Xen Server OS Upgrade & Hardware Upgrade
    Site Affected: XEN Node 2
    Expected Down time per VPS 5 Minutes
    Risk Classification: Medium
    Empire Hosting Master Ticket #: 550445

    Lowest Host/Empire Technology LLC Data center operations will be performing Hardware/OS Maint on XEN Node 2. We will be performing upgrade between 09:00 AM and 09:00 AM Pacific the week of 20 Dec . This will be a complete shutdown of Xen 2 while we upgrade the OS & Hardware.


    During this upgrade we will be Live Migrating All VPS servers to A different Xen Node so there will be 0 Downtime with the exception of bringing up the live instance on the new node "5 Minutes"

    -------------------------

    Update 24 Dec 2011 12:54

    All upgrades complete No issue

    Thank You Server Operations Team

  • Date - 12/19/2011 18:59 - 12/27/2011 00:00
  • Last Updated - 12/24/2011 12:54
Notice Of Network/Power Maintenance: (Resolved)
  • Priority - Critical
  • Affecting System - Power Distribution
  • Dear Lowest host Empire Technology LLC Customers,

    In order to serve you better, lowest host Empire Technology LLC is listing all upgrades and changes to our datacenters nationwide. Maintenance in other lowest host Empire Technology LLC Data Centers does not necessarily mean that your facility will be affected. All pertinent information can be found in the description for each maintenance action to be performed.

    Maintenance Scheduled for NOC (Cedar Falls )
    Maintenance Window: 04:00 PM to 11:59 PM PDT GMT -7 Friday Dec 23, 2011
    Maintenance Window Duration: 1 hour
    Maintenance Action: Network upgrades
    Site(s) Affected: Cedar Falls NOC
    Risk Classification: High
    Lowest host Empire Technology LLC Master Ticket #: 5877215

    Lowest host Empire Technology LLC operations will be performing maintenance on network infrastructure in NOC (Cedar Falls) this Friday, Dec 23, 2011. During this window, engineers will install Replace A Failed Power Strip in We Will be power Cycling the affected servers that & Pluging them into the new switch approximately during this maintenance window, customers may experience 10 15 Minutes of down time while your server is cut over to the new switch. We will perform this work between 04:00 PM to 11:59 PM PDT Time.


    Thank You,

    -------------------------

    Update 23 Dec 2011

    Power distribution switch Replaced No issues

  • Date - 12/23/2011 03:49 - 12/24/2011 00:00
  • Last Updated - 12/24/2011 12:52
Network Upgrades (Resolved)
  • Priority - Medium
  • Affecting Other - network infrastructure
  • Dear Lowest host Empire Technology LLC Customers,

    In order to serve you better, lowest host Empire Technology LLC is listing all upgrades and changes to our datacenters nationwide. Maintenance in other lowest host Empire Technology LLC Data Centers does not necessarily mean that your facility will be affected. All pertinent information can be found in the description for each maintenance action to be performed.

    Maintenance Scheduled for AVL01 (North Carolina )
    Maintenance Time: 01:00 AM to 02:00 AM Eastern Time (05:00 to 06:00 UTC/GMT) Sunday April 24, 2011
    Maintenance Window Duration: 1 hour
    Maintenance Action: Network upgrades
    Site(s) Affected: North Carolina NOC
    Risk Classification: Medium
    Lowest host Empire Technology LLC Master Ticket #: 573995

    Lowest host Empire Technology LLC operations will be performing maintenance on network infrastructure in North Carolina (AVL01) this coming Sunday, April 24, 2011. During this window, engineers will install an additional 48-port gigabit module in order to increase capacity to support future network growth. Additionally, they will be power-cycling another 48-port gigabit module that is presently in a minor error state. For approximately 3 minutes during this maintenance window, customers may experience momentary periods of packet loss or latency. We will perform this work between 01:00 AM and 02:00 AM Eastern.


    Thank You,

    Lowest host Empire Technology LLC
    NOC Support


    Risk Classifications:

    Risk classifications are based upon the probability of your experiencing a disruption in service.

    (Low) The probability of a disruption in service is minimal. The maintenance being performed involves basic infrastructure systems and should not affect normal operations. Although the risk of a service interruption is minimal, Lowest Host/Empire technology LLC requires this work be performed during the maintenance window as a matter of policy.

    (Medium) There is a possibility of a disruption in service. The maintenance being performed involves mission critical infrastructure systems. While redundant systems are the hallmark of our data center, we wanted to advise you in advance in the event the maintenance does not go as planned. We have taken all prudent precautions to eliminate the likelihood of a service interruption.

    (High) A service disruption should be anticipated. The maintenance being performed affects mission critical infrastructure systems. Should a disruption in service occur, Lowest Host/Empire technology LLC will move rapidly to limit this interruption to the absolute minimum amount of time? This classification is required for maintenance where the risk of a service outage is anticipated at being greater than 80%.

    ------

    no issure maint complete closing

  • Date - 04/24/2011 00:00 - 05/07/2011 14:40
  • Last Updated - 05/09/2011 18:15
Xen OS Maint Patch (Resolved)
  • Priority - Medium
  • Affecting Other - OS Security Patch & Software update (Xen Servers)
  • Dear Lowest Host/Empire Technology Customers

    In order to serve you better, Empire Hosting is listing all upgrades and changes to our datacenters nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.

    Maintenance Scheduled for North Carolina & Cedar Falls Data Centers
    Maintenance Time: 00:00 AM Tuesday to 0300 AM Pacific TimeSunday
    Maintenance Window Duration: 20 Min
    Maintenance Xen Server Maint patch
    Site Affected: Xen servers All (except Xen1)
    Risk Classification: Medium
    Empire Hosting Master Ticket #: 550598

    Lowest Host/Empire Technology LLC Data center operations will be performing OS Security Patch & Software update. We will be performing upgrade between 00:00 AM and 02:00 AM Pacific. Although, this is a normal maintenance patch we have classified this as high on the Xen 2 node only in North Carolina. As it has been reading an xen error sins 10:57 Pacific time this morning and we may elect to do this patch prior to Sunday morning to clear the error. Thank You,

    Sever Maint Operations Team

    ----------------------

    Upgrades complete no issues

    Closing

  • Date - 05/07/2011 14:40 - 05/09/2011 18:14
  • Last Updated - 05/09/2011 18:14
Cedar Falls Mild Latency (Resolved)
  • Priority - Critical
  • At approximately 10:00am PDT on 03-MAR-2011 Empire Technology began experiencing latency from our upstream providers in both (Cedar Falls). The issue is still under investigation with Internap and Level 3, our primary transit providers .

    Empire Technology Network Engineering is taking additional measures to re-route our traffic to resolve any continued latency from specific providers. If you are experiencing continued slow connections or an inability to reach your server hosted with Empire Technology in these locations, please check this posting for updates on our corrective measures.

    We appreciate your patience while we address this situation.

    update 10:29 AM PST

    Traffic has been rerouted

    Closing

  • Date - 03/03/2011 10:14 - 03/03/2011 10:31
  • Last Updated - 03/03/2011 10:31
Network upgrades North Carolina NOC (Resolved)
  • Priority - High
  • Dear Lowest Host/Empire Technology Customers

    In order to serve you better, Empire Hosting is listing all upgrades and changes to our datacenters nationwide. This does not necessarily mean that your facility will be affected. All pertinent information can be found in the description for each maintenance action to be performed.

    Maintenance Scheduled for North Carolina Data Center
    Maintenance Time: 12:30 PM Tuesday to 12:300 PM Eastern Time Thursday
    Maintenance Window Duration: 3 Days
    Maintenance Action: Network upgrades
    Site Affected: AVL01
    Risk Classification: High
    Empire Hosting Master Ticket #: 550547

    Empire Hosting s Network Technicians will be performing Maintenance On Our #1 forward public Switch Customers may experience packet loss or latency for a few minutes on each end of the window as engineers perform this maintenance. We will perform this work usual you may track all Maintenance activities at https://secure.empire-hosting.net/networkissues.php

    If you have any questions, or would like to determine the extent of impact any maintenance may have on your network, please contact Open a support request @

    https://secure.empire-hosting.net/submitticket.php


    Thank You,

    Datacenter Operations Team

  • Date - 03/01/2011 10:32 - 03/16/2011 20:43
  • Last Updated - 03/01/2011 09:26
Server Outage Server Outage Columbia shared (Resolved)
  • Priority - Low
  • Affecting Server - Columbia
  • Server Outage Wed Jan 5 2011 8:37 PDT

    We are having a Issue with our Columbia shared hosting server it is frozen and will not boot we are currently investigating the issue

    Update Wed Jan 5 2011 9:20 PDT
    We have diagnosed it to a bad RAM bank we have and corrected the issue
    this will remain downgraded to low while we continue to monitor it through the night

  • Date - 01/05/2011 20:39 - 01/06/2011 00:00
  • Last Updated - 01/08/2011 08:47
Network Outage North Carolina NOC (Resolved)
  • Priority - Medium
  • Network Outage Wednesday Dec 15 14:15 MST There has been a major outage feeding the North Carolina Data center. We are investigating at this time

    -----------------------------------------------------------------------------------------------------------------------------------------------

    Update 15 Dec 2010 15:25 MST we have got a quick answerer from the network engineers and they have isolated the issue and the network should be coming back on line. Please note we are still in the dark as to the actual issue as this is outside of our network when we have more information available we will post it.

    Update 15 Dec 2010 4:00 MST The network is back up We are we are leaving this network issue in a critical state as we still have no information on why the incident occurred in the first place.

    -----------------------------------------------------------------------------------------------------------------------------------------------

    Update 15 Dec 2010 18:00 MST forwarded Message from The data center Engineers

    As you are aware, we experienced a major network issue this afternoon that started at around 15:45 Eastern with a few BGP flaps followed by a full-on outage starting at 15:55. While service has been restored, we're not even close to closing out this issue. I am meeting with our network infrastructure team members right now to fully understand what transpired so I can provide you with the full details. We have a case open with Cisco TAC as well to bring additional insight into the events that transpired to cause an outage of this magnitude. Please look for an update from me within the next few hours.

    ------------------------------------------------------------------------------------------------------------------------------------------------

    Update 16 Dec 2010 11:36 MST

    With No updates as promised from our upsteam will are closing this issue Network was fully restored yesterday

    -------------------------------------------------------------------------------------------------------------------------------------------------

    Update 18 Dec 2010 0:841 MST

    Issue reopened

    ------------------------------------------------------------------------------------------------------------------------------------------------

    Update 18 Dec 2010 08:53 MST

    forwarded from our upstream:

    Dear Lowest Host Empire Technology LLC ,

    Here is an additional update for Friday, December 17th. In a few hours, our change management team will be sending out an emergency maintenance notification for this coming Sunday, December 19th beginning at 1:00 AM Eastern time. Working with Cisco TAC, our engineers believe we have narrowed down the cause of Wednesday’s outage to a known bug affecting the Cisco 6708 module when deployed in 7600 series routers. The 6708 module has 8 x 10 Gigabit ports that we use to cross uplink our edge to our core. All of the 10G connections on one particular module began to cycle up/down sporadically 10 minutes prior to the outage. While this up/down activity did not specifically cause any connectivity issues due to the redundancies in place, we believe that it was the ultimate trigger to the BGP failure between our edge and core which did cause the outage.

    We will be using this maintenance window to update to a later IOS version on all devices, add additional 10G modules to distribute the risk over multiple modules, and incorporate use of the 10G ports on the RSPs for additional redundancy using something other than a 6708 module – the only type we have used to date for connecting the edge to the core. As an aside, we use the ES20 module for connectivity from the core to our backbone nodes in Atlanta, GA and Ashburn, VA because it supports VPLS. The ES20 module is not affected by this bug.

    As always, we greatly appreciate your patience as we dissected the incident and worked through developing a concrete action plan. Many of you have emailed me personally with suggestions, and I do appreciate it. Keep them coming. I know we have a very long way to go to regain your trust, but that is my personal goal, and everything I will be focusing on between now and January 1st will revolve around that.

    I will write you on Monday after the maintenance has been completed to provide you with additional information.

    Sincerely,

    Jonathan Hoppe
    Chief Technology Officer

    --------------------------------------------------------------------------------------------------------------------------------------------------

    Update 18 Dec 2010 0900 MST

    In order to serve you better, Loesr Host/Empire Technology LLC is listing all upgrades and changes to our datacenters nationwide. Maintenance in other Data Center does not necessarily mean that your IDC facility will be affected. All pertinent information can be found in the description for each maintenance action to be performed.

    Maintenance Scheduled for North Carolina Data Center #2
    Maintenance Time: 01:00 AM to 05:00 AM Eastern Time (06:00 to 10:00 UTC/GMT) Sunday December 19, 2010
    Maintenance Window Duration: 4 hours
    Maintenance Action: Network upgrades
    Site Affected: AVL01
    Risk Classification: Moderate
    Master Ticket #: 555892

    Network operations will be performing IOS upgrades and routing changes to edge and core infrastructure this coming Sunday, December 19th in our AVL01 datacenter. During this window, engineers will reroute and fail over traffic to minimize Customer impact, however customers may experience periods of packet loss, latency or very brief service interruptions as engineers perform these upgrades. There will NOT be an extended outage during this maintenance window. We will perform this work between 01:00 AM and 05:00 AM Eastern.

    Thank You,

    Lowest Host /Empie Technolgy LLC Support


    Risk Classifications:

    Risk classifications are based upon the probability of your experiencing a disruption in service.


    (Medium) There is a possibility of a disruption in service. The maintenance being performed involves mission critical infrastructure systems. While redundant systems are the hallmark of our data center, we wanted to advise you in advance in the event the maintenance does not go as planned. We have taken all prudent precautions to eliminate the likelihood of a service interruption.

    -------------------------------------------------------------------------------------------------------------------------------------------------

    Update 19 Dec 03:42 MST

    all upgrades complete no issues

    closing this incident

  • Date - 12/15/2010 14:01 - 12/21/2010 04:23
  • Last Updated - 12/19/2010 02:43
Maintenance Scheduled for North Carolina (Resolved)
  • Priority - Medium
  • Affecting Other - infrastructure upgrades
  • Dear Lowest Host/Empire Technology Customers

    In order to serve you better, Empire Hosting is listing all upgrades and changes to our datacenters nationwide. This does not necessarily mean that your facility will be affected. All pertinent information can be found in the description for each maintenance action to be performed.

    Maintenance Scheduled for North Carolina Data Center
    Maintenance Time: 02:00 AM to 05:00 AM Eastern Time (07:00 to 10:00 UTC/GMT) Sunday November 21, 2010
    Maintenance Window Duration: 3 hours
    Maintenance Action: Network upgrades
    Site Affected: ATL01, AVL01
    Risk Classification: Medium
    Empire Hosting Master Ticket #: 550546

    Empire Hosting s Upstream Transit providers will be performing various infrastructure upgrades in order to increase the number of 10G links between Atlanta (ATL01) and Asheville (AVL01). During this window, engineers may reroute traffic from Asheville to Ashburn in order to minimize Customer impact. Customers may experience periods of packet loss or latency as engineers perform these upgrades. We will perform this work between 02:00 AM and 05:00 AM Eastern. Per usual you may track all Maintenance activities at https://secure.empire-hosting.net/networkissues.php

    If you have any questions, or would like to determine the extent of impact any maintenance may have on your network, please contact Open a support request @

    https://secure.empire-hosting.net/submitticket.php


    Thank You,

    Datacenter Operations Team



    Risk Classifications:

    Risk classifications are based upon the probability of your experiencing a disruption in service.

    (Medium) There is a possibility of a disruption in service. The maintenance being performed involves mission critical infrastructure systems. While redundant systems are the hallmark of our data center, we wanted to advise you in advance in the event the maintenance does not go as planned. We have taken all prudent precautions to eliminate the likelihood of a service interruption.
    Mission critical infrastructure systems. While redundant systems are the hallmark of our data center, we

    ============================

    Update Maint: concluded No impact

  • Date - 11/21/2010 00:00 - 11/21/2010 06:18
  • Last Updated - 11/21/2010 06:18
Maintenance Scheduled for North Carolina (Resolved)
  • Priority - Low
  • Affecting Other - semi-annual maintenance on our UPS
  • Dear Lowest Host/Empire Technology Customers

    In order to serve you better, Empire Hosting is listing all upgrades and changes to our datacenters this does not necessarily mean that your facility will be affected. All pertinent information can be found in the description for each maintenance action to be performed.

    Maintenance Scheduled for our North Carolina Data Center
    Maintenance Time: 10:00 AM to 3:00 PM Eastern Time (15:00 - 18:00 UTC/GMT) Saturday November 20, 2010
    Maintenance Window Duration: 5 hours
    Maintenance Action: Semi-Annual UPS Preventative Maintenance for Electrical Buss 1
    Site Affected: North Carolina NOC
    Risk Classification: Low
    Empire Hosting Master Ticket #: NA

    Empire Hastingss third party service provider will be performing semi-annual maintenance on our UPS systems affecting electrical buss 1 this coming Saturday, November 20, 2010 from 10:00 AM to 3:00 PM Eastern Time. The work being performed is routine and involves testing each UPS, checking components, battery voltages etc. The datacenter will be placed on stable generator power during bypass testing to ensure no interruption to the critical load. As an additional precaution, members of our electrical and facility operations teams will be onsite during this maintenance window to supervise the process.

    If you have any questions, or would like to determine the extent of impact any maintenance may have on your network, please contact Open a support request @

    https://secure.empire-hosting.net/submitticket.php

    Thank You,

    Datacenter Operations Team



    Risk Classifications:

    Risk classifications are based upon the probability of your experiencing a disruption in service.

    (Low) The probability of a disruption in service is minimal. The maintenance being performed involves basic infrastructure systems and should not affect normal operations. Although the risk of a service interruption is minimal, Empire Hosting requires this work be performed during the maintenance window as a matter of policy.

    =================================

    Update Maint copleate No issues

  • Date - 11/20/2010 00:00 - 11/20/2010 00:00
  • Last Updated - 11/20/2010 20:56
Xen Servers Maint Patch (Resolved)
  • Priority - High
  • Affecting Other - Maintenance Scheduled for Both Cedar Falls IA & Asheville NC data centers
  • Maintenance Scheduled for Both Cedar Falls IA & Asheville NC data centers
    Maintenance Time: 11:00 PM to 01:00 AM (PDT, GMT - 7:00) Sunday Oct 31, 2010
    Maintenance Window Duration: 20 Minutes
    Maintenance Action: OS Security Patch & Software upgrade
    Site Affected: Xen Servers ALL
    Risk Classification: High
    Lowest Host/Empire Technology LLC Master Ticket #: 9428

    Lowest Host/Empire Technology LLC Data center operations will be performing OS Security Patch & Software update. We will be performing upgrade between 01:00 AM and 02:00 AM Pacific. Although, this is a normal maintenance patch we have classified this as high all XEN Nodes have been on line for 900 + days with 0 down time (only brief outages to apply similar maintenance Patches) and the Hard Drives may go into a forced maintenance cycle. Should this happen the network status page will be upgraded to reflect the extended maintenance. (aprox 45 min 3 hours)

    If you have any questions, or would like to determine the extent of impact any maintenance may have on your service, please contact us at https://secure.empire-hosting.net/submitticket.php and reference the specific case number listed above.

    Thank You,

    Datacenter Operations Team

    Risk classifications

    (High) Risk classifications are based upon the probability of your experiencing a disruption in service longer than the posted time.

    -----------------------------------------------------------

    Update: Oct 31 2010 5:19 PM PDT

    updated all Centos 5 Images minimal, webmin, cPanel, cPanel DNS, DirectAdmin,

    -------------------------------------------------------------

    Update: Nov 1 2010 1:58 AM PDT

    All updates complete No issues

    closing

  • Date - 10/31/2010 09:51 - 11/01/2010 01:00
  • Last Updated - 11/01/2010 01:00
Maintenance Scheduled for Cedar Falls NOC (Resolved)
  • Priority - Low
  • Affecting System - Virtuozzo Server VPS 4 OS Security Patch & Software upgrade
  • Dear Lowest Host/Empire Technology LLC Customers

    Maintenance Scheduled for Cedar Falls NOC
    Maintenance Time: 01:00 AM to 02:00 AM (PDT, GMT - 7:00) Tues Oct 5, 2010
    Maintenance Window Duration: 20 Minutes
    Maintenance Action: OS Security Patch & Software upgrade
    Site Affected: Virtuozzo Server VPS 4
    Risk Classification: High
    Lowest Host/Empire Technology LLC Master Ticket #: 9428

    Lowest Host/Empire Technology LLC Data center operations will be performing OS Security Patch & Software update. We will be performing upgrade between 01:00 AM and 02:00 AM Pacific. Although, this is a normal maintenance patch we have classified this as high the VPS 4 NODE has been on line for virtually 937 days with 0 down time (only brief outages to apply similar maintenance Patches) and the Hard Drives may go into a forced maintenance cycle. Should this happen the network status page will be upgraded to reflect the extended maintenance. (aprox 45 min 3 hours)

    If you have any questions, or would like to determine the extent of impact any maintenance may have on your service, please contact us at https://secure.empire-hosting.net/submitticket.php and reference the specific case number listed above.

    Thank You,

    Datacenter Operations Team

    Risk classifications

    (High) Risk classifications are based upon the probability of your experiencing a disruption in service longer than the posted time.

    ============================

    Update: 5 Oct 2010

    After installing the Patch one of the Hard drives has faild it has since been replaced

    and the server is performing Normal Drive Miant

    =================================

    Update 5 Oct 2010 03:54 PDT

    The raid array is currently being rebuilt & Quotas are being repaird on client VPS's. We wil be downgrding this Maint action to Low as we do not foresee having any further issues we will leave this maint incident open until the raid array is rebuilt 100%

  • Date - 10/05/2010 00:00 - 10/27/2010 06:56
  • Last Updated - 10/05/2010 16:00
North Carlina NOC (Resolved)
  • Priority - Low
  • Affecting Other - Netwark Outage
  • Network Outage Sunday October 3 21:30 PST

    There has been a major outage feeding the North Carolina Data center. The Main Trunk Feed from the main NAP we do not have many details at this time
    Washington DC or Atlanta Portions of the Atlanta NAP are down as well. The issue is being investigated.

    I would only be speculating at the time if it was a fiber cut or main switch failure the Network engineers are currently rerouting traffic away from the main feed into the data center we will update your further when I have more information.

    =====================================

    Oct 2 2010 23:05

    The traffic has been rerouted the network has been restored

    Downgrading this issue to low

    ======================================

    From the Network engineers

    At 12:09AM EDT

    our Atlanta secondary path went down for scheduled and anticipated maintenance that was slated to be non-customer impacting, however at 12:12AM our primary

    Fiber to Atlanta and Ashburn went down as well, both served by a different provider. At 01:42AM, Ashburn connectivity was restored, and

    At 03:08AM Atlanta secondary/protect connectivity was restored. We are still working with our providers to fully understand the reason for this massive outage, and why our

    Atlanta primary fiber remains offline. As soon as we have further information, we will update you

    ===============================================================

    As of 9:46AM

    our Atlanta primary fiber is also back online completing full service restoration

    ===============================================================

    All Traffic routes have converged there should be no further issues we are closing this issue at this time

  • Date - 10/03/2010 00:00 - 10/04/2010 10:48
  • Last Updated - 10/04/2010 10:48
Maintenance Notification North Carilina NOC (Resolved)
  • Priority - Low
  • Affecting Other - Data Center Maint
  • Dear Lowest Host/Empire Technology LLC Customers

    Maintenance Scheduled for North Carolina NOC
    Maintenance Time: 01:00 AM to 02:00 AM (Eastern, GMT-4) Sunday August 29, 2010
    Maintenance Window Duration: 1 hour
    Maintenance Action: Network Grooming
    Site Affected: North Carolina NOC
    Risk Classification: Low
    Lowest Host/Empire Technology LLC Master Ticket #: 535923

    Lowest Host/Empire Technology LLC network operations will be performing low impact network grooming on our core infrastructure in Noth Carolina to remove configurations no longer in use as a result of previous upgrades. Customers should not experience any issues, however as a precautionary measure we are sending out this notice. We will perform this work between 01:00 AM and 02:00 AM Eastern.

    If you have any questions, or would like to determine the extent of impact any maintenance may have on your network, please contact us at https://secure.empire-hosting.net/submitticket.php and reference the specific case number listed above.

    Thank You,

    Datacenter Operations Team

    Risk Classifications:

    Risk classifications are based upon the probability of your experiencing a disruption in service.

    (Low) The probability of a disruption in service is minimal. The maintenance being performed involves basic infrastructure systems and should not affect normal operations. Although the risk of a service interruption is minimal, Lowest Host/Empire Technology LLC requires this work be performed during the maintenance window as a matter of policy.

  • Date - 08/29/2010 00:00 - 10/03/2010 22:09
  • Last Updated - 08/26/2010 07:41
Emergency server Maint (Resolved)
  • Priority - Low
  • Affecting Server - Columbia
  • Emergency Server maint

    Maintenance Scheduled for Columbia shared hosting server
    Maintenance Time: 15:10 - 15:50  PST 7-23-2010
     Maintenance Window Duration: 1 hour
    Maintenance Action: Hard drive failure & rebuild of raid array
    Site Affected: Columbia shared hosting server
    Risk Classification: Low
    Master Ticket #: NA

    we will be performing a  a Hard drive maintenance  on our Columbia shared hosting server to facilitate  post maint action to facilitate hard drive failure
     
     
    Thank You,

    -----------------------------------------------------------------------------------------


    Update 16:07 PDT  7-23-2010 
    the Maint is completed on schedule and  server was back on line at 3:50 PDT
    we will continue to monitor the server and the the rebuild of the raid array

    -----------------------------------------------------------------------------------

    Update  16:25  PDT 7-23-2010
    we are down grading this incident to low as we expect no further  issues


    -------------------------------------------------------------------------------------



    Risk Classifications:

    Risk classifications are based upon the probability of your experiencing a disruption in service.

    (Low) The probability of a disruption in service is minimal. The maintenance being performed involves basic infrastructure systems and should not affect normal operations. Although the risk of a service interruption is minimal, this requires this work be performed during the maintenance window as a matter of policy.

    (Moderate) There is a possibility of a disruption in service. The maintenance being performed involves mission critical infrastructure systems. While redundant systems are the hallmark of our data center, we wanted to advise you in advance in the event the maintenance does not go as planned. We have taken all prudent precautions to eliminate the likelihood of a service interruption.

    (High) A service disruption should be anticipated. The maintenance being performed affects mission critical infrastructure systems. Should a disruption in service occur, Lowest Host Empire Technology will move rapidly to limit this interruption to the absolute minimum amount of time. This classification is required for maintenance where the risk of a service outage is anticipated at being greater than 80%.

     

     

     

     

  • Date - 07/23/2010 15:33 - 08/26/2010 07:39
  • Last Updated - 07/23/2010 16:29
core routing infrastructure upgrades (Resolved)
  • Priority - High
  • Affecting System - Network Upgrades
  • Dear Lowest Host/Empire Technology Customers,

     

    Maintenance Scheduled for North Craolinia NOC
    Maintenance Time: 00:01 AM (Eastern, GMT-4) Monday July 5, 2010
    Maintenance Window Duration: 5 hours
    Maintenance Action: Network upgrades/maintenance
    Site Affected: NC NOC
    Risk Classification: High
     Master Ticket #: 527718

    Lowest Host Empire Technology network operations will be performing various core routing infrastructure upgrades in order to integrate North Carolina NOC into our new regional backbone. Customers may experience periods of packet loss or latency as engineers perform these upgrades. We will perform this work between 00:01 AM and 05:00 AM Eastern.

     

    Thank You,

    Datacenter Operations Team


    Risk Classifications:

     

    (High) A service disruption should be anticipated. The maintenance being performed affects mission critical infrastructure systems. Should a disruption in service occur, Lowest Host Empire Technology  will move rapidly to limit this interruption to the absolute minimum amount of time. This classification is required for maintenance where the risk of a service outage is anticipated at being greater than 80%.

     

    -----------------------

    update  closing issue   no  outages reported all maint complete

  • Date - 07/04/2010 05:26 - 07/05/2010 07:55
  • Last Updated - 07/05/2010 07:54
Network Incident Report: ALO (Cedar Falls) 27 (Resolved)
  • Priority - Low
  • Affecting Other - Modorate Packet Loss
  • EVENT START DATE: 27-06-2010

    EVENT START TIME: 22:00 CDT

    EVENT END DATE:

    EVENT END TIME: :

    EVENT IMPACT: Connectivity stability in the ALO-Cedar Falls data center impacting multiple customers.

    EVENT DETAILS: At 22:00 CDT 27-06-2010  NOC Techs observed connectivity issues impacting the Cedar Falls data center. Network Services is working to identify the source of the issue and a resolution. Updates will be provided regularly throughout this event and all efforts are being made to minimize the impact of this issue. Our apologies for the inconvenience.

     

    UPDATE: As of 22:40 Network Services is continuing to work to isolate the cause of the connectivity stability issues in Cedar Falls. Steps are being taken to isolate and mitigate the impact to the various segments of the network and to return stability as soon as possible. At this time there continues to be some instability that is manifesting itself for customers as latency/packet loss and loss of connectivity to various hosts. We will continue to update this posting as progress is made, and appreciate your patience while we work to resolve this issue.

     

    Update: As of 23:30 This has been Downgraded to Low

    This has been  identified a DDOS attack to another tenant that resides within our data center which is impacting portions of the Cedar Falls network. Efforts to isolate the source and targets of this attack are underway. Customers should begin seeing an improvement as these efforts continue. We appreciate your patience in this matter as we work to fully resolve the issue. We are downgrading  this issue to low as traffic has returned for the most part to normal and will continue to improve as the attack is being mitigated


    UPDATE: As of approximately 01:00 CDT, our uptream transit provider was able to identify the various sources and targets of a DDOS which was of significant size and scope to have impacted multiple portions of the Cedar Falls network. External sources of the attack and internal targets have been null routed to eliminate the impact to the remainder of our customer base in this facility. Overall network performance and stability has returned to normal as a result of these efforts.

    Again, we appreciate everyone's patience while we worked to identify and resolve this issue. If further updates are necessary they will be posted here.

     

  • Date - 06/27/2010 20:58 - 06/28/2010 05:21
  • Last Updated - 06/28/2010 00:00
Maintenance Action: Network upgrades/maintena (Resolved)
  • Priority - Low
  • Affecting System - Upstram Transit Maint:
  • Maintenance Scheduled for North Carolina NOC
    Maintenance Time: 00:01 AM (Eastern, GMT-4) Sunday June 27, 2010
    Maintenance Window Duration: 4 hours

    Site Affected: North Carolina NOC
    Risk Classification: Low
    Master Ticket #: 526847

    Our Upstream transit providers operations will be performing various low-impact network upgrades in order to optimize routes and increase capacity for future expansion activities. Customers may experience short periods of packet loss or latency as engineers perform these upgrades. We will perform this work between 00:01 AM and 04:00 AM Eastern.

     

    Thank You,

    Datacenter Operations Team
    )


    Risk Classifications:

    Risk classifications are based upon the probability of your experiencing a disruption in service.

    (Low) The probability of a disruption in service is minimal. The maintenance being performed involves basic infrastructure systems and should not affect normal operations. Although the risk of a service interruption is minimal,  requires this work be performed during the maintenance window as a matter of policy.

    Update SUN JUN 27 2010

    All upgrades Complete No issues reported  Closing

  • Date - 06/27/2010 00:00 - 06/27/2010 21:44
  • Last Updated - 06/27/2010 21:38
upstream Transit Provider circut maint: (Resolved)
  • Priority - Low
  • Affecting Other - Scheduled Maintenance
  • Server Maintenance Notice: To all in the Cedar Falls facility: Scheduled Maintenance: ALO01 (Cedar Falls) 05-30-2010 01:00 06:00 EST.

    EVENT START DATE: 05-30-2010

    EVENT START TIME: 01:00 EST (05:00 GMT)

    EVENT END DATE: 5-30-2010

    EVENT END TIME: 07:00 EST (11:00 GMT)

    EVENT IMPACT: ALO01 (Cedar Falls)

    On Sunday 5/30/2010 at 01:00 CST, our upstream Transit providers will be performing scheduled maintenance in the Cedar Falls data center to provide new customer aggregate connectivity. Customers in the Cedar Falls data center will experience brief periods of outage as customer connectivity is aggregated in a phased and controlled migration. Recovery for customers affected during this migration should be rapid and complete with no residual affect following resumption of normal connectivity.

    Update : 05-30-2010 02:25 EST (06:25 GMT)

    Maintenance has been completed on our network segment Network ticket will remain open but downgraded to low. As there may be some residual latency as routes converge.

     

  • Date - 05/30/2010 00:00 - 06/06/2010 15:14
  • Last Updated - 05/30/2010 00:27
Utility Transformer Replacement (Resolved)
  • Priority - Medium
  • Affecting Other - Maintenance Scheduled for Ashville NOC
  • Maintenance Scheduled for AVL01 (North Carolina NOC)
    Maintenance Time: 07:00 AM (Eastern, GMT-4) May 2, 2010
    Maintenance Window Duration: 4 to 6 hours hours
    Maintenance Action: Progress Energy Transformer Replacement
    Site Affected: AVL01 (Morth Carolina NIC)
    Risk Classification: Moderate
    Master Ticket #: 514649

    Progress Energy has informed us that they will need to replace a utility transformer behind AVL01 feeding our facility. No downtime is anticipated and all precautions have been implemented. We will switch to stable generator power prior to and for the duration of their maintenance activities. They will perform this activity tomorrow, Sunday May 2, 2010 starting at 7:00 AM and should complete at or around 12:00 Noon.

     

    Monday May 3 2010 (Update)

    All Maint completed without error No disruption

    Closing incident

  • Date - 05/02/2010 13:29 - 05/03/2010 08:40
  • Last Updated - 05/03/2010 08:40
Multiple IP DDOS flood (Resolved)
  • Priority - Critical
  • Affecting Other - Ashville NOC
  • Maintenance Unscheduled AVL01
    Maintenance Time: 20:50 PDT  
    Maintenance Window Duration: Until Resolved

    Maintenance Action:Latency  Multiple IP DDOS flood
    Site Affected: AVL01  CAB14
    Risk Classification: High
    Master Ticket #: 510304

     

    Currently our Asheville  NC Data Center is under a heavy sustained multiple IP DDOS flood

    Our Network Engineers' on site at the facility are working diligently to to mitigate the attack ant route the traffic either though the firewall or null routing the attacking IP's
    we will update this network ticket as we have moor information
    Datacenter Operations Team
    -------------------------------------------------------------------------------------------------------------------------------------------
     

     

    Identified and the affected server taken off Line. The affected IP block has been Null routed. The Flood traffic is still overwhelming our switch & Firewall  when the attacking Networks' Reroute & converge on the Null routed IP addresses the latency will improve

    Maintenance Unscheduled AVL01 (Update)
    Maintenance Time: 19:42 PDT
    Maintenance Window Duration: Until Resolved

    Maintenance Action:Latency  Multiple IP DDOS flood
    Site Affected: AVL01
    Risk Classification: High
    Master Ticket #: 479240

    The Netwowek engineers have Mitagated the attack though a DDOS Layer switch
    everything is back to nomal   and has been since  04:00 PDT, we will leave this network issue open until we reroute our netwok routes routes
    -------------------------------------------------------------------------------------------------------------------------------------------
     

    Maintenance Unscheduled AVL01 (update)
    Maintenance Time: 06:00  PDT  (12:00 GMT) 12-14-10
    Maintenance Window Duration: Until Resolved

    Maintenance Action:Latency  Multiple IP DDOS flood
    Site Affected: AVL01  CAB14
    Risk Classification: Low
    Master Ticket #: 510304

     Update The Distributed Denial of Service (DDOS) attack that started 20:50 PDT (02:00 GMT) 04-12-10 at Our North Carolina Facility Was under our control AT 04:00 (10:00 GMT ). 04-13-10We remained Under a DDOS Layer Switch (anti DDOS prevention) Until 14:00 PDT 20:00 (GMT) after that we allowed normal traffic to flow.

     During that Time a few clients Had access issues this was the exception not the rule as most did not see any anomalies. And a even fewer reported slowness while pulling data from other networks. This was unavoidable because while under the DDOS layer the inbound traffic must be filtered.We apologize for any inconvenience we had no control over the malicious acts of criminals who seem to have nothing better to do. The alternative would have been total network failure.

    ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

    Maintenance Unscheduled AVL01 (Update)
    Maintenance Time: 06:30  PDT  (12:30 GMT) 12-14-10
    Maintenance Window Duration: Until Resolved

    Maintenance Action:Latency  Multiple IP DDOS flood
    Site Affected: AVL01  CAB14
    Risk Classification: Low
    Master Ticket #: 510304

     Update 40 minutes after restoring traffic to normal the effected IP's of the attack were again under attack these IP's have been Null routed until 100% of the malicious traffic can be stopped. The respective Client(s) have been notified separately from this Ticket

     

    This ticket will remain open but reduced to a low priority as there should be no further impact on any other users in our North Carolina Facility.

  • Date - 04/12/2010 23:47 - 04/25/2010 08:21
  • Last Updated - 04/14/2010 06:46
(CedarFalls) core network upgrades (Resolved)
  • Priority - Medium
  • Affecting System - (CedarFalls) core network upgrades
  • Upstreem  Transit provider Maint Rescsheduled

    EVENT IMPACT: ALO01 (CedarFalls) core network upgrades, expected periods of increased latency during route propagation

    EVENT DETAILS:

    This is a rescheduled event from 21-02-2010**On Sunday, 28-02-2010, one of our transit providers will be performing emergency maintenance to address stability and redundancy issues for a core network link which services the ALO01 (Cedar Falls) datacenter.

    This maintenance is only significant to customers with services hosted out of the Cedar Falls facility and no service disruptions are expected. However, because the maintenance involves core network components this maintenance window is being announced. Customers may notice periods of increased latency during this maintenance window as route updates are applied.

  • Date - 02/18/2010 22:24 - 03/01/2010 05:22
  • Last Updated - 02/26/2010 14:06
Cisco Core Switch/Router Replacement (Resolved)
  • Priority - High
  • Affecting Other - Data Center I Equipment upgrade
  • Maintenance Time: 1 Oct 2009 – 31 December 2010

    Maintenance Window Duration: 1 year 3 months
    Actual Impact Window Duration: No impact
    Maintenance Upgrade of aging Cisco Switches
    Data center Site Affected: Cedar Fall NOC all network segments
    Risk Classification: High
    Master Ticket #: 473911

    Beginning October 1 2009 We will be doing a major overhaul Of our Network Infrastructure, Replacing all of our core switches with new Cisco Switches. This will affect all Customers with the exception of clients on the 208.91.131.XXX & 66.219.30.XX Network.

    Rather than have any Service disruption we have opted do a Live Migration to a a upgraded segment of our network, but unfortunately we can not port the IP's and move the servers to the new Rack/Switch without service disruption and our upstream will not allow us to move the IP's to a different location. So we will be moving affected clients & servers in Phases.

     

    July 1 2009  Phase I: Build Rack Space to accommodate All clients & servers.

    September 20 2009 Phase II: move all empty & non essential servers to new network segment (100% completed)

    October 1 2009 Phase III Move XEN VPS servers & cut to new network segment (100% complete)

    TBA in 2010 Phase IV: Move Virtuozzo servers

    TBA 2009/2010 Phase V: Move all shared & reseller services We will be retiring our current DNS servers

    TBA 2010 Phase VI: move all affected Dedicated servers ( 25% complete)

     

     

     

  • Date - 10/02/2009 15:53 - 02/18/2010 22:19
  • Last Updated - 12/09/2009 21:38
Emergencey Fiber Splice (Resolved)
  • Priority - Critical
  • Affecting Other - Asheville NOC
  • Maintenance Time: Between 01:00 AM and 06:00 AM (U.S. Eastern Daylight, GMT-4) September 15, 2009
    Maintenance Window Duration: 5 Hours
    Actual Impact Window Duration: 30 minutes
    Maintenance Action: Emergency Fiber Splice
    Datacenter Site Affected: Ashville NOC 208.91.XXX.XXX network segment
    Risk Classification: High
    Master Ticket #: 473911

    On Tuesday, September 15, 2009 at 01:00 AM through to 06:00 AM, Lowest Host/Empire Technology LLC fiber provider for Ashville NOC  will be performing an emergency fiber splice to replace a severely damaged section of fiber between Asheville and Atlanta. Customers will experience packet loss, latency, or other connectivity issues when traffic shifts to and from our other circuits.

    While we have taken every precaution to minimize complications during this event, there exists a certain amount of risk and the possibility of extended service disruption.

    We are committed to providing the highest possible level of service; therefore, we strive to increase client satisfaction and productivity through our proactive approach to customer notification. If you have questions or concerns in advance of this event,

    Thank You,

  • Date - 09/15/2009 12:44 - 10/02/2009 15:48
  • Last Updated - 09/14/2009 12:49
Network Distribution Upgrade Asheville NC NOC (Resolved)
  • Priority - Critical
  • Affecting System - Network Upgrades
  • On Sunday, August 2, 2009 at 00:01 AM through to 06:00 AM, 

    Empire Technology will be migrating customer VLANs from our current distribution routers to our new routers in order to take advantage of our new multi-10 Gigabit distribution infrastructure. This upgrade has been implemented in response to rapid growth of our network and in order to natively support 10 Gigabit and Cisco VSS 1440.

    Although this maint is  Marked as critical it will have little impact as the Asheville facility is new  and is posted just for informational purposes only.

  • Date - 08/02/2009 00:00 - 09/13/2009 18:51
  • Last Updated - 08/01/2009 11:09
New Circuit deployment at Cedar Falls NOC (Resolved)
  • Priority - Low
  • Affecting System - NetWork Maint
  • DATE: 8/06/2009
    START TIME: 0000 CDT
    ESTIMATED END TIME: 0400 CDT

    SERVICES/EQUIPMENT: New circuits on production network
    TYPE OF WORK: Circuit deployment
    IMPACT OF WORK: None expected

    On August 6, 2009 starting at 10:00 AM PDT until 2:00 AM PDT, we will be doing general maintenance and bringing new circuits into production to improve service for our customers on several of our core systems. Though none of the work is expected to affect or impair the flow of traffic or cause an outage, we are posting this announcement as a courtesy in the event a problem or an outage would occur. Systems affected with the scheduled maintenance are all internal and external BGP traffic in our Cedar Falls  Data Center.

  • Date - 08/06/2009 00:00 - 09/13/2009 18:51
  • Last Updated - 08/01/2009 10:56