On November 27th, 2012 the Village experienced a water main break on the transmission line coming from the elevated tank to the distribution system. The location of this break was located in Spring Rock Park, just east of the main parking lot, approximately 300 feet north of 47th street and the break occurred sometime between 2:00 am and 4:00 am.
Police dispatch received the first calls from residents without water at approximately 4:15 am. The Public Works on-call personnel were dispatched to the areas of the calls to investigate for a main break. After initial investigations found no signs of water near the reporting areas, crews began to work their way back towards the elevated tank at which point they found the break in the park at approximately 4:30 am. Water plant personnel and the Director of Municipal Services were notified shortly thereafter.
By the time the break was found much of the water in the tank had drained. Because of the remote location of the break and the hour at which it occurred there were no reports to Police dispatch until residents noticed low water pressure in their homes. The water pressure measured in the tank and system was around 24 psi. A boil order is not required unless water pressure drops below 20 psi, but to be cautious a CodeRED notification for a boil order was issued in the community at 6:33 am. The IEPA was also notified that morning per legal requirements.
The water main break in and of itself in this location is not particularly unusual and there have been three prior main breaks on this line in the past twenty years. Unfortunately, the alarm system in place to notify plant operators and the police dispatch of a drop in pressure in the elevated tank failed.
Two water sample sets were taken during the repair. The first sample set was taken prior to the refilling of the tank and the second sample set was taken after the repair had been completed. Sampling sites were located at the various schools throughout the community, Village Hall and Rec Center. Both sample sets came back clean 24 hours later indicating that there was no contamination and the boil order was lifted at approximately 2:00 pm on November 28th.
After the event an investigation was opened to determine the cause of the failed alarm for the water plant. The current alarm panel at the water plant is a basic ADT alarm panel, similar to one that would be found in a residential home. That panel should send a notification to the police dispatch if triggered. The ADT panel is connected to the existing alarm instrument panel in the water plant. Under normal circumstances if the ADT panel itself fails it would trigger a “trouble” signal to dispatch to be investigated.
The issue in this particular instance does not appear to have been the ADT panel itself. The existing alarm panel in the water plant dates from the 1950’s and the current connection from that equipment to the ADT panel is a custom solution since the two pieces of equipment are not designed to interact with each other. It appears that the communication between the water plant alarm panel and the ADT panel failed. The ADT panel was unfortunately unable to pick up this failed connection and was unable to issue a trouble alarm.
The Village is unsure how long the connection between the two pieces of equipment may have been off.
From our investigation staff does not believe that the failure or the break itself was related to the water plant construction. Work records show that there was no work in the area related to the alarm panels and the new alarms connected to the new SCADA system have no related or connecting equipment. At the time of the break the plant was not in operation and no construction was ongoing. Additionally, the break was located on the output or gravity side of the water tank. Any work at the plant that would have affected distribution would have more likely affected the line between the water plant and the tank, not the line exiting the elevated tank.
The ADT connection has since been repaired, but will also be replaced by a more comprehensive alarm through the new SCADA system being installed in the coming weeks as part of the R.O. construction.
Our post incident review of the CodeRED notification system also indicated very good success. Approximately 82% of all of the numbers in the CodeRED system were reached during the event. National average for reverse 911 systems is statistically between 72-76% for connection rates. Looking at our reports however, we did notice that several lines in town received an “Operator Intercept” error message. This message indicated that AT&T’s trunk lines were overloaded by the mass call out and rejected some calls. The Village has since done an “All Call” load test on the system and been in contact with AT&T to ensure the call numbers from CodeRED receive priority on their system.