Three Emergency Support Options for Industrial Control Systems

In this day and age, control systems are known for their reliability. A control system that gains a reputation for quality issues or failures will not survive in today’s demanding market. However, even the best control systems do occasionally experience events that lead to unplanned downtime. Often the cause is a factor that is external to the control system, such as a lightening strike or power outage. Other times the root cause is not within the control system at all but something connected to it, such as pressure or temperature transmitters, servos, etc. 

Unfortunately, the source of the problem and subsequent solution can be very difficult to determine in the moment, when there is a large amount of information to analyze and extremely high pressure to return to normal production. Unplanned downtime in industrial production and manufacturing environments is estimated to cost an average of $260,000 per hour. Additionally, unplanned maintenance is estimated to cost between 2 and 5 times the cost of planned maintenance. The total impact of such an outage can vary greatly depending on the nature of the operation but no matter how you look at it, unplanned downtime is usually a costly occurrence that most businesses would prefer to avoid.  

Because unplanned downtime of a control system is a relatively rare occurrence, in-house operation and maintenance personnel often lack experience in getting the system back online from these events. Therefore, when an unexpected control system outage does occur, it is critical to have access to the right experts to get you back up and running as quickly and efficiently as possible. Typically, this means having some level of agreement with a control service provider. Their support teams will have the most experience with the causes and solutions for a wide array of failure scenarios. 

With many providers offering different types and tiers of support, it can be challenging to determine the best option to meet your unique needs. If you purchase too much or too little support, you could end up paying more than needed or being underprepared to cope with any downtime situations. In today’s highly competitive markets, most companies cannot afford either situation. The remainder of this article provides guidance on the 3 main types of support available, the value of each type and what you should expect to pay. 

Every control system provider typically offers some form of technical phone support. This allows a customer to make a phone call or send an email to initiate a technical support case. These teams have experience across a wide range of problems and solutions, which makes them a great option to get questions answered or ask for advice. These teams tend to be the main interface for their company though, so they are often dealing with a larger volume of issues than the other types of support teams we will discuss. The larger the support provider is, the longer it tends to take for them to respond, especially to smaller customers. Working with this type of team to solve a problem is often a process that spans one or more days, requiring iterative requests from the support team for data that must be manually located, collected, and shared by the customer. In an emergent situation this team will often recommend sending a field service engineer to your site immediately, then work with you to gather information and triage the issue until someone arrives to take over. Unfortunately, field service callouts like these are the most expensive type of support due to travel time and expenses.

Technical phone support is often included with the price of your control system during the 1-2 year warranty period. However, this support usually has limits. For instance, the free support may only be offered during standard business hours rather than 24/7. It will also typically be limited only to problems that are specific to the equipment under warranty. This means it often will not extend to asking questions or for advice. You always have the option of purchasing additional coverage during the warranty period, which should cost considerably less than purchasing support after the warranty period. If you bundle this into a multi-year agreement then you should expect to get increasing discounts for each additional year you include in the agreement. 

If you don’t have any support agreement when the warranty period ends, some vendors may still help you for 1-2 hours free of charge, but if you require more assistance then you will need to submit a purchase order, which may delay finding a resolution. You will also pay a premium for this support as you will not get the benefit of any support agreement discounts. With a support agreement, many vendors will allow you to roll unused hours into the next year while some vendors may allow you to use hours on other types of support such as engineering modifications, health and cybersecurity assessments, etc. It is important to understand how your support provider operates before you decide what support to purchase.

The next type of support we will discuss is remote diagnostic support. This support typically includes some level of direct access to customer data that is only viewed periodically by the support provider to help solve a problem with your system. Additionally, some support teams may also use this data to provide a proactive report on the health of your system. The benefit is a significant reduction in the time it takes to determine the source of a problem, and the resulting solution, because the required data is available to support personnel immediately, without the need for iterative manual transactions. As a result, these teams can typically identify the proper solution within hours, down from days with traditional technical support teams. This is ideal for emergency support and will save time and money as compared to sending a field technician to your site. 

It is important to be aware of the ways in which this support can differ from vendor to vendor. In particular, it is critical to understand what data will be accessed, when it will be accessed, and how it will be used. You should always have a conversation with your service provider to understand the cybersecurity measures they put in place around any connection to your site or data. Cybersecurity is always evolving but at the time of this writing it is recommended that such a connection include access controls with two factor authentication, one or more firewalls, a VPN data tunnel with encryption, and complete traceability and logging capabilities. Some vendors may also provide a physical switch that you must place in the “on” position in order to allow access to your data, to ensure you have complete control over the connection to your site.

It is also important to note that some vendors may not set up any access to your data at all, but claim they have remote diagnostics by way of a live video connection. While such a setup certainly offers some benefit beyond traditional phone support, you will not get the full benefits of true remote diagnostic support. As you can see this type of support can vary significantly and it is important to make sure you understand what you are getting before choosing a provider. 

One final type of support that is gaining traction in this age of the fourth industrial revolution (4IR) is data monitoring. This type of support is characterized by continuous access to data in order to improve the ability to find and correct problems, ideally before they occur. Going back to the numbers mentioned at the beginning of this article, with the average cost of downtime estimated at $260,000 per hour and the average cost of unplanned maintenance estimated at 2 to 5 times the cost of planned maintenance, the value of identifying a problem and scheduling a correction before that problem occurs is undeniable. Even if an issue does occur, having instant access to historical data analysis trends across an aggregate of similar operations enables support experts to diagnose and solve your problem as quickly and efficiently as possible.

With data monitoring support it becomes even more important to understand what data is being collected and how it will be used.You will want to ensure that you know who in the support organization will have access to your data and that it will be handled according to best cybersecurity practices. You may want to ensure that you are not agreeing to allow your data to be sold to a third party. In most cases the support provider will want to use your data to improve their products and services as well as reduce costs for you and their other customers. As long as you are able to confirm that they will not be sharing any of your intellectual property then this arrangement can be mutually beneficial to all parties involved. When done right, data monitoring can be a very valuable service option worth your consideration. 

Now that we have covered the 3 main types of support that you would typically consider in the case of an emergency, let us spend some time on pricing. Obviously, prices will vary depending on the vendor and a number of other factors, but the guidance below should give you a general idea of what you can expect to pay for each of these services.

Technical phone support generally has the lowest hourly rate, typically between $200-400 per hour, but will also usually require more hours to solve your issue. Additionally, working with these teams alone will result in more site visits to solve emergency situations, which is the most costly form of support available due to time considerations and travel expenses. The cost of a site visit can range from $2,000-4,000 per day plus travel costs. You should expect to be charged for a minimum of 3 days – 1 day of travel each way plus 1 or more days at site – for a minimum of $6,000-12,000 per site visit. It is also not uncommon for it to take a technician 24-48 hours to get to your site, precious time that most organizations cannot afford to be down. It is therefore prudent that you limit this type of support to issues that cannot be solved without a physical site visit. 

For all other emergency situations, remote diagnostic support will likely be a faster and more cost-effective option.  This type of support is commonly sold as a set number of issues for which you will receive assistance. The price range for a single resolution is typically around $6,000-10,000 but you can expect that to go down as the number of resolutions you purchase per year increases. It is unlikely that you will find this type of service sold via an hourly rate. This is because the value provided must take into account not only the hours worked but also the hours and travel expenses saved. At a rate of $6,000-10,000 per resolution you will likely pay less than you would pay for a site visit and you can expect to save hours or days in identifying a solution to your problem.  

The last type of support is harder to nail down on pricing as it depends what data is being monitored and how it is being monitored. The data being monitored may be limited strictly to control system data or it may extend to the instruments and/or process. The volume of data that will be collected, stored, and analyzed, and therefore the underlying infrastructure required, can differ significantly depending on what you choose to monitor. The data may be monitored automatically by setting alerts to notify the support team when certain situations arise. In this case there can be several different modules available that use different algorithms to look for specific patterns such as anomaly detection or condition monitoring. Alternatively, the data may be monitored manually by a person or some combination of these two methods. Depending on the specific options you select you can expect to pay somewhere in the range of $3,000-9,000 per month. 

In all cases you should expect to pay less if you choose to purchase a service agreement. In the case of traditional technical support this could mean purchasing some number of hours up front or bundling this support with some other services, such as remote diagnostics or data monitoring. With remote diagnostics you should expect to pay less per resolution as the number of resolutions or other services you purchase at one time increases. With both of these types of support, as with data monitoring, you should receive a discount for purchasing multiple years of service at one time.

At the end of the day, the question you should ask to determine what type of support is the best option for you is how much your operation may end up losing if you run into an issue that cannot be solved by your local team. As mentioned in this article, the average cost per hour of downtime is $260,000 but the actual cost is different for every organization. If you have redundant units or your operation does not need to be running continuously, then traditional technical support coupled with site visits as needed may be the right choice for you. However, if you stand to lose tens or hundreds of thousands of dollars per day in the event of unplanned downtime, then the immediate nature of remote diagnostics or the preventative nature of data monitoring is likely worth the investment for your organization. Another way to look at it is to take an example - if it costs your operation an average of $100K per hour for unplanned downtime and you are currently operating at 99.9% uptime, that equates to 0.1% or roughly 9 hours of unplanned downtime per year, which would cost you $900K. If a data monitoring and service plan could cut your unplanned downtime to 4 hours, it would save your operation $500K. This certainly puts the the considerably smaller cost of a typical service package in perspective.