Jack Danielʼs Enlists EOSYS

Old and new often mix like oil and water, but sometimes the right emulsifying influence can smooth their relationship and get them to cooperate like whiskey and water.

Speaking of which, legendary distiller Jack Daniel’s in Lynchburg, Tenn., is well-known for its commitment to traditional manufacturing processes, which includes using water from the same iron-free spring since its founding in 1866, making its own barrels and charcoal, and securing the highest-grade corn, rye and malted barley.

At the same time, many of these tried-and-true practices are supported by more modern industrial controls, networks and other components on Jack Daniel’s six production lines for bottling, capping, labeling, packing and palletizing, which recently needed some revamping and reorganization to prevent downtime and other potential problems. Notably, powering off a main control panel had caused a multiple-line outage, which resulted in about 12 hours of downtime.

As a result, Jack Daniel’s enlisted EOSYS, a CSIA-certified system integrator in Smyrna, Tenn., to conduct a network assessment, develop a secure solution and network documentation, and help prevent any future interruptions.

Scan and assess

“We used SolarWinds’ Network Topology Mapper (NTM) to scan the network, and build the architecture where possible by leveraging Simple Network Management Protocol (SNMP) and Internet Control Message Protocol (ICMP),” explained Larry Grate, Director of Technology at EOSYS, who presented “CpWE helps Jack Daniel’s cut downtime, start The Connected Enterprise journey” this week at the Rockwell Automation TechED 2017 conference in Orlando, Fla.

“To get the most value, SNMP strings were recorded from all managed switches,” Grate continued. “When we scanned the production network and pulled wires, we found a lot of Ethernet that had just grown and grown, but wasn’t well organized. We also researched peer-to-peer messaging to determine the risk of failure with the existing network topology.”

EOSYS also discovered that the network overseeing Jack Daniel’s six production lines had:

• More than eight different manufacturers of managed and unmanaged switches
• Unmanaged switches that required physically walking down cabling or unplugging parts of the network and Internet protocol (IP) scanning using NTM or other tools
• Existing IP address schemes that wouldn’t scale well
• Switches that weren’t consistently configured for proper spanning-tree or Internet group management protocol (IGMP) management
• A majority of unmanaged switches

Multiple places where loss of power or maintenance could cause cascading line outage.

“This existing network architecture made maintenance difficult,” added Grate.” Operators couldn’t power-off unscheduled equipment as it was in the critical network path for running other equipment. Device failures caused multiple-line outages. Troubleshooting network issues was difficult due to a lack of consistency and switch-management functions.

“In addition, the network had been expanded with a convenient or lowest capital expenditure cost path, which increased operating costs. Existing network infrastructure wouldn’t support expansion or desired enterprise connections. Programming changes on off-shifts weren’t always well-documented, causing production losses on later shifts. Finally, it was that one specific event causing more than 12 hours of downtime that was really the last straw.”

CPwE to the rescue

To revamp and reorganize Jack Daniel’s network, EOSYS recommended that the distillery adopt Converged Plantwide Ethernet (CpWE) best practices. CpWE is a set of standards developed by Rockwell Automation and Cisco that network designers can use to securely connect operations technology (OT) networks to information technology (IT) and enterprise networks, and help users advance their journeys to The Connected Enterprise.

To follow CpWE’s best practices in constructing its new network, Grate reported that Jack Daniel’s would need:

• Scalable IP address scheme, separate VLAN for I/O, HMI, and management traffic
• Redundant communications such that no single point of failure on the network should result in loss of communication to the pallet handling area
• A new fiber backbone
• Fully managed switches employing Cisco IOS operating system
• Time synchronization via NTP
• Migration of PLC-5s for pallet handling to ControlLogix control systems to allow use of device level ring (DLR) topology
• Switches capable of network address translation (NAT) to allow migration of individual lines without requiring site-wide IP readdressing
• FactoryTalk AssetCentre deployment for file, configuration management and disaster recovery
• FactoryTalk View SE with Stratix aspect-oriented programming (AOP) implementation
• Configured FactoryTalk View ME for line-level alarming and monitoring at each PanelView graphic terminal main control panel (MCP).

“We installed 20-port Stratix 5700 switches for each MCP that have 100-MB fiber uplinks to the core and 1-GB interfaces to access switches online,” said Grate. “We also designed for security with multiple VLANs, including separately defined CIP and management VLANs; separately defined I/O and HMI VLANs on each line; dynamic host control protocol (DHCP) via the HMI VLAN for local network access on each line; and administratively disabled unused ports.”

Grate added that EOSYS’s network was approved with several significant changes, including:

• Use of Stratix 5400 at each MCP to allow for gigabit access and uplink
• The addition of Stratix wireless access points (WAP) for use with Apple iPads as clients
• Use of Rockwell Automation Intelligent Motor Control with its Connected Components Workbench software for VFD backup
• Additional drives that resulted in more required asset license.

FAT, deployment and benefits

Grate added that EOSYS’s factory acceptance test (FAT) at Jack Daniel’s consisted of thoroughly testing all of the new networks to make sure they’d perform reliably now and in the future.

Grate added that EOSYS installed Jack Daniel’s new network during its usual weekend closures. “We installed the new core and virtual host server in a new, locked network room,” he added.  “We upgraded pallet handling from PLC-5 to ControlLogix controllers, and installed the DLR with NAT to allow use of new IP addresses.

EOSYS upgraded additional lines the following weekend, and “then allowed operation for two weeks to gain confidence. In addition, we deployed FactoryTalk View SE for line monitoring and alarming, and used Stratix faceplates and situational awareness strategies. We also scheduled upgrades of the remaining lines on down weekends, and deployed FactoryTalk AssetCentre.”

As a result of all these network improvements, Grate reported that the distillery gained:

• Line separation, so powering down one line does not affect others
• Documentation to understand how things are connected for future upgrades
• Standardized IP address assignments and network infrastructure for when working with OEMs or corporate engineering
• Easier training, with only one switch to learn (instead of eight manufacturers as in the past)
• Better speed for troubleshooting, update rates of data, as well as not getting knocked off line
• No more random communication faults to I/O devices, which had led to line downtime
• Redundant structure preventing outages in the event of hardware failure
• FactoryTalk View SE and ME faceplates that improved diagnostic speed to repair with faster location of failed devices
• Robustness of the network for additional functions, such as OEE, historical data and remote access
• Ability to easily troubleshoot from both local ports and the Electrical & Instrumentation office
• One location in FactoryTalk AssetCentre for all PLC, HMI, VFD and switch configuration files

“This really increased everyone’s peace of mind because we’re no longer concerned if we have current VFD parameters or PLC programs,” added Grate. “Jack Daniel’s also has reporting to track changes to assets on the production floor.”

In the future, EOSYS and Jack Daniel’s plan to:

• Deploy an industrial demilitarized zone (iDMZ) between the core Stratix switches and the enterprise
• Work with the distillery’s corporate IT group to allow secure remote access to the iDMZ
• Configure jump servers in the iDMZ to control access to the process network
• Deploy engineering servers on the process network for remote management
• Deploy an automated OEE solution to improve maintenance and operations

​Read even more news happening at the Rockwell Automation TechEd 2017 event here: http://bit.ly/2syB7lY