SCADA Communications over Frame Relay Introducing the Broadcast Polling FRAD
Introduction to the BPF
DCB has developed a FRAD (Frame Relay Access Devices) for SCADA applications, the Broadcast Polling Frad (BPF). The DCB BPF operates point to point and point to multipoint. It can be used point to point in Frame Relay or non-Frame Relay networks. The BPF can be used over public (Phone Company) Frame Relay networks and over private Frame Relay networks. Some corporate IP routed networks include routers that have a Frame Relay encapsulation function, which also supports the BPF. The BPF provides multiple channels over a single communications link, which allows SCADA systems to operate at higher serial speeds and provide for transporting asynchronous SCADA protocols over Frame Relay.
The BPF is simple to set up and operate. The BPF relieves the SCADA host and RTUs from the burden of adding Frame Relay protocol. The BPF performs the task of transporting the asynchronous polling data over frame relay, and does it without requiring the user to do anything more than map the BPF port or ports to the DLCIs (logical circuit identifiers) that correspond to each remote site. The BPF is easy to set up using an asynchronous terminal or a PC terminal emulator such as Hyperterm. Commands are non-cryptic English, and the management port includes comprehensive help screens.
SCADA and Frame Relay
In the SCADA environment today, communications networks for SCADA systems are being forced off the traditional analog networks and SCADA networks are being forced to use higher speeds. The telephone companies are changing their networks from analog to all digital, and in the process the cost of digital facilities is falling below the price of analog networks. The typical digital network operates at higher speeds than the older analog networks. In addition to networks that are higher speed, the SCADA host computers and Remote Terminal Units (RTUs) are capable of higher speed operation. With more processor power, more computer memory, and newer SCADA polling protocols, more information is sent to the RTU and collected from the RTU, increasing the need for higher speed SCADA network operation. Frame Relay provides one of the network options for this higher speed operation.
Point to Point Private Frame Relay
The BPF can be used on a point to point circuit that is not frame relay, or can be used on a point to point circuit configured for frame relay protocol. On a circuit that is not frame relay, the BPF management is set to none. This is done via the BPF "FR" command. A DLCI must be set at each end of the circuit and the DLCI numbers must be the same, usually 16, the first legal DLCI number. The most frequent use of point to point operation on a non-frame relay link is for bench testing with back to back 56/64 kbps DSUs. A second use of the back to back links over non-frame relay lines is to combine two to four SCADA host ports over a single digital line through a four-port BPF.
The BPF will operate over non-frame relay circuits using point-to-point mode, as well as over frame relay circuits.
Point to Multipoint
The most common application of the BPF is to transport asynchronous multipoint polling over a frame relay network. This might involve moving a circuit from analog facilities to frame relay to reduce monthly costs, to increase speeds, or both, or this could be a new SCADA circuit. It is quite simple to set up a BPF for frame relay operation. At the host end, an async terminal or PC terminal emulator is connected to the management port. A Configure Map (CM) command is used to list the DLCI numbers associated with the BPF port. A BPF can have from 1 to 4 ports, and each port can have up to 40 DLCI numbers mapped to it. The frame relay management, LMI or Annex D does not need to be set since the default mode of the BPF will automatically detect whether the management is LMI or Annex D.
If there is only a single DLCI at the remote end, the only thing that needs to be set in the BPF is the speed. If the port speed is 9600 bps, even that does not need to be set, as 9600 bps is the default speed. The remote, or slave, BPF automatically detects LMI or Annex D management and automatically sets itself to the single DLCI that is assigned to the remote location. If there is just a single DLCI set, the BPF reads and uses that DLCI. When there is more than one DLCI set at the remote site the BPF needs to be mapped through the management port
Multiple host polling ports through a four-port BPF
Many SCADA systems have more than a single line of multipoint polling in operation. When there are dozens or more remote devices, the polling is usually split up so there are a dozen or fewer RTUs on a multipoint line. The BPF is available in a four-port version. Each of the four ports can be mapped to up to 40 DLCIs. Each port will be mapped to a different set of DLCIs, where the DCLI grouping on each BPF port represents a different multiport group. The following is an example, where there are four ports on the BPF, four RTUs per BPF port, and the addressing of the host computer is assumed to be 01, 02, 03 and 04:
BPF Port
Polling Addresses
DLCIs
Port 1
01,02,04,04
16,17,18,19
Port 2
01,02,03,04
20,21,22,23
Port 3
01,02,03,04
24,25,26,27
Port 4
01,02,03,04
28,29,30,31
The DLCIs are entered in the BPF through the management port using the Configure Map (CM) command. With this command, the BPF prompts for the port number and the DLCIs to be mapped to that port.
How the polling works
The BPF at the host has its asynchronous ports attached to the host SCADA computer ports. The BPF has timers to insure it takes in an entire poll from the SCADA host, without fragmenting the poll, and places the polling data into frame relay protocol packets. The frame relay packets are sent to the remote RTUs per the DLCI numbers assigned to the host BPF. Only one RTU will have an address to match the address polled by the host SCADA computer, and only that RTU responds to the poll. The remote BPF, just like the host BPF, has timers to insure it places an entire response from the RTU (up to 1024 characters per frame relay packet) into a single frame relay packet.
High Availability/Backup links
By using other ports on the BPF, external A/B switches, a meshed frame relay addressing scheme, or a combination of these, it is possible to set up a SCADA polling system with high availability backup links.
For point to point circuits, a backup host computer and a backup RTU can be setup on port 2 of the BPF. Port 1 functions as the main port. If the host or RTU fails, the polling system is moved from port 1 to port 2. Port 1 would be mapped for DLCI 16 at each end, port 2 mapped to DLCI 17 at each end.
For a point to point SCADA link with a backup host at a different location, each host location will have a unique DLCI. The common remote RTU location will have two DLCIs mapped to it. For example, the main host and the backup host will both have DLCI 16 assigned. The RTU will have two DLCIs, 16 and 17. Responses from the RTU will always be sent to both DLCI 16, the main host, and to DLCI 17, the backup host. The backup host will be activated only in the event the main host fails. While the main host is in operation, responses sent back to the backup host location are just ignored, as the backup host is not polling the RTU.
For point to multipoint with backup, one option is to have a backup host location, and have two RTUs at each remote site. The RTUs will be connected to four-port BPF units, port one connected to the primary RTU, port 2 connected to the backup RTU. Each host can be connected to a one port BPF (unless there are more than one multidrop lines in operation off the host SCADA computer). The mapping might look like that shown in the table below, illustrating a four-drop system with a backup host:
Backup Host
Primary Host
DLCI Assignments at each remote BPF
Port 1, DLCI 16,17,18,19
DLCI 16,17,18,19
Port 1, DCLI 16=primary host
Port 2, DCLI 17=backup host
In the above example, the two hosts can even be polling at the same time, since the hosts are at separate locations and the remote RTUs are on separate ports of the BPF. If the dual RTUs were both connected to a single port BPF, then the remote BPF port will have two DLCIs mapped and the RTU must be connected to the BPF using a sharing unit, such as the DCB MSU-4D.
Frame Relay Propagation Delay
DCB has been installing frame relay equipment for many years and often tests frame relay networks for propagation delay. The propagation delay times have been improving year by year as the switching equipment gets faster and the link speeds within the frame relay networks increase. Today the round trip propagation delay from a host to remote and back has fallen below 50 ms, sometimes better than the propagation delays of modems. In short, regarding propagation delay, frame relay networks will perform as well or better than modem linked systems.
What is Frame Relay
Frame relay is a synchronous HDLC protocol based network. Data is sent in HDLC packets, referred to as "frames". The diagram below of an HDLC frame may be familiar, since without adding specific definitions of how the Address, Control and CRC is used, the diagram is applicable to IBMs SDLC, to X.25, to HDLC, to Frame Relay, as well as other protocols.
Address
Control
Data
CRC Error Correction
The protocol is similar to that of an X.25 network, except all circuits are permanently assigned. X.25 circuits can be initiated and ended from the users' terminals. Frame relay relies on the customer equipment to perform end to end error correction. Each switch inside a frame relay network relays the data (frame) to the next switch. X.25, in contrast, performs error correction from switch to switch. The networks of today are sufficiently error free to move the burden of error correction to the end points. Most modern protocols do error correction anyway, protocols such as SDLC, HDLC, TCP/IP, stat mux protocols, etc.
Network Characteristic
Frame Relay
X.25
Propagation Delay
Low
High
Error Correction
None, done by the terminal equipment at each end of the link
Node to Node
Protocol family
HDLC
HDLC
Good for async data?
Yes
Barely acceptable, rather slow, with 1 second or more round trip delay
Good for polling protocols?
OK, sometimes requires "spoofing"
Slow with spoofing
Good for Lan file transfer?
Yes
Slow
Good for voice?
Standards under development
No
Ease of implementation
Easy
Difficult
Because of the way frame relay just passes blocks of data from switch to switch, propagation from customer end to customer end through the network is very fast. Propagation time in the DCB mux test at Wiltel (now MCI Worldcom) indicated a 70 millisecond round trip delay from Tulsa, Oklahoma to New York City and back. This is equal to or less than the propagation time through 9600 bps modems over the same distance. An X.25 network would experience a delay of a least one half second, and probably a second or more for the same distance.
Topology
Topology is the map, or visual layout of the frame relay network. A frame relay network topology is best viewed from several perspectives to appreciate fully understand the network and the equipment used to construct the network. Topological views include an overview "cloud" map, a logical connection map, perhaps a functional map, a map showing the detail equipment and channel links, an address map.
Overview "Cloud" Map of a Frame Relay Network
The map below is an overview where the frame relay switching is in the cloud in the center. The lines going out from the cloud represent the connections from the frame relay provider (phone company) to the customer premises. These are typically lines ranging in speed from 56000 bps to T-1 (1.544 Mbps) and faster. One or more DLCI numbers is assigned to each line end point. DLCI is discussed in more detail below. Over 900 DLCIs can be assigned, but frame relay networks rarely have that many, because a user does not need that many in most cases. There may also be limits due to constraints in the frame relay switching equipment. Also, a network that is too large can be a problem to maintain.
The frame relay network that is shown above as a cloud can also be shown as a connection of end points tied to a central site, or a number of end points that are all meshed together.
The Frame Relay Address Map
The end points to a central site is the traditional host to terminal model. The map below has 6 end points, just as in the previous map. Below, the end points are given meaning by defining one end as the host, the other 5 as the remotes. DLCI addressing could be assigned as follows:
Location
DLCI #
Host end
16,17,18,19,20
Remote 1
16
Remote 2
16
Remote 3
16
Remote 4
16
Remote 5
16
At the frame relay switch, all remotes are mapped back to the host. At the host, Host 16 is mapped to Remote 1, Host 17 to Remote 2, Host 18 to Remote 3, Host 19 to Remote 4, and Host 20 to Remote 6. In practice, data from the host for Remote 5 arrives at the frame relay switch, from the customer equipment, with the destination address of 20. The frame relay switch, using its internal lookup table, finds that the data is for Remote 5, a physical port on the frame relay switch, that has a local address of 16. In the frame relay switch, address 20 is replaced with address 16 and the data is then delivered to Remote 5. The packet received at Remote 5 has an address of 16 after the frame relay switch changed the address.
The frame relay network above can be shown as a traditional multidrop host to remote, with lines bridged in the middle, as shown below:
Frame relay also allows meshed networks. Since each endpoint can have one or more DLCI addresses, each user location can be connected to one other location, some locations, or all locations. If every location were connected to every other location, the network would be said to be "fully meshed". An illustration with all locations interconnected is below, next page.
Meshed network:
The fully meshed network mapping of DLCIs looks like this:
Location
DLCI #
Location 1
16,17,18,19,20
Location 2
16,17,18,19,20
Location 3
16,17,18,19,20
Location 4
16,17,18,19,20
Location 5
16,17,18,19,20
Location 6
16,17,18,19,20
This addressing can also be shown as a mapping table showing the addresses from the perspective of each one of the 6 locations. A frame relay switch requires this type of cross reference map. The switch uses the map to route the frame relay packets from one internal port of the switch to another port. The following table shows the local port number and the DLCI destination mapping that would be typical for a meshed network.
Local Port
DLCI Destination for the locations remote from the local port in column 1
Location 1
16 = Port 2
17 = Port 3<
18 = Port 4
19 = Port 5
20 = Port 6
Location 2
16 = Port 1
17 = Port 3
18 = Port 4
19 = Port 5
20 = Port 6
Location 3
16 = Port 1
17 = Port 2
18 = Port 4
19 = Port 5<
20 = Port 6
Location 4
16 = Port 1
17 = Port 2
18 = Port 3<
19 = Port 5
20 = Port 6
Location 5
16 = Port 1
17 = Port 2
18 = Port 3<
19 = Port 4
20 = Port 6
Location 6
16 = Port 1
17 = Port 2
18 = Port 3
19 = Port 4
20 = Port 5
Bandwidth Requirements
The BPF does not have the same bandwidth requirements that apply to LAN connections over frame relay. Most frame relay applications support LAN connections. In a LAN environment, there is typically a relatively high volume of traffic from remote offices back to headquarters locations. In these LAN environments, when there are several remote sites coming back to a central site, the bandwidth at the central site is in excess of the speeds at the remote locations. For example, if in the LAN environment there are 4 to 6 remote sites at 64 kbps, the central site may require 128 kbps.
With the BPF supporting polling SCADA systems, the bandwidth requirements are much less. For example, assume a 4-port BPF at the host, each port running 9600 bps and supporting 8 drops, for a total of 32 drops. The polling is half duplex, where the data is either an outbound poll or an inbound response, but not both at the same time on any single port. At most, there are only 4 remote sites responding simultaneously, combining for an aggregate data rate of 38400 bps (9600 x 4). Therefor, the host and all 32 remote sites have more than enough bandwidth if the local drops are only 56 or 64 kbps.
FRAD
Frame Relay Assembler/Disassembler. A DCB communications controller (BPF) that operates with frame relay protocol is a FRAD. When frame relay was first introduced most users and vendors thought of a FRAD only as a device for encapsulating LAN protocols for transmission through a frame relay network. This will change as more applications are moved onto frame relay networks.
PVC
Permanent Virtual Circuits. This contrasts with Switched Virtual Circuits (SVC). A connection to a network that allows connection and disconnection to various points is a switched virtual circuit. Circuits that are routed through software switching devices like X.25 pads and frame relay networks are virtual. A hard wired connection is a plain old circuit, not a virtual circuit.
DLC
Data Link Connections
DLCI
Data Link Connection Identifier. This number identifies a virtual circuit. For BPF applications, a head end DCB Broadcast Polling Frame may have 4 DLCIs associated with it, while each of 4 remotes would have just a single DLCI. The following illustrates how DLCIs might be assigned in this example:
Host Location
Host DLCI#
Remote DLCI#
Remote Location
Chicago<
16
16
Milwaukee
Chicago
17
16
Minneapolis
Chicago
18
16
St. Louis
Chicago
19
16
Detroit
In the above example, the Chicago host location might be a single 56 Kbps line into a DCB BPF. At each remote location there is a BPF. The DLCI numbers are not necessarily assigned in any order by the frame relay carrier, although it is commonly done.
The above example illustrates that DCLI numbers have significance at the users end point only. Chicago has 4 addresses, 16, 17, 18 and 19. At each remote, each location has the same number, 16. The telephone company providing the frame relay service has a frame relay switch that translates the addresses. When these permanent virtual circuits are established, the relationship of the physical ports is mapped and then assigned DLCI numbers.
The DCB BPF, when carrying SCADA information, does not have the same data intensive requirements. SCADA systems are polled systems, using the communications one way at a time, polling or getting a response. And the polls and the responses are usually short blocks when compared to the IP protocol used to connect LANs. Therefore, the BPF requires less bandwidth, and a 56 or 64 kbps line is usually sufficient for polling a dozen or more drops over a single frame relay circuit.
Special DLCI numbers
DLCI 0 (zero) and 1023 are reserved for management
DLCI 1 to 15 and 1008 to 1022 have been reserved for future use.
DLCI 992 to 1007 are reserved for layer 2 management of frame relay bearer service.
DLCI numbers 16 to 991 are available for subscribers for each user frame relay network
Management Protocols
Frame relay management protocols for exchanging information between a user FRAD and the frame relay network equipment. The management protocols are:
LMI - The original interim management protocol, uses DLCI 1023. LMI was specified by the Frame Relay Forum.
Annex D - An ANSI T1.617 management protocol standard, uses DLCI 1. Annex D was specified by the ANSI T1.617 specification.
Annex A - Nearly identical to Annex D, used in Europe.
The Status Inquiry (Keep Alive)
This is an inquiry the frame relay user must make to the network every 10 seconds. The time value can be 5 to 30 seconds. The default value is 10 seconds. If the user does not make this inquiry every 10 seconds, the carrier network equipment will generate an alarm. The network equipment default value is 15 seconds. That is, the user should make an inquiry every 10 seconds, and the network expects the request every 15 seconds. The 5 second difference is a buffer time that allows for a small delay. The frame relay network switch responds to the Status Inquiry with an acknowledgement.
Sending the inquiry tells the network that the user FRAD is in service. When a status inquiry is not sent, the network will generate an alarm. An alarm may result in a "pro active" call to the customer from the service vendor. The DCB frame relay products have a counter that counts the number of Status Inquiries sent to the frame relay network switch and how many were acknowledged by the network switch. This counter, which rolls over back to zero at 65536, provides a good indication of the quality of the circuit from the user location to the frame relay switch.
The Full Status Inquiry
The Full Status Inquiry gets more information from the frame relay network switch than is obtained in the Status Inquiry. Full status responses to the user from the network also return all of the active DLCI numbers. The frequency of sending the full status can range from 1, meaning every Inquiry is a Full Status Inquiry, to 255, where only 1 out of every 255 Inquiries is a Full Status Inquiry. The default value is 6, which means the user equipment sends 5 regular status inquiries, then the full status inquiry. The DCB BPF counts the number of Full Status Inquiries sent, the number received, and displays the DLCI numbers returned in the response to the Full Status Inquiry.
It is interesting to check the DLCI channel assignments reported by the frame relay vendors switch. In a many as a quarter of the frame relay customer networks DCB has worked on, the Full Status Inquiry response has returned a DLCI number that is not engineered into the system. Apparently, some DLCI number was set earlier or in error on a frame relay switch port and the DLCI was inadvertently left in operation. This extra DLCI does not usually cause any problems, but is confusing. If an extra DCLI is discovered on a frame relay circuit, the vendor should be notified so it can be removed.
Link Integrity Verification Timer
Indicates how frequently the FRAD should initiate a Status Inquiry Message. This is the 10-second timer referred to under the term "Status Inquiry".
FECN (Forward Explicit Congestion Notification)
This is a bit set in the frame relay protocol frame telling the customer recipient equipment that circuit congestion may exist in the direction from which the data was received.
BECN (Backwards Explicit Congestion Notification)
This is a bit set in the frame relay protocol frame telling the customer equipment getting the notification that circuit congestion may exist in the direction it (customer) is sending data.
CIR (Committed Information Rate)
The CIR is the bit per second rate that the network (Phone Company) agrees to carry over the circuit under normal conditions. Frame Relay networks will handle bursts above the CIR. The CIR is not the same as the access rate. The access rate is the speed of the link to the phone company network, usually a DSU speed like 56, 64, 128, 256 kbps, etc.
MIB
Management Information Base, a piece of information collected for an SNMP manager.
Agent
A device that collects MIB data to report them to SNMP managers. HPs Open View is an example of a manager.
SNMP
Simple Network Management Protocol. Standard management protocol used with above managers and agents.
Data Comm for
Business Inc.
2949 County Road 1000 E
Dewey, Il 61840
