Authors: John W. McCain, PE and Russell Straayer

The most commonly used wide area networking technology for SCADA systems is the venerable leased line and multi-drop modem. The infrastructure required to support this technology is fast becoming excessively costly as telephone carriers migrate to packet switched technology solutions. Current SCADA systems can be migrated to new Frame Relay wide area networks without costly deployment of new RTU and host equipment by using Frame Relay FRADS designed to support legacy SCADA protocols. This paper discusses the economics, practicality, and reliability of commercial Frame Relay networking for SCADA systems. Actual systems and products installed in the power industry are detailed in the examples.

For decades, the multi-drop leased line modem has been the backbone of the distributed SCADA communications world. Whenever a wide area network was required, no other technology could match the cost, reliability, and availability of those ubiquitous analog lines running at 300, then 1200, and now even the blazing speed of 9600 bits per second. For years, there has been little improvement in throughput or reliability. Multi-drop modem technology stalls out at 9600 bps. The cost of analog multi-drop lines has been an order of magnitude lower than point-to-point lines; and, most importantly, they were available in all areas of the country. Whether it's a cross-country pipeline, or sewer lift stations in a small city, this was the technology of choice. The only real alternatives were private, licensed radio or microwave links or point-to-point analog lines The radio networks required the SCADA user to maintain their own expensive (and sometimes fragile) infrastructure, and point-to-point analog lines often were prohibitively expensive.

But times change. Now, there are definite economic incentives for the telephone companies to phase out those old dedicated lines in favor of new "dumb" networks. And, with less regulation, many are doing so at a pace that leaves their SCADA customers looking for a fix and fast. Telcos are starting to un-tariff multi-drop and even point-to-point dedicated analog lines, and are often pricing them at ridiculously high rates. Luckily, technology marches on. The very technology that telcos are using to replace their dedicated interconnect network is the new star performer in SCADA wide area networks. That is the packet-switched fully meshed "dumb" network. This network may consist of Frame Relay or ATM (Asynchronous Transfer Mode) switches and links, and is most often accessed using Frame Relay Access Devices (FRADs). By "dumb network", we mean that unlike older networks with fixed routing hard-wired into the network, the end point information travels through the network with each packet of data, and every packet may use a different path. This Telco network is quite fault tolerant, and being a fully meshed network with little internal intelligence, is the most economical network for the Telco.

Along with the pressure from Telcos, another driving force behind the move to Frame Relay is the increased bandwidth required by more advanced SCADA systems. Faster computers, more data-intensive operations, and deeper integration often require more bandwidth to the RTUs of today's SCADA systems. By using the proper access devices, no software or hardware changes are required in existing RTUs or host controllers when moving a network from multi-drop modems to Frame Relay.

As I pointed out, multi-drop modem networks are quickly becoming obsolete because of several main reasons. Costs are rising due to additional telco expenses required to maintain copper circuits. The systems are simply no longer available from some telcos. The current generation of telco personnel are "digital-oriented" people, more versed in digital technologies including Frame Relay. There are fewer of the old guard analog-trained employees to provide the excellent service support found in the 70's through early 90's. And many modern SCADA systems require more bandwidth than the 9600 bps networks can provide.

There are several technologies that could replace that familiar network and are provided by various common carriers (telcos).

1. Wireless common-carrier networks such as CDPD and public wireless Internet.
2. Telco provided Asynchronous Transfer Mode (ATM) networks.
3. SONET, and other fiber technologies provided by telcos or privately owned.
4. Frame Relay, provided by telcos.
5. Private multi-point or point-to-point spread spectrum or licensed radio networks.
6. 6. XDSL, digital subscriber line in various incarnations.

Each of these has its pros and cons

1. Wireless common-carrier networks such as CDPD and public wireless Internet have not covered the country yet and are often missing in our most critical geographic areas. They tend to cover the cities and Interstates while ignoring the back roads and rural landscape. These also come with a per-packet cost or cost per Kb transferred. Usually available only in IP protocols, but, they are easy to implement where they fit. Unfortunately, bandwidth is severely limited often to 9600 bps or even slower.
2. Telco provided Asynchronous Transfer Mode networks. The access devices are too expensive for RTU applications. ATM networks usually provide much more bandwidth than required for SCADA and at a much higher cost. If a ultra-high bandwidth is needed say 100 Mbps, this might be the way to go. It would require IP type protocols and special access devices.
3. SONET, and other optical fiber-based technologies provided by telcos or privately owned. If telco provided, there is often a requirement for a substantial "contribution in aid to construction" (CIC) due to high installation cost. If privately owned, they require rights-of-way purchase and expensive installation. Regardless of the ownership, fiber technologies require expensive interface equipment although fiber has the capability of providing very high bandwidth. If the telco will provide an interface device at reasonable cost, and your RTUs speak IP over ethernet, this might be worth considering.
4. Private multi-point or point-to-point spread spectrum or licensed radio networks often require expensive initial investments as well as maintenance. The licensed spectrum is becoming scarce and quite valuable, and the spread spectrum (ISM) bands are rapidly becoming congested. The WANs we are discussing cover so much geography that low-power radios require repeaters, line-of sight, and other constraints that prevent radio from being an economical technology.
5. Frame Relay, provided by telcos. The most economical, reliable technology according to our analysis. Simple FRAD interface devices are available, no large up-front costs, and economical monthly costs that often don't reflect any city-to-city mileage contribute to low installed cost. This is by far the best choice for geographically dispersed wide area networks. In addition to those benefits, appropriate FRADS provide an economical interface device for legacy RTUs, easily add bandwidth (up to 1.5 Megabit speeds), and frame relay offers redundancy mechanisms that are extremely flexible. This redundancy preserves and usually improves reliability with easily configured backup routes and redundant sites.

Now that we have discussed the alternatives, it's time to focus on Frame Relay, look at some generic network examples, and talk about some specific DCB customer installations. We'll ignore the cost of host computer and RTUs since those are the same regardless of the WAN technology selected.

The FRAD (Frame Relay Access Device) used in all examples is the DCB Broadcast Polling Frad (BPF). The DCB BPF operates point to point and point to multi-point. It can be used point-to-point in Frame Relay or non-Frame Relay networks. The BPF can be used over public (telephone company) or private Frame Relay networks. Some corporate IP routed networks include routers that have a Frame Relay encapsulation function, which also supports the BPF. The BPF provides multiple channels over a single communications link, which allows SCADA systems to operate at higher serial speeds and provide for transporting asynchronous SCADA protocols over Frame Relay.

The BPF, as shown in Photo 1, is simple to set up and operate. The BPF relieves the SCADA host and RTUs from the burden of adding Frame Relay protocol. The BPF performs the task of transporting the asynchronous polling data over frame relay, and does it without requiring the user to do anything more than map the BPF port or ports to the DLCIs (logical circuit identifiers) that correspond to each remote site. The BPF is easy to set up using an asynchronous terminal or a PC terminal emulator such as Hyperterm. Commands are non-cryptic English, and the management port includes comprehensive help screens.

First up is the simple point-to-point example that we will build upon later. See Figure 1. In this case, there might be a single pump station located in the next county with a RTU that reports back to the host controller at your facility. Older technology would have used a simple modem link using low speed modems and a leased analog line. Depending upon the distance involved and imaginary lines (known as LATAs and company boundaries), the cost of this line today may be in the range of $100 to $500 per month with 9600 bps modems costing about $1800 per pair. The frame relay equivalent would use 56Kbps frame relay with a typical monthly cost of about $200, and our BPF SCADA FRAD with built-in DSU costing about $1600 per pair.

Figure 1

Next, we add two or more remotes to that network. See Figure 2. Let's go with about 3 remote sites, each with a single port RTU. That pushes us to multi-drop analog lines and polling modems. For this example, the RTUs could be at pipeline pump stations strung out in a line hundreds of miles long or power substations scattered around the state. The old method would be fast polling multi-drop modems at 9.6 Kbps. Again, costing about $1500 each since the fast-poll technology is more expensive.

Figure 2

The Frame Relay method starts to have some real cost advantage now. The remotes still cost about $800, but, now we are "sharing" a 56 Kbps line at the host site, and using a 56Kbps line at each remote. The per-location cost runs about $100 each for the frame relay drops, which is considerably lower than the per-drop cost of bridged analog lines. Since we have five times as much bandwidth (even at lower actual cost), we may transport more data, have more locations on a host port, or consider some redundancy in the data. It is possible to use a multi-port BPF FRAD and put multiple RTUs at a location, or tail-circuit RTUs where it's appropriate. Figure 3 shows a sample system using two host computers and a single Frame Relay network to support parallel SCADA systems.

Figure 3

When we add some redundancy to the equation, Frame Relay really shines. Figure 4 shows a system with a redundant host located at a different location. Each RTU answers and responds to both the main and backup host. It's up to the backup host to not control the system unless called upon. This can be either an automatic or manual operation, but in either case, it doesn't require any change by the telco. Since the BPF is available in a four port version, there could be up to four independent RTUs at each remote site, four host controllers, and up to 160 RTUs in the system. Most any combination of the network components could be redundant. Costs are similar to the other examples. Additional "paths" for redundant data to the same location are usually inexpensive, with a charge of a few dollars per month being typical for additional private virtual circuits (PVC).

Figure 4

These more complex systems illustrate several other Frame Relay advantages. Often, one modem failure in a multi-drop modem system will take the entire network down. With a Frame Relay system, the data is bridged or combined with software in the telco switches, so a single hardware failure or drop going down will only harm that drop not the entire system. Bandwidth improvement is another advantage. The drops are usually run at 56 or 64 Kbps. This is much better than the old 9.6 Kbps shared bandwidth modem option. In addition, the host site may have a T1 or fractional T1 link to the telco office. This allows multiple SCADA systems to share the incoming trunk for economy when redundancy isn't necessary.

It's not just theory. We've actually been installing these systems around the country for several years! A few examples

Customer: Gas and Electric Utility, OK

Project: Connect RTUs in remote power substations to control center. Approximately a dozen sites.

Customer: Power Authority

Partner: Southwestern Bell

Project: Power substation SCADA system. Approximately 30 remotes (and still growing).

Customer: Department of Water and Power

Project: Non-Frame Relay implementation of BPF. Installed several pairs of BPF units on 64Kbps DDS lines.

Customer: Pipeline

Project: Large natural gas pipeline distribution. Using BPF on a Frame Relay network having a fiber backbone.

Customer: Power Utility

Project: Long distance monitoring and control of peaking power plant located in Mississippi by control centers in Virginia. Includes multiple redundant systems.

Customer: Light and Power, MN.

Partner: QEI

Project: SCADA system to control peaking generator system. Includes single control location with multiple remotes.. growing to 25 sites.

Frame Relay is a proven technology with many advantages over the older multi-drop modem systems. Although there are some geographic areas where the telco hasn't moved forward into frame relay, for most of the country this is the multi-point network of choice. By using appropriate FRADs, legacy equipment as well as new SCADA equipment can be connected without regard to the protocols used. Network costs reduced, reliability is improved, and systems can grow easily. Frame Relay is probably in your SCADA future.

Copyright © 2000, John McCain. All rights reserved.

Additional information including tutorials, white papers, and product information is available from the Data Comm for Business, Inc. web site at http://www.dcbnet.com .

DCB customers often ask for a comparison of the reliability of Frame Relay versus analog phone lines. In our experience, the Frame Relay service has definitely been more reliable. The following evidence and technical information supports this improved reliability. We also note points of failure where analog circuits and Frame Relay have equivalent failure modes and rates of occurrence.

With analog circuits, the phone company must terminate the lines at the customer sites with the receive levels set to approximately -16 dBm, presuming a transmitting signal of 0 dBm. This attenuation is set in a Data Station Termination Unit. These are small boxes that typically have switches to set receive level at the customer site. Depending on the length of the cable from the central office to the customer, the signal might be just right, have to amplified or have to be attenuated (reduced). Signal levels that are too high or too low at installation time have occasionally been a problem with analog circuits. More often, the Data Station Termination Unit at the customer site has failed. This requires an on-site visit from the phone company, a trip they would rather not make, so the on-site visit is often delayed as long as possible while other failure possibilities are explored.

Another problem with analog circuits is accumulated noise. Each time an analog circuit is amplified, the noise is amplified along with the information. An analog repeater can not differentiate noise from data. This is similar to dubbing an audio tape. Each generation of an audio tape will have more noise than the previous generation. Analog amplifiers are located on the customer premise in the form of the Data Station Termination Unit and in the central office equipment, including the analog ports on carrier equipment and in analog bridges used for multi-point lines. Noise sources can be difficult to track down. At any point in the circuit path where an analog amplifier exists, additional noise can enter the system.

Digital circuits do not require levels to be set like analog circuits. And there is normally no Data Station Termination Unit required for a 56 or 64 Kbps digital circuit. Usually, the digital link to the customer site will either work or not work... no in-between conditions. Digital Dataphone System (DDS) circuits do not usually exhibit degraded behavior.

When digital circuits are regenerated at various points along the way from end to end, each regeneration is 100% new. There is no possibility for noise to be added through the regeneration process. If all the data is received correctly as 1s and 0s, it will be reproduced correctly. It is like making a copy of a CD or computer disk, where there is no degradation from one generation to the next.

To be fair in a discussion of analog circuits, they have become much more reliable in recent years. This is because carrier equipment has become nearly 100% digital. It is common today for an analog circuit to be analog only from the customer premise to the nearest central office. A circuit can go from coast to coast of the United States, perhaps 3000 miles, and be digital for all but a few thousand feet of cable from the central office to the customer. If a customer has the phone service coming to the premise over a T1 link, the analog portion of the link could be as short as a few hundred feet or less. When the analog portion of the link is that short, the reliability of analog circuits approaches the reliability of digital circuits.

Frame Relay circuits add another dimension of troubleshooting assistance to digital circuits. On a Frame Relay network, the customer equipment is required to send a Keep Alive data packet to the Frame Relay network (phone company's Frame Relay switch) every 10 seconds. If this Keep Alive does not show up as required, the Frame Relay network equipment sets an alarm. If a customer calls when a circuit fails, the Frame Relay network vendor can go to a console to look at the circuit. Often this console has a visual display that shows the customer equipment in various colors. A customer node that is up might show green, a customer node from which there are no Keep Alive packets may show up in red. The Frame Relay vendor instantly knows there is a problem and at which end of the circuit.

Copyright © 2000, John McCain. All rights reserved.

Appendix 2: Frame Relay How It Works

This appendix illustrates how to configure FRAD devices (The DCB BPF is used as an example) for commonly used topologies.

The BPF can be used on a point to point circuit that is not frame relay, or can be used on a point to point circuit configured for frame relay protocol. On a circuit that is not frame relay, the BPF management is set to none. This is done via the BPF "FR" command. A DLCI must be set at each end of the circuit and the DLCI numbers must be the same, usually 16, the first legal DLCI number. The most frequent use of point to point operation on a non-frame relay link is for bench testing with back-to-back 56/64 Kbps DSUs. A second use of the point-to-point links over non-frame relay lines is to combine from two to four SCADA host ports over a single digital line through a four-port BPF.

The BPF will operate over non-frame relay circuits using point-to-point mode, as well as over frame relay circuits.

The most common application of the BPF is to transport asynchronous multi-point polling over a public frame relay network. This might involve moving a circuit from analog facilities to frame relay to reduce monthly costs, to increase speeds, or both, or this could be a new SCADA circuit. It is quite simple to set up a BPF for frame relay operation. At the host end, an async terminal or PC terminal emulator is connected to the management port. A Configure Map (CM) command is used to list the DLCI numbers associated with the BPF port. A BPF can have from 1 to 4 ports, and each port can have up to 40 DLCI numbers mapped to it. The frame relay management does not need to be set since the default mode of the BPF will automatically detect whether the management is LMI or Annex D.

If there is only a single DLCI at the remote end, the only thing that needs to be set in the BPF is the speed. If the port speed is 9600 bps, even that does not need to be set, as 9600 bps is the default speed. The remote, or slave, BPF automatically detects LMI or Annex D management and automatically sets itself to the single DLCI that is assigned to the remote location. If there is just a single DLCI set, the BPF reads and uses that DLCI. When there is more than one DLCI set at the remote site the BPF needs to be mapped through the management port.

Many SCADA systems have more than a single line of multi-point polling in operation. When there are dozens or more remote devices, the polling is usually split up so there are a dozen or fewer RTUs on each multi-point line. The BPF is available in a four port version. Each of the four ports can be mapped to up to 40 DLCIs. Each port will be mapped to a different set of DLCIs, where the DCLI grouping on each BPF port represents a different multi-point group. The following is an example, where there are four ports on the BPF, four RTUs per BPF port, and the addressing of the host computer is assumed to be 01, 02, 03 and 04:

The DLCIs are entered in the BPF through the management port using the Configure Map (CM) command. With this command, the BPF prompts for the port number and the DLCIs to be mapped to that port.

The BPF at the host has its asynchronous ports attached to the host SCADA computer ports. The BPF has timers to insure it takes in an entire poll from the SCADA host, without fragmenting the poll, and places the polling data into frame relay protocol

packets. The frame relay packets are sent to the remote RTUs per the DLCI numbers assigned to the host BPF. Only one RTU will have an address to match the address polled by the host SCADA computer, and only that RTU responds to the poll.

The remote BPF, just like the host BPF, has timers to insure it places an entire response from the RTU (up to 1024 characters) into a single frame relay packet.

By using other ports on the BPF, external A/B switches, a meshed frame relay addressing scheme, or a combination of these, it is possible to set up a SCADA polling system with high availability backup links.

For point to point circuits, a backup host computer and a backup RTU can be setup on port 2 of the BPF. Port 1 functions as the main port. If the host or RTU fails, the polling system is moved from port 1 to port 2. Port 1 would be mapped for DLCI 16 at each end, port 2 mapped to DLCI 17 at each end.

For a point to point SCADA link with a backup host at a different location, each host location will have a unique DLCI. The common remote RTU location will have two DLCIs mapped to it. For example, the main host and the backup host will both have DLCI 16 assigned. The RTU will have two DLCIs, 16 and 17. Responses from the RTU will always be sent to both DLCI 16, the main host, and to DLCI 17, the backup host. The backup host will be activated only in the event the main host fails. While the main host is in operation, responses sent back to the backup host location are just ignored, as the backup host is not polling the RTU.

For point to multi-point with backup, one option is to have a backup host location, and have two RTUs at each remote site. The RTUs will be connected to four-port BPF units, port one connected to the primary RTU, port 2 connected to the backup RTU. Each host can be connected to a one port BPF (unless there is more than one multidrop line in operation at the host SCADA computer). The mapping might look like that shown in the table below, illustrating a four-drop system with a backup

host:

In the above example, the two hosts can even be polling at the same time, since the hosts are at separate locations and the remote RTUs are on separate ports of the BPF. If the dual RTUs were both connected to a single port BPF, then the remote BPF port will have two DLCIs mapped and the RTU must be connected to the BPF using a sharing unit, such as the DCB MSU-4D.

DCB has been installing frame relay equipment for many years and often tests frame relay networks for propagation delay. The propagation delay times have been improving year by year as the switching equipment gets faster and the link speeds within the frame relay networks increase. Today the round trip propagation delay from a host to remote and back has fallen below 50 ms, sometimes better than the propagation delays of modems. In short, regarding propagation delay, frame relay networks will perform as well or better than modem linked systems.

Frame relay is a synchronous HDLC protocol based network. Data is sent in HDLC packets, referred to as "frames". The diagram below of an HDLC frame may be familiar, since without adding specific definitions of how the Address, Control and CRC is used, the diagram is applicable to IBMs SDLC, to X.25, to HDLC, to Frame Relay, as well as other protocols.

The protocol is similar to that of an X.25 network, except all circuits are permanently assigned. X.25 circuits can be initiated and ended from the users' terminals. Frame relay relies on the customer equipment to perform end to end error correction. Each switch inside a frame relay network relays the data (frame) to the next switch. X.25, in contrast, performs error correction from switch to switch. The networks of today are sufficiently error free to move the burden of error correction to the end points. Most modern protocols perform error correction anyway, protocols such as SDLC, HDLC, TCP/IP, statistical multiplexer protocols, etc.

Because of the way frame relay just passes blocks of data from switch to switch, propagation from customer end to customer end through the network is very fast. Propagation time in the DCB multiplexer test at Wiltel (now MCI Worldcom) indicated a 70 millisecond round trip delay from Tulsa, Oklahoma to New York City and back. This is equal to or less than the propagation time through 9600 bps modems over the same distance. An X.25 network would experience a delay of a least one half second, and probably a second or more for the same distance.

Topology is the map, or visual layout of the frame relay network. A frame relay network topology is best viewed from several perspectives to appreciate fully understand the network and the equipment used to construct the network. Topological views include an overview "cloud" map, a logical connection map, perhaps a functional map, a map showing the detail equipment and channel links, an address map.

The map below is an overview where the frame relay switching is in the cloud in the center. The lines going out from the cloud represent the connections from the frame relay provider to the customer premises. These are typically lines ranging in speed from 56000 bps to T-1 (1.544 Mbps) and faster. One or more DLCI numbers are assigned to each line end point. DLCI is discussed in more detail below. Over 900 DLCIs can be assigned, but frame relay networks rarely have that many, because a user does not need that many in most cases. There may also be limits due to constraints in the frame relay switching equipment. Also, a network that is too large can be a problem to maintain.

Figure 5

Figure 5. Frame Relay Cloud Illustration

The frame relay network shown in Figure 5 as a cloud can also be shown as a connection of end points tied to a central site, or a number of end points that are all meshed together.

The end points to a central site is the traditional host to terminal model. The map below has 6 end points, just as in the previous map. Below, the end points are given meaning by defining one end as the host, the other 5 as the remotes. DLCI addressing could be assigned as follows:

At the frame relay switch, all remotes are mapped back to the host. At the host, Host 16 is mapped to Remote 1, Host 17 to Remote 2, Host 18 to Remote 3, Host 19 to Remote 4, and Host 20 to Remote 5. In practice, data from the host for Remote 5 arrives at the frame relay switch, from the customer equipment, with the destination address of 20. The frame relay switch, using its internal lookup table, finds that the data is for Remote 5, a physical port on the frame relay switch, that has a local address of 16. In the frame relay switch, address 20 is replaced with address 16 and the data is then delivered to Remote 5. The packet received at Remote 5 has an address of 16 after the frame relay switch changed the address.

The frame relay network above can be shown as a traditional multi-drop host to remote, with lines bridged in the middle, as shown below:

Frame relay also allows meshed networks. Since each endpoint can have one or more DLCI addresses, each user location can be connected to one other location, some locations, or all locations. If every location were connected to every other location, the network would be said to be "fully meshed". An illustration with all locations interconnected is below, next page. Meshed network:

The fully meshed network mapping of DLCIs looks like this:

This addressing can also be shown as a mapping table showing the addresses from the perspective of each one of the 6 locations. A frame relay switch requires this type of cross reference map. The switch uses the map to route the frame relay packets from one internal port of the switch to another port. The following table shows the local port number and the DLCI destination mapping that would be typical for a meshed network.

The BPF does not have the same bandwidth requirements that apply to LAN connections over frame relay. Most frame relay applications support LAN connections. In a LAN environment, there is typically a relatively high volume of traffic from remote offices back to headquarters locations. In these LAN environments, when there are several remote sites coming back to a central site, the bandwidth at the central site is in excess of the speeds at the remote locations. For example, if in the LAN environment there are 4 to 6 remote sites at 64 Kbps, the central site may require 128 Kbps.

With the BPF supporting polling SCADA systems, the bandwidth requirements are much less. For example, assume a 4-port BPF at the host, each port running 9600 bps and supporting 8 drops, for a total of 32 drops. The polling is half duplex, where the data is either an outbound poll or an inbound response, but not both at the same time on any single port. At most, there are only 4 remote sites responding simultaneously, combining for an aggregate data rate of 38400 bps (9600 x 4). Therefor, the host and all 32 remote sites have more than enough bandwidth if the local drops are only 56 or 64 Kbps.

Copyright © 2000, John McCain. All rights reserved.

Frame Relay Access Device (sometimes called Assembler/Disassembler). A DCB communications controller (BPF) that operates with frame relay protocol is a FRAD. When frame relay was first introduced most users and vendors thought of a FRAD only as a device for encapsulating LAN protocols for transmission through a frame relay network. This will change as more applications are moved onto frame relay networks.

Permanent Virtual Circuits. This contrasts with Switched Virtual Circuits (SVC). A connection to a network that allows connection and disconnection to various points is a switched virtual circuit. Circuits that are routed through software switching devices like X.25 pads and frame relay networks are virtual. A hard wired connection is not a virtual circuit.

Data Link Connection

Data Link Connection Identifier. This number identifies a virtual circuit. For BPF applications, a head end DCB Broadcast Polling Frame may have 4 DLCIs associated with it, while each of 4 remotes would have just a single DLCI. The following

illustrates how DLCIs might be assigned in this example:

In the above example, the Chicago host location might be a single 56 Kbps line into a DCB BPF. At each remote location there is a BPF. The DLCI numbers are not necessarily assigned in any order by the frame relay carrier, although it is common practice.

The above example illustrates that DCLI numbers have significance at the users end point only. Chicago has 4 addresses, 16, 17, 18 and 19. At each remote, each location has the same number, 16. The telephone company providing the frame relay service has a frame relay switch that translates the addresses. When these permanent virtual circuits are established, the relationship of the physical ports is mapped and then assigned DLCI numbers.

The DCB BPF, when carrying SCADA information, does not have the same data intensive requirements. SCADA systems are polled systems, using the communications one way at a time, polling or getting a response. And the polls and the responses are usually short blocks when compared to the IP protocol used to connect LANs. Therefore, the BPF requires less bandwidth, and a 56 or 64 Kbps line is usually sufficient for polling a dozen or more drops over a single frame relay circuit.

Special DLCI numbers

DLCI 0 (zero) and 1023 are reserved for management

DLCI 1 to 15 and 1008 to 1022 have been reserved for future use.

DLCI 992 to 1007 are reserved for layer 2 management of frame relay bearer service.

DLCI numbers 16 to 991 are available for subscribers for each user frame relay network

Frame relay management protocols for exchanging information between a user FRAD and the frame relay network equipment.

The management protocols are:

LMI - The original interim management protocol, uses DLCI 1023. LMI was specified by the Frame Relay Forum.

Annex D - An ANSI T1.617 management protocol standard, uses DLCI 1. Annex D was specified by the ANSI T1.617 specification.

Annex A - Nearly identical to Annex D, used in Europe.

This is an inquiry the frame relay user must make to the network every 10 seconds. The default value is 10 seconds. If the user does not make this inquiry every 10 seconds, the carrier network equipment will generate an alarm. The network equipment default value is 15 seconds. That is, the user should make an inquiry every 10 seconds, and the network expects the request every 15 seconds. The 5 second difference is a buffer time that allows for a small delay. The frame relay network switch responds to the Status Inquiry with an acknowledgement.

Sending the inquiry tells the network that the user FRAD is in service. When a status inquiry is not sent, the network will generate an alarm. An alarm may result in a "pro active" call to the customer from the service vendor. DCB frame relay products have a counter that counts the number of Status Inquiries sent to the frame relay network switch and how many were acknowledged. This counter, which rolls over to zero at 65536, provides a good indication of the quality of the circuit from the user location to the frame relay switch.

The Full Status Inquiry gets more information from the frame relay network switch than is obtained in the Status Inquiry. Full status responses to the user from the network also return all of the active DLCI numbers. The frequency of sending the full status can range from 1, meaning every Inquiry is a Full Status Inquiry, to 255, where only 1 out of every 255 Inquiries is a Full Status Inquiry. The default value is 6, which means the user equipment sends 5 regular status inquiries, then the full status inquiry. The DCB BPF counts the number of Full Status Inquiries sent, the number of responses received, and displays the DLCI numbers returned in the response.

It is interesting to check the DLCI channel assignments reported by the frame relay vendors switch. In as many as a quarter of the frame relay customer networks DCB has worked on, the Full Status Inquiry response has returned a DLCI number that is not engineered into the system. Apparently, some DLCI number was set earlier or in error on a frame relay switch port and the DLCI was inadvertently left in operation. This extra DLCI does not usually cause any problems, but is confusing. If an extra DCLI is discovered on a frame relay circuit, the vendor should be notified so it can be removed.

Indicates how frequently the FRAD should initiate a Status Inquiry Message. This is the 10-second timer referred to under the term "Status Inquiry".

This is a bit set in the frame relay protocol frame telling the customer recipient equipment that circuit congestion may exist in the direction from which the data was received.

This is a bit set in the frame relay protocol frame telling the customer equipment getting the notification that circuit congestion may exist in the direction it (customer) is sending data.

The CIR is the bit per second rate that the network provider (Phone Company) agrees to carry over the circuit under normal conditions. Frame Relay networks will handle bursts above the CIR. The CIR is not the same as the access rate. The access rate is the speed of the link to the phone company network, usually a DSU speed like 56, 64, 128, 256 Kbps, etc.

Management Information Base, a piece of information collected for an SNMP manager.

A device that collects MIB data to report them to SNMP managers. HPs Open View is an example of a manager.

Simple Network Management Protocol. Standard management protocol used with above managers and agents.

Copyright © 2000, John McCain. All rights reserved.

SCADA Frame Relay Products are listed at http://www.dcbnet.com/products.html#scada . Additional white papers and tutorials are available for download at HTTP://www.dcbnet.com/apnotes.html#scada .

Data Comm for Business Inc. 2949 County Road 1000 E Dewey, Il 61840	Voice: 217-897-6600 Toll Free: 800-4-DCB-NET Toll Free: 800-432-2638	Email: Contact Page Web: www.dcbnet.com Fax: 217-897-8023
All DCB web pages copyright ©1995- Data Comm for Business, All rights reserved. EtherPath®, EtherSeries®, EtherPoll®, EtherBridge® and EtherModem® are Registered Trademarks of Data Comm for Business, Inc.