Study on Partial-transit Inter-domain Routing over the TEIN2 Backbone

Zhonghui Li

Network Research Center of Tsinghua University

Beijing, China lizhh@cernet.edu.cn

Jilong Wang Network Research Center

of Tsinghua University Beijing, China

wjl@cernet.edu.cn

Bin Tian Network Research Center

of Tsinghua University Beijing, China

tianbin@cernet.edu.cn

Xing Li Network Research Center

of Tsinghua University Beijing, China

xing@cernet.edu.cn

Abstract—Based on the transit behavior, we categorize the Internet backbones into two types, global-transit backbone and partial-transit backbone. Partial-transit backbone represents the majority of R&E (Research & Education) backbones, which is liable to a number of inter-domain routing problems due to its particular characteristics. Through two routing cases in TEIN2 backbone, a typical R&E partial-transit backbone, we point out the main problems in partial-transit inter-domain routing, most of which are all relevant to inter-domain routing policy. After studying the related work on inter-domain routing policy, we propose a Distributed BGP Policy Simulation and Evaluation System, BGP-Grid. This solution is mainly out of consideration for operation and network engineering of TEIN2 backbone, which is proved to be practicable and effective during the three years’ TEIN2 operation. Keywords- BGP; partial-transit; inter-domain routing; routing policy

I. INTRODUCTION In terms of networking conception, Internet is regarded as the

composition of backbones and customer networks, among which Internet can provide transit and addressing service for the customer traffic. Unlike customer networks, backbones are mainly providing transit service for the customer traffic exchanged between customer networks.

There are several categorizations for Internet backbones. For instance, based on the routing hierarch, the backbones can be categorized into tier-1 backbone, tier-2 backbone etc. Moreover, based on the customer differentiation, the backbone can be categorized into R&E (Research & Education) backbone (the backbone providing transit service for research and education networks only) and non-R&E backbone (commercial backbone for commodity networks).

The primary purpose of R&E backbone is to provide efficient and high-performance transit service for R&E traffic generated by R&E customer applications, demonstrations and experiments among R&E networks, and the R&E backbone is only open to R&E networks, not commodity networks. As a result, R&E networks are supposed to have different requirements on the transit behavior, i.e. inter-domain routing policy, of backbones.

Hereby, a new transit-behavior-based categorization method for Internet backbones is introduced in this paper so as to carry out research on R&E-backbone-oriented inter-domain routing topics. Consequently, based on transit behavior, the backbones can be categorized into global-transit backbone and partial-transit backbone.

To clarify the differences between global-transit backbone and partial-transit backbone, here provides a comparison table (Table I).

Table I: Comparison between global-transit and partial-transit backbones

Global-transit Backbone Partial-transit Backbone Customer network

Commercial customer network oriented

R&E customer network oriented

Routing table Global routing table (full routing table) exchanged over backbone

Partial routing table (R&E routing table) exchanged over backbone

Primary objective

The reachability of customer networks

Optimizing routing for different applications and experiments, while the redundancy should be also considered

Routing policy implementation

Seldom apply fine tuning on inter-domain routing policy, and the applied inter-domain routing policy is coarsely granular (e.g. per peer)

Widely apply fine tuning on inter-domain routing policy, and the applied inter-domain routing policy is finely granular (e.g. per AS/community/prefix)

Interconnection mode

Globally open Locally constrained

The TEIN2 (the second generation of the Trans-Eurasia

Information Network) backbone is an intercontinental backbone connecting European R&E networks and Asia-Pacific R&E networks, which is a typical example of partial-transit backbone covering 11 NRENs (National Research and Education Networks) in Asia-Pacific area. The TEIN2 NOC is responsible for the operation of the TEIN2 backbone, on behalf of DANTE (Delivery of Advanced Network Technology to Europe) and the TEIN2 NRENs, and is operated by Tsinghua University. In this paper, a study on partial-transit inter-domain routing, especially

978-1-4244-7596-4/10/$26.00 ©2010 IEEE

for the TEIN2 backbone, is presented, which is mainly based on the operation and network engineering experience of the TEIN2 backbone from Tsinghua University.

The rest of this paper is organized as follows. Section 2 analyzes the main problems existing in partial-transit inter-domain routing. In Section 3, a summary of related work on inter-domain routing is outlined. Based on the operation experience of the TEIN2 backbone, a possible solution to partial-transit inter-domain routing is presented and described in Section 4 & 5. The summary and future works are concluded in Section 6.

II. PROBLEMS IN PARTIAL-TRANSIT INTER-DOMAIN

ROUTING To facilitate the understanding on problems existing in

partial-transit inter-domain routing, we will commence the analysis based on two typical cases during the operation of TEIN2 backbone. A. Taiwan Earthquake

As shown in Figure 1, the earthquake that occurred near Taiwan at the end of 2006 broke all the circuits between the TEIN2 backbone and Japan, which were the primary transit paths to Internet2.

At the same time, the circuit between the TEIN2 backbone and Korea as well as the circuit between Australia and USA were also broken, both of which were the backup transit paths to Internet2. Thus, after the earthquake, all the east-routed transit paths between the TEIN2 backbone and Internet2 were broken. Although GEANT2 (European R&E backbone) was regarded as another backup transit path (west-routed) between the TEIN2 backbone and Internet2, the traffic between the TEIN2 backbone and Internet2 was unable to traverse GEANT2 due to the selective transit routing policy applied inside GEANT2. As a result, the interconnection between the TEIN2 backbone and Internet2 was totally cut down by the earthquake.

Figure 1. The TEIN2 topology after Taiwan earthquake

After a short period of time, the circuit between Australia and

USA was restored first, followed by the circuit between the TEIN2 backbone and Korea, which meant two backup transit paths between the TEIN2 backbone and Internet2 were normalized as shown in Figure 2. Theoretically speaking, the traffic between the TEIN2 backbone and Internet2 should go through the circuit between the TEIN2 backbone and Korea which had the lower latency. However, the traffic between the TEIN2 backbone and Internet2 took Australia as preferred path due to inconsistent implementation of inter-domain routing policy among the TEIN2 partner networks, which introduced suboptimal routing problem (higher latency) between the TEIN2 backbone and Internet2.

Figure 2. Suboptimal routing after partial restoration of the TEIN2 circuits

B. CERNET International Cooperative Video Application

CERNET (China Education and Research Network) used to coordinate among the TEIN2 partners to carry out international cooperative video application. The participants included Japan, Korea and Thailand, all of which can be found in Figure 3. By default, the inbound traffic from the participants did not go through the TEIN2 backbone but the dedicated circuits among them.

Figure 3. Traffic paths before CERNET’s announcing more specific route

To take advantage of the TEIN2 backbone for particular

participants (e.g. Thailand), CERNET announced more specific route to the TEIN2 backbone so as to redirect Thailand’s inbound traffic from APAN-JP (Asia-Pacific Advanced Network, Japan node) to the TEIN2 backbone. However, as shown in

Figure 4, all the inbound traffic, including the inbound traffic from Japan and Korea, was redirected to the TEIN2 backbone, because Japan and Korea could also learn CERNET’s more specific route from the TEIN2 backbone, by which the suboptimal routing (higher latency and link congestion) was introduced.

Figure 4. Traffic paths after CERNET’s announcing more specific route

In the analysis of the two cases above, we can find several

problems in the partial-transit inter-domain routing as follows. Selective transit policy might lead to suboptimal routing

problem, and even weaken the redundancy of partial-transit backbone.

Due to limitation of funding, the stability of circuits over partial-transit backbone is not as high as that of global-transit backbone. Although the redundancy and optimization of transit paths have always been theoretically considered during the design of backbone topology, the suboptimal routing might be inevitable when more than one transit paths are broken.

The independence and non-transparence of each routing domain (backbone or customer network) might have side effect on global networks.

The main purpose of partial-transit backbone is to provide finely granular routing support to a variety of requirements from customer networks, so as to guarantee the effective and efficient transit service for all customer applications with different constraint condition, e.g. bandwidth, latency etc. However, in most cases, the accumulation of all finely granular routing amendments will not lead to global optimal routing, but might bring in unexpected problems.

Additionally, there are still some other problems in

partial-transit backbone which are not covered by the two cases. The private peering among customer networks of

partial-transit backbone is prevalent, which might introduce the asymmetrical routing problem.

In comparison with global-transit backbone, the IPv6 is more widely implemented over partial-transit backbone.

Because of the historical reason, a number of IPv6 virtual links (IPv6 tunnels) still exist, through which routing domains interconnect with each other without the limitation on geographical condition. Consequently, the traditional inter-domain routing policy amendment, which is mainly aim at physical link and geographical adjacency, might not be applicable for the peers interconnected with the IPv6 virtual link.

Therefore, the problems in partial-transit inter-domain routing

need to be studied and evaluated. Besides the problems caused by physical or personal reasons, most of the problems in partial-transit inter-domain routing are supposed to be triggered by one technical issue, inter-domain routing policy, which is exactly the main topic of this paper.

III. RELATED WORK The related study and work on inter-domain routing policy

are mainly focused on four topics. A. The influence of inter-domain routing policy on routing convergence.

The typical impact of inter-domain routing policy on routing convergence was described by T.G. Griffin et al. in [1], which pointed out the Stable Paths Problem (SPP) that might result in inter-domain routing protocol divergence. The solutions for SPP have been widely studied ([2, 3, 4, 5]), and one of latest solutions was proposed by S. Yilmaz et al in [6], i.e. Adaptive Policy Management (APM), which managed to resolve the problem in a distributed manner. The routing convergence problem caused by inter-domain routing policy is not involved in this paper. B. Conflict detection and avoidance of inter-domain routing policy.

Due to the independence and opacity of inter-domain routing policy deployment in individual AS (autonomous system), the conflict and dispute among different ASes are widely existent. Studies on the detection and avoidance of such problem were made. C. T. Ee et al. proposed a distributed mechanism that enforces a preference ordering in [7]. In [8], H.M. Guo et al. described another distributed mechanism that ASes dynamically adjust the preference of the routes involved. Moreover, S. Seetharaman et al. researched inter-domain policy violations in multi-hop overlay routes [9]. As a complementary mechanism, Internet Routing Registry (IRR) [10] was introduced and some tools have been developed for this purpose, including RAToolSet [11] and Nemecis [12] etc. All the studies aforementioned are focused on the inter-domain routing policy conflict, not the performance problems caused by inter-domain routing policy that the customer networks concern about, e.g. suboptimal routing. C. Abstraction and modeling for inter-domain routing policy.

In order to seek for the theoretical solution with

arithmetic-based principles to the problems led by inter-domain routing policy, the formalized description and modeling for inter-domain routing policy can provide mathematical foundation for the purpose. With respect to formalized description for inter-domain routing policy, IETF (Internet Engineering Task Force) designed a standard description language for routing policy, i.e. Routing Policy Specification Language (RPSL) [13]. The formalized modeling for inter-domain routing policy is still under research, and some progress has been achieved ([14, 15, 16]). Nevertheless, the formalized description and modeling are not mature enough to act as effective foundation of theoretical solution to inter-domain routing problems. D. Evaluation of inter-domain routing policy.

There are a number of inter-domain routing monitoring and analysis systems deployed over Internet, e.g. Routeview [17], but all of them are mainly focused on the monitoring and analysis of inter-domain route not inter-domain routing policy. N. Hu et al. proposed an ISP-oriented inter-domain routing system cooperative management framework (CMF) based on the self-organization method in [18]. Meanwhile, in [19], Y. He et al. provided a long-term solution to the difficult problem of AS-level routing evaluations. However, most of the studies are still limited in architecture design or system simulation, not the actual systems that can be used for the operation and network engineering of inter-domain routing over production backbone.

IV. SOLUTION OVER THE TEIN2 BACKBONE According to the conclusion of Section 3, an ideal and

practicable solution to problems in partial-transit inter-domain routing is unavailable yet, and the related study and research is supposed to be underway for a period of time.

As a production backbone, the TEIN2 can’t keep awaiting the ideal solution concluded from the related study and research in the future. Thus, to minimize the side effect caused by those problems, we propose a possible solution to the partial-transit inter-domain routing over the TEIN2 backbone, which is mostly out of operational consideration and based on technical feasibility of network engineering.

The solution we propose for the TEIN2 backbone is named as BGP-Grid.

V. BGP-GRID BGP-Grid is a Distributed BGP Policy Simulation and

Evaluation System, whose architecture is depicted in Figure 5. In this system architecture, each AS, as BGP-Grid participant, deploys and maintains one BGP-Grid Node (BGN), which will bring up BGP sessions with the BGP-Grid Nodes in other ASes, if its border routers are peering with those ASes. In other words, the BGP peerings among the BGP-Grid Nodes are same as the

actual BGP peerings among those ASes that the BGP-Grid Nodes are representing. Each BGP-Grid Node is managed by its own AS independently and brings up BGP peering with each other cooperatively, so we also call this architecture BGP-Grid. The main purpose of BGP-Grid is to simulate and evaluate the outcome of inter-domain routing policy design and amendment before applying on on-line routers, by which we can detect and rectify inter-domain routing problems over the TEIN2 backbone (e.g. suboptimal routing and asymmetrical routing problems) caused by inappropriate policy beforehand, so as to minimize the side effect on ongoing customer traffic and improve the inter-domain performance among TEIN2 networks.

Figure 5. System architecture of BGP-Grid

BGP-Grid has three important features as enumerated below. Independence: the participant’s BGP-Grid Node is only

maintained by the participant itself, and there is no need to look into the inter-domain routing policy of other participants.

Federation: the participants bring up BGP peerings with each other via the BGP-Grid Nodes, and the topology of peering is same as actual BGP peering among the participants, which is a cooperative federation of adjacency.

Simulation and Evaluation: the participants can simulate the amendment of inter-domain routing policy (even link outages) on BGP-Grid Nodes before applying it on on-line router, as it would be easier for participants to apply and coordinate inter-domain policy amendment on BGP Grid Nodes than on on-line routers among them. At the same time, the participants can generate virtual traffic among BGP-Grid Nodes to evaluate the outcome of amendment. Moreover, part of looking-glass feature is also introduced for simulation of the real peer-to-peer link latency and utilization among BGP peers.

There are two main system modules in BGP-Grid architecture

as shown in Figure 6. VTGCS (Virtual Traffic Generating & Capturing Server):

VTGCS is responsible for the virtual traffic initializing, generating, reverting, capturing, analyzing etc.

BGN (BGP-Grid Node): the core module of BGP-Grid, which is taking on several kernel tasks, e.g. peering with border routers of its own AS and external BGNs, simulating and evaluating BGP policy of its own AS, measuring actual peer-to-peer link latency and utilization between inter-domain border routers, updating per-hop record in virtual traffic packet, etc.

In addition, both VTGCS and BGN contain one or more embedded modules, GS-Tunnel (General Simulation Tunnel), which is a tunnel module developed for BGP-Grid particularly. GS-Tunnel is almost similar to point-to-point tunnel built in common operation systems, except that GS-Tunnel can provide support for point-to-multipoint tunnel, by which we can simulate BGP peering over multi-access networks, e.g. Ethernet, so as to make BGP-Grid suitable for most of scenarios over Internet.

Figure 6. System modules of BGP-Grid

The BGP-Grid Node is developed from Quagga routing

software, and has been deployed over TEIN2 backbone and some TEIN2 customer networks, which has been facilitating the detection and rectification of inter-domain routing problems over TEIN2 networks, as well as the optimization on inter-domain routing and performance among TEIN2 networks.

In order to verify the effectiveness of optimization on inter-domain routing and performance by BGP-Grid, we also made a simulation by establishing an inter-domain model with 3-8 ASes included, whose routing policy are independent and opacity to each other, and the peer-to-peer link latency (RTT) among ASes are from 50ms to 200ms. The comparison between the average end-to-end RTT before and after BGP-Grid simulation/evaluation/optimization is presented in Figure 7.

Figure 7. Simulation results

From Figure 7, we can find: Due to the deficiency of BGP routing protocol and

self-deployed policy (e.g. D-V algorithm, asymmetrical routing etc.), the average end-to-end RTT is increasing dramatically in comparison with the increase of interconnected ASes.

After the cooperative simulation/evaluation/optimization by BGP-Grid, the increase of average end-to-end RTT is in a reasonable linear manner, by which the overall performance of inter-domain routing is improved significantly.

VI. SUMMARY AND FUTURE WORKS

In comparison with global-transit backbone, partial-transit backbone has a few of special characteristics on routing requirements and behavior due to its R&E-interested background, which are more liable to some inter-domain routing problems than others.

Although a large number of studies and researches on inter-domain routing problems have been commenced or even concluded, few of them aimed at the particular scenarios of partial-transit backbone, and no ideal and practicable solution to the partial-transit inter-domain routing problem is available yet.

To alleviate the impact of such kinds of inter-domain routing problems, we proposed a possible solution which is mainly based on the experience in the TEIN2 backbone operation and network engineering, i.e. BGP-Grid, among TEIN2 networks. After 3-year operation of the TEIN2 backbone, BGP-Grid has been proved to be quite helpful for the mitigation of side effect caused by partial-transit inter-domain routing problems as well as the improvement of inter-domain routing performance among TEIN2 networks.

As emphasized in previous section, there is still no ideal solution to partial-transit inter-domain routing problems, so it is one of most important works for us in the future to continue the study and research on this topic. It is necessary to seek for the modeling of this kind of problem and academic solution to it.

In the meantime, as extension of the solution proposed in this

paper, we will carry on the improvement and enhancement for BGP-Grid, e.g. automatic synchronization of BGP policy from border router to BGN, solution to incontiguous BGP-Grid Node (across the AS without BGP-Grid node deployed).

REFERENCES [1] T.G. Griffin, F.B. Shepherd, and G. Wilfong. “The stable paths problem and

interdomain routing”, IEEE/ACM Transactions on Networking, 10(2): pp 232-43, 2002.

[2] T. G. Griffin, and G. Wilfong. “A safe path vector protocol”, Proceedings IEEE INFOCOM 2000. Conference on Computer Communications. Nineteenth Annual Joint Conference of the IEEE Computer and Communications Societies (Cat. No. 00CH37064), 2000(2): pp 490-499, 2000.

[3] L. Gao, T.G. Griffin, and J. Rexford. “Inherently safe backup routing with BGP”, Proceedings IEEE INFOCOM 2001. Conference on Computer Communications. Twentieth Annual Joint Conference of the IEEE Computer and Communications Society (Cat.No.01CH37213), pt.1, (1): pp 547-56, 2001.

[4] D. Pei, M. Azuma, D. Massey, L. Zhang, “BGP-RCN: Improving BGP convergence through root cause notification”, Elsevier Computer Networks Journal 48 (2) pp 175–194, 2005.

[5] A. Bremler-Barr, Y. Afek, and S. Schwarz, “Improved BGP convergence via Ghost Flushing”, Proceedings of the IEEE INFOCOM, 2003.

[6] S. Yilmaz, and I. Matta. “An Adaptive Management Approach to Resolving Policy Conflicts”, NETWORKING 2007, LNCS 4479, pp. 820–831, 2007.

[7] C. T. Ee, V. Ramachandran, B.G. Chun, K. Lakshminarayanan, and S. Shenker, “Resolving inter-domain policy disputes”, Computer Communication Review vol.37, no.4, pp 157-168, 2007.

[8] H.M. Guo, N. Yao, and Y.F. Ma, “A solution to inter-domain policy disputes”, IICT 2008: PROCEEDINGS OF CHINA-IRELAND INTERNATIONAL CONFERENCE ON INFORMATION AND COMMUNICATIONS TECHNOLOGIES 2008: pp 341-345, 2008.

[9] S. Seetharaman, and M. Ammar, “Inter-domain policy violations in multi-hop overlay routes: analysis and mitigation”, Computer Networks vol.53, no.1, pp 60-80, 2009.

[10] "Internet Routing Registry", http://www.irr.net/ [11] "RAToolSet", http://www.isi.edu/ra/RAToolSet [12] G. Siganos, and M. Faloutsos, “Analyzing BGP Policies: Methodology and

Tool”, IEEE, 0-7803-8355-9/04: pp 1641-1651, 2004. [13] C. Alaettinoglu, C. Villamizar, E. Gerich, D. Kessens, D. Meyer, T. Bates et

al., “Routing Policy Specification Language (RPSL)”, RFC 2622, 1999. [14] N. Feamster, R. Johari, and H. Balakrishnan. “Implications of Autonomy for

the Expressiveness of Policy Routing”, IEEE/ACM TRANSACTIONS ON NETWORKING, VOL. 15, NO. 6: pp 1266-1279, 2007.

[15] W. Mühlbauer, S. Uhlig, B. Fu, M. Meulle, O. Maennel, “In Search for an Appropriate Granularity to Model Routing Policies”, SIGCOMM’07, ACM 978-1-59593-713-1/07/0008: pp 145-156, 2007.

[16] R.X. Qu, H.X. Wang, and S.X. Cai, “Visualized inter-domain routing modeling language”, Computer Engineering vol.34, no.18: pp 148-150,

2008. [17] "University of Oregon RouteViews Project", http://www.routeviews.org/. [18] N. Hu, P. Zou, P.D. Zhu, and X. Liu, “Cooperative Management Framework

for Inter-domain Routing System”, ATC 2008, LNCS 5060: pp 567–576, 2008.

[19] Y. He, M. Faloutsos, S.V. Krishnamurthy, and M. Chrobak, “Policy-aware topologies for efficient inter-domain routing evaluations”, Proceedings IEEE INFOCOM: pp 291-295, 2008.