Matheion

MA-NET1 · Module 2

Packet core and networks of networks

Topics: Packet switching · Store-and-forward · Forwarding and routing · ISP hierarchy

Overview

Session 2 — The packet core and the network of networks

Last session, Mira's phone reached out to BayLink Access, her access ISP. Now we ask what happens on the far side of that doorway. Between her phone and the servers of Skylark Forecast lies a mesh of routers and links owned by many different organisations, including a large backbone called SpineNet. This session explains how that middle part moves data, why it chops messages into packets, and why the global arrangement of networks looks the way it does.

By the end you should be able to: contrast packet switching with circuit switching; explain store-and-forward; separate forwarding from routing; say where packet loss comes from; and explain, using simple counting, why access ISPs are not all wired directly to each other, and what peering, exchange points, and content-provider networks add.

Scroll Concepts → Examples → Quiz. Plan on roughly 55–65 minutes. Keep a pencil nearby for the small counting exercises in the examples; they are short, and they make the ideas stick.

Tip: if a paragraph feels dense, pause and say it back in your own words before moving on; that small habit saves time later.

Concepts

Core ideas — packets, routers, and the network of networks

Last session we stood at the edge and watched Mira's phone reach BayLink Access, her access ISP. Now we step through that doorway. Somewhere beyond it, her request for a forecast from api.skylark.example must find its way across many links and many organisations. The machines that do this relaying are not hosts; they belong to the network core. This session explains how the core moves data, and then zooms out to ask why the world's networks are arranged the way they are. Everything here, from the names to the numbers, is invented for practice.


Two ways to build a core: circuits and packets

Imagine you have to move a large number of people's messages between two cities. There are two sensible designs.

In the first design, before anyone speaks, you reserve a private lane for each conversation all the way from sender to receiver. While the conversation lasts, that lane is theirs alone, whether or not they are saying anything. Classic telephone networks worked this way, and the design is called circuit switching. The big advantage is predictability: once your lane is reserved, you get a steady rate with no surprises. The big disadvantage is waste: if you pause to think, your lane sits idle and nobody else can use it.

In the second design, nobody reserves anything. Each sender chops their message into small chunks and puts a destination label on each chunk. Every chunk is then sent into the network separately, and the relaying machines pass each chunk along toward its destination, one hop at a time, sharing the lanes with everybody else's chunks. This is called packet switching, and each labelled chunk is a packet. The Internet's core uses packet switching.


Why packet switching is a good fit for computers

The key observation is that computer traffic comes in bursts. Mira's app sits quiet for ten minutes, then sends a request, then receives a forecast within a fraction of a second, then goes silent again. If every such app held a reserved lane while idle, most of the capacity of the network would be sitting unused most of the time.

With packet switching, the capacity of a link is shared on demand. When Mira's phone is silent, her share of the link is automatically available to someone else's burst. This property is called statistical multiplexing: instead of cutting capacity into fixed slices for each user, we rely on the fact that, statistically, not everybody is active at once.

A small thought experiment makes the point concrete. Suppose a link can carry 6 Mbps. With circuit switching, if each user must be guaranteed 400 kbps, then the link can serve at most 15 users, because 15 times 400 kbps is 6 Mbps. If instead the users send in packets and each user is only active about a fifth of the time, then we can connect many more than 15 users: at any given moment only a few are active, and it would be unusual for more than 15 to speak at the same instant. The price we pay is that, on rare occasions, too many users burst at once and some packets must wait in line or be dropped. That price is the central trade-off of this whole topic.

Common confusion: Packet switching does not mean the network is "slower". It means the network is shared more efficiently. For bursty traffic it can serve many more users at about the same quality, and for steady, continuous traffic such as an ongoing phone call, circuit switching's guarantees can be more attractive.


A short word on how circuits share a link

A circuit-switched link still has to serve many circuits on one physical cable. Two classic tricks are worth a sentence each. Frequency-division multiplexing gives each circuit its own band of frequencies on the link, like radio stations sharing the air. Time-division multiplexing divides time into repeating frames and assigns each circuit certain slots within every frame, like a carousel where each rider owns particular seats. In both cases the share belongs to that circuit whether it is talking or not. The point to carry forward is not the details of either trick but the reserved nature of the share.

If the contrast still feels abstract, try this picture. Circuit switching is like booking a whole table at a restaurant for the evening: the table is yours even while you step outside for a phone call, and nobody else may sit there. Packet switching is like a buffet line: people take what they need when they need it, and the only cost of a busy evening is a short wait in line. Neither is "better" in general; the right choice depends on whether your visitors arrive steadily or in bursts.


How a packet travels: store-and-forward

Now back to packets. Suppose Mira's phone sends a packet that must pass through two routers, A and B, on its way to the server. A router is a machine in the core with several links attached to it. Its job is to receive packets on one link and send them out on another.

Almost all packet routers use a rule called store-and-forward: a router must receive the entire packet before it can begin sending the first bit of that packet onward. Think of a postal clerk who will not hand a parcel to the next truck until the whole parcel has been brought in from the previous one. The router stores the whole packet in memory and then forwards it.

Why does this matter? Because it means each hop adds a full "packet's worth" of transmission time. If a packet takes 2 milliseconds to be pushed onto one link, then a path of three links in a row, with store-and-forward at each router, takes about three times that, 6 milliseconds, just to push the bits out on each link (we are ignoring other delays for now). Session 3 turns this into a formula. For now, remember the picture: the packet moves in whole-packet jumps, one hop at a time.


Two jobs of the core: forwarding and routing

People often use the words routing and forwarding as if they were the same. They are two different jobs and mixing them up causes a lot of trouble later, so let us separate them cleanly with an analogy.

Imagine driving across a country. Planning your trip, working out which highways to take from start to finish, is one job. Passing through an intersection, deciding at this moment which exit to take, is another. The planning is slow and thoughtful; the intersection decision is quick and mechanical, and happens many times.

Forwarding is the intersection job. When a packet arrives at a router, the router reads the packet's destination label, looks it up in its forwarding table, and sends the packet out the corresponding outgoing link. The table is essentially a list of rules such as "destination labels starting like this go out link 2; those starting like that go out link 5". Forwarding must be fast, because a busy router handles enormous numbers of packets every second.

Routing is the planning job. It is the process by which the routers build those tables in the first place. Routers talk to each other using routing protocols, exchanging information about which destinations they can reach and through which neighbours, and each router uses that information to fill in or update its own forwarding table. Routing runs slowly and in the background, and it reacts to changes: if a link fails, routing notices and updates the tables so that future packets avoid the broken link.

In short: routing computes the table; forwarding uses the table. One is a background planning process spanning the whole network; the other is a per-packet local action inside a single router.

Common confusion: A router does not know the full path a packet will take. It only knows the next hop for each destination. Each router makes its own local choice, and the path emerges from many such choices, much like each intersection sign only points you to the next town.


Where packet loss comes from

A router has a limited amount of memory for holding packets that are waiting to go out on a link. This waiting area is a buffer, or queue. Normally packets spend only a short time in it. But suppose several incoming links all deliver packets destined for the same outgoing link at nearly the same moment. The outgoing link can only transmit one packet at a time, so the others must wait. If packets arrive faster than the link can send them, the queue grows. If it grows until the buffer is full, then the next arriving packet has nowhere to wait, and the router has to drop it. That dropped packet is a packet loss.

There are two other, rarer reasons for losing a packet: a bit error caused by noise that corrupts the packet so badly that a router or receiver throws it away, and a router or link failure. But the everyday cause on the Internet is the first one: queues that overflow.

Losing a packet is not a disaster for the network, because later layers can notice a missing piece and ask for it again. Still, loss costs time and wasted effort, so a network is happier when its queues stay short. We will learn exactly how queues grow and how to estimate them in the next session.


The big picture: how many ISPs, and how do they connect?

So far we looked at a single router and a single path. Now widen the camera. The Internet has a very large number of access ISPs, each serving a neighbourhood or a region, plus the networks of businesses, schools, and content providers. How are they all connected to each other so that Mira, on BayLink Access, can reach a server hosted anywhere?

The most obvious design would be a full mesh: connect every pair of access ISPs directly with its own link. It sounds fair and simple, but counting kills it. With N networks, a full mesh needs N times (N minus 1), divided by 2, links, because each pair needs exactly one. For 10 networks that is 45 links, which is manageable. For 40 networks it is 780. For thousands it is millions of links, each of which costs money to install, maintain, and run, and each of which must be separately negotiated between two companies. No access ISP could afford that, so full meshes do not happen.


Networks of networks: building it up in steps

A better design adds middlemen. Imagine each small access ISP pays a global or regional transit ISP, a bigger network, to carry its traffic to the rest of the Internet. This is a provider–customer relationship: the access ISP is the customer, the larger network is its provider, and the customer pays for the service. Now each access ISP needs only one link, to its provider, instead of dozens.

This is already much cheaper, but it introduces a new question: what if there are several competing providers? Then providers themselves must be connected to each other, otherwise customers of one provider could not reach customers of another. And since providers compete, a given region will often have several of them, each with its own customers. Providers in turn may buy service from still-larger networks. The result is a hierarchy: access ISPs at the bottom, regional networks above them, and a handful of very large, widely spanning backbone networks at the top that connect to one another. In this course, SpineNet plays the role of such a backbone.


Peering and exchange points

Paying a provider for every byte is expensive. Suppose two neighbouring access ISPs, BayLink Access and a rival, discover that many of their customers exchange traffic. Instead of each paying a provider to carry that traffic up and back down, they can connect their networks directly and carry each other's traffic. Such a direct arrangement is called peering. In a typical peering agreement the two networks do not charge each other for the traffic exchanged between their own customers, because both sides benefit about equally. Peering reduces the amount that must be paid to providers, and it often shortens the path, which helps speed.

Pairs of networks could set up many individual peering links, but a convenient shortcut exists: a shared building or facility where many networks bring a link and connect to one common switch. That facility is an Internet exchange point, or IXP. At an IXP, a network can set up peering with several others using a single physical connection instead of a separate cable for each. Many IXPs are run by independent, non-profit or cooperative organisations.

Importantly, peering is not transit. If BayLink peers with a rival, it can reach the rival's customers directly, but that does not mean the rival will carry BayLink's traffic onward to the rest of the world for free. Peering usually covers only the networks' own customers.


Content-provider networks

One more actor needs a place in the picture. Some organisations run enormous services that generate a large fraction of all traffic: video, forecasts, maps, software updates. To serve users quickly and to avoid paying transit for every byte, such organisations often build their own networks, spreading data centres and delivery nodes across many regions. They then connect those networks directly to access ISPs or to IXPs so that content reaches users over just a hop or two.

In our story, Northline Relay plays this role. Its delivery point edge.northline.example keeps copies of Skylark Forecast's pictures near BayLink's customers. When Mira's app requests an icon, the answer comes from a nearby Northline machine rather than from a faraway data centre. The benefits are speed, lower cost for everybody, and more control for the content provider over the path its traffic takes. The cost is a more complicated picture: the Internet is no longer a clean tree, but a tiered hierarchy overlaid with many shortcuts, with access ISPs, regional networks, backbones, peering links, exchange points, and content networks all connected in a messy web.


Pulling the session together

We began with two ways to build a core. Circuit switching reserves a path and a rate for each conversation, which gives predictability but wastes capacity when a conversation is quiet. Packet switching sends labelled chunks that share links on demand, which suits bursty traffic through statistical multiplexing, at the price of queueing and occasional loss. Routers use store-and-forward: they receive a whole packet before sending it on. Their two jobs are forwarding (per-packet table lookups) and routing (the slower process that builds the tables). Packet loss comes mostly from full buffers. Zooming out, a full mesh of access ISPs would need far too many links, so ISPs connect through providers, creating a hierarchy; peering and IXPs add cost-saving shortcuts; and content-provider networks bring data close to users.

In summary:

  • Circuit switching reserves capacity; packet switching shares it on demand.
  • Store-and-forward means a whole packet arrives at a router before it is sent on.
  • Forwarding looks up a table for each packet; routing builds the tables.
  • Buffers overflow when packets arrive faster than a link can send them, causing loss.
  • A full mesh of N networks needs N(N−1)/2 links, which is why providers and hierarchy exist.
  • Peering links and IXPs shortcut the hierarchy; content networks bring data close to users.

Next session turns the vague idea of "waiting in line" into precise delay components that you can compute.

Examples

Worked examples — counting, queues, and connections in Mira's corner of the network

These five stories use invented numbers. Try each question on paper first; then read the working line by line.


Example 1 — How many circuits fit, and what does sharing buy?

Scene. A small town hall has a single 9 Mbps link, which it can run in one of two ways. Under the circuit-switched plan, each connection is guaranteed 450 kbps. Under the packet-switched plan, connections just send packets as needed. The town hall wants to know how many staff devices can be supported.

Numbers. Link capacity: 9 Mbps = 9,000 kbps. Circuit share: 450 kbps. Under the packet plan, each device is active about 10% of the time, and when active it uses 450 kbps.

Working.

Circuit plan. Number of circuits = capacity ÷ share = 9,000 kbps ÷ 450 kbps = 20 circuits. A 21st device would be refused, even if all 20 existing devices were silent.

Packet plan, average load. Suppose 60 devices connect. Each is active 10% of the time, so the expected number of simultaneously active devices is 60 × 0.10 = 6. Expected demand = 6 × 450 kbps = 2,700 kbps = 2.7 Mbps. That is only 30% of the link's 9 Mbps.

Packet plan, worst case. The link is saturated only when more than 20 devices are active at the same instant (20 × 450 kbps = 9,000 kbps). With an average of 6 active, seeing more than 20 at once would be very unusual.

Interpretation. The circuit plan serves 20 devices with certainty. The packet plan serves 60 devices, three times as many, and it will usually run with plenty of spare room. The cost is that on rare occasions a rush of activity creates queues and possibly loss.

Takeaway. Packet switching wins when each user is idle most of the time. If every device were active nearly all the time, say 90%, the picture would reverse, and the reserved circuits would look more attractive.


Example 2 — One packet across store-and-forward hops

Scene. Mira's phone sends one packet toward api.skylark.example. The packet crosses three links, with two routers in between: phone → router R1 (BayLink) → router R2 (SpineNet) → server. Every link runs at 3 Mbps. The packet holds 6,000 bits. We ignore propagation, processing, and queuing for now and look only at the time to push the bits onto each link.

Working.

Step 1: time to push one packet onto one link. Transmission time = length ÷ rate = 6,000 bits ÷ 3,000,000 bits/s = 0.002 s = 2 ms.

Step 2: apply store-and-forward. The phone finishes sending the packet onto link 1 at t = 2 ms. R1 now has the entire packet, so it may begin sending onto link 2. R1 finishes at t = 2 + 2 = 4 ms. R2 has the whole packet at 4 ms and finishes sending onto link 3 at t = 4 + 2 = 6 ms.

Step 3: write the timeline.

t = 0 ms   phone starts sending on link 1
t = 2 ms   R1 has the whole packet; R1 starts sending on link 2
t = 4 ms   R2 has the whole packet; R2 starts sending on link 3
t = 6 ms   server has the whole packet

Interpretation. The packet needed 3 hops × 2 ms = 6 ms. If the packet were cut-through (starting to send before it was fully received), the finish time would be earlier, but store-and-forward routers do not do that.

Takeaway. With store-and-forward, each additional link adds one more packet-transmission time. A path of N links, equal rates, gives N × (L ÷ R).


Example 3 — A forwarding table at work

Scene. A router at SpineNet has four links numbered 1 to 4. Its forwarding table, written with invented destination labels for practice, is:

Destination label starts with   Outgoing link
41.                             1
41.88.                          2
57.                             3
anything else                   4

The router has a rule for resolving overlaps: the longest matching label wins, meaning the most specific rule is chosen. Three packets arrive with these destination labels: 41.12.5.9, 41.88.200.3, and 99.1.1.1.

Working.

Packet A, 41.12.5.9. It starts with 41., so it matches the rule for link 1. Does it start with 41.88.? No, its second part is 12. So link 1 is the only match among the first two rules, and the rule for 57 does not apply. Choose link 1.

Packet B, 41.88.200.3. It starts with 41., which matches link 1's rule. It also starts with 41.88., which matches link 2's rule. Both match, and the second is longer, so the more specific rule wins. Choose link 2.

Packet C, 99.1.1.1. It starts with neither 41. nor 57., so only "anything else" applies. Choose link 4.

Interpretation. The router never needed to know the whole journey. It consulted its table, found the outgoing link, and moved on. That is forwarding. Now suppose the link 2 cable is cut. The packets for 41.88. would be lost unless the table changes, and the process that updates the table (after neighbouring routers share news of the failure) is routing.

Takeaway. Forwarding is a quick local lookup, usually with a "most specific rule wins" tiebreak. Routing is the slower, network-wide activity that keeps the tables correct.


Example 4 — Counting links: mesh versus hierarchy

Scene. A region has 40 small access ISPs. A planner compares two designs. In Design M, every pair of access ISPs is directly connected. In Design H, the 40 access ISPs are divided evenly among 4 regional providers, with each access ISP connected to its own provider only, and the 4 regional providers connected in a full mesh among themselves.

Working.

Design M. Links = N(N−1)/2 with N = 40. So 40 × 39 ÷ 2 = 1,560 ÷ 2 = 780 links.

Design H, access links. Each of the 40 access ISPs has one link to its provider: 40 links.

Design H, provider mesh. With 4 providers, links = 4 × 3 ÷ 2 = 6 links.

Design H total. 40 + 6 = 46 links.

Comparison. 780 ÷ 46 is about 17, so the full mesh needs roughly seventeen times as many links.

Interpretation. The hierarchy trades some directness for enormous savings. A packet between two access ISPs under the same provider travels up one link and down another. A packet between ISPs in different regions travels up, across one provider-to-provider link, and down. Each trip is two or three hops, not one, but the network is affordable and manageable.

Now suppose two access ISPs under different providers exchange a great deal of traffic, and add one direct peering link between them. That link adds exactly 1 to the count, making 47, and it shortens the path for their traffic from three hops to one. This is why peering is attractive: a small addition removes heavy load from the providers above.

Takeaway. Mesh cost grows with the square of the number of networks, hierarchy cost grows roughly linearly, and selective peering adds shortcuts where they pay off.


Example 5 — A burst arrives and a buffer overflows

Scene. A BayLink router has one outgoing link that sends one packet every 2 ms. Its buffer can hold 5 packets waiting (this does not count a packet currently being transmitted). At one instant, a burst of 9 packets arrives, all headed for that outgoing link, and the link was idle just before.

Working.

Step 1: who gets in? The first packet finds the link idle and starts transmitting at once; it does not wait in the buffer. The next 5 packets fit in the buffer. That is 1 + 5 = 6 packets accepted. The remaining 9 − 6 = 3 packets find the buffer full and are dropped.

Step 2: how long do the accepted packets wait? Label the packets P1 to P6 in the order of acceptance. P1 starts at 0 ms and finishes at 2 ms. P2 starts at 2 ms. P3 starts at 4 ms, P4 at 6 ms, P5 at 8 ms, and P6 at 10 ms.

Step 3: queueing times. The queueing time of each packet equals the time it waits before starting transmission: P1 waits 0 ms, P2 waits 2 ms, P3 waits 4 ms, P4 waits 6 ms, P5 waits 8 ms, P6 waits 10 ms.

Step 4: the average wait among accepted packets. (0 + 2 + 4 + 6 + 8 + 10) ÷ 6 = 30 ÷ 6 = 5 ms.

Interpretation. A burst of nine packets lost three of them and made the others wait anywhere from 0 to 10 ms. If the router had a larger buffer, fewer packets would have been dropped, but the last ones would have waited even longer. So buffer size trades loss for delay.

Takeaway. Loss occurs when packets arrive faster than they can leave and the buffer fills. A buffer can absorb a brief burst, but it cannot create capacity.

Practice check — packet core and network of networks

Multiple-choice check from the Matheion app lesson. Pick an answer for each question, then submit.

  1. 1.A shared 12 Mbps link must give every circuit-switched connection a guaranteed 600 kbps. At most how many connections can be admitted?

  2. 2.Why does packet switching typically support more bursty users on the same link than circuit switching with guaranteed shares?

  3. 3.A packet of 8,000 bits crosses two store-and-forward routers, so there are three links in total, each running at 4 Mbps. Ignoring all delays except transmission, how long until the whole packet is at the destination?

  4. 4.Which statement correctly separates forwarding from routing?

  5. 5.A router's link sends one packet every 3 ms and its buffer holds 4 waiting packets. An idle link suddenly receives a burst of 10 packets. How many packets are dropped?

  6. 6.A planner wants to connect 30 access ISPs. How many links would a full mesh, with one direct link per pair, require?

  7. 7.Two neighbouring access ISPs notice heavy traffic between their own customers and connect their networks directly, without paying a provider to carry that traffic. What is this arrangement called, and what is a typical benefit?

  8. 8.Why might a content-delivery network such as Northline Relay place delivery nodes close to access ISPs and connect to them directly or at an exchange point?