Authors

Overview

This document describes a multi-zone firewall design for on-prem-to-VPC traffic on IBM Cloud VPC that inspects external traffic arriving from an on-premises network through Direct Link or Transit Gateway on its way to spoke application VPCs. The design builds on the same proven technology combination — Fortinet vFSA instances, IBM Cloud Route Mode Network Load Balancer (RMNLB), and FortiGate Session Life Support Protocol (FGSP) — that is covered in depth in the companion blog Fortinet vFSA HA with Route Mode NLB for Spoke-to-Spoke Traffic on IBM Cloud VPC. This document is an excellent reference and is strongly recommended reading before this one because it covers the foundational Active/Active vFSA + RMNLB + FGSP pattern, full FortiOS configuration details, and validated end-to-end test results in a single-zone environment.

This document addresses a complementary, but distinct use case. Where the spoke-to-spoke design handles east-west traffic between cloud workloads within IBM Cloud, this design is concerned with on-prem to VPC traffic—connections that originate outside IBM Cloud and must traverse a centralized, resilient firewall layer before reaching any spoke workload. That shift in traffic direction introduces two additional requirements: the firewall layer must span three availability zones because external traffic can target workloads in any zone, and the routing model must use tiered priority rules across those zones to keep traffic symmetric in steady state while providing predictable, automatic failover if a zone becomes unavailable. Both requirements are absent from the single-zone spoke-to-spoke scenario, which is why they are not covered there.

Use Case Comparison

The following table is provided as a quick orientation, not as a ranking. Each design is the right fit for its intended traffic pattern.

Dimension	Spoke-to-Spoke Design	This Design
Traffic direction	East-West — between IBM Cloud VPCs	North-South — on-prem through Direct Link or Transit Gateway to IBM Cloud spoke VPCs
Zones	Single zone	Three zones — Active/Active/Active
vFSA instances	2 (in one zone)	3 or more — one per zone, independently scalable
RMNLB instances	1	3 — one per zone, each with its own VIP
Routing model	Single ingress route table, flat static routes	Three custom route tables with tiered priority routing (priority 2 / 3 / 4) per zone
Zone-local return path	Not required — single zone	Required — Egress CR VIP per zone ties return traffic to the local zone's firewall
External connectivity	IBM Cloud Transit Gateway only	Direct Link and/or Transit Gateway for on-prem connectivity
Power Virtual Server workloads	Not in scope	Supported as spoke destinations

Architecture Diagram

This diagram shows the full on-prem-to-VPC traffic path. An on-premises client connects through Direct Link or Transit Gateway into the Transit VPC (Hub), where three independent RMNLB and vFSA pairs—one per availability zone—inspect and forward traffic to the target spoke VPC. Return traffic from the spoke follows the symmetric path back through the same zone's firewall using the Egress Custom Route VIP.

How the Routing Model Enforces Zone Symmetry

The key routing insight in this design is that both the forward path (on-prem → spoke) and the return path (spoke → on-prem) must be anchored to the same zone's firewall. If they are not, the stateful firewall in each zone will see only one direction of a session, which causes connection drops.

This is achieved through two complementary mechanisms:

• Ingress priority routing on the Transit VPC ensures that traffic arriving from on-premises for a Zone 1 spoke workload is steered to the Zone 1 RMNLB first (priority 2), falling back to Zone 2 (priority 3) or Zone 3 (priority 4) only if the preferred path is unavailable.
• Egress CR VIP per zone on the spoke subnets ensures that return traffic from a Zone 1 workload is also sent back through the Zone 1 RMNLB as the preferred next hop, keeping the full session on the same firewall.

When a zone fails, the RMNLB withdraws its route advertisement. IBM Cloud automatically promotes the next-priority route, shifting both forward and return paths to a healthy zone together. FGSP then ensures the receiving firewall already has the session state it needs to continue processing in-flight connections without disruption.

Traffic Flow Diagrams

Steady-State: On-Prem Client → Zone 1 Spoke (Forward Path)

On-Prem Client (10.0.0.4)
        │
        ▼  Direct Link / Transit Gateway
        │
        ▼  Transit VPC — Ingress Custom Route Table
           Destination: Spoke AZ1 prefix → RMNLB-Z1 (priority 2) ✓
        │
        ▼  RMNLB-Z1 selects vFSA-Z1
        │
        ▼  vFSA-Z1 inspects and forwards
        │
        ▼  Transit Gateway → Spoke VPC AZ1
        │
        ▼  Spoke VSI (192.168.101.4)

Steady-State: Zone 1 Spoke → On-Prem Client (Return Path)

Spoke VSI (192.168.101.4)
        │
        ▼  Spoke Egress Custom Route Table
           Destination: 10.0.0.0/8 → Egress CR VIP-Z1 (priority 2) ✓
        │
        ▼  RMNLB-Z1 selects vFSA-Z1  ← same firewall as forward path
        │
        ▼  vFSA-Z1 matches session, forwards
        │
        ▼  Transit VPC → Direct Link / Transit Gateway
        │
        ▼  On-Prem Client (10.0.0.4)

---

The Problem: Resilience Without Breaking Stateful Inspection

Stateful firewalls are designed around connection awareness. They track the full state of every TCP session so they can match request and return traffic to the same flow. Under normal conditions that works well because request and response traffic typically traverse the same firewall. In a distributed multi-zone design, that assumption can fail during an outage.

A resilient architecture therefore has to meet two goals at the same time:

• Keep traffic local and symmetric in steady state — request and response traverse the same firewall in the same zone,
• Continue forwarding traffic even when failover introduces asymmetric paths — where the return packet arrives at a different firewall than the one that created the session.

This is where RMNLB and FGSP work together:

• RMNLB steers traffic across zones using priority-based routing and withdraws route advertisements when a firewall fails, shifting traffic to a healthy zone.
• FGSP synchronizes session state across all three FortiGate instances so a surviving firewall can process return traffic for a session it did not originally create.

Neither mechanism alone is sufficient. RMNLB without FGSP would redirect traffic but drop asymmetric return flows. FGSP without RMNLB would synchronize sessions, but have no mechanism to shift traffic away from a failed zone.

Steady-State Traffic Behavior

When all components are healthy, traffic stays local to each availability zone. The routing priority model is designed specifically to enforce this.

Expected Characteristics in Steady State

• Symmetric routing: The same firewall handles both directions of every flow — request and response traverse the same vFSA in the same zone.
• Zone locality: Traffic for a Zone 1 spoke workload is handled by the Zone 1 RMNLB and vFSA. It does not cross zone boundaries unnecessarily.
• Full capacity utilization: All firewall instances inspect traffic simultaneously. This is Active/Active/Active, not Active/Standby.
• Low latency: Keeping traffic in-zone minimizes inter-zone traversal overhead.

Steady-State Traffic Patterns (On-Prem to VPC)

On-Prem Client → [Transit VPC AZ1] RMNLB-Z1 → vFSA-Z1 → Spoke AZ1 VSI
Spoke AZ1 VSI  → Egress CR VIP-Z1 → RMNLB-Z1 → vFSA-Z1 → On-Prem Client


On-Prem Client → [Transit VPC AZ2] RMNLB-Z2 → vFSA-Z2 → Spoke AZ2 VSI
Spoke AZ2 VSI  → Egress CR VIP-Z2 → RMNLB-Z2 → vFSA-Z2 → On-Prem Client


On-Prem Client → [Transit VPC AZ3] RMNLB-Z3 → vFSA-Z3 → Spoke AZ3 VSI
Spoke AZ3 VSI  → Egress CR VIP-Z3 → RMNLB-Z3 → vFSA-Z3 → On-Prem Client

This is the ideal operating condition—the routing priority model is specifically designed to produce this outcome. The Egress CR VIP per zone is the mechanism that ties the return path to the same zone as the forward path.

The Asymmetric Routing Challenge

During failover, traffic might enter through one zone's FortiGate and the return path might exit through a different zone's FortiGate. That creates an asymmetric flow.

For a traditional stateful firewall, this is a serious problem. If the return packet arrives at a different firewall than the one that established the session, the second firewall has no record of that connection and will drop it.

Typical symptoms when asymmetric routing is not handled:

• Connection failures and TCP resets
• Application timeouts that appear intermittent
• Sessions dropping specifically during or after a maintenance or failover event
• Traffic working in one direction but not the other

In this design, asymmetric routing is expected and handled, not prevented. The combination of RMNLB priority routing (which minimizes asymmetry in steady state) and FGSP (which makes asymmetry transparent during failover) means the application layer never sees these events.

Why FGSP Is Critical

FGSP is designed specifically for Active/Active firewall deployments where asymmetric routing is a known condition. Instead of requiring a single active device to handle both directions of every flow, it allows multiple FortiGate instances to remain active simultaneously while sharing session state.

How FGSP Preserves Sessions During Failover

Step 1 — Session creation in steady state:

On-Prem Client (10.0.0.4) → vFSA-Z1 → Spoke VSI (192.168.101.4)

• vFSA-Z1 creates the session entry.
• FGSP immediately synchronizes the session to vFSA-Z2 and vFSA-Z3.
• All firewall instances now know about the connection.

Step 2 — Failover shifts the return path:

Spoke VSI (192.168.101.4) → vFSA-Z2 → On-Prem Client (10.0.0.4)

• vFSA-Z1 has failed. RMNLB-Z1 withdraws its routes.
• Return traffic from the Spoke VSI now takes the priority-3 path through RMNLB-Z2 → vFSA-Z2.
• vFSA-Z2 looks up the synchronized session table, finds the session (originally created by vFSA-Z1), and processes the return packet correctly.

Step 3 — Session continues without interruption:

• Subsequent packets traverse any surviving firewall.
• Session state remains consistent across all active cluster members.
• The on-premises client sees no disruption. The TCP connection remains established.

In practice, FGSP is what makes the three-zone Active/Active/Active model viable for stateful inspection. Without it, any zone failure would break all in-flight sessions.

---

Target Architecture

Architecture Overview

The design follows a hub-and-spoke model with centralized On-Prem to VPC Traffic security inspection. All external traffic entering IBM Cloud from on-prem must pass through the Transit VPC firewall layer before reaching any spoke workload. Return traffic from spoke workloads follows the same path in reverse.

On-Prem Network (10.0.0.0/8)
          │
          ▼
  Direct Link  ──or──  Transit Gateway
          │
          ▼
  ┌─────────────────────────────────────┐
  │         Transit VPC (Hub)           │
  │  172.26.0.0/16 / 172.27.0.0/16 / 172.28.0.0/16  │
  │                                     │
  │  ┌──────────┐  ┌──────────┐  ┌──────────┐ │
  │  │ RMNLB-Z1 │  │ RMNLB-Z2 │  │ RMNLB-Z3 │ │
  │  │  vFSA-Z1 │  │  vFSA-Z2 │  │  vFSA-Z3 │ │
  │  │  (AZ1)   │  │  (AZ2)   │  │  (AZ3)   │ │
  │  └──────────┘  └──────────┘  └──────────┘ │
  └─────────────────────────────────────┘
          │
          ▼
  Transit Gateway
    ┌─────┴──────┐
    ▼            ▼
Spoke VPC-1    Spoke VPC-2
192.168.10x    192.168.20x
(AZ1/2/3)      (AZ1/2/3)

Note: Direct Link-based deployments are supported in place of Transit Gateway. Both VPC and Power Virtual Server workloads can co-exist as spoke destinations in this architecture.

Main Components

Component	Role
Direct Link / Transit Gateway	Connects on-premises networks to IBM Cloud
Transit VPC (Hub)	Hosts the centralized inspection layer; contains all RMNLB and vFSA resources
RMNLB (one per zone)	Route-based load balancer; steers traffic to the local zone's vFSA and withdraws route advertisements on firewall failure
FortiGate vFSA (one or more per zone, scalable)	Stateful firewall instance; member of the FGSP cluster. Additional instances can be added to a zone's RMNLB back-end pool without architectural changes.
FGSP cluster	Synchronizes session state across all three vFSA instances to handle asymmetric return paths after failover
Spoke VPCs	Application environments (production, development, shared services, Power Virtual Servers) connected via transit gateway
Egress Custom Routes	Zone-specific VIP routes on spoke subnets that steer return traffic back through the correct zone's RMNLB

Traffic Flow — Step by Step

Inbound (on-prem → Spoke):

1. On-premises client sends traffic to a Spoke VPC destination.
2. Traffic arrives at the Transit VPC via Direct Link or Transit Gateway.
3. The Ingress Custom Route Table (Transit Gateway or Direct Link source) matches the spoke destination prefix and forwards traffic to the preferred zone's RMNLB (priority 2 route).
4. RMNLB selects the vFSA back-end in that zone.
5. The vFSA inspects and forwards the traffic through the Transit VPC toward the Spoke VPC via Transit Gateway.
6. The packet arrives at the Spoke VSI.

Outbound / Return (Spoke → on-prem):

1. Spoke VSI sends the reply toward the on-premises source.
2. The Spoke Egress Custom Route Table matches the on-premises or remote prefix and forwards traffic to the Egress CR VIP for the local zone.
3. The RMNLB for that zone receives the traffic and routes it to the local vFSA.
4. The vFSA forwards the traffic back through the Transit VPC to Direct Link or Transit Gateway.
5. Traffic returns to the on-premises client.

---

Prerequisites

Before starting, ensure the following are available:

• IBM Cloud account with VPC, Transit Gateway, and Direct Link permissions
• Transit VPC (Hub) — 172.26.0.0/16 (AZ1), 172.27.0.0/16 (AZ2), 172.28.0.0/16 (AZ3) — with subnets in all three zones
• Spoke VPC-1 and Spoke VPC-2 — with subnets in all three zones
• Transit Gateway connecting Hub VPC and all Spoke VPCs
• Direct Link (or second Transit Gateway) connecting on-prem to Hub VPC
• Three Fortinet vFSA instances — one per availability zone — deployed as BYOL from the IBM Cloud catalog (see section vFSA Deployment from BYOL Tile (per Zone))
  • Each vFSA attached to two hub subnets: one management subnet and one data subnet
• Three RMNLB instances — one per zone — each in the data subnet of the corresponding zone
  • Route Mode NLBs require a service-to-service authorization policy before they can be created. In IBM Cloud IAM, grant VPC Infrastructure Services (Load Balancer for VPC) Editor access to VPC Infrastructure Services (Virtual Private Cloud). See Before you begin — NLB route mode for details.
• VSIs in the spoke VPCs (for testing)
• An on-prem VSI or test client with network access through Direct Link / Transit Gateway
• SSH or console access to each vFSA instance
• Valid Fortinet BYOL license tokens (one per vFSA)
• IBM Cloud VSI profile cx2-2x4 (2 vCPU, 4 GB RAM) recommended for each vFSA

Security group requirements for vFSA data interfaces (port2):

Direction	Protocol	Source	Destination	Port	Purpose
Inbound	TCP	On-prem CIDR (10.0.0.0/8)	Spoke prefixes	Any	Forwarded traffic
Inbound	TCP	Spoke prefixes	On-prem CIDR	Any	Return traffic
Inbound	TCP	Hub VPC CIDR	vFSA port2 IP	22	RMNLB health check
Inbound	ICMP	Hub VPC CIDR	vFSA port2 IP	—	Connectivity testing
Outbound	TCP/UDP/ICMP	Any	Any	Any	General forwarding

Important:

• IP spoofing must be enabled on each vFSA's port2 network interface in the IBM Cloud VPC console. Without this, same-interface (hairpin) routing will not work.
  
  
•  PSF (Protocol State Filtering) must be disabled on each vFSA.

---

Addressing Used in This Setup

VPC CIDRs

VPC	CIDR	Purpose
Transit VPC (Hub)	172.26.0.0/16 (AZ1 subnets), 172.27.0.0/16 (AZ2 subnets), 172.28.0.0/16 (AZ3 subnets)	Single Hub VPC; zone subnets drawn from each /16 prefix
Spoke VPC-1	192.168.100.0/22	Application workloads
Spoke VPC-2	192.168.200.0/22	Application workloads
On-prem	10.0.0.0/8	External source / destination

Notes:

• The Transit VPC (Hub) is a single IBM Cloud VPC spanning all three availability zones. The three /16 prefixes (172.26–28.0.0/16) are address ranges from which zone-specific subnets are carved — they are not separate VPCs.
• Power Virtual Server workloads can use CIDRs, such as 192.168.51.0/24 (AZ1), 192.168.52.0/24 (AZ2), 192.168.53.0/24 (AZ3). 
  
  

Subnets Used in This PoC

Transit VPC (Hub) — per zone:

Zone	Management Subnet	Data Subnet
AZ1 (us-south-1)	172.26.1.0/24	172.26.10.0/24
AZ2 (us-south-2)	172.27.1.0/24	172.27.10.0/24
AZ3 (us-south-3)	172.28.1.0/24	172.28.10.0/24

Spoke VPC-1 — per zone:

Zone	Subnet
AZ1	192.168.101.0/24
AZ2	192.168.102.0/24
AZ3	192.168.103.0/24

Spoke VPC-2 — per zone:

Zone	Subnet
AZ1	192.168.201.0/24
AZ2	192.168.202.0/24
AZ3	192.168.203.0/24

vFSA Interface IPs per Zone

Zone	Instance	port1 (Management)	port2 (Data / RMNLB back-end)	Floating IP
AZ1	vFSA-Z1	172.26.1.4	172.26.10.4	<FIP-Z1>
AZ2	vFSA-Z2	172.27.1.4	172.27.10.4	<FIP-Z2>
AZ3	vFSA-Z3	172.28.1.4	172.28.10.4	<FIP-Z3>

Substitute your actual IP addresses if they differ. Additional vFSA instances can be added to each zone's RMNLB back-end pool as needed.

RMNLB VIPs per Zone

Zone	RMNLB Name	Primary VIP	Secondary VIP	Subnet	Back-end
AZ1	rmnlb-hub-z1	172.26.10.6	172.26.10.7	172.26.10.0/24	172.26.10.4 (vFSA-Z1 port2)
AZ2	rmnlb-hub-z2	172.27.10.6	172.27.10.7	172.27.10.0/24	172.27.10.4 (vFSA-Z2 port2)
AZ3	rmnlb-hub-z3	172.28.10.6	172.28.10.7	172.28.10.0/24	172.28.10.4 (vFSA-Z3 port2)

Egress CR VIPs (used as next-hop in spoke egress route tables — recommended convention is the .100 address of each zone's data subnet):

Zone	Egress CR VIP
AZ1	172.26.10.100
AZ2	172.27.10.100
AZ3	172.28.10.100

---

Transit Gateway and Direct Link Setup

Two Transit Gateways are used in this Proof of Concept (PoC) — one connecting the Hub VPC to on-prem (acting as the external entry point) and one interconnecting the Hub VPC to each Spoke VPC. In a Direct Link deployment, replace the external Transit Gateway with your Direct Link connection.

Transit Gateway-External (On-Prem → Hub VPC)

Connect the on-premises network (or simulate it with a VPC acting as on-prem) to the Transit VPC.

• Add the Transit VPC (172.26.0.0/16, 172.27.0.0/16, 172.28.0.0/16) as connections.
• Add the on-prem / simulated on-prem VPC as a connection.
• Ensure route advertisement is enabled so that:
  • The Hub VPC learns the on-premises prefix (10.0.0.0/8 or more specific).
  • The on-premises network learns the Spoke VPC prefixes via the Hub.

Direct Link note: If using Direct Link instead, ensure the Transit VPC ingress routing table accepts traffic from the Direct Link source and that Spoke VPC prefixes are advertised back to on-prem via your BGP routing policy.

Transit Gateway-Internal (Hub VPC ↔ Spoke VPCs)

Connect the Hub VPC and both Spoke VPCs to a second Transit Gateway.

• Add connections for:
  • Transit VPC (Hub) — 172.26.0.0/16 (AZ1), 172.27.0.0/16 (AZ2), 172.28.0.0/16 (AZ3)
  • Spoke VPC-1 — 192.168.100.0/22
  • Spoke VPC-2 — 192.168.200.0/22
• Enable route advertisement on all connections so each Spoke VPC can reach the Hub and vice versa.

Validation

After setting up the Transit Gateways, validate connectivity before proceeding with firewall configuration.

From a Spoke VSI — verify route to on-prem (via Hub):

ip route get 10.0.0.4
# Expected: route via the spoke subnet gateway (Transit Gateway path through Hub)

From the on-prem VSI — verify route to a Spoke prefix:

ip route get 192.168.101.4
# Expected: route via the on-prem subnet gateway (Transit Gateway to Hub to Spoke)

At this stage, routing will reach the Hub, but no firewall is in path yet. End-to-end connectivity will only succeed after the vFSA, RMNLB, and routing tables are configured in later sections.

---

vFSA Deployment from BYOL Tile (per Zone)

Deploy one Fortinet vFSA instance per availability zone (three total). Each deployment is independent — repeat the steps below for AZ1, AZ2, and AZ3.

Deploying from IBM Cloud Catalog

1. Access the Catalog: Navigate to Fortinet FortiGate Next-Generation Firewall — Single VM

2. Configure Deployment:

   • Deployment target: IBM Cloud
   • Delivery method: Terraform (via Schematics)
   • Product version: Select available version (for example, 8.0.0 or 7.6.6)
   • Engine type: Schematics
   • Pricing plan: Bring your own license (BYOL)
     
     

3. Configure Workspace:

   • Name: for example, vfsa-hub-z1 (use z2, z3 for subsequent zones)
   • Location: Dallas (or your preferred region)
   • Resource group: Select your resource group
     
     

4. Set Input Variables — example values for AZ1:

   Variable	AZ1 Value	AZ2 Value	AZ3 Value
   cluster_name	vfsa-hub-z1	vfsa-hub-z2	vfsa-hub-z3
   region	us-south	us-south	us-south
   zone1	us-south-1	us-south-2	us-south-3
   profile	cx2-2x4	cx2-2x4	cx2-2x4
   vpc	Hub VPC name	Hub VPC name	Hub VPC name
   subnet1	172.26.1.0/24 subnet ID	172.27.1.0/24 subnet ID	172.28.1.0/24 subnet ID
   subnet2	172.26.10.0/24 subnet ID	172.27.10.0/24 subnet ID	172.28.10.0/24 subnet ID
   security_group	Hub VPC security group	Hub VPC security group	Hub VPC security group
   ssh_public_key	Your SSH key name	Your SSH key name	Your SSH key name
   ibmcloud_api_key	Your IBM Cloud API key	Your IBM Cloud API key	Your IBM Cloud API key
   user_data	Path to bootstrap config file (optional)	Path to bootstrap config file (optional)	Path to bootstrap config file (optional)

   • subnet1 maps to port1 (management).
   • subnet2 maps to port2 (data / RMNLB back-end).
   • user_data accepts a FortiOS bootstrap configuration file. Use this to pre-seed interface settings, static routes, or firewall policies at first boot. Leave empty if you prefer to apply configuration interactively after deployment.

5. Deploy:

   • Review the configuration and click Install.
   • Wait for the Schematics workspace to complete.
     
     

6. Repeat for AZ2 and AZ3: Repeat steps 1–5 with the corresponding zone values from the table.

Applying the BYOL License

After each vFSA instance is deployed, apply the BYOL license before proceeding.

1. SSH to the vFSA using the floating IP on port1:

   ssh admin@<FIP-Z1>
   

2. Apply the license token:

   execute vm-license <your-license-token>
   

3. The firewall will reboot automatically. After reboot, verify license status:

   get system status
   

   Expected output:

   Version: FortiGate-VM64-IBM v8.0.0,build0167,XXXXXX (GA.F)
   Operation Mode: NAT
   Current HA mode: standalone
   License Status: Valid
   

4. Repeat for vFSA-Z2 and vFSA-Z3.

IP Spoofing: After deployment, navigate to each vFSA instance's port2 network interface in the IBM Cloud VPC console and enable IP spoofing. This must be done at the VPC network interface level. It is separate from the FortiGate src-check setting.

PSF (Protocol State Filtering): On the same port2 network interface, disable Protocol State Filtering. PSF drops packets it considers out-of-state at the VPC level, which conflicts with asymmetric return flows handled by FGSP. This setting is independent of IP spoofing and must be explicitly disabled.

Management interface security group (port1): The BYOL catalog tile automatically attaches a Floating IP (FIP) to port1 for management access. Restrict inbound rules on the port1 security group to prevent unauthorized access:

Direction	Protocol	Source	Port	Purpose
Inbound	TCP	Your trusted admin IP (for example, 169.x.x.x/32)	443	HTTPS GUI
Inbound	TCP	Your trusted admin IP (for example, 169.x.x.x/32)	22	SSH CLI
Inbound	ICMP	Hub VPC CIDRs (172.26.0.0/16, 172.27.0.0/16, 172.28.0.0/16)	—	Connectivity testing
Outbound	TCP/UDP/ICMP	Any	Any	General

Do not expose port1 to 0.0.0.0/0. The FIP is reachable from the internet and should be locked to known management IP ranges only.

BYOL license validity: BYOL licenses have a fixed expiry date. After applying the license token (execute vm-license), note the expiry date shown in get system status and schedule renewal before expiry to avoid an enforcement grace-period that can limit throughput and new session creation.

---

vFSA FortiOS Configuration (per Zone)

The FortiOS configuration is nearly identical across all three zones. The only differences are the interface IP addresses, gateway IPs, and the static route destinations. The following examples use AZ1 values (172.26.10.x). Substitute AZ2 (172.27.10.x) and AZ3 (172.28.10.x) values for the other two instances.

Interface Configuration

Apply on each vFSA (SSH or console):

config system interface
    edit "port1"
        set vdom "root"
        set mode dhcp
        set allowaccess ping https ssh http
        set type physical
        set snmp-index 1
    next
    edit "port2"
        set vdom "root"
        set mode dhcp
        set allowaccess ping https ssh http
        set type physical
        set snmp-index 2
        set defaultgw disable
        set src-check disable
    next
end

Key settings:

• port1 — management interface with default gateway (used for FGSP peer communication and management access).
• port2 — data interface with no default gateway and src-check disable to allow hairpin routing (traffic enters and exits the same interface).
• SSH enabled on port2 is required for the RMNLB health check.

Static Routes

Traffic from on-prem needs to be forwarded to spoke VPC prefixes using port2. Add one static route per spoke VPC prefix.

vFSA-Z1 (AZ1):

config router static
    edit 1
        set dst 192.168.100.0 255.255.252.0
        set gateway 172.26.10.1
        set device "port2"
    next
    edit 2
        set dst 192.168.200.0 255.255.252.0
        set gateway 172.26.10.1
        set device "port2"
    next
    edit 3
        set dst 10.0.0.0 255.0.0.0
        set gateway 172.26.10.1
        set device "port2"
    next
end

vFSA-Z2 (AZ2): same routes with gateway 172.27.10.1

vFSA-Z3 (AZ3): same routes with gateway 172.28.10.1

The gateway IP is the IBM Cloud VPC subnet gateway for each zone's data subnet (always the first usable IP in the subnet, for example, 172.26.10.1 for 172.26.10.0/24, 172.27.10.1 for 172.27.10.0/24, 172.28.10.1 for 172.28.10.0/24).

Firewall Address Objects

config firewall address
    edit "ONPREM"
        set subnet 10.0.0.0 255.0.0.0
    next
    edit "SPOKE-VPC1"
        set subnet 192.168.100.0 255.255.252.0
    next
    edit "SPOKE-VPC2"
        set subnet 192.168.200.0 255.255.252.0
    next
end

Firewall Policies

All traffic in this design enters and exits via port2 (same-interface hairpin routing). Create policies for both directions.

config firewall policy
    edit 1
        set name "ONPREM-to-SPOKE"
        set srcintf "port2"
        set dstintf "port2"
        set action accept
        set srcaddr "ONPREM"
        set dstaddr "SPOKE-VPC1" "SPOKE-VPC2"
        set schedule "always"
        set service "ALL"
        set logtraffic all
    next
    edit 2
        set name "SPOKE-to-ONPREM"
        set srcintf "port2"
        set dstintf "port2"
        set action accept
        set srcaddr "SPOKE-VPC1" "SPOKE-VPC2"
        set dstaddr "ONPREM"
        set schedule "always"
        set service "ALL"
        set logtraffic all
    next
end

Tighten these policies by replacing "ALL" service with specific allowed protocols and ports after initial validation.

Routing Table Verification

After configuration, verify the routing table on each vFSA:

get router info routing-table all

Expected output on vFSA-Z1:

Routing table for VRF=0
S*      0.0.0.0/0 [10/0] via 172.26.1.1, port1, [1/0]
S       10.0.0.0/8 [10/0] via 172.26.10.1, port2, [1/0]
S       192.168.100.0/22 [10/0] via 172.26.10.1, port2, [1/0]
S       192.168.200.0/22 [10/0] via 172.26.10.1, port2, [1/0]
C       172.26.1.0/24 is directly connected, port1
C       172.26.10.0/24 is directly connected, port2

vFSA-Z2 expected routing table: same routes with gateway 172.27.10.1; connected networks 172.27.1.0/24 (port1) and 172.27.10.0/24 (port2).

vFSA-Z3 expected routing table: same routes with gateway 172.28.10.1; connected networks 172.28.1.0/24 (port1) and 172.28.10.0/24 (port2).

Verify system status:

get system status

Expected:

Version: FortiGate-VM64-IBM v8.0.0,build0167,XXXXXX (GA.F)
Operation Mode: NAT
Current HA mode: standalone configsync

---

RMNLB Setup per Zone

Deploy one Route Mode NLB per availability zone (three total). Each RMNLB is independent, lives in the data subnet of its zone, and has the local vFSA instance(s) as its back-end pool member(s).

Before you begin — IAM service-to-service authorization required: Route Mode NLBs require a service-to-service authorization policy before they can be created. Without it, the RMNLB creation will fail with a permissions error.

In IBM Cloud IAM → Authorizations, create the following authorization:

Field	Value
Source service	VPC Infrastructure Services — scope: Load Balancer for VPC
Target service	VPC Infrastructure Services — scope: Virtual Private Cloud
Role	Editor

This authorization must be in place once per account before any Route Mode NLB can be created. See Before you begin — NLB route mode for the full IBM Cloud documentation.

RMNLB Overview

Create each RMNLB with the following characteristics:

Setting	Value
Type	Private (Route Mode NLB)
Routing mode	Route Mode (not application load balancer)
Session stickiness	None — RMNLB distributes based on source IP hash
Pool algorithm	Weighted Forwarding (weighted_forwarding) — required for Route Mode NLB; other algorithms will not work
Subnet	Zone-specific data subnet (see section Subnets Used in This PoC)

Zone-by-zone summary:

Zone	Name	Primary VIP	back-end IP
AZ1	rmnlb-hub-z1	172.26.10.6	172.26.10.4 (vFSA-Z1 port2)
AZ2	rmnlb-hub-z2	172.27.10.6	172.27.10.4 (vFSA-Z2 port2)
AZ3	rmnlb-hub-z3	172.28.10.6	172.28.10.4 (vFSA-Z3 port2)

Back-end Pool and Health Checks

For each RMNLB, create a back-end pool with the zone's vFSA port2 IP as the sole member (scale out by adding more members):

Back-end pool configuration:

• Protocol: TCP
• Port: Any (RMNLB in route mode passes all protocols)
• Members: vFSA port2 IP for the zone

Health check configuration:

• Protocol: TCP
• Port: 22 (SSH — must be allowed on vFSA port2 and in security group)
• Interval: 5 seconds
• Timeout: 2 seconds
• Max retries: 2

If health checks are failing, verify:

1. SSH (port 22) is allowed inbound on the vFSA security group from the Hub VPC CIDR.
2. set allowaccess ssh is set on vFSA port2.
3. IP spoofing is enabled on the port2 VPC network interface.
4. PSF (Protocol State Filtering) is disabled on the port2 VPC network interface.
5. The back-end member IP matches the vFSA port2 IP (not port1).

Important Design Notes

• Routing table next-hop must point to the RMNLB VIP, not directly to the vFSA back-end IP. The RMNLB is what monitors health and withdraws routes on failure.
• The RMNLB VIP is what you use in all custom route table entries for this zone.
• When a vFSA fails its health check, the RMNLB automatically withdraws the route advertisement for the routes that reference its VIP. IBM Cloud then promotes the next-priority route in the routing table, shifting traffic to another zone's RMNLB.
• RMNLB does not pass ICMP by default in route mode. Use TCP-based tests (HTTP, SSH) for end-to-end validation.

---

Routing Tables with Priority Routing

This section is the most distinctive aspect of this design compared to the spoke-to-spoke blog. Three separate custom route tables are required, each using tiered priority values to enforce zone-local traffic preferences in steady state and automatic cross-zone failover when a zone becomes unavailable.

Why Priority Routing Matters

In the spoke-to-spoke design, a single RMNLB VIP is the next-hop for all spoke prefixes. There is only one zone and one firewall layer, so no preference needs to be expressed.

In this three-zone design, each spoke prefix is reachable through three different zone-local firewalls. Without priority routing, IBM Cloud would treat all three next-hops as equal-cost paths and could route traffic to any zone — making the return path unpredictable and breaking stateful firewall session symmetry.

Priority routing solves this by expressing a preference order per prefix per zone:

Priority Value	Meaning
2 (default)	Preferred — use this path in normal operation
3	Backup — use only if priority-2 path is unavailable
4	Last resort — use only if both priority-2 and priority-3 paths are unavailable

When an RMNLB detects that its vFSA back-end is unhealthy, it withdraws its advertised routes. IBM Cloud automatically activates the next-highest-priority route for the same destination.

Transit VPC Ingress Transit Gateway Custom Route Table

This route table is attached to the Transit VPC's ingress routing table for traffic arriving from Transit Gateway (both external on-prem traffic and internal spoke-to-spoke traffic that flows through the Hub).

Traffic source: Transit Gateway

Zone	Destination	Next-Hop	Advertise	Priority	Purpose
AZ1	10.0.0.0/8 (on-prem)	RMNLB-Z1 (172.26.10.6)	Yes	2 (default)	Preferred path for on-prem traffic via AZ1 FW
AZ2	10.0.0.0/8 (on-prem)	RMNLB-Z2 (172.27.10.6)	Yes	2 (default)	Preferred path for on-prem traffic via AZ2 FW
AZ3	10.0.0.0/8 (on-prem)	RMNLB-Z3 (172.28.10.6)	Yes	2 (default)	Preferred path for on-prem traffic via AZ3 FW
AZ1	0.0.0.0/0	RMNLB-Z1	Yes	2 (default)	Default route — covers all spoke traffic via AZ1
AZ2	0.0.0.0/0	RMNLB-Z2	Yes	2 (default)	Default route — covers all spoke traffic via AZ2
AZ3	0.0.0.0/0	RMNLB-Z3	Yes	2 (default)	Default route — covers all spoke traffic via AZ3
AZ1	Egress CR VIP-Z1 (172.26.10.100)	RMNLB-Z1	Yes	2 (default)	Zone 1 return path — preferred
AZ2	Egress CR VIP-Z1 (172.26.10.100)	RMNLB-Z2	Yes	3	Zone 1 return path — backup via AZ2
AZ3	Egress CR VIP-Z1 (172.26.10.100)	RMNLB-Z3	Yes	4	Zone 1 return path — last resort via AZ3
AZ2	Egress CR VIP-Z2 (172.27.10.100)	RMNLB-Z2	Yes	2 (default)	Zone 2 return path — preferred
AZ3	Egress CR VIP-Z2 (172.27.10.100)	RMNLB-Z3	Yes	3	Zone 2 return path — backup via AZ3
AZ1	Egress CR VIP-Z2 (172.27.10.100)	RMNLB-Z1	Yes	4	Zone 2 return path — last resort via AZ1
AZ3	Egress CR VIP-Z3 (172.28.10.100)	RMNLB-Z3	Yes	2 (default)	Zone 3 return path — preferred
AZ1	Egress CR VIP-Z3 (172.28.10.100)	RMNLB-Z1	Yes	3	Zone 3 return path — backup via AZ1
AZ2	Egress CR VIP-Z3 (172.28.10.100)	RMNLB-Z2	Yes	4	Zone 3 return path — last resort via AZ2

How this works end-to-end:

The Egress CR VIP is a dedicated unused IP from each zone's data subnet. The recommended convention is to use the .100 address (for example, Z1: 172.26.10.100, Z2: 172.27.10.100, Z3: 172.28.10.100). Using a consistent, predictable offset per zone avoids confusion and accidental conflicts. Spoke workloads in each zone send their outbound (return) traffic to their zone's VIP as the next-hop. When that VIP-destined traffic arrives back at the Transit VPC, this routing table matches it and sends it to the correct RMNLB — priority 2 (local zone RMNLB) in steady state, or priority 3/4 if that zone is unavailable.

Transit VPC Ingress Direct Link Custom Route Table

This route table is attached to the Transit VPC's ingress routing table for traffic arriving from Direct Link (on-prem → cloud). If you are using Transit Gateway for the external connection instead of Direct Link, skip this table and rely on Section 9.2.

Traffic source: Direct Link

Zone	Destination	Next-Hop	Advertise	Priority	Purpose
AZ1	192.168.101.0/24 (Spoke VPC-1, AZ1)	RMNLB-Z1	Yes	2 (default)	Preferred — AZ1 FW for AZ1 workloads
AZ2	192.168.101.0/24	RMNLB-Z2	Yes	3	Backup — AZ2 FW for AZ1 workloads
AZ3	192.168.101.0/24	RMNLB-Z3	Yes	4	Last resort — AZ3 FW for AZ1 workloads
AZ2	192.168.102.0/24 (Spoke VPC-1, AZ2)	RMNLB-Z2	Yes	2 (default)	Preferred — AZ2 FW for AZ2 workloads
AZ3	192.168.102.0/24	RMNLB-Z3	Yes	3	Backup
AZ1	192.168.102.0/24	RMNLB-Z1	Yes	4	Last resort
AZ3	192.168.103.0/24 (Spoke VPC-1, AZ3)	RMNLB-Z3	Yes	2 (default)	Preferred — AZ3 FW for AZ3 workloads
AZ1	192.168.103.0/24	RMNLB-Z1	Yes	3	Backup
AZ2	192.168.103.0/24	RMNLB-Z2	Yes	4	Last resort
AZ1	192.168.201.0/24 (Spoke VPC-2, AZ1)	RMNLB-Z1	Yes	2 (default)	Preferred
AZ2	192.168.201.0/24	RMNLB-Z2	Yes	3	Backup
AZ3	192.168.201.0/24	RMNLB-Z3	Yes	4	Last resort
AZ2	192.168.202.0/24 (Spoke VPC-2, AZ2)	RMNLB-Z2	Yes	2 (default)	Preferred
AZ3	192.168.202.0/24	RMNLB-Z3	Yes	3	Backup
AZ1	192.168.202.0/24	RMNLB-Z1	Yes	4	Last resort
AZ3	192.168.203.0/24 (Spoke VPC-2, AZ3)	RMNLB-Z3	Yes	2 (default)	Preferred
AZ1	192.168.203.0/24	RMNLB-Z1	Yes	3	Backup
AZ2	192.168.203.0/24	RMNLB-Z2	Yes	4	Last resort

Spoke VPC Egress Custom Route Table

This route table is applied to each subnet in each Spoke VPC. It steers outbound traffic (toward on-prem or other spokes) to the Egress CR VIP for the local zone's RMNLB.

Traffic source: VPC zone (applied per subnet)

Zone	Destination	Next-Hop (Egress CR VIP)	Advertise	Action	Priority
AZ1 subnets	10.0.0.0/8 (on-prem)	172.26.10.100 (VIP-Z1)	No	Deliver	2 (default)
AZ1 subnets	192.168.200.0/22 (Spoke VPC-2)	172.26.10.100 (VIP-Z1)	No	Deliver	2 (default)
AZ1 subnets	Local VPC prefixes	—	No	Delegate	2 (default)
AZ2 subnets	10.0.0.0/8 (on-prem)	172.27.10.100 (VIP-Z2)	No	Deliver	2 (default)
AZ2 subnets	192.168.200.0/22 (Spoke VPC-2)	172.27.10.100 (VIP-Z2)	No	Deliver	2 (default)
AZ2 subnets	Local VPC prefixes	—	No	Delegate	2 (default)
AZ3 subnets	10.0.0.0/8 (on-prem)	172.28.10.100 (VIP-Z3)	No	Deliver	2 (default)
AZ3 subnets	192.168.200.0/22 (Spoke VPC-2)	172.28.10.100 (VIP-Z3)	No	Deliver	2 (default)
AZ3 subnets	Local VPC prefixes	—	No	Delegate	2 (default)

Notes:

• The Egress CR VIP (172.26.10.100 for AZ1, 172.27.10.100 for AZ2, 172.28.10.100 for AZ3) is a dedicated unused IP from the respective zone's RMNLB subnet. It does not need to be assigned to any instance. The RMNLB sees traffic destined to this VIP and routes it to the healthy vFSA back-end.
• A default route (0.0.0.0/0) can be used instead of specific spoke/on-prem prefixes to simplify configuration.
• Delegate action for local prefixes ensures that traffic within the same VPC does not get unnecessarily forwarded to the firewall.
• Apply the correct zone's egress route table to subnets in that zone. AZ1 subnets use VIP-Z1, AZ2 subnets use VIP-Z2, AZ3 subnets use VIP-Z3.
• Apply the same pattern to Spoke VPC-2 subnets, replacing 192.168.200.0/22 with 192.168.100.0/22 as the destination for the other spoke.

---

FGSP Configuration for 3-Node Session Synchronization

FortiGate Session Life Support Protocol (FGSP) enables session state sharing across all three vFSA instances. In a three-zone Active/Active/Active design, this is what allows a surviving firewall to continue processing return traffic for a session that was originally established on a failed firewall in a different zone.

FGSP Benefits in a 3-Zone Cluster

• Asymmetric traffic handling: After a zone failover, return traffic might arrive at a different vFSA than the one that created the session. FGSP ensures the receiving vFSA has the session state to process it correctly.
• Zero-disruption failover: Existing TCP sessions survive firewall instance failures without resetting.
• NAT session sync: NAT translations are synchronized so return flows match correctly.
• Connectionless session sync: UDP and ICMP flows are also synchronized.
• Configuration sync: FortiOS policy configuration can be kept consistent across peers with standalone-config-sync enable.

FGSP Configuration per Node

Each vFSA has the same standalone-group-id and a unique group-member-id. Each node lists the port1 (management) IPs of its two peers.

On vFSA-Z1 (AZ1) — group-member-id 1:

config system standalone-cluster
    set standalone-group-id 1
    set group-member-id 1
    config cluster-peer
        edit 1
            set peerip 172.27.1.4
        next
        edit 2
            set peerip 172.28.1.4
        next
    end
end


config system ha
    set session-pickup enable
    set session-pickup-connectionless enable
    set session-pickup-expectation enable
    set session-pickup-nat enable
    set standalone-config-sync enable
    set override disable
end

On vFSA-Z2 (AZ2) — group-member-id 2:

config system standalone-cluster
    set standalone-group-id 1
    set group-member-id 2
    config cluster-peer
        edit 1
            set peerip 172.26.1.4
        next
        edit 2
            set peerip 172.28.1.4
        next
    end
end


config system ha
    set session-pickup enable
    set session-pickup-connectionless enable
    set session-pickup-expectation enable
    set session-pickup-nat enable
    set standalone-config-sync enable
    set override disable
end

On vFSA-Z3 (AZ3) — group-member-id 3:

config system standalone-cluster
    set standalone-group-id 1
    set group-member-id 3
    config cluster-peer
        edit 1
            set peerip 172.26.1.4
        next
        edit 2
            set peerip 172.27.1.4
        next
    end
end


config system ha
    set session-pickup enable
    set session-pickup-connectionless enable
    set session-pickup-expectation enable
    set session-pickup-nat enable
    set standalone-config-sync enable
    set override disable
end

FGSP Configuration Details:

Parameter	Value	Description
standalone-group-id	1	Same on all nodes — identifies the cluster
group-member-id	1, 2, 3	Unique per node
peerip	Port1 IP of each peer	FGSP uses port1 (management) for peer communication
session-pickup enable	—	Enables TCP session synchronization
session-pickup-connectionless enable	—	Enables UDP/ICMP session sync
session-pickup-nat enable	—	Syncs NAT translation tables
standalone-config-sync enable	—	Keeps FortiOS config in sync across peers
FGSP protocol	UDP 708	Session sync traffic
Heartbeat	UDP 703	HA heartbeat between peers

FGSP Verification

Check FGSP synchronization status on any node:

diagnose sys session sync

Expected output (vFSA-Z1):

sync_ctx: sync_started=1, sync_tcp=1, sync_others=1,
sync_expectation=1, sync_nat=1, stdalone_sesync=1, asymmetric_traffic_control=0.
sync: create=NNN:0, update=NNN, keepalive=0, delete=0:0
recv: create=NNN:0, update=NNN, keepalive=0, delete=0:0
udp pkts: send=NNNN, recv=NNNN
sync_filter:
    1: peerid=2, peerip=172.27.1.4
    2: peerid=3, peerip=172.28.1.4

Key indicators to check:

• sync_started=1 — FGSP is active.
• stdalone_sesync=1 — standalone session sync is enabled.
• sync: create=NNN — sessions are being sent to peers (count increases with traffic).
• recv: create=NNN — sessions are being received from peers.
• peerip values match the port1 IPs of the other two vFSA nodes.
• No errors in recv_err, sz_err, or ses_convert_err.

Check HA mode:

get system status

Look for:

Current HA mode: standalone configsync

Monitor synchronized sessions:

diagnose sys session filter src 10.0.0.4
diagnose sys session list

Sessions should appear on all three vFSA nodes even when only one is actively forwarding the traffic — confirming cross-zone session synchronization is working.

---

End-to-End Test: On-Prem VSI to Spoke Server

Test Setup

Start a simple HTTP listener on a Spoke VSI in Zone 1 (AZ1):

# On Spoke VPC-1, AZ1 VSI (192.168.101.4)
python3 -m http.server 8080

Ensure the Spoke VSI security group allows inbound TCP port 8080 from the on-prem CIDR (10.0.0.0/8).

Traffic Test and Expected Output

From the on-prem VSI (10.0.0.4):

curl -v http://192.168.101.4:8080/

Expected output — successful on-prem-to-vpc traffic traversal:

*   Trying 192.168.101.4:8080...
* Connected to 192.168.101.4 (192.168.101.4) port 8080 (#0)
> GET / HTTP/1.1
> Host: 192.168.101.4:8080
> User-Agent: curl/7.81.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
* HTTP 1.0, assume close after body
< HTTP/1.0 200 OK
< Server: SimpleHTTP/0.6 Python/3.10.12
< Date: <timestamp>
< Content-type: text/html; charset=utf-8
< Content-Length: NNN
<
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" ...>
...
* Closing connection 0

Spoke server log (confirms source IP is the on-prem client):

ubuntu@spoke-vsi-z1:~$ python3 -m http.server 8080
Serving HTTP on 0.0.0.0 port 8080 (http://0.0.0.0:8080/) ...
10.0.0.4 - - [DD/Mon/YYYY HH:MM:SS] "GET / HTTP/1.1" 200 -

This confirms that the on-prem client IP (10.0.0.4) is preserved through the firewall — the vFSA is forwarding the packet, not performing NAT.

Run multiple requests to verify active load distribution across zones:

# On on-prem VSI — run 6 requests
for i in {1..6}; do
    echo "===== Request $i ====="
    curl -s http://192.168.101.4:8080/ > /dev/null && echo "OK"
    sleep 0.5
done

vFSA Session Verification

After running the test, verify that sessions are being processed and synchronized across all three vFSA nodes.

On vFSA-Z1 — check active sessions:

diagnose sys session filter src 10.0.0.4
diagnose sys session filter dst 192.168.101.4
diagnose sys session list

Expected session output:

session info: proto=6 proto_state=01 duration=2 expire=3598 timeout=3600
  src: 10.0.0.4:NNNNN dst: 192.168.101.4:8080
  origin-dir: org=10.0.0.4:NNNNN reply=192.168.101.4:8080
  reply-dir: org=192.168.101.4:8080 reply=10.0.0.4:NNNNN
  state=log may_dirty npu

On vFSA-Z2 and vFSA-Z3 — verify synchronized sessions:

diagnose sys session filter src 10.0.0.4
diagnose sys session list

Sessions originally created on vFSA-Z1 should also appear on vFSA-Z2 and vFSA-Z3 (with potentially a short FGSP sync delay), confirming that FGSP cross-zone synchronization is active.

Live packet capture on the vFSA handling AZ1 traffic:

diagnose sniffer packet port2 "host 10.0.0.4 and host 192.168.101.4 and tcp and port 8080" 4

Expected: bidirectional traffic visible on port2 (SYN, SYN-ACK, PSH/data, FIN) with the on-prem source IP and spoke destination IP, confirming the On-Prem to VPC traffic path through the firewall.

---

Failover Scenarios

Scenario 1: Single-Zone Firewall Failure

Condition: vFSA in one zone fails or becomes unreachable.

What happens:

1. RMNLB detects back-end failure via health check (TCP port 22 timeout after max retries).
2. RMNLB withdraws its advertised route entries.
3. IBM Cloud promotes the next-highest-priority route (priority 3) in the routing table.
4. Traffic for affected prefixes is now forwarded to a healthy zone's RMNLB.
5. FGSP ensures the surviving vFSA nodes have session state for existing connections.

Impact:

• New sessions: Routed immediately to a healthy zone — no impact.
• Existing sessions: FGSP-synchronized sessions continue without interruption on the surviving firewalls.
• Sessions in flight during the failover convergence window may briefly traverse different firewalls for forward and return paths — FGSP handles these transparently.

Recovery: When the failed vFSA recovers, RMNLB detects it as healthy, re-advertises the priority-2 route, and traffic automatically returns to the preferred zone.

Scenario 2: Complete Zone Failure

Condition: An entire availability zone (compute, network, storage) becomes unavailable.

What happens:

• All resources in the zone (RMNLB, vFSA, subnets) become unreachable.
• RMNLB route advertisements for the failed zone are withdrawn.
• Traffic redistributes to the two healthy zones using their priority-3 and priority-4 routes.
• The FGSP cluster operates with two active members.

Impact: Capacity is reduced to two-thirds. Existing sessions continue via FGSP on the surviving two nodes.

Scenario 3: Two-Zone Failure

Condition: Two zones fail simultaneously.

What happens:

• All traffic concentrates on the single remaining zone (priority-4 routes activate for affected prefixes).
• FGSP operates with a single member (no active peer sync possible).
• Significant capacity reduction — plan capacity for this scenario if it is a business continuity requirement.

Scenario 4: FGSP Session Walk-Through During Failover

A concrete example of FGSP in action during a single-zone failure:

1. Steady state: Client (10.0.0.4) connects to Spoke VSI (192.168.101.4). Traffic flows through vFSA-Z1. The session is synchronized to vFSA-Z2 and vFSA-Z3.

2. Failure: vFSA-Z1 fails. RMNLB-Z1 withdraws routes. IBM Cloud activates the priority-3 path via RMNLB-Z2.

3. Return traffic: The response from Spoke VSI (192.168.101.4) now arrives at vFSA-Z2. vFSA-Z2 looks up its session table (populated by FGSP sync), finds the session, and processes the return packet correctly.

4. Client view: The TCP session continues. The client sees no interruption beyond a brief pause while RMNLB detects the failure and routes shift.

---

Summary and Key Takeaways

What This Design Adds Beyond the Spoke-to-Spoke Design

This document extends the foundational vFSA + RMNLB + FGSP pattern from a single-zone, east-west design to a three-zone, on-prem-to-VPC traffic design. The core technology is the same; the operational envelope is broader:

Capability	Spoke-to-Spoke	This Design
Traffic direction	East-West (cloud internal)	North-South (on-prem to cloud)
Zone coverage	Single zone	Three zones — Active/Active/Active
Firewall instances	2	3 (one per zone, independently scalable)
Load balancers	1	3 (one per zone)
Routing model	Flat static routes	Tiered priority routing (2/3/4)
Failover mechanism	RMNLB + FGSP	RMNLB route withdrawal + priority promotion + FGSP
External connectivity	Not applicable	Direct Link and/or Transit Gateway

Critical Configuration Checklist

Before declaring the design production-ready, verify each item:

• [  ] Three vFSA instances deployed from BYOL catalog tile — one per zone
• [  ] BYOL license applied and validated (get system status shows License Status: Valid)
• [  ] BYOL license expiry date noted and tracked — renew before expiry to avoid throughput throttling
• [  ] IP spoofing enabled on port2 VPC network interface for each vFSA
• [  ] PSF (Protocol State Filtering) disabled on port2 VPC network interface for each vFSA
• [  ] src-check disable set on port2 of each vFSA
• [  ] Static routes to spoke and on-prem prefixes via port2 on each vFSA
• [  ] Firewall policies for ONPREM-to-SPOKE and SPOKE-to-ONPREM on each vFSA
• [  ] Three RMNLB instances deployed — one per zone, in correct data subnet
• [  ] RMNLB health checks passing (green members) on all three RMNLBs
• [  ] Transit VPC Ingress Transit Gateway custom route table created with tiered priorities
• [  ] Transit VPC Ingress Direct Link custom route table created (if using Direct Link)
• [  ] Spoke VPC egress custom route tables created per zone with correct Egress CR VIPs
• [  ] FGSP 3-node cluster configured on all vFSA instances
• [  ] FGSP sync verified (diagnose sys session sync shows sync_started=1)
• [  ] End-to-end test: curl from on-prem VSI to Spoke VSI returns HTTP 200
• [  ] Session verification: session visible on vFSA handling the flow with correct source/destination IPs
• [  ] Failover test: stop vFSA in one zone — confirm traffic resumes via backup zone

---

This design was validated on FortiOS v8.0.0 (build 0167) on IBM Cloud VPC using cx2-2x4 VSI profiles.