Repository navigation
Feature/active active lbaas - #13
volehuy1998 wants to merge 34 commits into
Conversation
|
OCTAVIA ACTIVE/ACTIVE IMPLEMENTATION TECHNICAL REVIEW Date: 28 September 2026 Community design reference: Related distributor design:
This report reviews the correctness of the Octavia Active/Active implementation developed for OpenStack Caracal 2024.1. The implementation is based on the historical Octavia community Active/Active design and source-level references, but it is not a direct cherry-pick of a complete upstream implementation. The historical code was written against significantly older Octavia APIs, TaskFlow structures, database models, provider interfaces, and Amphora agent APIs. The implementation therefore ports the original architectural concepts into the current Octavia 2024.1 codebase. Two distributor deployment modes are implemented:
The distributor is a Nova VM created by Octavia inside the OpenStack cloud. This mode is intended for environments where no dedicated external distributor hardware is available.
The distributor is an existing physical server, virtual appliance, or dedicated device outside the Nova lifecycle. In the current implementation, the external distributor is an Ubuntu server running Open vSwitch, OVN controller, an Octavia distributor agent, and a custom OVN VIF-plugging mechanism.
The original community design uses two layers of load balancing. The first layer is the Distributor. The second layer consists of multiple active Amphorae running HAProxy. A simplified topology is: Client An additional Amphora exists as an in-cluster standby. For N=2 active Amphorae, the expected topology is therefore: 2 x IN_CLUSTER The Distributor does not replace HAProxy. Its primary purpose is to select which active Amphora receives a new connection. HAProxy inside the selected Amphora then performs the actual load balancing toward backend pool members. This is an important distinction. The architecture is not equivalent to placing two independent full load balancers in parallel. The Distributor performs lightweight frontend connection distribution, while the Amphora performs the normal Octavia HAProxy processing.
The implementation correctly follows the N+1 Amphora concept. With: active_active_desired_amphorae = 2 Octavia creates: IN_CLUSTER Only Amphorae with the IN_CLUSTER role are registered in the Distributor forwarding group. The IN_CLUSTER_STANDBY Amphora is not included in normal packet distribution. This is consistent with the community design. During failure handling, the intended behavior is:
This preserves the topology: N active Amphorae plus one standby Amphora. The implementation therefore correctly represents the fundamental N+1 concept of the original Active/Active design.
The Active/Active model requires multiple Amphorae to use the same VIP. However, the Amphorae must not independently claim ownership of the VIP through normal ARP behavior. The implementation adds ARP suppression for Active/Active Amphorae. This is consistent with the original architecture. The Distributor owns the frontend forwarding responsibility and selects the destination Amphora by rewriting Layer 2 information. The Amphora still receives traffic addressed to the VIP at Layer 3, but the Ethernet destination MAC identifies the selected Amphora. Conceptually: IP destination: Ethernet destination: This allows several active Amphorae to process the same VIP without using the normal Active/Standby VRRP ownership mechanism.
The current OVS Distributor implementation uses an OpenFlow select group. The forwarding behavior is approximately: priority=200,ip,nw_dst= The group contains one bucket for each active Amphora. Each bucket performs approximately: set destination MAC = selected Amphora MAC The group uses: type=select This design is technically reasonable. The purpose is not to NAT the connection inside the Distributor. The IP destination remains the VIP. Only Layer 2 addresses are changed so that the packet is delivered to a selected active Amphora. The use of dp_hash is also appropriate for this design because OVS can consistently distribute flows based on a hash rather than making a random decision for every packet. The current implementation therefore follows the primary forwarding idea of the community Distributor design.
The Distributor is designed as a one-legged forwarding device. The packet can enter and leave through the same frontend interface. For this reason, the OpenFlow action: output:in_port is intentional. In Open vSwitch, ordinary output to the numeric port that was also used as the packet input port is suppressed. The special in_port output action exists specifically to allow explicit hairpin forwarding back through the ingress port. Therefore, replacing: output:in_port with an ordinary: output: would be incorrect. The implementation's use of output:in_port is consistent with Open vSwitch hairpin semantics.
There is one major architectural difference between the current implementation and the historical community design. The original Active/Active Distributor design intended the Distributor primarily for ingress traffic. Conceptually: Request: Client Response: Backend The return traffic was intended to bypass the Distributor. This is a form of direct return architecture. The current OVN-based implementation does not use this exact model. During testing, direct Amphora return traffic caused problems with the OVN conntrack/NAT path. In particular, the return path could cause subsequent packets to be treated as invalid by OVN conntrack because the two directions did not follow the same logical connection path. The implementation therefore adds a reverse hairpin rule. Conceptually: Request: Client Response: Amphora The reverse OpenFlow rule rewrites: destination MAC = recorded gateway MAC and hairpins the packet back through the Distributor frontend. This is a deliberate adaptation to the OVN environment. It should not be described as identical to the original community direct-return datapath. The implementation is better described as: Active/Active Amphorae behind a centralized one-legged OVS Distributor with symmetric OVN forwarding. The consequence is important. The Distributor now participates in both traffic directions. Therefore its forwarding capacity can become a throughput limitation, and its failure affects the entire load balancer.
The Internal Distributor VM mode is currently the implementation that most closely follows the historical community reference architecture. For every Active/Active load balancer, Octavia creates: 1 Distributor VM For example: LB-A plus: Amphora-A3 = standby A second load balancer creates a separate Distributor: LB-B plus: Amphora-B3 = standby This provides an important isolation property. Each load balancer has its own: Distributor VM This architecture naturally supports overlapping tenant networks. For example, the following is valid: Tenant A: Tenant B: Because each Distributor VM is isolated, the identical IP addresses do not collide. This makes Internal Distributor VM mode structurally safer for multi-tenant OpenStack environments.
Although Internal Distributor VM mode is architecturally reasonable, it is not yet production-complete. The first major limitation is Distributor availability. The topology currently contains only one Distributor per load balancer. Therefore: Active Amphora A failure: Active Amphora B failure: Distributor failure: The Distributor therefore remains a single point of failure. The community design also recognized this limitation. The initial implementation allowed a single Distributor, while Distributor redundancy was considered an additional availability requirement. The current code does not yet provide full Distributor HA.
The implementation can wait for the Distributor agent during provisioning. However, provisioning-time connectivity is not equivalent to continuous Distributor health monitoring. A production implementation should continuously detect at least: Distributor process failure The current implementation contains runtime-state reconciliation mechanisms, but it does not yet provide a complete equivalent of a dedicated Distributor health-management architecture. This distinction is important. A Distributor that was reachable during load-balancer creation is not guaranteed to remain healthy later.
Another significant issue is compute placement. The Active/Standby Octavia implementation contains anti-affinity logic for Amphora placement. The current Active/Active flow does not appear to provide equivalent anti-affinity behavior for all Active/Active Amphorae. Without anti-affinity, Nova could schedule: Active Amphora A on the same compute host. If that compute host fails, the entire N+1 Amphora topology disappears simultaneously. This defeats the failure-domain objective of Active/Active. Production deployment therefore requires explicit scheduling rules so that the Amphorae are distributed across appropriate failure domains.
The historical community Active/Active design discusses synchronization of HAProxy persistence state between active Amphorae. This is important when load balancer session persistence is used. If persistence information exists only on one Amphora, a connection or client that is later distributed to another Amphora may lose its persistence mapping. HAProxy supports peer synchronization mechanisms for stick tables. The current implementation does not yet demonstrate a complete implementation of this community persistence-state synchronization design. Therefore successful basic HTTP or TCP balancing does not prove that all Octavia session-persistence behaviors are correctly supported in Active/Active mode.
The idea of an external Distributor is valid. The historical Distributor design explicitly allowed the possibility of external vendor-specific hardware. However, the current external implementation should be understood as a local extension. The historical specification defines the Distributor role and forwarding behavior, but it does not provide the complete generic external-device lifecycle currently implemented in the branch. The current external peer is not simply an IP address representing some hardware appliance. It currently needs to behave as an OpenStack networking participant. It runs: Open vSwitch Therefore the current external implementation is effectively an external Linux OVS/OVN Distributor appliance.
The external mode successfully demonstrates several important control-plane concepts. Octavia can: create an external Distributor database record allocate the frontend port request binding of the frontend logical port to the external OVN chassis communicate with the Distributor agent using mTLS send VIP state to the Distributor register active Amphora information skip Nova Distributor VM creation The external peer can: appear as an OVN chassis receive Geneve traffic create local OVS connectivity for the requested frontend logical port maintain persistent Distributor state install OpenFlow forwarding state These results demonstrate that using an external distributor instead of a Nova-created VM is technically possible.
The current external implementation contains a major architectural issue. The configuration contains a single: external_distributor_id The implementation reuses this value as the Distributor ID for every Active/Active load balancer handled by that external peer. At the same time, each load balancer creation attempts to create a Distributor database row. The Distributor ID is the database primary key. This creates a conflict between two different concepts: physical Distributor device identity and per-load-balancer Distributor instance identity These must not be the same abstraction. For example, one physical external device may serve: LB-A The physical device identity should remain constant. However, each load balancer still requires its own unique Distributor attachment/state object. The current model effectively attempts to represent all of them using one Distributor ID. That model is not correct for a shared external Distributor.
The external architecture should separate the physical device from the per-load-balancer attachment. Conceptually: ExternalDistributorDevice id Then: DistributorAttachment-A unique distributor attachment ID DistributorAttachment-B unique distributor attachment ID This allows one physical distributor device to host many isolated load-balancer datapaths without reusing the same database primary key.
The current external implementation uses a shared OVS bridge: br-dist The custom VIF plugger effectively creates connectivity similar to: br-int br-int br-int This is significantly different from the isolation model described by the historical Distributor specification. The historical design expects load-balancer interfaces not to route or leak traffic between independent load balancers. A shared br-dist removes an important per-LB isolation boundary.
The shared external bridge becomes especially dangerous in a multi-tenant OpenStack environment. OpenStack allows overlapping address spaces. For example: Tenant A: Tenant B: OVN can distinguish these tenants because each packet belongs to a different logical datapath. The current Distributor state, however, uses the VIP address as a major identifier. Conceptually: state["vips"][vip] The OpenFlow group ID is also derived from the VIP. If both tenants have: 10.0.0.100 then the external shared Distributor can no longer distinguish those two load balancers using the VIP alone. The OpenFlow match also primarily uses: nw_dst= without sufficient logical-network context. Therefore: Tenant A 10.0.0.100 and: Tenant B 10.0.0.100 can collide inside the shared external bridge. This is a production-blocking issue for a public cloud where overlapping tenant address space is expected.
The implementation also configures the OVS bridge hardware address from the load-balancer frontend MAC. This is safe when every load balancer has its own bridge. For example: br-aa-LB-A br-aa-LB-B However, with a shared external bridge: br-dist creating LB-B can change the bridge hardware address that was previously configured for LB-A. This again demonstrates that the current OVS design was originally safer when applied to a per-load-balancer bridge. The same issue extends to Linux-side VIP addresses, gateway resolution, and neighbor state when several overlapping tenant networks share one Linux network context.
The external Distributor should preserve per-load-balancer isolation even if the same physical server hosts many load balancers. A possible design is: OVN br-int Each per-LB bridge should have its own: frontend interface The state key should use the load balancer UUID or Distributor attachment UUID, not only the VIP address. The group ID should also be derived from a unique load-balancer identifier rather than only the VIP. For example: group_id = hash(load_balancer_uuid) or an equivalent collision-safe allocation mechanism. If Linux L3 operations are required, stronger isolation using network namespaces or VRFs should also be considered. This is especially important for overlapping tenant IP addresses and gateway addresses.
Both current modes still contain a fundamental limitation: one Distributor handles the complete Active/Active Amphora cluster. The Amphora layer is Active/Active. The Distributor layer is not. Therefore the implementation currently provides: Amphora redundancy but does not yet provide: Distributor redundancy A failure of one active Amphora can be recovered using the standby. A failure of the only Distributor can interrupt the whole load balancer. This must be stated explicitly when describing the architecture. The implementation should not be described as eliminating all central bottlenecks or all single points of failure.
Internal Distributor VM: The implementation creates a dedicated Distributor VM per load balancer. This architecture naturally provides strong per-LB isolation. Overlapping tenant addresses are not inherently problematic because every load balancer receives a separate Distributor networking context. This mode is currently structurally closer to the historical community reference implementation. Its main remaining problems are: Distributor single point of failure External Distributor: The concept itself is valid. The external peer control-plane integration is technically promising. However, the current shared external implementation has architectural problems when one external device serves multiple load balancers. The main issues are: reusing one external Distributor ID as a per-LB database primary key one frontend_port_id/frontend_mac representation per Distributor row shared br-dist datapath VIP-only state indexing VIP-derived group identity overlapping tenant IP collisions shared Linux neighbor/gateway context shared bridge MAC behavior insufficient per-LB network isolation The external mode therefore requires architectural changes before it should be considered correct for a general multi-tenant OpenStack cloud.
Historical Internal Distributor testing produced meaningful evidence. A previous validation window demonstrated: two active IN_CLUSTER Amphorae one IN_CLUSTER_STANDBY an OVS select group containing both active Amphora MAC addresses real frontend traffic reaching the Distributor return-path counters increasing 100 out of 100 HTTP requests returning HTTP 200 responses being served through both active forwarding paths This proves that the core Active/Active concept was operational during that historical test window. However, this should not be interpreted as proof of complete production readiness. Later audits showed: operating_status OFFLINE timed-out requests degraded infrastructure during some tests incomplete L4 reproduction incomplete failover validation The current documentation correctly distinguishes historical successful dataplane evidence from current health.
The external peer has demonstrated successful control-plane integration. Observed successful steps included: external OVN chassis registration frontend logical-port binding mTLS communication between Octavia and the external Distributor agent VIP state stored on the external peer control-plane Distributor registration However, the latest complete end-to-end validation could not proceed because the single-node Nova environment did not have sufficient capacity to create the required Active/Active Amphora set. Therefore external mode currently has: control-plane evidence but not yet a complete fresh end-to-end Active/Active dataplane validation. This distinction must be preserved.
The implementation is not simply an arbitrary custom Active/Active design. Its fundamental concepts are derived from the historical Octavia community Active/Active architecture. The following parts are technically well aligned with that design: N active Amphorae plus one explicit standby IN_CLUSTER and IN_CLUSTER_STANDBY roles active-only Distributor registration ARP suppression on active Amphorae lightweight Distributor forwarding Layer 2 Amphora selection OVS select groups dp_hash-based flow distribution one-legged Distributor forwarding standby promotion during active-member failure The implementation intentionally diverges from the original direct-return design in one important area: return traffic is hairpinned through the Distributor to maintain a valid OVN conntrack path. This adaptation is technically understandable for the current OVN architecture, but it increases the importance and performance impact of the Distributor.
The Internal Distributor VM implementation is the more technically mature of the two deployment modes. It follows the original community architecture closely enough to provide a credible Active/Active Amphora implementation and has historical real-traffic evidence showing that the basic datapath works. It should still be considered an experimental implementation rather than production-ready because Distributor HA, continuous health management, anti-affinity, persistence-state synchronization, failure recovery, upgrade behavior, and wider validation remain incomplete. The External Distributor concept is also valid and consistent with the extensibility anticipated by the historical Octavia Distributor design. However, the current implementation should not yet be considered correct for a shared multi-tenant production Distributor. The physical external device identity must be separated from the per-load-balancer Distributor identity. Per-load-balancer datapath isolation must also be restored. The highest-priority external-mode corrections are therefore:
Internal Distributor VM mode: The architecture is fundamentally sound for continued development and validation. The core Active/Active Amphora and Distributor datapath has previously worked with real traffic. It is not yet production-complete. External Distributor mode: The external-device concept is valid. The current control-plane implementation proves that an external OVN/OVS Distributor can be integrated with Octavia. However, the current shared-device data model and shared datapath design contain important multi-load-balancer and multi-tenant isolation problems. Those architectural issues should be corrected before further production-readiness claims are made. In short: The core Active/Active Amphora design is valid. The Internal Distributor VM implementation is a credible port of the historical community architecture. The OVN return-path hairpin is a deliberate local adaptation rather than the original direct-return behavior. The external Distributor concept is valid, but the current implementation requires a redesign of per-LB identity and isolation before it can safely serve multiple OpenStack tenants and load balancers. |
|
OCTAVIA ACTIVE-ACTIVE DISTRIBUTOR IMPLEMENTATION REVIEW Review scope This review assesses the architecture and evidence recorded in the supplied beginner guide and implementation report. The supplied files do not contain the complete source repository or a full source diff. Therefore, this is not a line-by-line audit of GitHub PR #13 or the complete Octavia implementation. According to the implementation report, the local work is a manual port to Octavia's unmaintained/2024.1 code base. It was not a direct cherry-pick of the historical community changes. The lab used OpenStack-Helm Caracal on a single node. This differs from the Kolla-Ansible multinode deployment using OVN 24.03.2, so lab results must not be treated as production validation for that deployment. Executive conclusion The active-active design is technically plausible. Multiple Amphorae run HAProxy concurrently, while a Distributor forwards incoming VIP traffic to an active Amphora and preserves flow affinity. The N+1 model means N active Amphorae plus one standby Amphora. It does not mean that the Distributor has a redundant standby. The supplied evidence does not establish that the current implementation is healthy or production-ready:
The appropriate verdict is: prototype-level implementation with historical functional evidence, but current end-to-end acceptance remains open. Design described by the community blueprint The Distributor answers ARP requests for the VIP and forwards traffic to a selected Amphora. The packet keeps the VIP as its destination IP while the Distributor rewrites Ethernet addresses for delivery to the Amphora. The Amphora then runs HAProxy and forwards traffic to the backend pool. In the community topology, response traffic from the Amphora goes directly to the gateway and bypasses the Distributor. The Distributor is therefore a lightweight forwarding stage, not a second HAProxy proxy. The blueprint also describes a single Distributor as an initial implementation step, with Distributor high availability left as a separate concern. Internal Distributor VM In this mode, Octavia creates a Distributor VM in Nova. The Distributor VM runs OVS and the Distributor agent. The report's expected topology is two active Amphorae, one standby Amphora, and a Distributor VM for the load balancer. This is consistent with the software reference design, but it does not reduce the cloud's compute requirements. It adds a Distributor VM to the Amphora resources. If the cloud is short of CPU, RAM, or disk, this mode can make provisioning harder. The single Distributor VM also remains a failure point and a possible throughput limit. The report records a historical 100/100 HTTP success result for this mode. It also records a later audit in which the preserved load balancers were OFFLINE and FIP requests timed out. The historical result shows that the path worked during that earlier test window; it does not prove current health or successful failover. External Distributor peer or device The local external mode uses an existing machine outside Nova's server lifecycle. However, the implementation described in the report is not a generic integration for any vendor appliance. The external peer runs Linux, OVS, ovn-controller, an OVN VIF plugger, and the Octavia Distributor agent. It joins the OVN environment as a chassis and receives traffic through the OVN Geneve data path. This mode can remove the Distributor VM's compute requirement from the OpenStack cloud. It does not remove the Amphora resource requirement, and it does not remove the Distributor from the data path. The reported configuration uses one external peer, so it remains a single failure point unless the physical device supplies its own HA and the integration keeps the Distributor state consistent across its HA members. The report's external-mode evidence proves control-plane connectivity and VIP state setup. It does not prove that a client request traversed the external peer, reached an Amphora, and returned successfully. A hardware appliance that cannot run or provide equivalent OVN chassis, OVS forwarding, VIF-plugging, and agent behavior will need a separate vendor-specific integration. Datapath and implementation findings
The local implementation adds a reverse flow that matches packets with the VIP as their source, rewrites Ethernet addresses, and uses IN_PORT for hairpin forwarding. The report says this was added to keep return traffic on the OVN conntrack path in the lab. OVS supports IN_PORT as the explicit way to send a packet back through its ingress port. This makes the hairpin action syntactically valid, but it does not prove that the complete OVN path, Neutron port security, conntrack handling, and return traffic are correct. This change also means the local implementation's return path differs from the community topology, where Amphora response traffic bypasses the Distributor. Because response traffic may traverse the Distributor in this implementation, its capacity and performance effect should be measured in both directions.
The code excerpt in the report uses: The value 4294967296 is 0x0000000100000000. In the OVS selection parameter, the upper 32 bits select the datapath hash algorithm and the lower 32 bits provide the hash basis. The value shown selects the symmetric hash over the IP 5-tuple with a zero basis. This can keep packets in the same TCP flow on the same Amphora, provided the target OVS and datapath behave consistently. It does not implement the blueprint's source-IP-only default. Different TCP connections from the same client IP can be distributed to different Amphorae because transport ports are part of the hash. This may affect TLS session reuse or other state kept locally by HAProxy. The implementation should document this policy or make the affinity mode configurable.
The community Distributor design keeps the cluster size fixed during a single Amphora failure and replaces the failed Amphora with the standby in the corresponding bucket. This limits remapping of flows assigned to healthy Amphorae. The report's code excerpt rebuilds buckets from a sorted list of Amphora MAC addresses and does not show explicit bucket IDs or preserved positions. The excerpt is not enough to establish that bucket placement remains stable during promotion. The full code and a live failover test must prove that replacing one Amphora does not unnecessarily change the bucket assignment for the other active Amphora.
The report records a trace that matched the VIP flow but returned "no live bucket." It suggests that zero source or destination MAC values in the synthetic trace may have contributed to the result. That explanation is not established by the evidence. Packet fields determine hash-based bucket selection. Bucket liveness is a separate question and may depend on watch_port or watch_group state. The group dump, group statistics, trace, bridge name, and interface state need to be captured at the same time. The report excerpt does not show watch fields, so the trace needs to be reconciled with the actual group state before the Distributor data path can be accepted.
Both the internal VM and the reported external mode use one Distributor endpoint for the load balancer. If that Distributor fails, the Amphora cluster can remain healthy while client traffic to the VIP stops. This is a separate availability domain from Amphora failover. A redundant Distributor pair, synchronized configuration, and a tested VIP or routing failover mechanism are required if the service objective includes Distributor failure tolerance. Validation status by mode Internal Distributor VM:
External Distributor peer:
Required acceptance tests Run the internal VM and external peer modes as separate test cases. For each mode:
A separate L3/ECMP design can move distribution into the network fabric using BGP anycast routes and ECMP, rather than forwarding through a Distributor VM or peer. It is a different architecture, not an automatic property of the external Distributor mode. The current L3 specification also identifies Floating IP support as requiring additional Neutron work, so it should not be treated as a drop-in replacement for the existing FIP topology. Final verdict The internal VM design follows the software Distributor direction described by the community and has historical end-to-end success evidence. It consumes additional cloud resources and remains a single Distributor failure domain. The external peer design can move the Distributor compute workload outside Nova, but the implementation is specific to an OVS/OVN external chassis and has not yet passed end-to-end traffic validation. It does not support arbitrary hardware appliances without a compatible integration. The active-active Amphora concept is sound, but the implementation should remain classified as experimental until a fresh healthy validation run proves traffic, affinity, failover, and restart recovery for each mode. Production approval also requires a source-level review of the complete diff and testing on the target Kolla-Ansible and OVN/OVS versions. References Octavia active-active topology specification: Octavia active-active Distributor specification: Open vSwitch ovs-ofctl manual: Open vSwitch OpenFlow FAQ: Octavia L3 active-active Distributor specification: |
No description provided.