Skip to content

Navigating a BGP Zombie Outbreak on Juniper Routers

It would be appreciated if you could help me continue to provide valuable network engineering content by supporting my non-profit solitary efforts. Your donation will help me conduct valuable experiments. Click here to donate now.

This article has been published on the APNIC blog as well.

Border Gateway Protocol (BGP) routing issues in general, can be a headache for network engineers. However, when those issues start exhibiting zombie-like behaviour, it’s time to take a closer look. This is exactly what I have observed on some Juniper routers in production at AS48635 running as edge routers (responsible for IP Transit, PNIs and IXP peering). This phenomenon is not only perplexing but can also impact traffic engineering efforts or in some rare cases, cause network disruptions.

In this article, I will share my experience with BGP zombie routes on Junos OS (JunOS).

However, please note that personally I am not a big Juniper user or fan, my home base has always been FRRouting (Linux) and MikroTik (also Linux with minimal vendor abstraction), so I may not necessarily have complete knowledge of JunOS quirks.

Background

At AS48635, one of the projects I worked on was cleaning up redundant route objects on IRR and performing route aggregation, that is, exporting only aggregates wherever possible to our external peers.

In our Juniper routers, this translates to:

  • Automatic generation of up-to-date prefix lists for our route policies using bgpq3.
  • Properly configuring the route aggregation feature.
  • The BGP peer export policy should only export aggregates, even if more specifics exist in the local routing table on a given router.

Below is a real-life configuration sample from one of the affected routers in production to give the readers some context. Please note, only IPv4 config is shared in this article, as IPv6 is identically configured, with the exact same end result (Zombie routes).

#Up-to-date aggregate-only prefix list#
policy-options {
    }
    prefix-list as48635-v4 {
        2.57.56.0/22;
        5.157.80.0/21;
    }
#BGP Policy logic#
policy-statement as6453-v4-out {
    term prefixes-out {
        from {
            family inet;
            prefix-list as48635-v4;
        }
        then accept;
    }
    term else-reject {
        then reject;
    }
}
#Route Aggregation#
routing-options {
    aggregate {
        defaults {
            as-path {
                origin igp;
            }
        }
        route 2.57.56.0/22 discard;
        route 5.157.80.0/21 discard;
    }
    generate {
        route 0.0.0.0/0 discard;
    }
#BGP Peer#
group ebgp-6453-v4 {
    type external;
    import [ XYZ ];
    export as6453-v4-out;
    remove-private {
        all;
    }
    peer-as 6453;
    multipath multiple-as;
    neighbor 80.231.152.77 {
        description TATA-v4;
    }
}

Context (Expectation vs Reality)

Based on the configuration sample shared, we expected to only see aggregates and not more-specific routes to be advertised to our peer. However, the following is what we observed:

daryllswer@edge-juniper-router> show route advertising-protocol bgp 80.231.152.77   
inet.0: 909266 destinations, 3698169 routes (909000 active, 2 holddown, 317 hidden)
  Prefix		  Nexthop	       MED     Lclpref    AS path
* 2.57.56.0/22            Self                                    I
* 5.157.80.0/21           Self                                    I
* 5.157.80.0/22           Self                                    I
* 5.157.86.0/23           Self                                    I

As we can see, for reasons I could not understand, our Juniper router in this example (multiple routers showed the same behaviour) is still exporting more specifics instead of only the aggregates to our peer. I understand from the configuration example shared that the router should only explicitly advertise aggregates, even if more specifics exist in the local table.

Debugging

The first step I took was to verify the route aggregation was working correctly and indeed it was:

daryllswer@edge-juniper-router> show route 2.57.56.0/22 
inet.0: 909307 destinations, 3698305 routes (909043 active, 0 holddown, 317 hidden)
+ = Active Route, - = Last Active, * = Both
2.57.56.0/22       *[Aggregate/130] 1w4d 11:15:19
                      Discard
daryllswer@edge-juniper-router> show route 5.157.80.0/21 
inet.0: 909328 destinations, 3698589 routes (909064 active, 0 holddown, 317 hidden)
+ = Active Route, - = Last Active, * = Both
5.157.80.0/21      *[Aggregate/130] 1w4d 11:19:45
                      Discard

The second step I tried was to clear the BGP session with the affected peer(s) (clear bgp neighbor 80.231.152.77), but that did not resolve the issue.

The third step I tried was deactivating the BGP peer completely, and then re-enabling it, but that didn’t resolve the issue either.

The fourth step I tried was to verify that no legacy config on the router could be responsible for the issue and config was clean as expected.

daryllswer@edge-juniper-router> show configuration | display set | match 5.157.*
set routing-options aggregate route 5.157.80.0/21 discard
set policy-options prefix-list as48635-v4 5.157.80.0/21

The final step I tried was a fresh reboot on all our Juniper edge routers, and voilà! Zombie routes disappeared.

daryllswer@edge-juniper-router> show route advertising-protocol bgp 80.231.152.77    
inet.0: 909222 destinations, 3940003 routes (908958 active, 0 holddown, 317 hidden)
  Prefix		  Nexthop	       MED     Lclpref    AS path
* 2.57.56.0/22            Self                                    I
* 5.157.80.0/21           Self                                    I

JunOS versions:

Different routers ran different JunOS versions and all of them were affected by the same issue. Although of course, these versions may not be the latest recommended versions, it does show that the issue is impacting multiple versions. Versions in use at the time of this issue include:

  • 17.3R3.10
  • 17.3R3-S7.2
  • 18.4R1-S6.3
  • 21.4R1-S1.6
  • 21.4R3.15

Conclusion

In my experience working with alternative vendors like MikroTik (RouterOS) for a few years, I have never seen a BGP zombie route issue on their platform.

Based on my understanding of the configuration logic, debugging steps taken to resolve the issue and my personal previous experience with alternative vendors, I’m inclined to believe that the issue I have observed is likely a Juniper-specific bug, especially based on the fact that a reboot fixed it with no further configuration changes being made.

If this is indeed a Juniper-specific bug, this could also mean that a large part of Zombie routes in the Default-free zone (DFZ) could be due to the bug as many Tier 1, 2 and 3 networks (where zombie routes often are propagated) in the world also use Juniper equipment and one wouldn’t expect them to just reboot a router because of Zombie routes, so hence would leave Zombie routes lingering for a long time in the DFZ.

In short, a reboot should never be required to remove (or prevent) zombie routes in production.

If someone reading this post is from Juniper or someone with extensive knowledge of JunOS knows what the reason and preventive solution could be for this issue, please feel free to contact me or via any of my professional social media profiles (DMs are open).

Update, 30th March 2023

Some people have suggested using the ‘from protocol aggregate’ parameter in the policies as a mitigation/workaround. While this could in theory mitigate the issue, it also creates a problem with our internal policy logic because in this case we have downstream BGP customers, and their prefixes are part of as48635-v4 / as48635-v6 prefix list and for obvious reasons, we do not ‘aggregate’ customer prefixes. The as48635-vX prefix list is generated based on our AS-SET and hence this workaround isn’t ideal for us.

At the same time, it still doesn’t answer the question of why a reboot fixed the issue just fine.

Published inDC/EnterpriseNetworking

4 Comments

  1. Hi,

    In your export policy, did you try using a prefix-list-filter with exact match instead? A prefix filter alone will allow more specific routes.

    Eg.

    set policy-statement as6453-v4-out term prefixes-out from prefix-list-filter as48635-v4 exact

    Then remove the existing prefix-list from the term.

    • Hi

      The prefix list by default seems to be an exact match already, even though Juniper doesn’t explicitly say it here.

      We’ve been using prefix list on many other routers in different ASes with no problems for exporting only aggregates.

      Another user also tested the config and confirmed, there’s nothing wrong with using prefix list directly as I did.

      It’s also based on the simple logical fact that if a reboot fixed the issue, then the issue isn’t in the config. If it was the config, then why would a reboot fix the issue?

  2. Cathal Mooney Cathal Mooney

    Yep your logic is sound there – if a reboot fixes it it’s not about the config! Very odd what’s happening alright. Interesting discussing with you 🙂

    • Indeed, the logic is sound. Approximately 8-ish Juniper employees saw this blog post at least once, via social media (I know via Analytics). However, knowing Juniper (or any other big vendor), they are unlikely to reach out to me or fix this bug, unless I report it via JTAC (which isn’t going to happen).

      For this type of reason (money/greed – Hello Flex Licensing), I extensively specialise in alternative vendors like MikroTik – Don’t need to pay $3k to get a bug fixed within a few weeks or months at the most. I strongly believe in open source networking 😉

      Anyway, like-wise, interesting discussion it was 🙂

Leave a Reply

Your email address will not be published. Required fields are marked *

Daryll Swer's network engineering blog.