Skip to content

Multi-WAN setups with retail ISPs [Part 2 – Implementation (Using RouterOS)]

In part one of this series, I talked about the theoretical aspects, pros and cons of a Multi-WAN setup. So now in part two, I will show how one can implement this setup in their network setup and this will be a lengthy article.

Requirements

  • A router that is running either enterprise or open-source network OS
    • Like RouterOS or Cisco IOS for enterprise
    • Or like VyOS or pfSense for open-source
  • More than one uplink to the same ISP or a different ISP
  • Minimum 1GbE on all your devices’ interfaces/ports
  • Bridging the ISP’s CPE device to ensure there is no double/triple/quadruple NAT situation

Implementation

  • For this example, I will use RouterOS v6 stable
    • Hardware: RB450Gx4
    • Only IPv4 config will be covered due to the lack of proper IPv6 support in RouterOS v6 stable at the time of this article, but it would essentially be the same thing
    • I will be using PCC and Nth together in order to achieve bandwidth aggregation

Whatever is shown here using RouterOS can be replicated more or less on any other network OS like VyOS etc. Go through the vendor’s documentation if you are not on RouterOS.

  • Assumptions
    • I will assume you have already taken care of the basic configuration such as securing the router, firewalling, NATting etc
      • Regarding NAT, use masquerade instead of src NAT, the reason being it clears the conn_track table if an interface goes down and hence we can achieve sub 0ms failover effect
    • I will assume you have some basic idea of computer networking and basic ideas on the routing
    • I will assume you have read MikroTik’s documentation on PCC, Nth, Mangle etc
    • Only two uplinks (relevant to the example here)

Let us begin

Step 1 we create two default routes for each uplink with distance attribute to enable the failover effect (meaning if one link goes down, the next immediate link will be used next)

/ip route
###ISP1 has lower distance and hence is the primary link###
add check-gateway=ping comment="Default Route for ISP1" distance=1 gateway=pppoe-out1

add check-gateway=ping comment="Default Route for ISP2" distance=2 gateway=pppoe-out2

###If you have more than two uplinks, you simply increase the distance as required like this:
add check-gateway=ping comment="Example Route for ISP3" distance=3 gateway=pppoe-out3###

Step 2, we need to take care of MTU for the WAN interfaces which is applicable to any networking device or OS in this world. If your uplink is using DHCP Client/Static IP address, then by default this is already taken care of with 1500 MTU.

However with PPPoE, this is not the case and on RouterOS, there are what a user call ghost bytes. I will discuss PPPoE MTU in a future article, but for now, all that you need to do is set the underlying ethernet interface’s actual MTU to 1520 meaning the interface to which your uplink is connected to.

Figure-1 (In this case, ether1 and ether2 are the underlying ethernet interfaces)

Step 3, leave MTU, MRU, MRRU as blank to enable auto-negotiation, RouterOS will automatically find the correct MTU value set by your ISP. Along with MRU.

Figure-2 (PPPoE Client MTU Config)

Step 4, now we get started with the multi-WAN rules that will enable load-balancing and bandwidth aggregation without breaking HTTP/HTTPS connections

###Set Passthrough=no to reduce CPU usage for rules that do not need to be re-validated once they've been processed###
###connection-mark=no-mark to prevent re-marking of already marked connections and hence waste CPU cycles###

/ip firewall mangle
###Incoming connections through ISP1 must leave through ISP1###
add action=mark-connection chain=prerouting connection-mark=no-mark in-interface=pppoe-out1 new-connection-mark=ISP1_conn passthrough=no

###Incoming connections through ISP2 must leave through ISP2###
add action=mark-connection chain=prerouting connection-mark=no-mark in-interface=pppoe-out2 new-connection-mark=ISP2_conn passthrough=no

###I am assuming a 50/50 split ratio between the two ISPs#

###We are using dst-address-list=!not_in_internet && dst-address-type=!local to prevent marking LAN-to-LAN traffic###

###We will use PCC to handle HTTP/HTTPS traffic with "both-addresses" attribute to reduce chances of connections being marked more "randomly" which would break the connections as then connections would go through ISP1 and ISP2 more "randomly" and break###

###For old school HTTP/HTTPS traffic###
###50% going to ISP1###
add action=mark-connection chain=prerouting connection-mark=no-mark dst-address-list=!not_in_internet dst-address-type=!local dst-port=80,443 in-interface-list=LAN new-connection-mark=ISP1_conn passthrough=yes per-connection-classifier=both-addresses:2/0 protocol=tcp
###50% going to ISP2###
add action=mark-connection chain=prerouting connection-mark=no-mark dst-address-list=!not_in_internet dst-address-type=!local dst-port=80,443 in-interface-list=LAN new-connection-mark=ISP2_conn passthrough=yes per-connection-classifier=both-addresses-and-ports:2/1 protocol=tcp

###For new school HTTP3 traffic aka QUIC###
###50% going to ISP1###
add action=mark-connection chain=prerouting connection-mark=no-mark dst-address-list=!not_in_internet dst-address-type=!local dst-port=80,443 in-interface-list=LAN new-connection-mark=ISP1_conn passthrough=yes per-connection-classifier=both-addresses-and-ports:2/0 protocol=udp
###50% going to ISP2###
add action=mark-connection chain=prerouting connection-mark=no-mark dst-address-list=!not_in_internet dst-address-type=!local dst-port=80,443 in-interface-list=LAN new-connection-mark=ISP2_conn passthrough=yes per-connection-classifier=both-addresses-and-ports:2/1 protocol=udp

###If you have a third uplink, then the split ratio would be 3/0, 3/1, 3/2###

###Now we will use Nth for non HTTP/HTTPs traffic in order to acheieve bandwidth aggregation###

###50% going to ISP1###
add action=mark-connection chain=prerouting connection-mark=no-mark dst-address-list=!not_in_internet dst-address-type=!local in-interface-list=LAN new-connection-mark=ISP1_conn nth=2,1 passthrough=yes
###50% going to ISP2###
add action=mark-connection chain=prerouting connection-mark=no-mark dst-address-list=!not_in_internet dst-address-type=!local in-interface-list=LAN new-connection-mark=ISP2_conn nth=2,2 passthrough=yes

###Now we will send the marked connections to their appropriate routing table###

###For our marked/split traffic###
add action=mark-routing chain=prerouting connection-mark=ISP1_conn in-interface-list=LAN new-routing-mark=to_ISP1 passthrough=no
add action=mark-routing chain=prerouting connection-mark=ISP2_conn in-interface-list=LAN new-routing-mark=to_ISP2 passthrough=no

###For the incoming traffic from WAN###
add action=mark-routing chain=output connection-mark=ISP1_conn new-routing-mark=to_ISP1 out-interface=pppoe-out1 passthrough=no
add action=mark-routing chain=output connection-mark=ISP2_conn new-routing-mark=to_ISP2 out-interface=pppoe-out2 passthrough=no

###Now Finally we add the required routing tables###
/ip route
add check-gateway=ping comment="Load Balancing Route to ISP 1" distance=1 gateway=pppoe-out1 routing-mark=to_ISP1
add check-gateway=ping comment="Load Balancing Route to ISP 2" distance=1 gateway=pppoe-out2 routing-mark=to_ISP2

That’s it, you now have aggregated bandwidth capability, load balancing capability and HTTP/HTTPS stability using RouterOS.

For P2P networking (assuming all the uplinks have a public IP), you’d need to use a script like this for UPnP to work correctly (enable it on only one WAN interface and let the script handle the rest).

Bonus Material

  • Use RFC 6890 to build the not_in_internet address list
  • RouterOS will automatically use the main routing table (default routes) should any uplink go down, so the “marked connections” will automatically be routed through whichever uplink is available even though for instance they are marked for ISP2, and ISP2 is down

Side Note:

In part one, you may have noticed the upload speed is lower than what I claimed and below is the explanation for that.

  • ISP1’s MTU = 1460, MRU = 1500
  • ISP2’s MTU = 1500, MRU = 1500

The upload bandwidth never reached 300Mbps due to ISP1’s MTU of 1460 which resulted in packet fragmentation and hence affected throughput performance and was able to reach only an average of 170Mbps for upload. ISP2’s MTU is a straight 1500 and was able to reach its advertised speed which I verified by looking at the link rate on the router itself. Since MRU for both links were 1500, download bandwidth was able to reach its advertised speed on both links.

This is why MTU should be taken care of to prevent issues.

Published inNetworking

16 Comments

  1. Hi Daryll

    Thanks for sharing this. This is working great!

    I am a newbie to Mikrotik world & not a networking guy.

    I had trouble with accessing my webserver from outside with an earlier config based on some guide. But your config solved this issue.

    Wanted your help on few things of you may please?

    1. I don’t have the not_in_internet_list setup & have excluded it from Mangle rules. Still the config works. How can that be? Am I missing something here.(I can share my config with you if needed)

    2. What would be the hairpin NAT config so that LAN traffic can access webserver internally?

    3. I have set up dst-nat for port forwarding on one of the pppoe interface. Can I add the same for my other pppoe interface & make it high availability for users accessing my services?

    Thanks in advance.

    Cheers,
    Ashish

  2. Thanks for the response Daryll

    I have to convey that for the first time I was able to understand rules & their working apart from the not_in_internet list.

    1. I am not able to understand which addresses should be in that list. Does !not_in_internet means in_internet based on plain logic? I have configured DHCP range on 10.10.10.0/24 for LAN interface(Ether-5) & rest are the IP’s assigned by ISP which are handled in PPPOE dialling interfaces.

    2. Will setup Hairpin NAT based on the guide

    3. I have opened up the WAN interface and found out that my ISP is redirecting port 80 to their Mikrotik switch login page 😐 Have dropped them an email for correction. Will update on this once it is resolved from ISP’s end. One good thing is, I will now try to set MTU as 1520 on 1 ISP for Jumbo Frame support(based on your both articles as I now know about ISP’s hardware)

    • 1. ALL the subnets from RFC6890 should be added to the list.

  3. Thanks for the info Daryll. Added the list. Not sure how to test. But looks good, IMHO.

    I have a strange observation with MTU. Whatever MTU I set 1480, 1500, 1520 it reverts to 1500 / 1480 for one ISP & other ISP’s PPPOE authentication fails if MTU is changed from 1480. Is it supposed to work this way?

    • Follow exactly what I did in the article. I clearly mentioned to not specify the MTU manually on the PPPoE interface.

      And even then your ONT needs to support jumbo frames.

  4. ชื่อ นามสกุล ชื่อ นามสกุล

    guy I have zero knowledge of microtik. I set up configuration by your following instruction. And I still have a problem with some https: website(just some of them just like http://www.teenee.com Its long time loading) and some application such as LINE messenger(sometime can’t download picture and files).
    please advice if you have any solution that can fix this issue
    and can I use per-connection-classifier=srt-addresses for all classifier
    I read from some article and it mention that it can be solve https problem right?

    • Without seeing your configuration, I can’t tell what you did wrong.
      Dump the config with:
      /export file myconfig

      Email it to [email protected]

  5. ชื่อ นามสกุล ชื่อ นามสกุล

    thank you very much for your fast reply here is myconfig

    # dec/16/2021 15:28:37 by RouterOS 6.47.10
    # software id = ZS4S-B9IN
    #
    # model = RB4011iGS+
    # serial number = F03A0F603BBD
    /interface bridge
    add name=bridge-lan
    /interface ethernet
    set [ find default-name=ether1 ] name=ether1-ais
    set [ find default-name=ether2 ] name=ether2-cat
    set [ find default-name=ether8 ] name=ether8-nas
    set [ find default-name=ether9 ] name=ether9-nas
    set [ find default-name=ether10 ] name=ether10-office
    /interface ethernet switch port
    set 0 default-vlan-id=0
    set 1 default-vlan-id=0
    set 2 default-vlan-id=0
    set 3 default-vlan-id=0
    set 4 default-vlan-id=0
    set 5 default-vlan-id=0
    set 6 default-vlan-id=0
    set 7 default-vlan-id=0
    set 8 default-vlan-id=0
    set 9 default-vlan-id=0
    set 10 default-vlan-id=0
    set 11 default-vlan-id=0
    /interface wireless security-profiles
    set [ find default=yes ] supplicant-identity=MikroTik
    /ip pool
    add name=dhcp_pool0 ranges=10.10.2.200-10.10.2.250
    /ip dhcp-server
    add address-pool=dhcp_pool0 disabled=no interface=bridge-lan name=dhcp1
    /ppp profile
    add name=pppoe-profile-cat on-down=":local removeRoute do={ \r\
    \n /ip route remove [find comment=\$wan]\r\
    \n}\r\
    \n\r\
    \n:local gw [/ip route print as-value where gateway=\$\"remote-address\"];\
    \r\
    \n:if ((\$gw->0->\"comment\") != \"\") do={\r\
    \n \$removeRoute wan=(\$gw->0->\"comment\")\r\
    \n}" on-up=":local createRoute do={\r\
    \n /log info message=\"Add route \$remoteAddress\"\r\
    \n /ip route add dst-address=\$dstAddress gateway=\$remoteAddress scope=10\
    \_target-scope=10 comment=(\$wan)\r\
    \n /ip route add dst-address=0.0.0.0/0 gateway=\$gateway scope=30 target-s\
    cope=10 distance=\$distance check-gateway=ping routing-mark=nrm-cat commen\
    t=(\$wan)\r\
    \n}\r\
    \n\r\
    \n:local wanArr {\r\
    \n {\r\
    \n \"remoteAddress\"=\$\"remote-address\";\r\
    \n \"dstAddress\"=8.8.8.8/32;\r\
    \n \"gateway\"=8.8.8.8;\r\
    \n \"distance\"=2;\r\
    \n \"wan\"=\"pppoe-out-cat\"\r\
    \n };\r\
    \n}\r\
    \n\r\
    \n:local intf1 [/interface get \$interface];\r\
    \n:foreach w in=\$wanArr do={\r\
    \n if ((\$w->\"wan\") = (\$intf1->\"name\")) do={\r\
    \n \$createRoute remoteAddress=(\$w->\"remoteAddress\") dstAddress=(\$w->\
    \"dstAddress\") gateway=(\$w->\"gateway\") distance=(\$w->\"distance\") wa\
    n=(\$w->\"wan\")\r\
    \n }\r\
    \n}"
    /interface pppoe-client
    add disabled=no interface=ether2-cat name=pppoe-out-cat password=71560 \
    profile=pppoe-profile-cat [email protected]
    /interface bridge port
    add bridge=bridge-lan interface=ether8-nas
    add bridge=bridge-lan interface=ether9-nas
    add bridge=bridge-lan interface=ether10-office
    /ip address
    add address=10.10.2.99/8 interface=bridge-lan network=10.0.0.0
    add address=192.168.1.2 interface=ether1-ais network=192.168.1.1
    /ip cloud
    set ddns-enabled=yes
    /ip dhcp-server lease
    add address=10.10.2.31 client-id=1:c4:2c:3:7:eb:98 mac-address=\
    C4:2C:03:07:EB:98 server=dhcp1
    add address=10.10.2.5 client-id=1:f4:4d:30:64:eb:75 mac-address=\
    F4:4D:30:64:EB:75 server=dhcp1
    add address=10.10.2.10 client-id=1:1c:69:7a:68:e0:d7 mac-address=\
    1C:69:7A:68:E0:D7 server=dhcp1
    add address=10.10.2.12 client-id=1:f4:4d:30:64:e4:c9 mac-address=\
    F4:4D:30:64:E4:C9 server=dhcp1
    add address=10.10.2.14 client-id=1:f0:79:59:8d:97:73 mac-address=\
    F0:79:59:8D:97:73 server=dhcp1
    add address=10.10.2.3 client-id=1:f4:4d:30:64:e7:f9 mac-address=\
    F4:4D:30:64:E7:F9 server=dhcp1
    add address=10.10.2.16 client-id=1:1c:87:2c:63:2d:5b mac-address=\
    1C:87:2C:63:2D:5B server=dhcp1
    add address=10.10.2.9 client-id=1:f4:4d:30:64:dd:3f mac-address=\
    F4:4D:30:64:DD:3F server=dhcp1
    add address=10.10.2.6 client-id=1:f4:4d:30:64:e7:37 mac-address=\
    F4:4D:30:64:E7:37 server=dhcp1
    add address=10.10.2.8 client-id=1:f4:4d:30:64:e6:e8 mac-address=\
    F4:4D:30:64:E6:E8 server=dhcp1
    add address=10.10.2.7 client-id=1:f4:4d:30:64:ee:1c mac-address=\
    F4:4D:30:64:EE:1C server=dhcp1
    add address=10.10.2.205 client-id=1:f4:4d:30:64:f0:b5 mac-address=\
    F4:4D:30:64:F0:B5 server=dhcp1
    add address=10.10.2.11 client-id=1:f4:4d:30:64:e9:45 mac-address=\
    F4:4D:30:64:E9:45 server=dhcp1
    add address=10.10.2.13 client-id=1:f4:4d:30:64:eb:69 mac-address=\
    F4:4D:30:64:EB:69 server=dhcp1
    add address=10.10.2.2 client-id=1:f4:4d:30:64:d0:b mac-address=\
    F4:4D:30:64:D0:0B server=dhcp1
    add address=10.10.2.50 client-id=1:1c:69:7a:6c:9c:34 mac-address=\
    1C:69:7A:6C:9C:34 server=dhcp1
    add address=10.10.2.100 client-id=1:0:11:32:fe:e0:65 mac-address=\
    00:11:32:FE:E0:65 server=dhcp1
    /ip dhcp-server network
    add address=10.0.0.0/8 dns-server=8.8.8.8 gateway=10.10.2.99
    /ip dns
    set servers=8.8.8.8,8.8.4.4,1.1.1.1
    /ip firewall address-list
    add address=10.0.0.0/8 list=LAN
    /ip firewall filter
    add action=accept chain=input disabled=yes dst-port=8291 protocol=tcp
    /ip firewall mangle
    add action=mark-connection chain=prerouting comment=\
    "INCOMING CONNECTIONS MARK AS ncm-ais,ncm-cat" connection-mark=no-mark \
    in-interface=ether1-ais new-connection-mark=ncm-ais passthrough=no
    add action=mark-connection chain=prerouting connection-mark=no-mark \
    in-interface=pppoe-out-cat new-connection-mark=ncm-cat passthrough=no
    add action=mark-connection chain=prerouting comment=\
    "PCC TO HANDLE HTTP/HTTPS " connection-mark=no-mark disabled=yes \
    dst-address-list=!LAN dst-address-type=!local dst-port=80,443 \
    in-interface=bridge-lan new-connection-mark=ncm-ais passthrough=yes \
    per-connection-classifier=src-address:2/0 protocol=tcp
    add action=mark-connection chain=prerouting connection-mark=no-mark disabled=\
    yes dst-address-list=!LAN dst-address-type=!local dst-port=80,443 \
    in-interface=bridge-lan new-connection-mark=ncm-cat passthrough=yes \
    per-connection-classifier=src-address:2/1 protocol=tcp
    add action=mark-connection chain=prerouting comment=\
    "PCC TO HANDLE HTTP3 TRAFFIC AKA QUIC" connection-mark=no-mark \
    dst-address-list=!LAN dst-address-type=!local dst-port=80,443 \
    in-interface=bridge-lan new-connection-mark=ncm-ais passthrough=yes \
    per-connection-classifier=src-address:2/0 protocol=udp src-port=""
    add action=mark-connection chain=prerouting connection-mark=no-mark \
    dst-address-list=!LAN dst-address-type=!local dst-port=80,443 \
    in-interface=bridge-lan new-connection-mark=ncm-cat passthrough=yes \
    per-connection-classifier=src-address:2/1 protocol=udp src-port=""
    add action=mark-connection chain=prerouting comment="NTH FOR NON HTTP/HTTPS" \
    connection-mark=no-mark dst-address-list=!LAN dst-address-type=!local \
    in-interface=bridge-lan new-connection-mark=ncm-ais nth=2,1 passthrough=\
    yes
    add action=mark-connection chain=prerouting connection-mark=no-mark \
    dst-address-list=!LAN dst-address-type=!local in-interface=bridge-lan \
    new-connection-mark=ncm-cat nth=2,2 passthrough=yes
    add action=mark-routing chain=prerouting comment="MARKED/SPLIT TRAFFIC" \
    connection-mark=ncm-ais dst-address-type="" in-interface=bridge-lan \
    new-routing-mark=nrm-ais passthrough=no
    add action=mark-routing chain=prerouting connection-mark=ncm-cat \
    dst-address-type="" in-interface=bridge-lan new-routing-mark=nrm-cat \
    passthrough=no
    add action=mark-routing chain=output comment="INCOMING TRAFFIC FROM WAN" \
    connection-mark=ncm-ais dst-address-type="" new-routing-mark=nrm-ais \
    out-interface=ether1-ais passthrough=no
    add action=mark-routing chain=output connection-mark=ncm-cat \
    dst-address-type="" new-routing-mark=nrm-cat out-interface=pppoe-out-cat \
    passthrough=no
    /ip firewall nat
    add action=masquerade chain=srcnat out-interface=ether1-ais
    add action=masquerade chain=srcnat out-interface=pppoe-out-cat
    add action=masquerade chain=srcnat src-address=10.0.0.0/8
    add action=dst-nat chain=dstnat dst-port=10390 protocol=tcp to-addresses=\
    10.10.2.50 to-ports=5900
    add action=dst-nat chain=dstnat dst-port=10391 protocol=tcp to-addresses=\
    10.10.2.31 to-ports=5900
    add action=dst-nat chain=dstnat disabled=yes dst-port=10392 protocol=tcp \
    to-addresses=10.10.2.99 to-ports=8291
    /ip firewall service-port
    set ftp disabled=yes
    set tftp disabled=yes
    set irc disabled=yes
    set h323 disabled=yes
    set sip disabled=yes
    set pptp disabled=yes
    set udplite disabled=yes
    set dccp disabled=yes
    set sctp disabled=yes
    /ip route
    add check-gateway=ping comment=pppoe-out-cat distance=2 gateway=8.8.8.8 \
    routing-mark=nrm-cat
    add check-gateway=ping comment="static ether1-ais" distance=1 gateway=1.1.1.1 \
    routing-mark=nrm-ais
    add comment="static ether1-ais" distance=1 dst-address=1.1.1.1/32 gateway=\
    192.168.1.1 scope=10
    add comment=pppoe-out-cat distance=1 dst-address=8.8.8.8/32 gateway=\
    100.70.192.1 scope=10
    add comment=pppoe-out-cat distance=1 dst-address=8.8.8.8/32 gateway=\
    100.70.192.1 scope=10
    /ip service
    set telnet disabled=yes
    set ftp disabled=yes
    set www disabled=yes
    set ssh disabled=yes
    set api disabled=yes
    set api-ssl disabled=yes
    /ip upnp
    set enabled=yes
    /ip upnp interfaces
    add interface=ether1-ais type=external
    add interface=pppoe-out-cat type=external
    add interface=bridge-lan type=internal
    /system clock
    set time-zone-name=Asia/Bangkok
    /system identity
    set name="JPAC GATEWAY"

    • Your config is convoluted and wrong, why would you disable NAT Traversal helpers? What’s the point of VLAN 0’s? What’s with removeRoute? Where’s not_in_internet address list?

      I would reset it to null config and start again from scratch.

  6. ชื่อ นามสกุล ชื่อ นามสกุล

    thank you for reply guy
    not_in_internet is
    /ip firewall address-list
    add address=10.0.0.0/8 list=LAN

    removeRoute is from script to auto connect ppp and add route guy here is full script
    i add in ppp profile

    ############PPP UP#######################
    :local createRoute do={
    /log info message=”Add route $remoteAddress”
    /ip route add dst-address=$dstAddress gateway=$remoteAddress scope=10 target-scope=10 comment=(“D”.$wan)
    /ip route add dst-address=0.0.0.0/0 gateway=$gateway scope=30 target-scope=10 distance=$distance check-gateway=ping comment=(“D”.$wan)
    }

    :local wanArr {
    {
    “remoteAddress”=$”remote-address”;
    “dstAddress”=8.8.8.8/32;
    “gateway”=8.8.8.8;
    “distance”=1;
    “wan”=”pppoe-out-cat”
    };
    }

    :local intf1 [/interface get $interface];
    :foreach w in=$wanArr do={
    if (($w->”wan”) = ($intf1->”name”)) do={
    $createRoute remoteAddress=($w->”remoteAddress”) dstAddress=($w->”dstAddress”) gateway=($w->”gateway”) distance=($w->”distance”) wan=($w->”wan”)
    }
    }
    ############PPP UP#######################

    ############PPP DOWN#######################
    :local removeRoute do={
    /ip route remove [find comment=$wan]
    }

    :local gw [/ip route print as-value where gateway=$”remote-address”];
    :if (($gw->0->”comment”) != “”) do={
    $removeRoute wan=($gw->0->”comment”)
    }
    ############PPP DOWN#######################

    i have no idea about disable NAT Traversal
    i was setup its from null config guy

    • Where’s RFC6890 subnets in not_in_internet? Why are you using scripts when routes can failover based on reachability? This is NOT null config!

      Reset the router and start from scratch.

  7. ชื่อ นามสกุล ชื่อ นามสกุล

    Why are you using scripts when routes can failover based on reachability?
    Its dynamic ip and i have to use script guy i have no problem with pppoe and static ip from wan router

    Where’s RFC6890 subnets in not_in_internet?
    10.0.0.0/8 is this right guy?
    /ip firewall address-list
    add address=10.0.0.0/8 list=LAN
    i was name it as LAN guy

    here is myscript after i cut all /r/n make its easy to read guy thanks


    # dec/16/2021 15:28:37 by RouterOS 6.47.10
    # software id = ZS4S-B9IN
    #
    # model = RB4011iGS+
    # serial number = F03A0F603BBD
    /interface bridge
    add name=bridge-lan
    /interface ethernet
    set [ find default-name=ether1 ] name=ether1-ais
    set [ find default-name=ether2 ] name=ether2-cat
    set [ find default-name=ether8 ] name=ether8-nas
    set [ find default-name=ether9 ] name=ether9-nas
    set [ find default-name=ether10 ] name=ether10-office
    /interface ethernet switch port
    set 0 default-vlan-id=0
    set 1 default-vlan-id=0
    set 2 default-vlan-id=0
    set 3 default-vlan-id=0
    set 4 default-vlan-id=0
    set 5 default-vlan-id=0
    set 6 default-vlan-id=0
    set 7 default-vlan-id=0
    set 8 default-vlan-id=0
    set 9 default-vlan-id=0
    set 10 default-vlan-id=0
    set 11 default-vlan-id=0
    /interface wireless security-profiles
    set [ find default=yes ] supplicant-identity=MikroTik
    /ip pool
    add name=dhcp_pool0 ranges=10.10.2.200-10.10.2.250
    /ip dhcp-server
    add address-pool=dhcp_pool0 disabled=no interface=bridge-lan name=dhcp1
    /ppp profile
    add name=pppoe-profile-cat
    on-down=":local removeRoute do={
    /ip route remove [find comment=$wan]
    }
    :local gw [/ip route print as-value where gateway=$"remote-address"];
    :if (($gw->0->"comment") != "") do={
    $removeRoute wan=($gw->0->"comment")
    }"
    on-up=":local createRoute do={
    log info message="Add route $remoteAddress"
    /ip route add dst-address=$dstAddress gateway=$remoteAddress scope=10 target-scope=10 comment=($wan)
    /ip route add dst-address=0.0.0.0/0 gateway=$gateway scope=30 target-scope=10 distance=$distance check-gateway=ping routing-mark=nrm-cat comment=($wan)
    }
    :local wanArr {
    {
    "remoteAddress"=$"remote-address";
    "dstAddress"=8.8.8.8/32;
    "gateway"=8.8.8.8;
    "distance"=2;
    "wan"="pppoe-out-cat"
    };
    }
    :local intf1 [/interface get $interface];
    :foreach w in=$wanArr do={
    if (($w->"wan") = ($intf1->"name")) do={
    $createRoute remoteAddress=($w->"remoteAddress") dstAddress=($w->"dstAddress") gateway=($w->"gateway") distance=($w->"distance") wan=($w->"wan")
    }
    }"
    /interface pppoe-client
    add disabled=no interface=ether2-cat name=pppoe-out-cat password=71560 profile=pppoe-profile-cat [email protected]
    /interface bridge port
    add bridge=bridge-lan interface=ether8-nas
    add bridge=bridge-lan interface=ether9-nas
    add bridge=bridge-lan interface=ether10-office
    /ip address
    add address=10.10.2.99/8 interface=bridge-lan network=10.0.0.0
    add address=192.168.1.2 interface=ether1-ais network=192.168.1.1
    /ip cloud
    set ddns-enabled=yes
    /ip dhcp-server lease
    add address=10.10.2.31 client-id=1:c4:2c:3:7:eb:98 mac-address=C4:2C:03:07:EB:98 server=dhcp1
    add address=10.10.2.5 client-id=1:f4:4d:30:64:eb:75 mac-address= F4:4D:30:64:EB:75 server=dhcp1
    add address=10.10.2.10 client-id=1:1c:69:7a:68:e0:d7 mac-address=1C:69:7A:68:E0:D7 server=dhcp1
    add address=10.10.2.12 client-id=1:f4:4d:30:64:e4:c9 mac-address= F4:4D:30:64:E4:C9 server=dhcp1
    add address=10.10.2.14 client-id=1:f0:79:59:8d:97:73 mac-address=F0:79:59:8D:97:73 server=dhcp1
    add address=10.10.2.3 client-id=1:f4:4d:30:64:e7:f9 mac-address= F4:4D:30:64:E7:F9 server=dhcp1
    add address=10.10.2.16 client-id=1:1c:87:2c:63:2d:5b mac-address=1C:87:2C:63:2D:5B server=dhcp1
    add address=10.10.2.9 client-id=1:f4:4d:30:64:dd:3f mac-address= F4:4D:30:64:DD:3F server=dhcp1
    add address=10.10.2.6 client-id=1:f4:4d:30:64:e7:37 mac-address= F4:4D:30:64:E7:37 server=dhcp1
    add address=10.10.2.8 client-id=1:f4:4d:30:64:e6:e8 mac-address= F4:4D:30:64:E6:E8 server=dhcp1
    add address=10.10.2.7 client-id=1:f4:4d:30:64:ee:1c mac-address=F4:4D:30:64:EE:1C server=dhcp1
    add address=10.10.2.205 client-id=1:f4:4d:30:64:f0:b5 mac-address= F4:4D:30:64:F0:B5 server=dhcp1
    add address=10.10.2.11 client-id=1:f4:4d:30:64:e9:45 mac-address= F4:4D:30:64:E9:45 server=dhcp1
    add address=10.10.2.13 client-id=1:f4:4d:30:64:eb:69 mac-address=F4:4D:30:64:EB:69 server=dhcp1
    add address=10.10.2.2 client-id=1:f4:4d:30:64:d0:b mac-address=F4:4D:30:64:D0:0B server=dhcp1
    add address=10.10.2.50 client-id=1:1c:69:7a:6c:9c:34 mac-address=1C:69:7A:6C:9C:34 server=dhcp1
    add address=10.10.2.100 client-id=1:0:11:32:fe:e0:65 mac-address= 00:11:32:FE:E0:65 server=dhcp1
    /ip dhcp-server network
    add address=10.0.0.0/8 dns-server=8.8.8.8 gateway=10.10.2.99
    /ip dns
    set servers=8.8.8.8,8.8.4.4,1.1.1.1
    /ip firewall address-list
    add address=10.0.0.0/8 list=LAN
    /ip firewall filter
    add action=accept chain=input disabled=yes dst-port=8291 protocol=tcp
    /ip firewall mangle
    add action=mark-connection chain=prerouting comment="INCOMING CONNECTIONS MARK AS ncm-ais,ncm-cat" connection-mark=no-mark in-interface=ether1-ais new-connection-mark=ncm-ais passthrough=no
    add action=mark-connection chain=prerouting connection-mark=no-mark in-interface=pppoe-out-cat new-connection-mark=ncm-cat passthrough=no
    add action=mark-connection chain=prerouting comment="PCC TO HANDLE HTTP/HTTPS " connection-mark=no-mark disabled=no dst-address-list=!LAN dst-address-type=!local dst-port=80,443 in-interface=bridge-lan new-connection-mark=ncm-ais passthrough=yes per-connection-classifier=src-address:2/0 protocol=tcp
    add action=mark-connection chain=prerouting connection-mark=no-mark disabled=no dst-address-list=!LAN dst-address-type=!local dst-port=80,443 in-interface=bridge-lan new-connection-mark=ncm-cat passthrough=yes per-connection-classifier=src-address:2/1 protocol=tcp
    add action=mark-connection chain=prerouting comment="PCC TO HANDLE HTTP3 TRAFFIC AKA QUIC" connection-mark=no-mark dst-address-list=!LAN dst-address-type=!local dst-port=80,443 in-interface=bridge-lan new-connection-mark=ncm-ais passthrough=yes per-connection-classifier=src-address:2/0 protocol=udp src-port=""
    add action=mark-connection chain=prerouting connection-mark=no-mark dst-address-list=!LAN dst-address-type=!local dst-port=80,443 in-interface=bridge-lan new-connection-mark=ncm-cat passthrough=yes per-connection-classifier=src-address:2/1 protocol=udp src-port=""
    add action=mark-connection chain=prerouting comment="NTH FOR NON HTTP/HTTPS" connection-mark=no-mark dst-address-list=!LAN dst-address-type=!local in-interface=bridge-lan new-connection-mark=ncm-ais nth=2,1 passthrough=yes
    add action=mark-connection chain=prerouting connection-mark=no-mark dst-address-list=!LAN dst-address-type=!local in-interface=bridge-lan new-connection-mark=ncm-cat nth=2,2 passthrough=yes
    add action=mark-routing chain=prerouting comment="MARKED/SPLIT TRAFFIC" connection-mark=ncm-ais dst-address-type="" in-interface=bridge-lan new-routing-mark=nrm-ais passthrough=no
    add action=mark-routing chain=prerouting connection-mark=ncm-cat dst-address-type="" in-interface=bridge-lan new-routing-mark=nrm-cat passthrough=no
    add action=mark-routing chain=output comment="INCOMING TRAFFIC FROM WAN" connection-mark=ncm-ais dst-address-type="" new-routing-mark=nrm-ais out-interface=ether1-ais passthrough=no
    add action=mark-routing chain=output connection-mark=ncm-cat dst-address-type="" new-routing-mark=nrm-cat out-interface=pppoe-out-cat passthrough=no
    /ip firewall nat
    add action=masquerade chain=srcnat out-interface=ether1-ais
    add action=masquerade chain=srcnat out-interface=pppoe-out-cat
    add action=masquerade chain=srcnat src-address=10.0.0.0/8
    add action=dst-nat chain=dstnat dst-port=10390 protocol=tcp to-addresses=10.10.2.50 to-ports=5900
    add action=dst-nat chain=dstnat dst-port=10391 protocol=tcp to-addresses=10.10.2.31 to-ports=5900
    add action=dst-nat chain=dstnat disabled=yes dst-port=10392 protocol=tcp to-addresses=10.10.2.99 to-ports=8291
    /ip firewall service-port
    set ftp disabled=yes
    set tftp disabled=yes
    set irc disabled=yes
    set h323 disabled=yes
    set sip disabled=yes
    set pptp disabled=yes
    set udplite disabled=yes
    set dccp disabled=yes
    set sctp disabled=yes
    /ip route
    add check-gateway=ping comment=pppoe-out-cat distance=2 gateway=8.8.8.8 routing-mark=nrm-cat
    add check-gateway=ping comment="static ether1-ais" distance=1 gateway=1.1.1.1 routing-mark=nrm-ais
    add comment="static ether1-ais" distance=1 dst-address=1.1.1.1/32 gateway=192.168.1.1 scope=10
    add comment=pppoe-out-cat distance=1 dst-address=8.8.8.8/32 gateway=100.70.192.1 scope=10
    add comment=pppoe-out-cat distance=1 dst-address=8.8.8.8/32 gateway=100.70.192.1 scope=10
    /ip service
    set telnet disabled=yes
    set ftp disabled=yes
    set www disabled=yes
    set ssh disabled=yes
    set api disabled=yes
    set api-ssl disabled=yes
    /ip upnp
    set enabled=yes
    /ip upnp interfaces
    add interface=ether1-ais type=external
    add interface=pppoe-out-cat type=external
    add interface=bridge-lan type=internal
    /system clock
    set time-zone-name=Asia/Bangkok
    /system identity
    set name="JPAC GATEWAY"

  8. ชื่อ นามสกุล ชื่อ นามสกุล

    guy after i disabled this 2 mangle rule

    add action=mark-connection chain=prerouting comment=”PCC TO HANDLE HTTP/HTTPS ” connection-mark=no-mark disabled=no dst-address-list=!LAN dst-address-type=!local dst-port=80,443 in-interface=bridge-lan new-connection-mark=ncm-ais passthrough=yes per-connection-classifier=src-address:2/0 protocol=tcp

    add action=mark-connection chain=prerouting connection-mark=no-mark disabled=no dst-address-list=!LAN dst-address-type=!local dst-port=80,443 in-interface=bridge-lan new-connection-mark=ncm-cat passthrough=yes per-connection-classifier=src-address:2/1 protocol=tcp

    as above in example is
    ###For old school HTTP/HTTPS traffic###
    ###50% going to ISP1###

    its works without any problem guy i don’t understand this

  9. The issue I see with this kind of traffic identification is that you never know in what port is HTTP/HTTPS going to be in every network, so you may break some sites that use a non-standard http/https ports (80,443), another issue I see is that nth isn’t really doing a proper per packet load sharing because packets are being marked by connections and the linux iptables mangle engine will respect that over any packet routing mark, if it really was doing a per-packet load sharing you will be experience lots of protocol breaks because they’re not expecting connections to come from different path or ip address, they will ask? you don’t have an stablished connection with me?, so drop the previous connection because will timeout and stablish a new one, and that is basically a retransmission.

    • I’ve been using this setup for years and never saw a problem with non-standard HTTP/HTTPs ports, as anyways the Nth rules mark the connection.

      The Nth here is not per-packet load balancing, it is simply preferred to traditional PCC as it is more randomised on a per-packet basis for connection marking and hence gives us better load distribution than just traditional PCC. Which can be proven by multi-threaded download/upload.

      If you want true native per-packet load balancing, you need your own ASN with a multi-homing IP Transit setup.

      You’re welcome to use any variation of the original setup, but for me, I’m using this config in all production cases and no reported cases have been reported for broken traffic or protocols. You can also use WireShark on the WAN interfaces and see that no traffic breaks/drops with my config.

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.