8                             
                                                                                                                                                                                                                                                                              888                           
                   S SSXSSSXX%S@X%X@X%X@          8 X X@S@X@X@X@X@@@@@@@@@@@@@8             XXX@X@X@X@@@@@@@@@@@@8@8@8@8            888888888888888888888888888888                                                     88888888888888888888                   888888888888    88@8                          
                    %SX               X S         @S%@                       8XS            8@X@                       @@@            888                       888                                                    8888               888                 88       88X    888888                        
                       X%               SSS       XSX  @                       8S@          8X  8X                      @@@@            8@8                       888                                                  88  88               888               88       888    88  88tX                      
                         S                % S     S%X   X                        8%X        8X    8                       XX@            @@8@                      8888                                                88    88              888              88       888    88    8888                    
                           S                SSX   %X@     S                       8S@@      8      8S                       8@@            @X@                       888                8                              88      8               888            88       888    88      888                   
                            X S S S S S S XS 8X    @S      % SSX8 SSXSSSXSSSXX%S@X88X@@     8S       8%XX@X@X@X@X@@@@@@@@@@@XXS@             X@X@X@X@@@@@@@@@@@@8@8@88@888            88888                            88       88              888           8        8@8    88        888                 
                             X                @    S                %S                      @        8               8       8X%             8@X               @X                    8888@8@8                          88         88             88           88       888    8          8888               
                            .S  .S            S    XS                S                      XS       @               8       8SS             8X8               XX                     @8X@@8X88                        88           8            88           88       888    88           888              
                            :S    :S          S    XX                S                      S        X               @       8 X             8X                8@                       @@@X8X8X@                      88            88          88           88       888    88             888            
                            ;S      :        .S   .S                 S                       X       %               X       @S              8S8               8X                         @@SXX8S@@                    88              88       888           88       888888888              8888          
                            tS       ;S      ;S   :X        .        S                       S                       S       X%@             8SS               8S                           X@XS8S@@@                  8@8               8888888888           88       88      88              88@          
                            %X         tS    tS   ;X%       :       .S S S S                 S                       %       S%X             @SS               8S                             8XS8S8@@X                  @8@               888                88       8       88 8             88          
                            SX           %   %X   tS        ;       :X X X X S               S                               %X@             X @               8                                8X8S8%XX@                 @@@@              8888              88       8       8   88           88          
                            XX             S SX   %8.       t       ;        :X S            S                                XX             SSS               @S                                  8SXSXSXS            XX   @X@               @88             88       8       8     88         88          
                            @X              X8;   SS%       %       t          ;S.          :S                                S               XS               X                                     8S8X8SXS          8@XX   8@@               @@@           8@       8       8       8        88          
                            8@               X8   X8t       S       %            tS;        ;S       :                        X%              SSX              SS                                      8@@X88 X        8X8 @   8X@S              @@@@         @8       8       8        88      88          
                            8X               @S   @XS       X       S             %@;S      tS       ;               .        XS              XS  S             S                                       8S8SSSSX8      8X  8X@   8XS               8@@        @X       @       8          88    88          
                            8X               8@   888       @       XXXXXXSXSXSXS@St8:%     %X       t               :       .S               SS                S                                     @ XSS8 SX        8@    8S@   8S@               8@@      XX       @       @            88  88          
          8@8X8X8@8X8X8@8X8@8@               8X   8XS       8       @8                      SX       %               ;       :X               XS     S          S                                   X XSX88S           8X     8S@8  8X@               88@8    8@       X       @              @@88          
           8XSXS@S@SXXXX@S8888               8X   888       8       8X                      XX       S               t       ;X%             .S        S        S                                 % @X 88XS            @S       8 X XX@8 X X@S@X@X@X@@8@8@%   8X       8@@888@XXX              @@@          
            8888             8               8@   888       8       8X                      @X       X               %       tS              :X         .       S                                SS@ %88X              XX         @                      8S   8S       8@@X@XS8 8%              @@          
              8X8            8               88   888       8       8X                      @X       @               S       %8.             tX%          :    .S                              X% S8888                %           X                     8    8S       8SS       8X@            8@          
                8@8          8               8@   888       8       8@                      8@       @               X       SS%             %X            ;S  ;S                            X S 888                    X            S                   @S   8        8S          8S@          8@          
                 8@8X        8               8X   8@8       8       8X8X8@8X8               8X       8               @       X8t   XXSXSX%SS@S88             tXS                           S% 888S                      S              %                 X    @S       8SS          8X@         8X          
                   8X8X      8               8X   8@8       8@8X8X8X8%XXX@XX888             8X8      8               8       @XS     X@SX8%8;8 :SSX%SSX%S%XtXS:8 %                      :X S 888                        S                S              SSX   X S      @ S            8S        8X          
                     8X8     8               8X     8@8     8                 8@8            888     8               8     88S        @8%8                      S8:%                 ;S:@ 8 88                           X                 S           SSX     S%XS    XSX              8 X     8           
                       8@8   8               88      8@8X   8                  8@8X            8@8   8               8   88S8           8S8@                      SXS                tXS 8 8                               S                          XX         %X    SSX               @ S    @S          
                        8@8X 8               8@        8X8  8                    8X8             8@8 8               8  888               8X8                       XXX                %8                                   .S                      X%              S   S                  XSX X S          
                          8X8X               8X          8@8X                     8X8X            8@8@               8@8@                   8@8                      @8SX                                                    :X S               S S%%                X @%8                   S @8           
                            8S8X8@8X8X8@8X8X8S@           8X8X8X8@8X8X8@8X8X8@8X8X8@@@8S            8S8@8X8X8@8X8X8@8SX                      888X8X8@8X8@8X8@8X8@8@88S@8XS                                                     ;S S.S S S % S S X S                    XS                     %  S          
                                                                                                                                                                                                                                                                         X                       S          
                                                                                                                                                                                                                                                                                                            
                                                                                                                                                                                                                                                                                                            
                                                                                                                                                                                                                                                                                                            
                                                                                                                                                                                                                                                                                                            
                                                                                                                                                                                                                                                                                                            

NanoPi R4S: Implementing a Transparent Network Monitor

=============================================================


Now that hobbyist ARM boards come with multiple 1GbE network ports, they start to have application in the network realm. The NanoPi R4S is equipped with up to 4GB of RAM, and a Rockchip 3399 64bit CPU which has 6 cores @ 1.8GHz, readily capable of being a homebrew router or transparent proxy. Let's focus on how to implement a transparent network monitor for starters. Onboard you will be generating Netflow and SNMP that can be consumed by ntopng to monitor your local network. You will be able to see top talkers, flows, device fingerprints and more. All from a convenient web interface with historical lookback. You can deploy behind or infront of the router, between a modem and the router, or between a single machine while staying invisible to Layer 3.


Things you will need:

******************


  • NanoPi R4S (1GB/2GB/4GB memory)
  • MicroSD Card (8GB+)
  • MicroSD Card Reader
  • USB-C wall power adapter
  • USB to Ethernet Jack
  • 3x Cat5e / Cat6 Cables
  • Ancillary PC to flash the MicroSD card (*PC1)
  • Intermediate level of Linux knowledge
  • (optional NanoPi R4S Acrylic Case / USB to UART cable for debugging)


Notes:

******


  • In this tutorial, I am using an additional ethernet jack via USB to provide an out of band network. This is to ensure that we are not inspecting ntopng traffic in addition to what we want to monitor.



Steps:

******

                                                X.                                                  
                                              : 8    @                                            
                                           t88..;888XX@88                                           
                                        @:@t :@8%@@ @ SSt8 %                                        
                                       X@ ::%:tX88@8888@  88;%                                      
                                        t8@8;X8SSX88@88@ ;:@;@8                                     
                                :      .%:88@@@8X88@888 :X .t@Xt8                                   
                             :S:88   @S8@8 t8.8@8@88@St%8S88;S8@ ;S                                 
                           t8 X88X8X. %88:@%8S@.88t%8888@%@88XS8;88%:                               
                        S@@8SS   8;S8X888888:S8:XXXXS888t@8S%888:%                              
                       ::SX88S;%@%X X SXX8888Stt.:X8@8X888t8888888t @8S@                            
                        ;.8@8@X 8S8X%S8 S X88 :@8t8X8S%;%8888888:8@8SX888                           
                       @.SX88@S8@@88X88.;%8X8888@88@ 8X@888S88..88S%S8@8X8                          
                    SX888t@X:@888X8X:%Xt8X888@8888 @X88@8%888.88.:.::;;88888                        
                 t8;8X888888888XX@S;S8X@8S@S@888@8XSSS;888%8SS88@t88 88X888S@                     
              ttttt..88888888SX.8%%SX88S888t%@8888888%8%8@88@8S@;88;8%8%8;888X8:8                   
           ;ttttttt%%t@88.88@.@%8888S8S888:88888@88888@S8t888;8S@8.:88t88X8S88t88@                  
       ;;;t;;tttttt%%%tS8%8 S 888@S8:8t.88@888888SX88888%XSt88t888X@:t%8888:8;8t8:8X                
       88;;t;tttttttt%t;8;:8.@88X.888S8@@8888888@88X@8888@%8X8S8X.88X888t88X8;.::@ 8.8  %           
    S8@ ..:;tt;tttttttt%;t8X.888%8.XX@88@888SXX88tX8XS88888@%8%888XtX8:8%8;X8tSt 8 tXX X8           
 8;@8;SS :%S;tttttttttt8tS :@8tt8:8S%%tS8@8@88@%X8XS@8SX8888@t@@t@8%;8@.%SX8S8%.;8XS8@%S8%          
888%S@ %t.:;;;t;ttttt% t:St;8@8:X888.8%t;8X8@88@%%88%88S@888X%888;8X8 ;%SSX@@@@8:;;S@%@8XS.         
   8.X%.:8t@%:;t;%S8@S%8S888 X;:%X8888S8S;S8@8888@tX88888XSS8@t8@ %;tttSSX@@@88Xt%8X88XXS888%       
   88.8.888S.S;@ S@%;%%8S. X888:t@8X;.88t%S:@88888888@8S%8S;S.:t:;:;%%%SX@8888@@@S;@@X88@88@8S      
    8@X8%:tX8:;8X8tS88X8 S88%88888t@88X%8X8S%t888888SS@@:@:;XS;:;::ttSXX@8@888888@@8%X88t88 88X8    
      X:888@;X8X%88.8888X888X88@888@X888%88 8@t%8X:X@8%8.SX88%88:%S%SSXX@@88888888888X SX@8SX.S8:%  
       SX8888S8:.@8%X @;8:X@8@8%X..X.S;X;8888S@X88XS:S88X%888X8X.%tSSSXX@8888888888@88@S:@X%S8@88XS 
         X8S tt;8S8.8X:888 X8X. 88@  8:@888X.8.@@St@%@@888S;: 8888;;%XX88888888888888888X8  %888@88 
           88@S@888 X8 8%S8S88 X;8888X@88:..@88;Xt;t%X@88888888888 ;X%@@88888888888888@X8%8@tt88    
            @@888;@8;;% 8888@%X8.%8X @8 .%  .@t8;t%SXXX888888@::S@8;;8X88888888888888@S .@S@8X8X    
             888@@88@8 8 ;t@@.8: S.@@88:8 8X@.;%t%SSXX88888888@88@8Xt 8:@888888888@t8SX8S.8SX       
                X8@X;8X.;8@ @@ @t.8:8:XS.:tt;ttSSXXX@8@888888X@888%t;.8S888888X:XS8@88:         
                X888t888  :;8.%8%t8;8@88 ;8t@;tt%SXSX@@@8@88888@88X;88S;@8S8SXX@St@88@:;X           
                  t.88;8t%X%.8%88. 888t.8;@%X@;tSSSXX@@888888888X88@XS  8888S:@%888@8t              
                   88. t8t  @8S8 .%888@  8X8S8tSSXX@8X888888888888XXt8@X88%X888X88                
                      @888;@%  8X.88X8;;88@@%S%t%XX@@88@8888888888Xt;X ;;8 @X8@t                  
                      888X@8X88 8 %t@ :SSt%;@888XS;%@88@8888888888X% X.%S.8@8:.S                    
                        %.8S:t8@88@88@%:t;@:8  .S:  XX8888888888%%tS%S% 88;SX8S                     
                           @Xt:X.X%t8XS8S 8;S.t.; :: 88888888@%tS8%@88;S@@8X                        
                           88S.888888@@;8:;;X 8@%X8S:8@8@SS@8%8@8Xt%@                             
                             88;;8@@S.SX8  :88X8  %X8SS 8t@X8@@8                                
                              S@88888 ;@ .:.;@88.%.@:8t: 8SX8@8@8@.                                 
                                 S@ttS::;:S8X88:%8 S8 ;t88S88%                                    
                                 %@8@Xtt@8t@X8 8Xt:8 .;8S 8XX @                                     
                                   8%:88@88.:@:8t %XS@8;:8                                        
                                    %8St@%8t88..:;S.tSX.XX                                          
                                      8t8S:S;::.::;t.@X@;                                           
                                       %8X@@;8;:;S%t                                                
                                          @%.:%t8 %                                                 
                                          .: .                                                      

  • Assemble your NanoPi RS.

  • Connect your MicroSD Card Reader to *PC1 (or use the onboard one, if available.)

  • Download Balena Etcher on *PC1 [X]

  • Download Armbian Buster on *PC1 [X]

  • Flash the Armbian Buster image using Balena Etcher or dd to the MicroSD Card. Be sure to decompress the .xz image and flash the resulting .img file.

  • Insert the flashed MicroSD card into the NanoPi R4S. Connect the USB to Ethernet Jack to the NanoPi R4S. Connect the USB jack to your network that has DHCP enabled. Login to your router's administration page and find the DHCP leases, or a list of connected devices and their respective IP addresses. Now connect your NanoPi R4S to power using the USB-C connector. It will take up to 2minutes to boot up, but you should see a new DHCP lease issued to your device. If you do not, try some of the following steps:
    * Power cycle the NanoPi while using the onboard ethernet port.
    * Reflash the MicroSD card with Armbian Buster.
    * Attach a USB to UART cable to the board, and open a serial console to check for errors.

  • Assuming all went well, ssh to the IP discovered in step #6.

  • user: root
    password: 1234
    
       $ ssh [email protected]
       The authenticity of host '192.168.1.2 (192.168.1.2)' can't be established.
       ECDSA key fingerprint is SHA256:O+a++zzltzlr63zp1DQn+08y0THRbwnX9HfN3wyrehc.
       Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
       Warning: Permanently added '192.168.1.2' (ECDSA) to the list of known hosts.
       [email protected]'s password:
        _   _ ____  _   ____  _  _  ____
       | \ | |  _ \(_) |  _ \| || |/ ___|
       |  \| | |_) | | | |_) | || |\___ \
       | |\  |  __/| | |  _ <|__   _|__) |
       |_| \_|_|   |_| |_| \_\  |_||____/
       
       Welcome to Armbian 21.05.1 Buster with Linux 5.10.35-rockchip64
       
           
    Upon logging in you will be prompted to do some setup. Choose a new password & shell. Generate your locale.

  • Now inspect, and note your network devices. The enx000xxxxxxxx interface will be your USB jack. enp1s0 and eth0 (may be named eth1) will be your interfaces you will be bridging together.

    
       root@nanopi-r4s:~# ifconfig
       enp1s0: flags=4163  mtu 1500
               ether 2e:2a:9f:de:ad:be  txqueuelen 1000  (Ethernet)
               RX packets 771158  bytes 91913945 (87.6 MiB)
               RX errors 0  dropped 0  overruns 0  frame 0
               TX packets 1385036  bytes 1345777980 (1.2 GiB)
               TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
       
       enx000ec6c9ddc6: flags=4163  mtu 1500
               inet 192.168.1.2  netmask 255.255.255.0  broadcast 192.168.1.255
               inet6 fe22::411f:42a6:d34d:b33f  prefixlen 64  scopeid 0x20
               ether 00:0e:c6:c9:de:ad  txqueuelen 1000  (Ethernet)
               RX packets 652526  bytes 550076502 (524.5 MiB)
               RX errors 0  dropped 0  overruns 0  frame 0
               TX packets 257966  bytes 42645158 (40.6 MiB)
               TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
       
       eth1: flags=4163  mtu 1500
               ether 68:27:19:a5:be:ef  txqueuelen 1000  (Ethernet)
               RX packets 1385646  bytes 1346350291 (1.2 GiB)
               RX errors 0  dropped 19  overruns 0  frame 0
               TX packets 770843  bytes 89234143 (85.1 MiB)
               TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
               device interrupt 35
       
       lo: flags=73  mtu 65536
               inet 127.0.0.1  netmask 255.0.0.0
               inet6 ::1  prefixlen 128  scopeid 0x10
               loop  txqueuelen 1000  (Local Loopback)
               RX packets 861733  bytes 90523009 (86.3 MiB)
               RX errors 0  dropped 0  overruns 0  frame 0
               TX packets 861733  bytes 90523009 (86.3 MiB)
               TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
          


  • Without creating a persistent rule, eth0 will toggle between being named eth0 and eth1 on reboot. Let's fix that, create a file with nano /etc/udev/rules.d/70-persistent-net.rules and replace the MAC address in ATTR{address} with the MAC of your eth0 or eth1 in the following:

  • SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="68:27:de:ad:be:ef", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth*", NAME="eth0"

  • If you have an eth1 and no eth0, reboot at this point. Then edit your network configuration as follows, renaming the network devices to those discovered in ifconfig.
    nano /etc/network/interfaces

  • source /etc/network/interfaces.d/*
    auto lo
    iface lo inet loopback
    allow-hotplug eth0
    allow-hotplug enp1s0
    iface eth0 inet manual
    iface enp1s0 inet manual
    allow-hotplug enx000ec6c9ddc6
    iface enx000ec6c9ddc6 inet dhcp
    
    auto br0
    iface br0 inet manual
      pre-up ifconfig eth0 down
      pre-up ifconfig enp1s0 down
      pre-up brctl addbr br0
      pre-up brctl addif br0 eth0
      pre-up brctl addif br0 enp1s0
      pre-up ifconfig eth0 0.0.0.0
      pre-up ifconfig enp1s0 0.0.0.0
      pre-up ifconfig br0 0.0.0.0
      post-down ifconfig eth0 down
      post-down ifconfig enp1s0 down
      post-down ifconfig br0 down
      post-down brctl delif br0 eth0
      post-down brctl delif br0 enp1s0
      post-down brctl delbr br0

    This configuration ensures that our onboard network interfaces are left unconfigured and only bound to the bridge which possess no IP address on the network.

  • Disable & stop NetworkManager. Let's rely on the legacy network configuration system since NetworkManager is not respecting /etc/network/interfaces in Armbian. At this point we will only be able to access our NanoPi through the out of band network. If it becomes unreachable, fixing any errors will mean connecting through the serial port or mounting the MicroSD under another system to fix our configuration.

  • root@nanopi-r4s:~# systemctl disable NetworkManager
       Removed /etc/systemd/system/dbus-org.freedesktop.nm-dispatcher.service.
       Removed /etc/systemd/system/multi-user.target.wants/NetworkManager.service.
       Removed /etc/systemd/system/network-online.target.wants/NetworkManager-wait-online.service.
       

  • ifup br0 to bring up the interface.

  • Install ntopng dependencies.

  • apt-get install libmariadb-dev libmariadb3 libzmq3-dev libsnmp-dev \
    librrd-dev libexpat1-dev libreadline5 libsqlite3-dev libmaxminddb-dev \
    libcurl4-openssl-dev libpcap-dev libjson-c-dev redis-server autoconf \
    libtool gcc pkg-config


  • Clone & build ntopng, and nDPI.

  • git clone https://github.com/ntop/ntopng && cd ntopng \
    && git clone https://github.com/ntop/nDPI.git \
    && cd nDPI && ./autogen.sh && ./configure && make -j6 && cd ../ \
    && ./autogen.sh && ./configure && make -j6


  • Ensure redis is running.

  • systemctl start redis-server


  • Let's create a systemd service for ntopng. nano /etc/systemd/system/ntopng.service

  • [Unit]
    Description=ntopng
    After=network.target
    
    [Service]
    Type=simple
    User=root
    Group=root
    WorkingDirectory=/root/ntopng
    ExecStart=/root/ntopng/ntopng --dont-change-user
    Restart=on-failure
    
    [Install]
    WantedBy=multi-user.target

  • Enable and start ntopng systemctl enable ntopng && systemctl start ntopng now you can login to the web interface at http://your_ip:3000 and set a new password. Connect the physical ports to the network we want to monitor and you will see flows should start being collected.


  • Thoughts:

    *************


    Now that you are able to see what your network is doing, you will be able to troubleshoot issues easier (or understand why your network is talking to AS8003.) Keep in mind it can be a source of dropped packets, if you are running it inline and not on a mirrored port. I have not (yet) seen the NanoPi introduce issues on an 1GB home network. If we wanted to scale this up to 10GB we could repeat this process on beefier hardware that supports DPDK.