On linux, by default, even when you have multiple network interfaces, you can only have 1 default gateway. This wasn’t working out for me because I had traffic coming in from network 2 routed back to the originator through network 1 and dropped because return packets were coming from an unknown source. Using the old route tool, you are only able to specify a specific gateway for a specific network, but this wouldn’t work out because traffic from both network 2 and 1 came from the internet. We couldn’t segment on that. So what to do?

iproute2 to the rescue!

You can set up rules that traffic from a specific IP address on a specific interface is routed through a separate gateway than the default gateway. Here’s the code I used:

echo "1 rt2" >> /etc/iproute2/rt_tables
ip route add dev tun0 src table rt2
ip route add default via dev tun0 table rt2
ip rule add from table rt2
ip rule add to table rt2 is network 2. is the 2nd IP address on network 2. is network 2 gateway.

This should work without disrupting regular traffic on the default gateway. Only traffic received from the secondary NIC/IP will be routed to the secondary gateway.

More reading…

When you have multiple network interfaces on a linux box, you can link them all together using a network bridge. This connects them all at the data-link level (OSI layer 2). To do this, use the brctl command.

brctl addbr br0
brctl addif eth0
brctl addif eth1
brctl addif eth2

This turns your linux box essentially into a hub. I don’t know how smart linux routes different packets – it’d be great if linux keeps some sort of MAC routing table.

I recently worked on making OpenWRT work with a 3G usb modem. Here’s how I did it – hopefully it’ll save you some hours of frustration…

My Setup:

RouterStation Pro
NovaTel MC760Virgin Mobile’s Version
Virgin Mobile’s Broadband2Go – conveniently $40/month – no contract
OpenWRT – Backfire 10.03.1-rc3 – openwrt-ar71xx-ubnt-rspro-squashfs-factory.bin

I chose the RS Pro because it was the most powerful router that I could find (being a work project, I wasn’t pinching pennies). The novatel MC760 supposedly had good linux support so I was crossing my fingers and hoping that the setup would work for me. Virgin Mobile’s no contract was attractive for development. Also, the MC760 is supported on the Sprint and Verizon networks, so when this goes into production, we could easily (theoretically :-P) switch over to one of those ISPs if they proved to be more reliable or cheaper with a contract.

You have to activate your modem first on a windows or OS X machine. It uses some web interface and you put in some numbers that is stored into the modem. Do that first.

Install Latest OpenWRT:
Wrote about that already… Read Upgrading Routerstation Pro Firmware

Detecting USB Device:
# stores repo list in /tmp/ - you must run this everytime you want to use opkg install after a reboot
opkg update
opkg install kmod-usb-uhci kmod-usb-serial kmod-usb-storage usbutils sdparm comgt
opkg install microcom # for testing

The MC760 has this annoying CDROM drive of drivers that must be ejected before the real modem will pop up. If you do:
Bus 002 Device 003: ID 1410:5031 Novatel Wireless

You want to see 1410:6002. To get that to work:
/usr/bin/sdparm --command=eject /dev/`dmesg|grep CD-ROM -A1 |grep "type 5"|tail -1|awk '{ print $6 }'`
# searches dmesg for the CDROM device to eject

Make sure it has changed to 6002. If not, your modem is not enabled. After this, time to get the serial device working.
rmmod usbserial
insmod usbserial vendor=0x1410 product=0x6002

dmesg should show:

usbserial_generic 2-1:1.0: generic converter detected
usb 2-1: generic converter now attached to ttyUSB0

Now let’s check if the serial modem works…

microcom -D/dev/ttyUSB0

The screen will look like it’s frozen, type ATZ and hit enter. If you receive an OK, that means we’re doing good! Type ~x to quit.

Dialing the Modem:

We now have to setup the pppd client to dial in and get IP networking going.

mkdir /etc/ppp/peers/

make the following files:


#idle 300
maxfail 0
user internet
password internet
connect ‘/usr/sbin/chat -t3 -f /etc/ppp/peers/virgin_chat’


OK ‘ATQ0 V1 E1 S0=0 &C1 &D2 +FCLASS=0′

Let me highlight a few things about the config files… #777, that’s the Sprint, Verizon, and Virgin Mobile dial in number. The username and password is useless, you can put anything in there. Make sure your /dev/ttyUSB* device is the correct device. The persist command will make pppd try indefinitely to reconnect.

After setting the config files, try pppd call virgin. Check logread to see if the connection worked. If it did, you should see:

Aug 19 02:58:02 OpenWrt daemon.notice pppd[1261]: pppd 2.4.4 started by root, uid 0
Aug 19 02:58:05 OpenWrt daemon.info pppd[1261]: Serial connection established.
Aug 19 02:58:05 OpenWrt daemon.info pppd[1261]: Using interface ppp0
Aug 19 02:58:05 OpenWrt daemon.notice pppd[1261]: Connect: ppp0 <--> /dev/ttyUSB0

ifconfig should show a ppp0 device and ping will work.

Sharing the internet with the other computers:
If you try pinging from a computer connected to the LAN ports, you’ll notice that NAT is not working and you get a nasty “Destination Net Unreachable”. This is because we haven’t configured this ppp0 device to work with openwrt’s UCI so iptables does not know to NAT packets to the ppp0 device.

Let’s make ppp0 the default WAN device.

Note: If you want to do load balancing and more complex stuff, you’re on your own.. :-P But if you figure it out, please let me know. I would be interested to know how you set it up.

/etc/config/network – make these changes:

#config interface wan
# option ifname eth0
# option proto dhcp

config interface wan
option ifname ppp0
option proto none

After this, run the following:

ifup wan
#Unfortunately, this wipes out the ip address info for the ppp0 device, so you will need to reset your ppp0 device.
kill `pidof pppd`
pppd call virgin

DNS Settings:

If you didn’t notice, DNS was down. You can fix that by going to /etc/config/dhcp and changing resolvfile to option resolvfile ‘/tmp/resolv.conf.ppp’.

Make Settings Permament:

First, let’s make the usbserial device automatically load.

echo usbserial vendor=0x1410 product=0x6002 > /etc/modules.d/60-usb-serial

There are many ways to start the connection on boot, some make /etc/init.d/ scripts, or bind it to when the usb modem becomes active (in /etc/hotplug.d/usb) or use openwrt network start scripts (we sorta can’t since openwrt isn’t managing our connection). I am paranoid so I made a cronjob that checks every minute if the modem is connected. I also bound this same script to the usb modem inserted hook.



# Copyright (C) 2010 bashusr.com
case “$ACTION” in
/etc/scripts/initmodem.sh &

mkdir /etc/scripts and add this:


## bashusr.com
if kill -0 `cat /tmp/initmodem.lock` 2> /dev/null; then

echo $$ > /tmp/initmodem.lock

if (pidof pppd > /dev/null); then
if (/usr/sbin/lsusb|grep -q ‘Novatel’) && !(/usr/sbin/lsusb|grep -q ‘1410:5031′); then
echo “`date` – modem was or is unplugged! killing pppd…” >> /tmp/novatel_modem.log
kill `pidof pppd`

if (/usr/sbin/lsusb | grep -q ‘1410:5031′); then
while (/usr/sbin/lsusb | grep -q ‘1410:5031′); do
sleep 5
if ! [ -e “/dev/`dmesg|grep CD-ROM -A1 |grep “type 5″|tail -1|awk ‘{ print $6 }’`” ]; then
echo “`date` – /dev/`dmesg|grep CD-ROM -A1 |grep “type 5″|tail -1|awk ‘{ print $6 }’` device doesn’t exist! do something
sleep 5
echo “`date` – trying to eject” >> /tmp/novatel_modem.log
/usr/bin/sdparm –command=eject /dev/`dmesg|grep CD-ROM -A1 |grep “type 5″|tail -1|awk ‘{ print $6 }’` 2>> /tmp/novatel_modem

while !(/usr/sbin/lsusb | grep -q ‘1410:6002′); do
echo “`date` – waiting for modem device to come online” >> /tmp/novatel_modem.log
sleep 2


if (/usr/sbin/lsusb | grep -q ‘1410:6002′); then

if (pidof pppd); then
echo “`date` – pppd already running, not dialing again…” >> /tmp/novatel_modem.log
echo “`date` – novatel modem online… dialing!” >> /tmp/novatel_modem.log
ifup wan
ifconfig ppp0 down
pppd call virgin

and last but not least, crontab -e and add

* * * * * /etc/scripts/initmodem.sh

Now you should be able to do anything to your router and if it turns back on, it should connect to the internet by itself. I tried rebooting, pulling the power supply, pulling out and plugging back in the USB modem… as long as it is on and the usb modem is connected it works! I hope this works for you too!

References & Resources:
chatfile from dd-wrt docs (ironically, I couldn’t get this to work under dd-wrt, but their config information was useful!)
Sprint Linux Config Guide
http://josefsson.org/openwrt/dongle.html – this guy’s postings were most useful.

“Seb” wins for putting the instructions for upgrading the firmware in the Routerstation Pro in one simple place.


If you’re wondering (like I did), which version to use (jffs2, squashfs, factory, or sysupgrade?) I found the right one to use is the squashfs version (because of this) and the factory version. The sysupgrade version is for you to upload into /tmp and use the sysupgrade command. Unfortunately, that doesn’t work on the kamikaze version so you’ll have to tftp it at least once. Make sure you select the factory version if you’re doing tftp!! I haven’t tried, but the sysupgrade version could brick your router if you don’t install it from sysupgrade.

The latest version at the time of this posting is backfire 10.03.1-rc3:


Simple ISO Image Creator For OS X

| September 28th, 2010

I needed to convert CDs to ISO for use on my VirtualBox VMs. There are many tutorials and applications out there to do just this, but none I found was as simple as ISOlater. I originally tried using hdiutil (like this guy), but the ISO image would not mount for me after I created the file. ISOlater is a nice simple drag and drop solution and worked for me without any issues. The program is written in applescript with a wrapper application and calls dd and hdiutil. Check it out here:


Masquerade NAT OpenVPN

| September 23rd, 2010

Not a full tutorial, but pointers to how I set up Masquerade NAT OpenVPN.

Primary Reading: http://www.secure-computing.net/wiki/index.php/OpenVPN/Routing

OpenVPN Host: has a network that needs to NAT into OpenVPN client network. Internal IP:; VPN IP:
OpenVPN Client: NAT to internal network(s). VPN IP:; Internal network:

You HAVE to use client/server – no static or bridging! So you have to use TUN (makes sense since you’re doing routing here).

Start simple – Try to get the host talking with the client’s internal network before attempting, then add the rest of the network. Remember to diagnose the small pieces before trying to make sure the whole thing works.

Ok, finally commands needed:

You definitely need to set ccd and iroute. In the example, set in the server.conf:

push “route″

In ccd/client, place:

iroute # let’s openvpn know to route packets to this client

That’s all you should have to do on openvpn. The rest is routing… Let’s start on the client…

enable net.ipv4.ip_forward=1 in /etc/sysctl.conf (and if you don’t want to restart, echo 1 > /proc/sys/net/ipv4/ip_forward)

Add the masquerading rule to the iptables:

iptables –table nat –append POSTROUTING –out-interface eth0 -j MASQUERADE
iptables –append FORWARD –in-interface eth1 -j ACCEPT

Ping a client internal IP address from the openvpn host such as troubleshoot troubleshoot troubleshoot…

After you get that going, let’s allow the whole openvpn host network to communicate to the client network. Enable IP forwarding on the openvpn host computer  with the same enable net.ipv4.ip_forward=1 in /etc/sysctl.conf (and if you don’t want to restart, echo 1 > /proc/sys/net/ipv4/ip_forward).

On your router (in network), add a routing table such as:

network: netmask: gateway:

Test from another computer and that should conclude the setup…. It annoying took me several hours to come to these very simple commands.

Of course, don’t use 192.168.x.x – it’s just too crowded and you risk having overlaps unless you know for sure no one will ever connect remotely from a public network that is using one of these IPs. You are better off with a seldom used private block such as

Such a pity (and no wonder they disbanded), I was working with opensolaris sendmail trying to configure it to work with a smarthost that required authentication. I found out there is no SASL auth support on the default sendmail binary. Looks like you have to compile it yourself… I didn’t do so, but this guy seems to have made it work.

OpenVPN TAP Bridge

| September 4th, 2010

Again, we assume you have the initial config working

We’ll be creating a bridge from the client to the local network.

First make sure bridge-utils is installed:

sudo apt-get install bridge-utils
Let's get that default config file out again:
sudo bash -c 'gzip -dc /usr/share/doc/openvpn/examples/sample-config-files/server.conf.gz > /etc/openvpn/bridge-vpn.conf'
sudo cp /usr/share/doc/openvpn/examples/sample-scripts/bridge-* /etc/openvpn/

Like the other site to client (Road warrior) config, you need to enable ip_forward, refer to my previous post.

Configure bridge-vpn.conf and change the following:

#vim syntax :-)
:%s/dev tun/dev tap0/cg
# for example, server-bridge

Edit bridge-start editing the same VPNSERVERIP, netmask, and broadcast address in this file. I had to add to the end of the script:

route del $gw
route del default gw $gw $eth
route add default gw $gw $br

because my routing tables was empty when br0 would come online, I don’t know why.

I also added service network-manager restart in bridge-stop to restore my eth0 settings.

After this, I created a client.conf and the appropriate client keys and set dev tun too. I connected to the system and wala! pinging other computers in the local network works.

Original Reference: OpenVPN Ethernet Bridging

This post assumes you have a working vpn connection as outlined in my last post and you are using a TUN device. A properly set TAP VPN automatically has access to whatever network the host VPN server has access to.

Site to Client (Road Warrior) Setup:

First thing to do is make sure ip_forwarding is enabled on the server.

grep ip_forward /etc/sysctl.conf # ubuntu (maybe debian specific?), it’s somewhere else on redhat and other distros

If it is not enabled, enable it and do a

echo 1 > /proc/sys/net/ipv4/ip_forward

If not, you’ll have to restart your system for the /etc/sysctl.conf settings to take effect.

Next, add push “route MYNETWORK MYSUBMASK” on the server.conf, for example push “route″ if you are using PKI.

This will NOT work if you’re using a static key because static key clients do not honor push commands. If you’re using static, you have to add route MYNETWORK  MYSUBMASK to the client.conf.

You’re done if the vpn server happens to also be your default gateway/router for your network such as a DD-WRT, IPCop, or PFSense. If you’re like me, and your VPN server is different from your router, go into your router and add a route so that traffic can go back to your client. The interfaces are different, but you effectively have to add:

route add -net VPNNETWORK netmask VPNSUBMASK gw VPNSERVERIP dev lan0

Ping an internal lan ip address from a vpn client and it should work… If not, double check your routes and make sure you established a path to the internal network and back to the client. Remember, even if a connection starts from a client, it still is a 2 way connection! There needs to exist a path for packets to be sent back to the client (usually… =P unless you want to only send ping requests and get no replies!).

OpenVPN Installation

| September 3rd, 2010

UPDATE: I realize I cannot cover everything in one post, so I’m breaking my openvpn discussions into several sessions as I work on openvpn. This first session will just be about installing and connecting client(s) to one server without any routing.

I keep forgetting how to configure OpenVPN and it’s so annoying reading the same HOWTO a dozen times! The problem is I only setup VPN once a year and I forget it afterwards, so this post is partially for my own personal reference.

sudo apt-get install openvpn

Decision #1 (out of many!) – PKI (Public Key Infrastructure) or Static keys..

PKI:  More complex to setup, but more secure, supports multiple clients
Static: Very simple to setup (1 command), insecure (all traffic could be decrypted if the static.key file is lost), only supports 1 client

Static Keys Setup:

The openVPN tutorial is simple enough to follow so I am just going to link it here. You can add commands such as port 443 or proto tcp-server and route, but be warned! Push commands do not work from the server! You need some tls-mode which is not available using a static key. I wasted some time trying to debug this and wondered why my push commands were not being honored until I explicitly placed pull in the client config and found out:

Options error: Parameter –pull can only be specified in TLS-mode, i.e. where –tls-server or –tls-client is also specified.

Public Key Infrastructure (PKI) Setup:

# setup CA & key-generating server
sudo mkdir /etc/openvpn/easy-rsa/
sudo cp -r /usr/share/doc/openvpn/examples/easy-rsa/2.0/* /etc/openvpn/easy-rsa/
sudo chown bashusr: /etc/openvpn/easy-rsa

# create keys
vi /etc/openvpn/easy-rsa/vars # set defaults for state, country, email, etc.
source vars

./build-key-server server

./build-key no_password_client
./build-key-pass client_with_password

cd keys
cp ca.crt server.crt server.key dh1024.pem /etc/openvpn/

scp ca.crt no_password_client.crt no_password_client.key remoteclient:/etc/openvpn/

# you can find this configuration file online here too
sudo bash -c ‘gzip -dc /usr/share/doc/openvpn/examples/sample-config-files/server.conf.gz > /etc/openvpn/server.conf’

Choices choices… At this point, you (or I?) have to decide whether to do ethernet bridging or routing, here’s a simple rundown:

TUN: Ethernet Routing (different networks), good for site to site, client to site, or site to client connections

TAP: Ethernet Bridging (same network), good for client to site – simplified setup (setup instructions)

(Note to self, you’ll need to use TUN if you want to do any routing!)

Go through the config file reading the descriptions to each line. Remember, this is just a reference… you should have a general idea what you want to do with the IP addresses, if not… read the full howto! You shouldn’t be here! After you’re done, create a corresponding client.conf and install it on the client. Then test the configuration using:

openvpn –config CONFIGFILE.conf

If the connection establishes, you’re all set! Now, make sure if you’re behind a router, that you forward the appropriate ports so you can connect from outside your network. Also, setup the service (on ubuntu, you don’t have to do anything) so it runs automatically on startup.

/etc/init.d/openvpn start

Stay tuned for future posts on VPN routing…