A tale of Docker and Linux ConnMan
Recently I came across a painful issue while using my development system with Docker. I spent an awful lot of time figuring out the exact issue. This is my attempt to document the fix hoping it helps someone in the future.
The problem definition:
Every time I start a Docker container, my internet connection breaks.
And that’s it. Now added frustration is I cannot even google when the issue occurs. I have to stop the container to get my internet back.
What a Gem!
To add more fun, I’m consulting a company to set up their CI/CD process where every service runs in a Docker container. Not a good situation to be in huh.
So I started with what a normal engineer would do -
Google it!
Indeed. But most of the search results are about the Internet not working inside the containers. Which is not our problem. I could find few solutions like this. But unfortunately, that did not fix it for me.
So at that point, it was clear I’ll need to be more clear about explaining my problems to Google and do some homework first.
Firstly I checked what are the interfaces that are up. I used
ip link show
for this. This tells me what are the interfaces available — physical or virtual.
I can see
- wlan0 which is how I’m connected to the internet using wifi
- docker0 which is a bridge Docker creates by default
- vethxxxx which is the bridge that Docker creates for every container that runs in bridge mode. Nothing suspicious so far.
Now we know that Docker behind the hood uses iptables to manage the networking. So it must update my route table, let’s check what my route table looks like.
Oh, now that’s where it is wrong. I have an extra route that acts as a default gateway with the destination as 0.0.0.0 . This must cause all my traffic to receive by this interface and route to nowhere.
We have a lead! Again, going back to Google! Now we have found someone who has been through this. The question is available in stack overflow.
I tried this solution, but again it did not solve my problem. Sigh. But now that this answer gives us a hint that ConnMan is something that might be a problem and not the Docker itself.
So I went ahead to ConnMan’s project page to find if there is any known documentation around this. And thanks to the amazing people who document common errors, I found that this is a known issue with ConnMan. Following is the excerpt from the documentation to save you a click:
If something like Docker is creating virtual interfaces Connman may attempt to connect to one of these instead of your physical adapter if the connection drops. A simple way of avoiding this is to blacklist the interfaces you do not want to use. Connman will by default blacklist interfaces starting with vmnet, vboxnet, virbr and ifb, so those need to be included in the new blacklist as well.
So I went ahead, updated the config and restarted connman, and voila, the issue was fixed.
Such a relief!
Originally published at https://sitaram.substack.com.
