Strange network issues using docker with AS6706T

[jakob]

I recently bought an AS6706T, to use it as a storage system as well as a small server managing some services here within the network. I am using docker for all the different services running (about 11 containers).

Everything seemed to work okay in the beginning, however I quickly started to notice some strange behavior regarding the network of the spawned containers. There are essentially 3 different kinds of problems, that seem to occur especially, if a container is changed and restarted, after the docker daemon has done its initial startup:

1. Spontaneously sometimes some containers don't have outbound network access at all. It seems no DNS resolution is possible then from within the container.
2. Even though a corresponding port forwarding onto a local host port is configured and visible in the docker status output as well as a netstat call, the container is still not reachable and no network connection is accepted on the configured port. (This seems to happen sometimes, but much more seldom with containers running host network\_mode as well. Which I find mildly disturbing)
3. The strangest thing however is, that after restarting a whole bunch of containers with different exposed ports, but same internal ports (eg 1234 -> 80, 1235 -> 80, 1236 -> 80, ...), the container mappings are mixed up, meaning, that the container reachable at 1234 is randomly reachable at 1236 and the container under 1236 is reachable at 1235 and so on. The docker output however states the correct port mapping, only the connections are directed to the wrong container.

Especially the last issue is something I have never seen in this way and makes using the whole docker setup mostly impossible, as it causes me to not being able to update or restart seperate images, without restarting the whole docker daemon.

I really hope somebody here has an idea, about what is going on and how to try and fix it, as I am not able to figure it out further by myself. Unfortunately other sources of information came up empty in this matter as well.

Looking forward to all your ideas about this. Thanks in advance.

[Nazar78]

You can try assign static DNS to each container with the `--dns 8.8.8.8` switch.

If the container seems to lose its network, think it's best to get into the container itself by to troubleshoot the network activities.

For #3 that seems strange, I've never encountered it before. How did you check? Perhaps telnet or netcat or curl to the IP:exposed-ports (or even Bridged-IP:internal-ports from its host) could tell better about its replies.

Frankly I'm using mostly hosted network for all my containers so I don't have to manage the port mappings, unless of course there's no way to change the default port in the container that's already been occupied by the host.

[jakob]

You can try assign static DNS to each container with the --dns 8.8.8.8 switch.

If the container seems to lose its network, think it's best to get into the container itself by to troubleshoot the network activities.

Thanks for the idea, I already did that, the container can without a problem initiate connections to other containers in the same docker network and is being routed to the internet, as it should be. However incoming connections on the defined port seem to be simply not terminating within this container. I would assume it is related to #3

For #3 that seems strange, I've never encountered it before. How did you check? Perhaps telnet or netcat or curl to the IP:exposed-ports (or even Bridged-IP:internal-ports from its host) could tell better about its replies.

I have different services with a webinterface, which is exposed on 8080 within the container. I am mapping those to different ports on the host, but after the described phenomenon the wrong services are answering.

Furthermore I got a setup, where different containers spawn a ssh/sftp Server combination. They are mapped to different host ports as well. I can see that sometimes, once this “switch” happened the incoming connections are directed to the wrong container.

I have managed and setup a lot of docker setups over the years, but never had such behavior. I am, as well, have absolutely no idea what this could be.

I simply hope someone has an idea on how to solve or analyse this further, as in the current state this is quite unusable for me .

Frankly I'm using mostly hosted network for all my containers so I don't have to manage the port mappings, unless of course there's no way to change the default port in the container that's already been occupied by the host.

Unfortunately this is not a real possibility for me here, as I only partially control the ports within the containers. Furthermore I got some container stacks, which only have one public service within a container and a bunch of other services, which only need to talk to each other. I don’t want to expose them to the host network.

[Nazar78]

Understood, it could be a bug. You could try check if the docker-proxy process got defunct after a specific container exits, `sudo netstat -natp|grep docker-proxy`.

You could also try stop all the containers, then start them up one by one whilst checking if the issue exist for that particular container. If it does then it's a confirm bug which should be reported. Else for the time being you can do a workaround by setting all the containers switched to `--restart no`, then do a few seconds delay loop iteration to start all the containers, sample below written top of my mind, do re-check and change them accordingly to your needs:

Code: Select all

sudo vi /usr/local/etc/init.d/S99docker-start-containers.sh

Code: Select all

#!/bin/sh

secsDelay=5

case "$1" in
        start)
                for i in `docker container ls -a|tail -n+2|awk '{print $NF}'`;do
                        echo -ne "\nStarting $i in ${secsDelay}secs..."
                        sleep ${secsDelay}
                        docker start $i>/dev/null && echo "OK!"
                done
        ;;
        *)
                echo "Usage: $0 <start>"
        ;;
esac

Code: Select all

sudo chmod +x /usr/local/etc/init.d/S99docker-start-containers.sh

[jakob]

Understood, it could be a bug. You could try check if the docker-proxy process got defunct after a specific container exits, `sudo netstat -natp|grep docker-proxy`.

Did that as one of the first measures to see, what’s going on. The processes seem to work okay, but somehow don’t seem to do the right thing.

You could also try stop all the containers, then start them up one by one whilst checking if the issue exist for that particular container. If it does then it's a confirm bug which should be reported. Else for the time being you can do a workaround by setting all the containers switched to `--restart no`, then do a few seconds delay loop iteration to start all the containers, sample below written top of my mind, do re-check and change them accordingly to your needs:

Thanks for the idea, but even though this might help during boot, it does not help at all with the fact, that I effectively can’t Update containers and roll them out again, as the port mappings are either mixed up then, or aren’t working at all.

I am kind of irritated, that I seem to be the only person out there with such a problem. I can’t even find anything similar without the asustor association. It’s all very weird.

I know all of this sounds, like I simply configured stuff wrong. Let me however emphasize, that this is far from being my first server and/or docker setup. I am quite sure it is not a configuration issue. At least no obvious one. That’s why I am stumped about the reactions and behavior of the system. So if anybody has any idea, I would be glad to give things a try. Thanks.