Deploying to private servers with "Bitbucket" Pipelines and Tailscale

Tailscale is a Wireguard (r) based networking tool that allows access to your private resources. It creates encrypted tunnels between your resources with almost zero-config and works out-of-the-box.

We use it extensively to manage our servers behind the firewall.

"Bitbucket" ise our Git provider of choice and with "Bitbucket" Pipelines we automate our deployments and utilize any workflow we want.

Tailscale comes handy when one needs to access restricted servers. However it's a bit tricky when you try to access restricted servers with Tailscale from pipeline containers as Tailscale does not work directly inside the container (with tailscale up command), the reasons for this are:

• Pipeline servers are actually containers.
• These containers do not have the necessary modules in their kernel.
• (for "Bitbucket" Pipelines) we are not authorized to run it with root.

For all these reasons, we can only run Tailscale in "userspace networking mode", which is Tailscale's special mode for containers and does not require additional permissions.

When we run in this mode, it gives us a SOCKS5 (or HTTP) proxy for Tailnet access, since it doesn't have permissions to manage your containers' network settings. You can route your SSH traffic needed for deployment through these proxies.

A Tailnet is your private network. When you log in for the first time to Tailscale on your phone, laptop, desktop, or cloud VM, a tailnet is created.

After you complete your Tailscale installation and log in, you can create your proxy with the following command:

tailscaled --tun=userspace-networking --socks5-server=localhost:1055 &

Step-by-step "Bitbucket" Pipelines Setup

We need to install Netcat on the container so that SSH can exit through the SOCKS5 proxy.

...
script:
	- echo "Install netcat for SSH tunneling thru socks5"
	- apt-get update
	- apt-get install netcat -y

SSH needs to be configured to use the SOCKS5 proxy, there are two ways to do this, the first is to write it to ~/.ssh/config :

...
script:
	...
	- echo "Configure SSH client"
	- echo "Host *" >> ~/.ssh/config
	- echo "StrictHostKeyChecking no" >> ~/.ssh/config
	- echo "ProxyCommand nc -X 5 -x 127.0.0.1:1080 %h %p" >> ~/.ssh/config
	- chmod 600 ~/.ssh/config

Here we have instructed the container to use the SOCKS5 proxy running on port 1080 for all SSH connections, yet the proxy is not up.

The second way to connect to the SSH server with the SOCKS5 proxy is to pass a -o parameter to the SSH command, for example

ssh -o StrictHostKeyChecking=”no” -o ProxyCommand=“nc -X 5 -x 127.0.0.1:1080 %h %p"

This is preferable to writing to a file, but the first method makes the code more readable.

Then we install Tailscale, start the daemon in user-space networking mode with limited user privileges (without root) and log in to our Tailnet with the tailscale up command.

...
script:
	...
	- echo "Install & configure Tailscale"
	- curl -fsSL https://tailscale.com/install.sh | sh
	- tailscaled --tun=userspace-networking --state=tailscaled.state --socks5-server=127.0.0.1:1080 &
	- tailscale up --authkey $TAILSCALE_AUTH_KEY

$TAILSCALE_AUTH_KEY is a predefined value in "Bitbucket" Pipelines settings. You can define it under repository variables.

Normally (when installing on our own machine) Tailscale asks us to authenticate with the service provider of our choice in a browser window, but since this is not possible in pipelines, we need to set it up as Ephemeral Node.

Ephemeral nodes are temporary nodes and do not need any additional authentication because they receive API key with --authkey parameter. They are automatically deleted from the tailnet shortly after the container goes down or when tailscale logout is issued.

The connection is now established; our SSH is set to use SOCKS5 and the proxy is up. Deploy commands can be executed.

...
script:
	...
	- echo "Write target command to a stub file"
	- echo "ls -la" >> command.txt
	- echo "Run commands remotely on the target SSH server"
	- ssh $USER@$HOST < command.txt

In this example, we wrote the deploy commands to a file first, to make the code more readable; these commands could have been sent after the SSH command.

After the deployment is complete, we can logout Tailscale so that the temporary machine is immediately dropped from the machine list. If we don't do this, it will be automatically dropped shortly after the container is down and the network is offline.

...
script:
	...
	- tailscale logout

Sample bitbucket-pipelines.yml File

pipelines:
  branches:
    master:
      - step:
          name: Deploy to Kubernetes
          deployment: production
          image: atlassian/default-image:3
          script:
            - echo "Install netcat for SSH tunneling thru socks5"
            - apt-get update
            - apt-get install netcat -y
            - echo "Configure SSH client"
            - echo "Host *" >> ~/.ssh/config
            - echo "StrictHostKeyChecking no" >> ~/.ssh/config
            - echo "ProxyCommand nc -X 5 -x 127.0.0.1:1080 %h %p" >>
              ~/.ssh/config
            - chmod 600 ~/.ssh/config
            - echo "Install & configure Tailscale"
            - curl -fsSL https://tailscale.com/install.sh | sh
            - tailscaled --tun=userspace-networking --state=tailscaled.state
              --socks5-server=127.0.0.1:1080 &
            - tailscale up --authkey $TAILSCALE_AUTH_KEY
            - echo "Write target command to a stub file"
            - echo "ls -la" >> command.txt
            - echo "Run commands remotely on the target SSH server"
            - ssh $USER@$HOST < command.txt
            - echo "Removing container from network"
            - tailscale logout

Using "Exit Node" to Access Restricted Resources

Sometimes the destination server may only allow certain static IPs without hosting Tailscale on it. In such cases, requests over HTTP may need to be sent using an exit node (i.e. over a specific static IP).

pipelines:
  branches:
    master:
      - step:
          name: Migrate to Test
          deployment: test
          script:
            - echo "Install & configure Tailscale"
            - curl -fsSL https://tailscale.com/install.sh | sh
            - tailscaled --tun=userspace-networking --state=tailscaled.state
              --socks5-server=127.0.0.1:1080 &
            - tailscale up --authkey $TAILSCALE_AUTH_KEY
              --exit-node=$TAILSCALE_EXIT_NODE
            - echo "Set up HTTP proxy for all outgoing HTTP requests"
            - export http_proxy=socks5://127.0.0.1:1080
            - export HTTP_PROXY=socks5://127.0.0.1:1080
            - export https_proxy=socks5://127.0.0.1:1080
            - export HTTPS_PROXY=socks5://127.0.0.1:1080
            - ....
            - tailscale logout