This post is build up on my previous post about Ultronex, a slack bot rewrite in Elixir. In this post I will walk through how a simple slack bot expanded to include an API, reverse proxy , container monitor and log management solution all on a cheap Hetzner VPS using cloud native tech Terraform, Packer, Ansible and Docker. If you have gone through the original post you would know that Ultronex was packaged as a container for easier development and it helped me to move my locally developed code to cloud and not expect any nasty surprises doing deployments manually. I will walk through the components of the app and how they were implemented and how they work together using docker-compose . In the following parts I will show how to provision cloud services using Terraform, Packer and Ansible.
API
One of the first things I added to the bot was an API to monitor health check.
Request: curl -v http://localhost:8080/ultronex/heartbeat
Response: {"msg":"I don't have a heart but I am alive!"}
status: 200As being a slack bot what good is it if it can’t send messages to slack, hence an API endpoint support to send msgs to slack was added
Password protected endpoint
Request: curl --request POST \
--url http://localhost:8080/ultronex/slack \
--header 'Accept: */*' \
--header 'Accept-Encoding: gzip, deflate' \
--header 'Authorization: Basic username:password' \
--header 'Cache-Control: no-cache' \
--header 'Connection: keep-alive' \
--header 'Content-Length: 81' \
--header 'Content-Type: application/json' \
--header 'Host: localhost:8080'
--header 'cache-control: no-cache' \
--data '{"msg": {\n "channel": "U08MG2700",\n "text": "test",\n "payload": "testing",\n "title": "Hola"\n \n}\n \n}'
Response: {
"id": "e21f9216-43b4-45d9-978b-de94da273654",
"msg": {
"channel": "U08MG27P0",
"text": "test",
"title": "Hola"
},
"status": "triggered"
}
status: 200With time I added more endpoints to have visibility over what patterns were being tracked and some stats around how many messages were received, processed and responded to.
Password protected endpoint
Request: curl -v -H"Authorization: Basic username:password" http://localhost:8080/ultronex/stats
Response: {"uptime":"2019-10-16 16:01:05.571045Z","total_msg_count":90075,"total_attachments_downloaded":64992,"replied_msg_count":32,"forwarded_msg_count":14}
status: 200Password protected endpoint
Request: curl -v -H"Authorization: Basic username:password" http://localhost:8080/ultronex/track
Response: {"stark":"U08MG273O"}
status: 200
Recently I added an endpoint to track the slack errors of websocket termination and rate limiting due to increasing volume of messages
Password protected endpoint
Request: curl -v -H"Authorization: Basic username:password" http://localhost:8080/ultronex/external
Response: {"errors":{"Slack error : remote closed":12}}
status: 200DNS
I took the API to cloud using a free DNS from freedns.afraid.org. Its an amazing service where you can get a free DNS and point it to you cloud server and not bother to remember the IP address again. To keep the IP in sync with the domain name you picked just add a cron script provided by them.
sleep 40 ; wget — no-check-certificate -O — https://freedns.afraid.org/dynamic/update.php?<token> >> /tmp/freedns.log 2>&1 &SSL
Once you have a domain in today's age a web service that is not on HTTPs is a crime. With LetsEncrypt getting a free cert for your DNS is as simple as a one line command and just follow the steps and you should have a certificate on your machine.
certbot certonly — standaloneContainer Monitoring
You will find a lot of tools out there for docker monitoring some are based on popular open source tools like grafana and prometheus but nothing comes close to open source weavework scope. It not only shows you all your containers and how they are connected. It also gives you metrics about them and your host.
The magic doesn’t end here, once you have your stack up and running it allows you to control individual container from the web console. You can enter the containers for debugging purpose, pause it or even stop it.
Log Management
Once you are in cloud, log management becomes crucial and going to the server to tail a log becomes cumbersome specially when you now have so many components running. Two of the most popular log management stacks are ELK (Elasticsearch, Logstash, Kibana) and EFK (Elasticsearch,Fluentd, Kibana). Elasticsearch can be resource intensive and and I needed a solution to avoid multiple dashboards that is when I stumbled across another lesser known tool Loki. Loki is a tool from Grafana and uses Promtail another tool to push logs in to Grafana where you can stream the logs and query them.
Reverse Proxy
Once you have all these services running from an API to log manager to container monitor. Connecting them to your DNS to route traffic requires to use a reverse proxy. For reverse proxy I chose HAProxy as its open source and has a ton of features others don’t have, to name a few its got built in healthchecks, DDoS protection, stats page and a promethues compliant stats exporter api. Reverse proxy allowed to redirect all incoming requests from HTTP to HTTPS and monitor all services’ uptime. It also allowed to shield all the apps from unnecessary traffic.
With the Prometheus compliant stats api its easy to hookup tons of stats in to Grafana and have a detail look in to its performance.
At the moment all of this is running on a Hetzner CX11 which is just 1vCPu and 2GB or ram. Serving close to 43rpm and peaking to 80rpm on the API and this excluding close to 200msgs per minute that the slack bot process in the application and all with under 18% CPU usage.
In the following parts I will go through how to provision the instance in hetzner and how to build images programmatically to avoid manual configuration of your cloud instance. I will share terraform and packer code for doing so in Hetzner and Scaleway and why I chose to switch back to Hetzner from Scaleway.
All the code for this part is available online at
