📰 Good News app. Backend in Golang behind Traefik reverse proxy with HTTPS available.

Prerequisites & Idea, project and database structure and API endpoints.

Hello guys 🖖

I am glad to see you on the first article out of six in this chapter. The whole chapter is dedicated to show you how to build a backend using Golang and make it available through https requests. And among articles, there are going to be some interesting topics which could be useful in developing almost every backend. I am going to repeat those topics as I have briefly mentioned about them in the introductory article:

• Deploying a backend written in Go within a tiny Docker image.
• Parsing websites with Colly.
• Exploiting Mongo as a database.
• Using Traefik (it is a great reverse proxy made by guys from Continious) as a server with built-in Let’s encrypt certificates and DNS challenges managing tools.
• Publishing all of the above on Digital Ocean.

I hope that they seem interesting and you are going to finish all the articles in one breath. However it is going to be a long chapter but very informative.

If you are not familiar with what I am going to implement in this series of chapters, I would recommend you to read the introductory article:

📰 Good News app. Golang + Flutter + Hummingbird = 🖤

All chapters of a “book”:

1. 📰 Good News app. Backend in Golang behind Traefik reverse proxy with https available.
2. [in progress] 📰 Good News app. Flutter for rapid mobile applications development.
3. [in progress] 📰 Good News app. Hummingbird as a promising replacement for frontend frameworks.

And here are articles of current chapter:

1. Prerequisites & Idea, project and database structure and API endpoints.
2. Project creation, go modules & GIN (beautiful framework) integration.
3. Colly usage.
4. MongoDB setup using official Golang driver.
5. Running all together locally with Docker and Docker Compose & Traefik v2.0 configuration.
6. Publishing to Digital Ocean, Let’s Encrypt and DNS Challenge configuration.

So let’s start!

In this article I am going to tell you about some prerequisites before we proceed to coding and explain the idea of the project in more details that contains database structure and API endpoints.

All the code with detailed comments is published in my Github repository that you can use to check current steps during the post progress if you misunderstand anything 🙂

Prerequisites

There are some prerequisites for you in order to start following steps in this article.

I assume that you are at least a bit familiar with Golang, already have it installed on your machine, have written some small programs and understand what GOPATH is. If not, please follow to the Golang official website and try it out (useful links: 1 & 2), it is AWESOME! If you are looking for the easy way to install Golang and using MacOS, please use Homebrew packages manager (I am using Golang 1.12). There is no need for you to be familiar with GIN or Colly but if you are, that would be great.

Also it is necessary Docker to be installed on your local machine in order to launch a Golang backend and MongoDB behind Traefik reverse proxy. To make installation process smooth, please follow Docker installation guide.

Besides that, if you would like to publish everything you will follow during this chapter, you have to have Digital Ocean account and register a domain. If you don’t have it, it is okay, I will have my backend published and running so you can check it out and use it in future chapters where we will develop mobile applications using Flutter. So last step from contents is optional.

For your convenience and best experience, I would recommend you to use VSCode and install Go Extension for VSCode that is officially supported by Microsoft.

Idea, project and database structure and API endpoints

Idea or service that I have chosen for this series of articles and going to implement step by step is basically a simple news reader app. Why? Well, I have a few websites with news which I usually read during the first 30 minutes when I come to the coworking place and it is a pain for me to open each website, so I decided to write my own backend and mobile apps to solve that problem. Yes, it would be much easier to find RSS feeds of those websites, download special RSS feed reader and use it but I can not miss a chance to learn something new and share it with the world. So that is why I have decided to work on this app.

Basically in order to solve this problem, as I mentioned earlier, we will need to write our own backend that is going to parse those websites every 3 minutes, save latest news to a database which were not added yet and return needed count of news by calling special URL or endpoint. The last step is needed to build a mobile application and make latest news available within an app and show them in a list. As you can see, it is a pretty primitive backend but the main goal here is to get knowledge how to put all those things together and make it work.

The project structure is going to have following folders, please notice that it is only my desire to create such structure for this project:

/good-news-backend
    /controllers - where our controller(s) will be located
    /crawler     - files for sites parsing
    /db          - db access and manipulating files
    /images      - static images
    /models      - files for describing our models
    /server      - server configuration files
    /utils       - some utils files
    main.go      - starting point for the project
+ some other files which we are going to create during development

Database that we are going to use as mentioned earlier is MongoDB. Why? Because it is pretty easy to start with and is used by me in almost all of my projects because of its NoSQL nature and documents store. If you are not familiar with it, please spend 10 minutes to read about it.

Before talking about database structure of this project, I am going to tell you a bit about NoSQL collections and documents as you already know what they are. So each database has its collections which contain documents. Documents, by its NoSQL nature, could have different content, for example, one document in a collection can contain information about person credentials and other document within the same collection could be person’s address information. I don’t think it is a good practice to do like this. So as a summary, we should create a collection with the same structure of documents. For instance, if we have collection named Persons then all documents should contain First Name, Last Name, Date of Birth, Salary (optional field, could be presented or not).

In this project we are going to have a few collections which we are going to use. Logically we have news, news sources (sites where we will gather news from) and news types. So for each type we create a collection and collections will be news, news_source and news_types.

News collection is simply an array of news objects (documents) which we are going to parse from sites and add them to the collection every three minutes if there are new ones. If you already would like to check how they look like, please open this link in your browser — https://api.good-news.ggc.team/v1/news/?count=10. Structure of each document is following:

[
    {
        _id: <string>,
        title: <string>,
        preamble: <string>,
        time_added: <int64>,
        link: <string>,
        news_type: <NewsType>,
        news_source: <NewsSource>
    },
    ...
]

For news sources we also have an array of objects. If you would like to check it, please follow this link — https://api.good-news.ggc.team/v1/news/sources. And here is a structure of a news source:

[
    {
        _id: <string>,
        name: <string>,
        image_url: <string>,
        types: <array of NewsType>
    },
    ...
]

And as you might guessed we have exactly the same situation with news types which are going to be an array of objects. Please follow this link —https://api.good-news.ggc.team/v1/news/types, if you would like to check how it looks like. It has the following structure:

[
    {
        _id: <string>,
        name: <string>,
        type: <string>
    },
    ...
]

News sources and types are going to be pre-filled to a database when we will launch our backend first time.

As you may have noticed, I already have three running API endpoints which give us a needed information in response. So they are:

• /v1/news/?count=<count_number> — returns last added news to the database so we can show them in our mobile apps in the future. <count_number> is optional and by default it is 50.
• /v1/news/sources — returns news sources so we can manage and show correct information in our mobile apps.
• /v1/news/types — returns news types. They are pre-filled to a database as news sources.

I am glad that you have read this far 🙌 Below you will find the link to the next article.

📰 Good News app. Backend in Golang. Project creation, Go Modules & GIN integration.

If you have any comments or suggestions, please feel free to write them down or email me at batr@ggc.team 🙂

If you would like to know when I post new articles, follow me on twitter 🐦

ʙ ᴀ ᴛ ʀ