📰 Good News app. Backend in Golang. MongoDB setup using official Golang driver.

It is time to start working with MongoDB!

I am glad to see you on the fourth article out of six in this chapter.

If you are not familiar with what I am going to implement in this series of chapters, I would recommend you to read the introductory article:

📰 Good News app. Golang + Flutter + Hummingbird = 🖤

All chapters of a “book”:

1. 📰 Good News app. Backend in Golang behind Traefik reverse proxy with https available.
2. [in progress] 📰 Good News app. Flutter for rapid mobile applications development.
3. [in progress] 📰 Good News app. Hummingbird as a promising replacement for frontend frameworks.

And here are articles of current chapter:

1. Prerequisites & Idea, project and database structure and API endpoints.
2. Project creation, go modules & GIN (beautiful framework) integration.
3. Colly usage.
4. MongoDB setup using official Golang driver.
5. Running all together locally with Docker and Docker Compose & Traefik v2.0 configuration.
6. Publishing to Digital Ocean, Let’s Encrypt and DNS Challenge configuration.

So let’s start!

In this article, we will start working on configuration of connection to MongoDB using official Golang driver that has been recently released. Also we will implement saving and gathering mechanisms of news which we have parsed in the previous article as well as pre-filling database with news sources and news types right before server launch. However we are not going to run any code in this article, but we will do all steps in order to launch MongoDB in Docker in the next article.

MongoDB configuration

We are going to launch MongoDB in a Docker container. While being inside a Docker container, MongoDB will be exposed on a specific url that has following schema: mongodb://mongo_db_user:mongo_db_pswd@mongo:27017/mongo_db_name. As we can see, in order to connect to MongoDB from our code we will need to know mongo_db_user, mongo_db_pswd and mongo_db_name. This information will be set in .env file to avoid having our credentials inside the code. It is also convenient for the cases if some information would change, we will need to set new values only inside .env file and won’t directly touch the code so no need to rebuild whole program.

Besides that information, we will need to provide mongo_admin and mongo_admin_pswd to a MongoDB instance in order to make sure our database won’t be accessible for bad people who would like to steal our information. We will use this information to connect to MongoDB through the shell and set it in .env file as well.

So this is how our .env file should look like now:

API_GIN_DEBUG_MODE=true
MONGO_ADMIN=GGCTeamBatr
MONGO_ADMIN_PSWD=MySuperSecretPassword
MONGO_DB_USER=suuuper_user
MONGO_DB_PSWD=soop3r_U$eR_PSWD
MONGO_DB_NAME=good_news_db

You can put your own credential information instead of mine but when we will be running some code in shell or other places, please make sure to put your credentials.

Based on this information, we can generate a url in order to connect to MongoDB. We will pass it through Docker environment variables in the next article and for now, you should know that in our code we will need only database name and url to connect. So this data should be placed in utils/utils.go where we manage our environment variables. Please make sure that now your EnvVariables struct looks like this:

type EnvVariables struct {
    DebugMode bool
    MongoDBName string
    MongoDBURL  string
}

db/ folder structure

As you might have noticed, I have created a separate folder (package) for crawlers in the previous article so I want to keep all logic separately, divided, let’s say, to microservices. I am going to do the same with accessing, writing and reading information to/from a database.

So as an entry point of the database service, I will create service.go file under db/ folder where all corresponding files will located as well. Whole process of Mongo client initialisation, establishing a connection to the Mongo database port and some extra helpful functions will take place in service.go file. Likewise, we will create separate files for managing access (reads/writes) to each collection (news, news_sources and news_types) of the database as well as for pre-filling news sources and types purposes.

So the step that we have to do before writing any code for database service is to download official MongoDB driver.

> go get go.mongodb.org/mongo-driver/mongo

After this step is done, create service.go file under db/ folder as mentioned before and paste the code below.

Please, make sure that you import utils package as well because my folders’ names would be different from yours.

In this file, we create Service struct that contains a pointer to mongo client instance and then we declare a variable of Service struct that is initialised in InitService() function.

Whole work and communication with MongoDB driver is based on contexts. You can read more about contexts in official Golang documentation. Basically, context is used like a timeout or deadline or a channel to indicate stop working and return. So you can see that we create a context and pass it to Connect() function of mongo driver. I have created a separate function CTX() to avoid verbose code because context.WithTimeout is going to be used multiple times in our code. After establishing a connection to our database instance, we ping to it to ensure that everything is set up. So if everything is fine, we initialise our Service’s client. Initialisation method will be called right before we launch (run) our server as well as action of pre-filling information of news sources and types. GetClient() function is used to return a pointer to a service variable that could be used outside of db package. CTX() function, as mentioned earlier, is used to create a context.WithTimeout to avoid verbose code while working with MongoDB driver. Collection() is used to easily create a mongo collection instance so we can use it for different collections. InsertManyOptionsOrdered() is used to return options for mongo when we will be inserting many values of news, sources and types. As you can see, it always return true in order to make sure that mongo will ignore cases when we add information, such as news, with the same _id.

So before writing any code for operating read/write actions for news, let’s start from pre-filling news sources and types information. Create seeds.go file under db/ folder and paste the code below.

It is pretty primitive, right? Yes, it is made with intention to simplify the logic of pre-filling and fillNewsSources() and fillNewsTypes() functions’ implementation will take place in separate files. Please, create newsSources.go and newsTypes.go files under db/ folder.

Please, import models package in both files. So as you can see, both files newsSources.go and newsTypes.go have a pretty same structure. In the beginning, you can see that we declare a constant with corresponding collection name. Then we have functions that pre-fill information to the database which are invoked in seeds.go file. As you can see in those functions, we create a collection instance for corresponding collection in the database, then we generate a context that is being used in InsertMany action. After that we create an array, append all needed information about news sources and types which are predefined in models package. And then we invoke InsertMany function of mongo that is going to insert those values to MongoDB.

As we want our news sources and types to be returned when requesting /v1/news/sources and /v1/news/types, we have to create a function for each of them that is going to be called in a corresponding method in NewsController. So you can see NewsSourcesGet() and NewsTypesGet() functions which are going to do it. I think the code is pretty intuitive except of iterations with cursor. First, we create a corresponding collection instance, generate a context and invoke Find() method to get all documents from the database. Find() method returns a cursor object with which you can make several actions but one that we are interested in is .Next() that is looping over while there is still elements in returned documents. Then we decode each value and add it to an array. One thing that has to be done is closing our cursor.

And now it is time to talk about actions which should be performed on news. In order to that, create news.go under db/ folder and paste the code below.

Don’t forget to import models package!

So for news, we have to create two functions which are going to perform saving and reading actions to/from database. Obviously, NewsInsert() function takes care about saving news to the database. All lines of this function should be easy to understand because most of them we have done earlier except of one thing where we have to cast our news objects to []interface{} type. I don’t know why it is so but mongo driver asks to do so.

However, NewsGet() function is more interesting, especially options which we pass while performing Find() function. Well options.SetSort(bson.D{{“time_added”, -1}}) says that our news should be sorted in descending order by time_added property. And options.SetLimit(count) tells mongo to return only count number of news. That’s it, other lines of code are simple and exactly what we have done before.

So now as we finished working on the code of database service, we have to initialise our database service and call pre-filling method of news sources and types before we start our server. So open server/server.go file and change the code you have now based on what’s below.

Please, import crawler, db and utils packages.

As you can see we have added two lines of database initialisation and filling information and changed the port to :80. That is done for the future article when we will run the project in Docker.

So now as we have initialised our database service, we can start using it in the code, in particular to save our news. We have to add just 2 lines of code in crawler/crawler.go in the end of each crawling iteration. So below is how your crawler/crawler.go should look like.

Don’t forget to import db and models packages!

Also I assume that you have copied two other crawlers, that is why I initialise them when creating crawlers array.

And it is time to change our NewsController so it returns actual information for corresponding request. This is how it should look like.

And please, don’t forget to import db package.

So in Get() method that is claiming to return last news with count number, first thing we do is taking care of count parameter, then we get db client instance and trying to gather news with given count from mongo database. If there are any errors on our path we take care of them by returning needed status code and error message. Hopefully, we won’t face any of them. GetSources() and GetTypes() have almost the same code except of invoking corresponding method of db service to get needed information.

Please, run go build in the terminal window to ensure that everything is being built correctly and there is no errors.

That’s it for this article. We are not going to test our code as we have been doing in the end of previous articles because I prefer using Docker for launching MongoDB, it’s much easier than managing all configuration on your local machine, trust me, I don’t want you to feel that pain. So stay tuned and move to the next article!

I am glad that you have read this far 🙌 Below you will find the link to the next article.

📰 Good News app. Backend in Golang. Docker Compose & Traefik v2.0.

If you have any comments or suggestions, please feel free to write them down or email me at batr@ggc.team 🙂

If you would like to know when I post new articles, follow me on twitter 🐦

ʙ ᴀ ᴛ ʀ