Go Modules Tutorial


Go Modules Tutorial

Original link

  Go modules is a officially announced project dependency solution in the Go language. Go modules (formerly known as vgo) were officially released in Go1.11. They are ready for Go1.14 and can be used for production. Go officially encourages all users to migrate from other dependency management tools to Go modules.

  And Go1.14, finally officially released today, Go official call you use:

Go 1.14 Introduction

  So in this article today, I will bring you the *Ultimate Getting Started for Go modules, and welcome everyone to discuss it together.

What is Go Modules

  Go modules is a dependency solution for the Go language. It was released in Go1.11, grew up in Go1.12, enriched in Go1.13, and officially recommended for production in Go1.14.

  Go modules is currently integrated in Go’s toolchain. As long as Go is installed, Go moudles can be used naturally, and the emergence of Go modules has also resolved several common controversial issues before Go1.11:

-Go has a long history of dependency management issues.
-Retire existing GOPATH usage models.
-Other dependency management tools in the unified community (providing migration capabilities).

Those bits of GOPATH

  We mentioned that one of the problems solved by Go modules is to “eliminate” GOPATH, but what is GOPATH? Why did you use GOPATH before Go1.11 and gradually recommend using Go modules after Go1.11? Do you recommend GOPATH again?

What is GOPATH

  Let’s first look at the first question, what is GOPATH, we can enter the following command to view:

$ go env
GOPATH="/Users/eddycjy/go"
...

  After we enter the go env command line, we can see the result of the GOPATH variable. We enter the directory to view it, as follows:

go
├── bin
├── pkg
└── src
    ├── github.com
    ├── golang.org
    ├── google.golang.org
    ├── gopkg.in
    ....

  The GOPATH directory contains three subdirectories, namely:

-bin: Stores the compiled binary files.
-pkg: stores pre-compiled object files to speed up subsequent compilation of the program.
-src: stores all .go files or source code. When writing Go applications, packages and libraries, they are usually stored in the path $ GOPATH/src/github.com/foo/bar.

  Therefore, when using GOPATH mode, we need to store the application code in a fixed $ GOPATH / src directory, and if you executego get to pull external dependencies, it will be downloaded and installed automatically in the $ GOPATH directory.

Why deprecate GOPATH mode

  Store the .go file or source code under$ GOPATH / src of GOPATH. We can call it the“ GOPATH ”mode. This mode seems to have no problem, so why do we discard Use it, see the following reasons:

There is no concept of version control in GOPATH mode, and it has fatal flaws, which will cause at least the following problems:

  • When executing go get, you cannot convey any expectation of version information, that is, you cannot know which version you are currently updating, nor can you pull the specific version you expect by specifying.
  • When running Go applications, you cannot guarantee that other people and the third-party libraries you expect to rely on are the same version, that is to say, in the management of project dependent libraries, you cannot guarantee that all the dependent versions are consistent.
  • You can’t deal with different versions of v1, v2, v3, etc., because the import path in GOPATH mode is the same, all are github.com/foo/bar.
  • Go language officially promoted Go modules (formerly vgo) from Go1.11. Go1.1 is no longer recommended to use GOPATH. Go modules are also becoming stable, so it is not necessary for new projects to continue to use GOPATH mode.

Product in GOPATH mode

  Go1 was released on March 28, 2012, and Go1.11 was officially released on August 25, 2018 (data source: GitHub Tag). In this gap time, there is no such thing as Go modules. In the early days, it may be okay to say that because it has just been released and is not used by many people, there is no obvious exposure, but there are more and more people using Go language in the later period. What should we do?

  At this time, a large number of dependency solutions gradually emerged in the community, making it difficult to choose, including a model of the vendor directory that we are familiar with, and this type of dependency management, which was once considered an “official” dep. tool.

  But why dep is not becoming official propaganda, it is actually because as Russ Cox and other members of the Go team continue to discuss in depth, it is found that some details of dep seem to be more and more unsuitable for Go, so the official adopted another proposal. The way to advance, the result of the solution was to release vgo (the predecessor of Go modules, you do n’t need to understand it), and it finally evolved into the Go modules we now see. It also officially entered in Go1.11. Go’s toolchain.

  So instead of “products in the GOPATH mode”, history provides important lessons for the current, so Go modules appeared.

Basic use of Go Modules

  After a preliminary understanding of the past and present life of Go modules, we officially entered the use of Go modules. First, we will create a Go modules project from scratch (in principle, the directory created should not be placed in GOPATH).

Commands provided

  In Go modules, we can use the following commands:

command function
go mod init initialize new module in current directory
go mod download download modules to local cache
go mod tidy add missing and remove unused modules
go mod graph print module requirement graph
go mod edit edit go.mod from tools or scripts
go mod vendor make vendored copy of dependencies
go mod verify verify dependencies have expected content
go mod why explain why packages or modules are needed

Provided environment variables

  There are the following common environment variables in Go modules, we can view them with the go env com


$ go env
GO111MODULE="auto"
GOPROXY="https://proxy.golang.org,direct"
GONOPROXY=""
GOSUMDB="sum.golang.org"
GONOSUMDB=""
GOPRIVATE=""
...

GO111MODULE

  Go provides an environment variable GO111MODULE as a switch for Go modules, which allows setting the following parameters:

-auto: Enable Go modules as long as the project includes a go.mod file. Currently it is still the default in Go1.11 to Go1.14.
-on: Enable Go modules. The recommended setting will be the default value in future versions.
-off: Disable Go modules, not recommended.

GO111MODULE history

  You may notice that the name GO111MODULE is more “strange”. In fact, there are often such staged variables in Go. The name GO111MODULE represents the variables added to Go in 1.11 for Module.

  For example, in the Go 1.5 version, a system environment variable GO15VENDOREXPERIMENT was also released, which is used to enable support for the vendor directory. At that time, the default value was not enabled, and it was only experimental. It then changed the default value to ON in Go1.6 version, and finally became official, and the GO15VENDOREXPERIMENT system variable exited the stage of history.

  In the future, GO111MODULE, a system environment variable, will also face this problem. It will also be adjusted to the default value of on. ), Then remove the support of GO111MODULE, we guess that GO111MODULE should be removed in Go2, because if you remove the support of GO111MODULE directly, there will be compatibility problems.

GOPROXY

  This environment variable is mainly used to set the Go module proxy, and its role is to enable Go to pull the module version from the traditional VCS method and pull directly through the mirror site.

  The default value of GOPROXY is: https://proxy.golang.org,direct, this has a very serious problem, that is,proxy.golang.org is not accessible in China, so this will directly jam you The first step, so you must set the domestic Go module proxy when you start Go modules, and execute the following command:

$ go env -w GOPROXY=https://goproxy.cn,direct

  The value of GOPROXY is a , separated list of Go module agents. It allows multiple module agents to be set. If you do not want to use it, you can also set it to “off”. This will prevent Go from being used in subsequent operations Any Go module proxy.

what is direct

  In the value just set, we can find the “direct” flag in the value list. What does it do?

  In fact, “direct” is a special indicator that instructs Go to go to the source address of the module version to fetch (such as GitHub, etc.). The scenario is as follows: When the last Go module proxy in the value list returns a 404 or 410 error , Go automatically tries the next one in the list. When it encounters “direct”, it goes back to the source, that is, it goes back to the source address to grab. When it encounters EOF, it terminates and throws an error like “invalid version: unknown revision …”

GOSUMDB

  Its value is a Go checksum database, which is used to ensure that the module version data that has been pulled has not been tampered with when pulling module versions (whether from the source site or through the Go module proxy). There may be tampering and it will be suspended immediately.

  The default value of GOSUMDB is: sum.golang.org, which is not accessible in China, but GOSUMDB can be proxied by the Go module proxy (see: Proxying a Checksum Database for details).

  So we can solve it by setting GOPROXY, and the module proxy goproxy.cn we set up previously can support the proxysum.golang.org, so after setting GOPROXY, you don’t need to care too much.
  In addition, if there is a custom requirement for the value of GOSUMDB, it supports the following formats:

  • 格式 1:<SUMDB_NAME>+<PUBLIC_KEY>。
  • 格式 2:<SUMDB_NAME>+<PUBLIC_KEY> <SUMDB_URL>。

  It can also be set to “off”, which prevents Go from verifying the module version in subsequent operations.

GONOPROXY/GONOSUMDB/GOPRIVATE

  These three environment variables are used in current projects that rely on private modules, such as your company’s private git repository, or private libraries in github, which are all private modules and must be set. Pull failed.

  In more detail, it is the scenario when you rely on a module that is not accessible by the Go module agent specified by GOPROXY or the Go checksum database specified by GOSUMDB.

  It is generally recommended to set GOPRIVATE directly, and its value will be used as the default value of GONOPROXY and GONOSUMDB, so the best recommended posture is to use GOPRIVATE directly.

  And their values are a module path prefix separated by a comma “,”, that is, you can set multiple, for example:

$ go env -w GOPRIVATE="git.example.com,github.com/eddycjy/mquote"

  Once set, modules prefixed with git.xxx.com and github.com/eddycjy/mquote are considered private modules.

If you don’t want to reset it every time, we can also use wildcards, for example:

$ go env -w GOPRIVATE="*.example.com"

  In this way, all subdomains with a module path of example.com (for example: git.example.com) will not go through the Go module proxy and Go checksum database. Note that example.com itself is not included.

Open Go Modules

  At present, Go modules are not enabled by default, so Go provides the environment variable GO111MODULE as a switch for Go modules, which allows the following parameters to be set:

  • auto: Enable Go modules as long as the project includes a go.mod file. Currently it is still the default in Go1.11 to Go1.14.
  • on: Enable Go modules. The recommended setting will be the default value in future versions.
  • off: Disable Go modules, not recommended.

  If you are not sure what your current value is, you can execute the go env command to see the results

$ go env
GO111MODULE="off"
...

  If you need to change the value of GO111MODULE, it is recommended to set it through the go env command:

 $ go env -w GO111MODULE=on

  However, it should be noted that if the corresponding system environment variable has a value (set it), go env does not support overwriting, otherwise the following error message will appear:warning: go env -w GO111MODULE = .. . does not override conflicting OS environment variable.

Or you can achieve this by directly setting the system environment variables (writing the corresponding .bash_profile file):

$ export GO111MODULE=on

Initialize the project

  After starting Go modules, we need to create a sample project for demonstration, execute the following command:

$ mkdir -p $HOME/eddycjy/module-repo 
$ cd $HOME/eddycjy/module-repo

  Then initialize Go modules as follows:

$ go mod init github.com/eddycjy/module-repo
go: creating new go.mod: module github.com/eddycjy/module-repo

  When executing the go mod init command, we specified the module import path asgithub.com/eddycjy/module-repo. Next we create the main.go file in the root directory of the project, as follows:

package main

import (
    "fmt"
    "github.com/eddycjy/mquote"
)

func main() {
    fmt.Println(mquote.GetHello())
}

  Then execute the go get github.com / eddycjy / mquote command in the project root directory, as follows:

$ go get github.com/eddycjy/mquote 
go: finding github.com/eddycjy/mquote latest
go: downloading github.com/eddycjy/mquote v0.0.0-20200220041913-e066a990ce6f
go: extracting github.com/eddycjy/mquote v0.0.0-20200220041913-e066a990ce6f

View go.mod file

  When the project is initialized, a go.mod file is generated, which is the most important identifier necessary for the Go modules project to be enabled. It is also the identifier when the GO111MODULE value is auto. It describes the current project (that is, the current module). Meta information, each line starts with a verb.

  After we have just initialized and simply pulled, we look at the go.mod file again, the basic content is as follows:

module github.com/eddycjy/module-repo

go 1.13

require (
    github.com/eddycjy/mquote v0.0.0-20200220041913-e066a990ce6f
)

  For further explanation, we simulate the following:

module github.com/eddycjy/module-repo

go 1.13

require (
    example.com/apple v0.1.2
    example.com/banana v1.2.3
    example.com/banana/v2 v2.3.4
    example.com/pear // indirect
    example.com/strawberry // incompatible
)

exclude example.com/banana v1.2.4
replace example.com/apple v0.1.2 => example.com/fried v0.1.0 
replace example.com/banana => example.com/fish
  • module: used to define the module path of the current project.
  • go: It is used to identify the Go language version of the current module. The value is the version when the module was initialized. At present, it is only an identification function.
  • require: used to set a specific module version.
  • exclude: Used to exclude a specific module version from use.
  • replace: Used to replace one module version with another module version.

  In addition, you will find that there is an indirect logo behind example.com / pear. The indirect logo indicates that the module is an indirect dependency, that is, in the import statement in the current application, there is no explicit reference to this module. It may be that you first manually pull it down with go get, or it may depend on the modules you depend on. There are several situations.

View go.sum file

  After pulling the module dependencies for the first time, you will find that there is an extra go.sum file that lists all the module versions that the current project directly or indirectly depends on, and indicates the SHA-256 hash value of those module versions In the future, Go will ensure that the module versions that the project depends on will not be tampered with.

github.com/eddycjy/mquote v0.0.1 h1:4QHXKo7J8a6J/k8UA6CiHhswJQs0sm2foAQQUq8GFHM=
github.com/eddycjy/mquote v0.0.1/go.mod h1:ZtlkDs7Mriynl7wsDQ4cU23okEtVYqHwl7F1eDh4qPg=
github.com/eddycjy/mquote/module/tour v0.0.1 h1:cc+pgV0LnR8Fhou0zNHughT7IbSnLvfUZ+X3fvshrv8=
github.com/eddycjy/mquote/module/tour v0.0.1/go.mod h1:8uL1FOiQJZ4/1hzqQ5mv4Sm7nJcwYu41F3nZmkiWx5I=
...

  We can see that a module path may have the following two types:

github.com/eddycjy/mquote v0.0.1 h1:4QHXKo7J8a6J/k8UA6CiHhswJQs0sm2foAQQUq8GFHM=
github.com/eddycjy/mquote v0.0.1/go.mod h1:ZtlkDs7Mriynl7wsDQ4cU23okEtVYqHwl7F1eDh4qPg=

  The h1 hash is the Go module that unpacks the zip file of the target module version, and then hashes all the files in the package in order, and then composes their hash results into a total hash value according to a fixed format and algorithm.

  The h1 hash and the go.mod hash exist either at the same time or only go.mod hash. Under what circumstances will there be no h1 hash, that is, when Go thinks that a module version is definitely not needed, it will omit its h1 hash, and there will be no h1 hash, only go.mod hash.

View global cache

  We just successfully pulled the github.com/eddycjy/mquote module, and the results of the pull are cached in the$GOPATH/pkg/mod and $GOPATH/pkg/sumdb directories, and in the mod directory It will be stored in the format github.com/foo/bar, as follows:

mod
├── cache
├── github.com
├── golang.org
├── google.golang.org
├── gopkg.in
...

  It should be noted that only one copy of the data of the same module version is cached, and all other modules are shared and used. If you want to clean up all cached module version data, you can execute the go clean -modcache command.

Go get behavior under Go Modules

  When pulling project dependencies, you will find that the pulling process is divided into three major steps, namely, finding (finding), downloading (downloading), and extracting (extracting), and the pulling information is divided into three sections :

Go Modules Version Info

  It should be noted that the commit time of the pulled version is based on the UTC time zone, not the local time zone. At the same time, we will find that the version pulled by our go get command is v0.0.0, because we directly execute The version obtained by go get -u does not specify any version information, and it is selected by Go modules according to internal rules.

go get pull behavior

  Just now we used the go get command to pull the new dependencies, then what functions does go get provide, the common pull commands are as follows:

command function
go get Pulling dependencies will perform a specified pull (update) and will not update other modules that it depends on.
go get -u Updating an existing dependency will force an update of all other modules it depends on, excluding itself.
go get -u -t ./… Update all directly and indirectly dependent module versions, including those used in unit tests.

  Then I want to choose how to implement the specific version, as follows:

command function
go get golang.org/x/text@latest Pull the latest version. If there is a tag, it will be used first.
go get golang.org/x/text@master Pull the latest commit of the master branch.
go get golang.org/x/text@v0.3.2 Pull the commit with tag v0.3.2.
go get golang.org/x/text@342b2e Pulling a commit with a hash of 342b231 will eventually be converted to v0.3.2.

Go get version selection

  Let’s review the go get github.com/eddycjy/mquote we pulled, and the result isv0.0.0-20200220041913-e066a990ce6f. Looking at the behavior of go get mentioned above, you may still There will be some doubts, what is the version selection rule of go get without specifying any version, that is whygo get pulls v0.0.0, when is it Will pull tags with normal version number. Actually this needs to distinguish between two cases, as follows:

  • The pulled modules have release tags:
    • If there is only a single module, then take the tag with the largest major version number.
    • If there are multiple modules, the corresponding module path is calculated, and the tag with the largest major version number is used (the module path of the submodule tag will have a prefix requirement)
  • The module being pulled has not been published tags:
    • By default, the commithash of the latest commit of the master branch is taken.
    • No tags have been posted

No published tags

  So why does it pull v0.0.0, becausegithub.com / eddycjy / mquote does not publish any tags, as follows:

没有发布过 tags

  Therefore, it takes the commit time and commithash of the latest commit of the master branch by default, which is 20200220041913-e066a990ce6f, which belongs to the second case.

Have tags

  When the project has published tags, there are also multiple modes, that is, there is only a single module and multiple modules. We unified the display with multiple modules, because in the case of multiple modules, a single module is already included. Used, as shown below:

有发布 tags

  In this project, we have added two tags: v0.0.1 and module / tour / v0.0.1. At this time you may be wondering, why is it so strange to tag module / tour / v0.0.1, does this have any purpose?

  In fact, it is the tag expression of multiple modules of Go modules in the same project. Its main directory structure is:

mquote
├── go.mod
├── module
│   └── tour
│       ├── go.mod
│       └── tour.go
└── quote.go

  You can see that there is a go.mod file in the root directory of the mquote project, and a go.mod file in the module / tour directory. The corresponding relationship between module import and version information is as follows:

tag mod url function
v0.0.1 github.com/eddycjy/mquote mquote repository v 0.0.1 version
module/tour/v0.01 github.com/eddycjy/mquote/module/tour v0.0.1 version of the submodule module / tour under the mquote repository

Import main and submodules

  In combination with the above, if you pull the main module, you still execute the following command as usual:

$ go get github.com/eddycjy/mquote@v0.0.1
go: finding github.com/eddycjy/mquote v0.0.1
go: downloading github.com/eddycjy/mquote v0.0.1
go: extracting github.com/eddycjy/mquote v0.0.1

  If you want to pull the submodule, execute the following command:

$ go get github.com/eddycjy/mquote/module/tour@v0.0.1
go: finding github.com/eddycjy/mquote/module v0.0.1
go: finding github.com/eddycjy/mquote/module/tour v0.0.1
go: downloading github.com/eddycjy/mquote/module/tour v0.0.1
go: extracting github.com/eddycjy/mquote/module/tour v0.0.1

  We compare the pull of the main module and the submodule, and you will find that the pull of the submodule will be an extra step. It will first find github.com / eddycjy / mquote / module, then continue to calculate, and finally pull tomodule / tour.

Go Modules import path description

Import paths for different versions

  In the previous module pull and reference, you will find that our module import paths are github.com/eddycjy/mquote andgithub.com/eddycjy/mquote/module/tour, it seems that there is nothing special.
  Actually not. In fact, Go modules omit the version number when the main version number is v0 and v1, and the main version number needs to be explicitly specified if the main version number is v2 or above, otherwise there will be conflicts, and its tags and modules The approximate correspondence of the import path is as follows:

tag 模块导入路径
v0.0.0 github.com/eddycjy/mquote
v1.0.0 github.com/eddycjy/mquote
v2.0.0 github.com/eddycjy/mquote/v2
v3.0.0 github.com/eddycjy/mquote/v3

  In short, when the major version numbers are v0 and v1, you do not need to include the major version information in the module import path. After the v1 version, that is, v2, you must add the major version number at the end of the module import path. , When quoting, it needs to be adjusted to the following format:

import (
    "github.com/eddycjy/mquote/v2/example"
)

  Also ignoring major version numbers v0 and v1 is mandatory (not optional), so each package has only one clear and canonical import path.

Why ignore major version numbers for v0 and v1

-1. The reason for ignoring the v1 version in the import path is: considering that many developers create packages that will never change once they reach the v1 version, this is officially encouraged, and it is not considered that all these developers unintentionally release v2 Should be forced to have a clear v1 version suffix, which will cause the v1 version to become “noisy” and meaningless.

-2. The reason for ignoring the v0 version in the import path is that according to the semantic version specification, these versions of v0 have no compatibility guarantee at all. The need for an explicit v0 version of the identity does not help much to ensure compatibility.

Semantic versioning for Go Modules

  We keep referring to the version number in the use of Go Modules, which is essentially called the “semantic version”, assuming our version number is v1.2.3, as follows:

  Its version format is “major version number. Minor version number. Revision number”.

Go Modules Version 1

  • Major version number: when you make incompatible API changes.
  • Minor version number: Added functionality when you made backwards compatible.
  • Revision number: when you made backwards compatibility fixes.

  Assume that you are the previous version number or special case, you can append the version information to the “major version number. Minor version number. Revision number” as an extension, as follows:

Go Modules Version 2

  So far, we have introduced the two types of version number methods supported by Go modules. When we release a new version to tag, we need to pay attention to it. Otherwise, version numbers that do not follow the semantic version rules cannot be pulled.

Go Modules minimal version selection

  Now we already have a module and a published tag, but a module often depends on many other modules, and different modules will likely rely on different versions of the same module when they depend on it, as shown below (from Russ Cox):

Go Modules Version 3

  In the above dependencies, module A depends on module B and module C, while module B depends on module D, module C depends on modules D and F, module D depends on module E, and different versions of the same module also depend on the corresponding Different versions of the module. So how does Go modules choose the version at this time, and which version is chosen?

  According to the proposal, we can learn that Go modules will sort out the list of dependent versions of each module, and finally get a build list, as shown below (from Russ Cox):

Go Modules Version 4

  We see rough list and final list. The difference between the two is the module D (v1.3, v1.4) that is repeatedly referenced. The final list uses the v1.4 version of module D. The main reasons are:

  • Semantic version control: Because the v1.3 and v1.4 changes of module D are minor version number changes, under the constraints of the semantic version, v1.4 must be backward compatible with v1.3 , So it ’s considered non-destructive and compatible.

  • Specification of the module import path: the major version number is different, the module import path is different, so if there is an incompatibility, the major version number will change, and the module import path will naturally change, so it will not be the same as the first Conflicting foundations.

Go.sum file to submit

  In theory, both go.mod and go.sum files should be submitted to your Git repository.

  Suppose we do not upload the go.sum file, it will cause everyone to execute Go modules related commands, and a new go.sum will be generated, that is, it will be pulled to the upstream again. It may be tampered when pulling Yes, there will be great security risks, and the verification content with the baseline version (the first person submitted, the expected version) is lost, so the go.sum file needs to be submitted.

Summarize

  So far we have introduced the past and present life of Go modules, the basic use and the behavior conversion of the go get command in Go modules mode. At the same time, we have performed common multi-version import paths, semantic version control, and minimum version selection rules for multiple modules Gave a rough introduction.

  The growth and development of Go modules has gone through a certain process. If you are a new reader, you can start a project based on Go modules directly. If you have an old project, then it is time to consider switching over. Go1.14 is ready Ready and recommended for you to use.

Seference

wiki/Modules
wiki/vgo
Full of Go Modules and goproxy.cn


Author: Kyle Liu
Reprint policy: All articles in this blog are used except for special statements CC BY 4.0 reprint polocy. If reproduced, please indicate source Kyle Liu !
 Previous
Istio Bookinfo Task in Minikube Istio Bookinfo Task in Minikube
Kubernetes 的安装非常复杂,本文将会介绍 Deployment YAML 名词解释。
2020-05-17
Next 
Go Modules 终极入门 Go Modules 终极入门
Go modules 是 Go 语言中正式官宣的项目依赖解决方案,Go modules(前身为vgo)于 Go1.11 正式发布,在 Go1.14 已经准备好,并且可以用在生产上(ready for production)了,Go 官方也鼓励所有用户从其他依赖项管理工具迁移到 Go modules。
2020-02-29
  TOC