Collaboration and Crawling W/ Golang - Google's Go Language

This is an introduction to Daniel and git

• working in windows on the terminal / bash / shell

  • as opposed to command prompt / cmd / dos prompt / powershell

• add an ssh public key to github

• working in windows

• git, and our git repo

adding Daniel as a collaborator to the github repo

Daniel clones the repo for our class

• cloning and forking explained

• how branches work with this

An example of a merge conflict and how it gets resolved. Using branches helps you avoid merge conflicts.

• Commands used in video

  • show branches git branch

    • all branches git branch -a

  • create a new branch git checkout -b <branch name>

before creating a branch, get your current state of the remote repository

• git pull

when you create a branch it’s created off of whatever branch you’re already on

• checkout out a branch from master git checkout <branch name>

• on github

• on terminal

  • get onto the branch you want to merge INTO, then PULL, then merge

  • merge a branch git merge <branch name>

It is always a good idea to make things current before you work on branches. Do this with a GIT PULL. Once you have done that, switch to the master branch, then delete whatever branches you no longer want. Make sure you have merged any changes you want from a branch before you delete it.

• delete a branch git branch -d <branch name>

Prefixing branches with an issue number from github is a useful practice.

• • create branch git checkout -b <branch name>

    • see branches git branch

    • see all branches git branch -a

  • work on a branch git checkout <branch name>

  • see status of branch git status

  • stage files git add -A

    • see staged files git status

  • commit files git commit -m <some message>

  • switch back to master branch git checkout master

  • merge your branch git merge <branch name>

  • push your code to remote repo git push [origin master]

  • delete your branch git branch -d <branch name>

  • clean up cache git remote prune origin

An introduction in a class is a common practice. This lets the students learn about the other students in the course. In this hands-on exercise, we will introduce ourselves.

In this exercise, you will merge

FROM

• the master on your fork

TO

• master on the repo from which you forked

  • ( https://github.com/GoesToEleven/golang-project )

fork this repo

• https://github.com/GoesToEleven/golang-project

• on the branch master, go into the folder “002-hands-on-exercises/01-about”

• make a copy of the file “000-TEMPLATE-about-me.txt”

• rename your copy to

  • “<your github username>.txt”

• in the file, to the extent that you feel comfortable, introduce and share about yourself

• stage this file

• commit this file

  • provide a descriptive commit message

• push this commit to your fork

• create a pull request to branch master on this repo

  • https://github.com/GoesToEleven/golang-project

Note: the structure of this file will allow us to later parse the information into other forms of storage, eg, take it into a database; do things with it. Second note: if for some reason the template is no longer there, simply provide a description.

In this exercise, you will merge

FROM

• the branch (off master) on your fork

TO

• master on the repo from which you forked

  • ( https://github.com/GoesToEleven/golang-project )

At your fork of this repo https://github.com/GoesToEleven/golang-project do the following

• create a branch titled “myhumble”

  • switch to the branch

• in the folder “002-hands-on-exercises/02-humble” create a file called “<your github username>-humble.txt”

• in the file, share what “being humble” means to you

• stage this file

• commit this file

  • provide a descriptive commit message

• push this file to your fork

• create a pull request to branch master on this repo

  • https://github.com/GoesToEleven/golang-project

• git remote

• git remote show origin

• git remote add upstream <ssh url>

• git fetch upstream

• git checkout master

  • confirm with

    • git branch -a

• git pull

• two ways to pull from FORKED repo

  • git rebase upstream/master

    • modifies history

    • can cause issue if people have FORKED your FORK

  • git merge upstream/master

    • doesn’t modify history

    • safer if people have forked your fork

• now push to your FORKED repo on github - either the first or, if needed, the second

  • git push

  • git push -f origin master

Notes from StackOverflow:

In your local clone of your forked repository, you can add the original GitHub repository as a "remote". ("Remotes" are like nicknames for the URLs of repositories - origin is one, for example.) Then you can fetch all the branches from that upstream repository, and rebase your work to continue working on the upstream version. In terms of commands that might look like:

# Add the remote, call it "upstream": git remote add upstream https://github.com/whoever/whatever.git # Fetch all the branches of that remote into remote-tracking branches, # such as upstream/master: git fetch upstream # Make sure that you're on your master branch: git checkout master # Rewrite your master branch so that any commits of yours that # aren't already in upstream/master are replayed on top of that # other branch: git rebase upstream/master

If you don't want to rewrite the history of your master branch, (for example because other people may have cloned it) then you should replace the last command with git merge upstream/master. However, for making further pull requests that are as clean as possible, it's probably better to rebase.

If you've rebased your branch onto upstream/master you may need to force the push in order to push it to your own forked repository on GitHub. You'd do that with:

git push -f origin master

You only need to use the -f the first time after you've rebased.

source

https://stackoverflow.com/questions/7244321/how-do-i-update-a-github-forked-repository

Here is Todd’s solution

In this exercise, you will merge

FROM:

• the master on your fork

TO

• master on the repo from which you forked

  • ( https://github.com/GoesToEleven/golang-project )

At your fork of this repo https://github.com/GoesToEleven/golang-project in the folder “002-hands-on-exercises/03-purpose” do the following

• create a branch titled “mypurpose”

  • switch to the branch

  • create a file called “<your github username>-purpose.txt”

    • share what you believe gives purpose and meaning to human life

  • stage this file

  • commit this file

    • provide a descriptive commit message

  • merge this branch to the master branch ON YOUR FORK

• create a pull request from branch master on your fork to branch master on this repo

  • https://github.com/GoesToEleven/golang-project

Here is Todd’s solution

Delete your branches.

• git branch -d <branch name>

Once we accept your pull request (which might take a few months), you can delete your branch on your fork on github, and then prune locally.

• git remote prune origin

An overview of go modules.

• GOPATH will always still work for backwards compatibility

• Modules is the new recommended way of doing things however

Understanding package management and dependencies. Different names, mostly the same meaning:

• packages

• libraries

• groupings of other people’s code (OPC)

• groupings of your own code

“A module is a collection of related Go packages. Modules are the unit of source code interchange and versioning. The go command has direct support for working with modules, including recording and resolving dependencies on other modules. Modules replace the old GOPATH-based approach to specifying which source files are used in a given build.”

• source - at the terminal: go help modules

Using other people’s code comes with risks. Great article by Russ Cox:

• Our Software Dependency Problem

  • “Note that while the go command makes adding a new dependency quick and easy, it is not without cost. Your module now literally depends on the new dependency in critical areas such as correctness, security, and proper licensing, just to name a few.”

    • THIS APPLIES TO ALL DEPENDENCIES / USING OTHER PEOPLE’S CODE

• npm “left-pad” story

  • direct dependencies

  • indirect dependencies

Modules are usually one-to-one with a repo, though there are ways to put multiple modules in one repo.

• A module is a collection of Go packages stored in a file tree with a go.mod file at its root. The go.mod file defines the module’s module path, which is also the import path used for the root directory, and its dependency requirements, which are the other modules needed for a successful build. Each dependency requirement is written as a module path and a specific semantic version.

  • NAMESPACING

    • not: you forgot someone’s name (spaced on their name)

    • is: you have a unique name to identify different things

      • people example

      • code example

• As of Go 1.11, the go command enables the use of modules when the current directory or any parent directory has a go.mod, provided the directory is outside $GOPATH/src. (Inside $GOPATH/src, for compatibility, the go command still runs in the old GOPATH mode, even if a go.mod is found.

• GO COMMAND DOCUMENTATION

  • ctrl + f → “module”

• Starting in Go 1.13, module mode will be the default for all development.

  • source

• Golang blog

  • this section of the curriculum draws from, and in some cases closely parallels, these go blog articles:

  • Using Go Modules Tyler Bui-Palsulich and Eno Compton

  • Migrating to Go Modules Jean de Klerk

  • Module Mirror and Checksum Database Launched Katie Hockman

Create a new, empty directory somewhere outside $GOPATH/src.

• go mod init <name spacing, eg, example.com/hello>

file

code

hello.go

package hello

func Hello() string {

return "Hello, world."

}

hello_test.go

package hello

import "testing"

func TestHello(t *testing.T) {

want := "Hello, world."

if got := Hello(); got != want {

t.Errorf("Hello() = %q, want %q", got, want)

}

}

• go test

The go.mod file only appears in the root of the module. Packages in subdirectories have import paths consisting of the module path plus the path to the subdirectory.

• cat go.mod

The primary motivation for Go modules was to improve the experience of using (that is, adding a dependency on) code written by other developers. Go resolves imports by using the specific dependency module versions listed in go.mod.

• Let's update our hello.go to import rsc.io/quote and use it to implement Hello

file

code

hello.go

package hello

import "rsc.io/quote"

func Hello() string {

return quote.Hello()

}

hello_test.go

package hello

import "testing"

func TestHello(t *testing.T) {

want := "Hello, world."

if got := Hello(); got != want {

t.Errorf("Hello() = %q, want %q", got, want)

}

}

• go test

• cat go.mod

NOTES:

• if you import a package/library (call it what you will) not yet tracked by go.mod, by default:

  • go adds it to go.mod, using the latest version.

  • “Latest” is defined as

    • latest tagged stable (non-prerelease) version, or

    • latest tagged prerelease version, or

    • latest untagged version.

• see the version of rsc.io/quote

  • cat go.mod

  • only direct dependencies are recorded in the go.mod file

• two types of dependencies

  • DIRECT dependency

  • INDIRECT dependency

• see all direct and indirect dependencies

  • go list -m all

In addition to go.mod, the go command maintains a file named go.sum containing the expected cryptographic hashes of the content of specific module versions:

• cat go.sum

go.sum ensures that future downloads of modules retrieve the same bits as the first downloads. This ensures the modules your project depends on do not change unexpectedly, whether for malicious, accidental, or other reasons.

• Both go.mod and go.sum should be checked into version control.

DISCUSSION

• Why check go.mod and go.sum into version control?

DISCUSSION & EXPLORATION

• Modules already downloaded are cached locally (in $GOPATH/pkg/mod).

With Go modules, versions are referenced with semantic version tags. A semantic version has three parts:

• major.minor.patch

For example, for v0.1.2,

• major version is 0

• minor version is 1

• patch version is 2

Here is a description of each part:

• MAJOR

  • backwards incompatible changes added

• MINOR

  • backwards compatible changes added

• PATCH

  • backwards compatible bug fixes

Learn more about semantic versioning.

DISCUSSION

• If we see a dependency go from v1.8.9 to v2.0.0, what should we do?

Let's walk through a couple minor version upgrades. Then, we’ll consider a major version upgrade. From the output of ...

• go list -m all

... we can see we're using an untagged version of golang.org/x/text. The golang.org/x/text version is something like this ...

• v0.0.0-20170915032832-14c0d48ead0c

… and is an example of a pseudo-version, which is the go command's version syntax for a specific untagged commit. Let's upgrade to the latest tagged version - either of these commands work (and we will see how we can do @v1.3.1 in the next video):

• go get golang.org/x/text

• go get golang.org/x/text@latest

Each argument passed to go get can take an explicit version. The default is @latest which resolves to the latest version as defined earlier. Now let’s test that everything still works:

• go test

What do our dependencies look like now?

• go list -m all

The golang.org/x/text package has been upgraded to the latest tagged version. What does our go.mod file look like now?

• cat go.mod

The indirect comment indicates a dependency is not used directly by this module, only indirectly by other module dependencies. See go help modules for details.

See all of your direct and indirect dependencies

• go list -m all

Now let's try upgrading the rsc.io/sampler minor version.

• go get rsc.io/sampler@latest

• go test

Uh, oh! The test failure shows that the latest version of rsc.io/sampler is incompatible with our usage. Let's list the available tagged versions of that module.

• go list -m -versions rsc.io/sampler

rsc.io/sampler v1.99.99 should have been backwards-compatible with rsc.io/sampler v1.3.0, but bugs or incorrect client assumptions about module behavior can both happen. Let’s get a different version

• go get rsc.io/sampler@v1.3.1

Each argument passed to go get can take an explicit version

• the default is @latest which resolves to the latest version as defined earlier.

• or you can specify the version like the above @v1.3.1

Let's add a new function to our package:

file

code

hello.go

package hello

import (

"rsc.io/quote"

quoteV3 "rsc.io/quote/v3"

)

func Hello() string {

return quote.Hello()

}

func Proverb() string {

return quoteV3.Concurrency()

}

hello_test.go

package hello

import "testing"

func TestHello(t *testing.T) {

want := "Hello, world."

if got := Hello(); got != want {

t.Errorf("Hello() = %q, want %q", got, want)

}

}

func TestProverb(t *testing.T) {

want := "Concurrency is not parallelism."

if got := Proverb(); got != want {

t.Errorf("Proverb() = %q, want %q", got, want)

}

}

• go test

• cat go.mod

• go list -m all

• go list -m rsc.io/q...

Semantic import versioning

• you can have different versions of the same package / library / code (call it what you will)

• Each different major version (v1, v2, and so on) uses a different module path

  • starting at v2, the path must end in the major version

  • this gives incompatible packages (different major versions) different names

  • you can only use one of each major version - example:

    • rsc.io/quote

    • rsc.io/quote/v2

    • rsc.io/quote/v3, and so on.

  • it is impossible for a program to build with both rsc.io/quote v1.5.2 and rsc.io/quote v1.6.0

  • allowing different major versions of a module (because they have different paths) gives module consumers the ability to upgrade to a new major version incrementally.

    • The ability to migrate incrementally is especially important in a large program or codebase.

Let's complete our conversion from rsc.io/quote to only rsc.io/quote/v3. If we look at the documentation of the package using go doc:

• go doc rsc.io/quote/v3

Because of the major version change, we should expect that some APIs may have been removed, renamed, or otherwise changed in incompatible ways. Reading the docs, we can see that Hello has become HelloV3.

file

code

hello.go

package hello

import "rsc.io/quote/v3"

func Hello() string {

return quote.HelloV3()

}

func Proverb() string {

return quote.Concurrency()

}

hello_test.go

package hello

import "testing"

func TestHello(t *testing.T) {

want := "Hello, world."

if got := Hello(); got != want {

t.Errorf("Hello() = %q, want %q", got, want)

}

}

func TestProverb(t *testing.T) {

want := "Concurrency is not parallelism."

if got := Proverb(); got != want {

t.Errorf("Proverb() = %q, want %q", got, want)

}

}

• go test

We've removed all our uses of rsc.io/quote, but it still shows up in go list -m all and in our go.mod file:

• cat go.mod

• go list -m all

Building a single package, like with go build or go test, can easily tell when something is missing and needs to be added, but not when something can safely be removed. Removing a dependency can only be done after checking all packages in a module, and all possible build tag combinations for those packages. An ordinary build command does not load this information, and so it cannot safely remove dependencies. The go mod tidy command cleans up these unused dependencies:

• go mod tidy

• cat go.mod

• go list -m all

If you are already using dependency manager like dep or glide or something else, the command go mod init will automatically convert your dependency management to go modules. These are the older dependency management tools that automatically convert to go modules. To convert,

• navigate to the root of your project

• go mod init

• go mod tidy

Note that other dependency managers may specify dependencies at the level of individual packages or entire repositories (not modules), and generally do not recognize the requirements specified in the go.mod files of dependencies. Consequently, you may not get exactly the same version of every package as before, and there's some risk of upgrading past breaking changes. Therefore, it's important to follow the above commands with an audit of the resulting dependencies. To do so, run

• go list -m all

Upgrade or downgrade to the correct version using go get as needed.

• A module mirror is a special kind of module proxy that caches metadata and source code in its own storage system, allowing the mirror to continue to serve source code that is no longer available from the original locations. This can speed up downloads and protect you from disappearing dependencies.

• The Go team maintains a module mirror, served at proxy.golang.org, which the go command will use by default for module users as of Go 1.13. If you are running an earlier version of the go command, then you can use this service by setting GOPROXY=https://proxy.golang.org in your local environment.

• GOPRIVATE

source & video: https://www.youtube.com/watch?v=KqTySYYhPUE

A “git tag” allows you to tag a particular commit with some name. With golang modules, you tag should follow semantic versioning. Here are the commands to use:

• create a tag

  • git tag <tag name>

• show a list of all the tags

  • git tag

• push a tag

  • git push --tags

  • git push origin <tag name>

Releasing major versions in go modules

You can alias one package name with another package name

• go mod init <add your name space>

  • creates a new module, initializing the go.mod file that describes it.

• go build, go test

  • and other package-building commands add new dependencies to go.mod as needed.

• cat go.mod

  • shows the contents of go.mod file

• go list -m all

  • prints the current module’s dependencies.

    • direct

    • indirect

• go list -m -versions <package name>

  • lists all of the versions of a package

• go get <package name>

  • changes the required version of a dependency (or adds a new dependency).

  • example: go get rsc.io/sampler@v1.3.1

  • defaults to @latest

• go mod tidy

  • removes unused dependencies.

  • adds dependencies for other platforms

• go doc <package name>

  • shows us the documentation of a package.

  • example: go doc fmt

• tags

  • create a tag

    • git tag <tag name>

  • show a list of all the tags

    • git tag

  • push a tag

    • git push --tags

    • git push origin <tag name>

• create a go module

• import rsc.io/quote

  • use Glass() from rsc.io/quote

• run your program

• look at go.mod

  • What version of rsc.io/quote are you using?

• Do you have any indirect dependencies?

  • go list -m all

• look at go.sum

Working with the same code that you built in hands-on exercise #1:

• go get v3.1.0 of rsc.io/quote

• Does your code still run?

• What is the folder structure of v3.1.0 on github?

• run these commands

  • go list -m all

  • go mod tidy

  • go list -m all

• https://github.com/golang/go/wiki/Modules#releasing-modules-v2-or-higher

• from v3.1.0 of rsc.io/quote

  • use the function Glass()

• from v2.0.1 of rsc.io/quote

  • use the function Opt()

• repo #1

  • create a github repo called quotes

  • clone your repo

  • add go module

  • add a function Favs() []string

  • add a git tag version of v0.1.0

  • push your code

• repo #2

  • create a new project with a go module

    • this can be on your computer and does not need to be on github

  • from your github repo #1 “quotes”

    • use the function Favs() []string

  • run your code

  • look at

    • go.mod

    • go.sum

Summary

Using “git checkout” to check files out from HEAD on my local computer / my local clone

also covered:

• .gitignore

• git fetch

also covered:

• .gitignore

• git fetch

Here are some things to think about when crawling:

• First thing to think when wanting to crawl a website: Is there an API from the website you want to crawl that provides the data you want?

• Second thing to think about - How do you get the data from the web page sent to you?

  • robots.txt

    • good to look at

    • tells you what you can do at the site

• introducing colly

  • http://go-colly.org/

• look at the examples

• look at the documentation

• css class names?

• wow! colly is free!?!

• did this get all of the tweets?

• anything else?

• two versions of twitter

  • logged in

  • incognito

• Daniel’s solution - overview

  • found a good library

  • looked at the examples

    • how much of your code was just copy-pasta of the example?

• terminology disambiguation revisited

  • module

    • “In software, a module is a part of a program. Programs are composed of one or more independently developed modules that are not combined until the program is linked. A single module can contain one or several routines. (2) In hardware, a module is a self-contained component.”

      • https://www.webopedia.com/TERM/M/module.html

    • IN GO

      • A “module” or “go module” is

        • namespacing for your workspace.

        • dependency management

          • direct dependencies

          • indirect dependencies

          • versioning

      • can have 1+ packages

      • usually 1-to-1 with a repo

  • package

    • “A package is a namespace that organizes a set of related classes and interfaces. Conceptually you can think of packages as being similar to different folders on your computer. You might keep HTML pages in one folder, images in another, and scripts or applications in yet another.”

      • https://docs.oracle.com/javase/tutorial/java/concepts/package.html

    • IN GO

      • higher level grouping of code

      • scope in go:

        • universe

        • package

        • file

        • code block

          • function

          • curlies

          • Go is lexically scoped using blocks: The scope of a predeclared identifier is the universe block. The scope of an identifier denoting a constant, type, variable, or function (but not method) declared at top level (outside any function) is the package block. The scope of the package name of an imported package is the file block of the file containing the import declaration. The scope of an identifier denoting a method receiver, function parameter, or result variable is the function body. The scope of a constant or variable identifier declared inside a function begins at the end of the ConstSpec or VarSpec (ShortVarDecl for short variable declarations) and ends at the end of the innermost containing block. The scope of a type identifier declared inside a function begins at the identifier in the TypeSpec and ends at the end of the innermost containing block.

            • source: golang spec

  • library

    • “In computer science, a library is a collection of non-volatile resources used by computer programs, often for software development. These may include configuration data, documentation, help data, message templates, pre-written code and subroutines, classes, values or type specifications.”

      • https://en.wikipedia.org/wiki/Library_(computing)

    • IN GO

      • standard library

        • which is filled with packages

      • code you can use from others