tweeeetyのぶろぐ的めも

アウトプットが少なかったダメな自分をアウトプット<br>＼(＾o＾)／

【GAE】goappコマンドについて簡単にまとめてみた

go golang gcp GAE

はじめに

GAE goな環境で開発しているときに、

$ goapp serve app.yaml

という感じで使うわけですが、そもそもgoappってなんぞやという事を軽くまとめてみました。

アジェンダ

  1. goappコマンドの簡単な紹介
  2. goappコマンドとは
  3. goappコマンドのinstall
  4. goapp serveコマンドの挙動

1. goappコマンドの簡単な紹介

goappコマンドは、GAE goな環境で開発する際に使うコマンドです。

例えば、以下はローカルでGAEアプリを起動する際の例です

$ goapp serve app.yaml
INFO     2018-05-02 14:45:49,257 devappserver2.py:120] Skipping SDK update check.
INFO     2018-05-02 14:45:49,353 api_server.py:274] Starting API server at: http://localhost:51983
INFO     2018-05-02 14:45:49,360 dispatcher.py:270] Starting module "Hoge" running at: http://localhost:8080
INFO     2018-05-02 14:45:49,365 admin_server.py:152] Starting admin server at: http://localhost:8000
WARNING  2018-05-02 14:45:49,367 devappserver2.py:215] No default module found. Ignoring.
/usr/local/Caskroom/google-cloud-sdk/latest/google-cloud-sdk/platform/google_appengine/google/appengine/tools/devappserver2/mtime_file_watcher.py:182: UserWarning: There are too many files in your application for changes in all of them to be monitored. You may have to restart the development server to see some changes to your files.
  'There are too many files in your application for '

2. goappコマンドとは

goappコマンドとは、自分なりの言葉で表現すると

Google Cloud SDKの一部である
gcloudでインストールするコンポーネントの1つに含まれるコマンド

です。（文章がわかりずらくてすみません）

gcloud/SDK/コンポーネントあたりについては別に書いてみたのでご参考ください。

【GAE】Google Cloud SDKとgcloudとコンポーネント（components）とgoappをおさらい

以降で、goappについて掘り下げます

GAEとgo

GAE goな環境で開発すると記載しましたが、
GAEはgoを内包しています。

ためしにversionを見てみます。
PCにインストールしたgoとversionが違う事がわかります

# goappのversionと
# 内包されているgoのversionを確認
$ goapp version
go version 1.8.5 (appengine-1.9.68) darwin/amd64

# PCのgoのversionを確認
$ go version
go version go1.6.2 darwin/amd64

goappのコマンド

helpを見てみるとgoそのもののhelpと似ています

goapp help
$ goapp help
Go is a tool for managing Go source code.

Usage:

  goapp command [arguments]

The commands are:

  serve       starts a local development App Engine server
  deploy      deploys your application to App Engine
  build       compile packages and dependencies
  clean       remove object files
  doc         show documentation for package or symbol
  env         print Go environment information
  bug         start a bug report
  fix         run go tool fix on packages
  fmt         run gofmt on package sources
  generate    generate Go files by processing source
  get         download and install packages and dependencies
  install     compile and install packages and dependencies
  list        list packages
  run         compile and run Go program
  test        test packages
  tool        run specified go tool
  version     print Go version
  vet         run go tool vet on packages

Use "goapp help [command]" for more information about a command.

Additional help topics:

  c           calling between Go and C
  buildmode   description of build modes
  filetype    file types
  gopath      GOPATH environment variable
  environment environment variables
  importpath  import path syntax
  packages    description of package lists
  testflag    description of testing flags
  testfunc    description of testing functions

Use "goapp help [topic]" for more information about that topic.
go help
$ go help
Go is a tool for managing Go source code.

Usage:

  go command [arguments]

The commands are:

  build       compile packages and dependencies
  clean       remove object files
  doc         show documentation for package or symbol
  env         print Go environment information
  fix         run go tool fix on packages
  fmt         run gofmt on package sources
  generate    generate Go files by processing source
  get         download and install packages and dependencies
  install     compile and install packages and dependencies
  list        list packages
  run         compile and run Go program
  test        test packages
  tool        run specified go tool
  version     print Go version
  vet         run go tool vet on packages

Use "go help [command]" for more information about a command.

Additional help topics:

  c           calling between Go and C
  buildmode   description of build modes
  filetype    file types
  gopath      GOPATH environment variable
  environment environment variables
  importpath  import path syntax
  packages    description of package lists
  testflag    description of testing flags
  testfunc    description of testing functions

Use "go help [topic]" for more information about that topic.

goのラッパー的な要素に＋αして
GAE goな環境で開発する際に便利な
servやdeployやbugなどを加えたといった感じです。

3. goappコマンドのinstall

goappコマンドは直接インストールするわけではありません。

2. goappコマンドとは に記載しましたが、
gcloudコンポーネント群のうち、app-engine-goというコンポーネントに含まれるコマンドです。

ここではgoappについてのみ記載します。
gcloudについては公式がわかりやすいです
gcloud の概要

# インストールされる場所を確認
$ gcloud info | grep Installation
Installation Root: [/usr/local/Caskroom/google-cloud-sdk/latest/google-cloud-sdk]
Installation Properties: [/usr/local/Caskroom/google-cloud-sdk/latest/google-cloud-sdk/properties]

# gcloud経由でapp-engine-goをインストールする
$ gcloud components install app-engine-go

# 確認すると
# /usr/local/Caskroom/google-cloud-sdk/latest/google-cloud-sdk/platform/google_appengine
# というパスに入っている
$ ls -l /usr/local/Caskroom/google-cloud-sdk/latest/google-cloud-sdk/platform/google_appengine/*goapp*
-rwxr-xr-x+ 1 tweeeety  tweeeety  5109  5  2 22:08 /usr/local/Caskroom/google-cloud-sdk/latest/google-cloud-sdk/platform/google_appengine/goapp

# パスを追加
$ export PATH=$PATH:/usr/local/Caskroom/google-cloud-sdk/latest/google-cloud-sdk/platform/google_appengine

# 実行権限がないので追加
$ chmod +x /usr/local/Caskroom/google-cloud-sdk/latest/google-cloud-sdk/platform/google_appengine/goapp

# 使えるようになる
$ goapp version
go version 1.8.5 (appengine-1.9.68) darwin/amd64

app-engine-goでインストールされるもの

/usr/local/Caskroom/google-cloud-sdk/latest/google-cloud-sdk/platform/google_appengine 配下には
すでにいくつかのリソースが配置されてますが、
gcloud components install app-engine-go によって以下が追加されます。

-rw-r--r--+  1 tweeeety  tweeeety     5949  3 17 16:55 LICENSE.golang
-rw-r--r--+  1 tweeeety  tweeeety    17738  3 17 16:59 RELEASE_NOTES.golang
-rw-r--r--+  1 tweeeety  tweeeety      248  3 17 16:55 VERSION.golang
-rwxr-xr-x+  1 tweeeety  tweeeety  3679424  3 17 17:00 go-app-stager
-rw-r--r--+  1 tweeeety  tweeeety     4798  3 17 16:59 goapp
-rw-r--r--+  1 tweeeety  tweeeety     4798  3 17 16:59 godoc
-rw-r--r--+  1 tweeeety  tweeeety     4798  3 17 16:59 gofmt
drwxr-xr-x+  2 tweeeety  tweeeety       68  3 17 17:01 gopath
drwxr-xr-x+ 14 tweeeety  tweeeety      476  5  2 21:30 goroot-1.6
drwxr-xr-x+ 14 tweeeety  tweeeety      476  5  2 21:30 goroot-1.8
drwxr-xr-x+ 14 tweeeety  tweeeety      476  5  2 21:30 goroot-1.9

まぁ、go関連で納得という感じですね

4. goapp serveコマンドの挙動

goappというかgoapp serveの挙動です。

help

まずはhelpを見てみます。

$ goapp serve --help

usage: serve [serve flags] [application_dir | package | yaml_files...]

Serve launches your application on a local development App Engine server.

The argument to this command should be your application's root directory or a
single package which contains an app.yaml file. If you are using the Modules
feature, then you should pass multiple YAML files to serve, rather than a
directory, to specify which modules to serve. If no arguments are provided,
serve looks in your current directory for an app.yaml file.

The -host flag controls the host name to which application modules should bind
(default localhost).

The -port flag controls the lowest port to which application modules should
bind (default 8080).

The -use_mtime_file_watcher flag causes the development server to use mtime
polling for detecting source code changes, as opposed to inotify watches.

The -admin_port flag controls the port to which the admin server should bind
(default 8000).

The -clear_datastore flag clears the local datastore on startup.

The -debug flag builds the binary with debug information so that gdb or delve
can be used (default false).

This command wraps the dev_appserver.py command provided as part of the
App Engine SDK. For help using that command directly, run:
  ./dev_appserver.py --help

最後に書いてありますが、dev_appserver.py のラッパーであることがわかります。
また、オプションでportなどが指定できます。

goapp コマンドの中身

中身はpythonで書かれています。
100行ちょいなので読めるレベルです。

#!/usr/bin/env python2.7                                                                                                                                                                                                                                     
#
# Copyright 2011 Google Inc. All rights reserved.
# Use of this source code is governed by the Apache 2.0
# license that can be found in the LICENSE file.

"""Convenience wrapper for starting a Go tool in the App Engine SDK."""
from __future__ import print_function

import argparse
import os
import sys 

SDK_BASE = os.path.abspath(os.path.dirname(os.path.realpath(__file__)))


def FixGooglePath():
  """Adds the python libraries vendored with the SDK to sys.path.

  This allows us to run gotool.py directly and import the dependencies under the
  google/ directory.
  """
  # pylint: disable=g-import-not-at-top
  import wrapper_util
  # pylint: enable=g-import-not-at-top
  sys.path = wrapper_util.Paths(SDK_BASE).v1_extra_paths + sys.path
  if 'google' in sys.modules:
    google_path = os.path.join(os.path.dirname(__file__), 'google')
    google_module = sys.modules['google']
    google_module.__path__.append(google_path)
    if not hasattr(google_module, '__file__') or not google_module.__file__:
      google_module.__file__ = os.path.join(google_path, '__init__.py')


FixGooglePath()


# pylint: disable=g-import-not-at-top
from google.appengine.api import appinfo_includes
from google.appengine.tools.devappserver2.go import goroots
# pylint: enable=g-import-not-at-top


def GetArgsAndArgv():
  """Parses command-line arguments and strips out flags the control this file.

  Returns:
    Tuple of (flags, rem) where "flags" is a map of key,value flags pairs
    and "rem" is a list that strips the used flags and acts as a new
    replacement for sys.argv.
  """
  parser = argparse.ArgumentParser(add_help=False)
  parser.add_argument(
      '--dev_appserver', default=os.path.join(SDK_BASE, 'dev_appserver.py'))
  parser.add_argument(
      '--print-command', dest='print_command', default=False,
      action='store_true')
  namespace, rest = parser.parse_known_args()
  flags = vars(namespace)
  rem = [sys.argv[0]]
  rem.extend(rest)
  return flags, rem


def GetAppYamlPaths(gopath, app_path):
  """Generate possible paths to app.yaml.

  Args:
    gopath: String path to gopath directory.
    app_path: String path to the app.

  Yields:
    Possible paths for app.yaml from app_path up to the gopath root.
  """
  gopath = os.path.abspath(gopath)
  app_path = os.path.abspath(app_path)
  # If we were given an app.yaml, return only it
  if app_path.endswith('.yaml'):
    yield app_path
    raise StopIteration()
  # Always include the app_path, even if not on gopath
  yield os.path.join(app_path, 'app.yaml')
  # But only walk up if we're still on the gopath
  while app_path.startswith(gopath) and app_path != gopath:
    app_path = os.path.abspath(os.path.join(app_path, '..'))
    yield os.path.join(app_path, 'app.yaml')


def _ParseAppYaml(file_name):
  with open(file_name, 'r') as f:
    return appinfo_includes.Parse(f)


def GetGoRoot(gopath, argv):
  """Find the goroot.

  Args:
    gopath: String path to gopath directory.
    argv: A list of strings that looks like sys.argv. The last element of this
      list is checked to see if it's a valid path and, if so, is included in the
      search.

  Returns:
    The path to the correct goroot for the project.
  """
  # First, look for an app.yaml starting in the pwd
  paths = [os.path.realpath(os.path.curdir)]
  # Check if the last cli arg was a path, and look there as well
  if len(argv) > 2:
    app_path = argv[-1]
    if not app_path.startswith('-'):
      app_path = app_path.rstrip('...')
      paths += [os.path.realpath(app_path)]
  for path in paths:
    for app_yaml_path in GetAppYamlPaths(gopath, path):
      if os.path.exists(app_yaml_path):
        app_yaml = _ParseAppYaml(app_yaml_path) 
        if hasattr(app_yaml, 'api_version'):
          return os.path.join(SDK_BASE, goroots.GOROOTS[app_yaml.api_version])

  # Default to the goroot for go1
  return os.path.join(SDK_BASE, goroots.GOROOTS['go1'])


if __name__ == '__main__':
  vals, new_argv = GetArgsAndArgv()
  tool = os.path.basename(__file__)

  # Set a GOPATH if one is not set.
  if not os.environ.get('GOPATH'):
    os.environ['GOPATH'] = os.path.join(SDK_BASE, 'gopath')
  os.environ['APPENGINE_DEV_APPSERVER'] = vals['dev_appserver']

  # Find the correct goroot for the app and make sure we're running the given
  # tool from there.
  goroot = GetGoRoot(os.environ['GOPATH'], sys.argv)
  os.environ['GOROOT'] = goroot
  tool = os.path.join(goroot, 'bin', tool)

  # Remove env variables that may be incompatible with the SDK.
  for e in ('GOARCH', 'GOBIN', 'GOOS'):
    os.environ.pop(e, None)

  # Replace the path to this file with the path to the underlying tool
  new_argv[0] = tool
  if vals['print_command']:
    print(' '.join(new_argv))
  else:
    os.execve(tool, new_argv, os.environ)

if __name__ == '__main__':がメインです。

tool, new_argv, os.environらを組み立てて
最後のos.execveで実行しています。

os.execveは、現在のプロセスを置き換える形で新たなプログラムを実行します。
os.execve(path, args, env)

os.execveの前に登場する各変数をdumpしてみるとこんな値が入ってます

tool

/usr/local/Caskroom/google-cloud-sdk/latest/google-cloud-sdk/platform/google_appengine/goroot-1.8/bin/goapp

vals

{'dev_appserver': '/usr/local/Caskroom/google-cloud-sdk/latest/google-cloud-sdk/platform/google_appengine/dev_appserver.py','print_command': False}

new_argv
['/usr/local/Caskroom/google-cloud-sdk/latest/google-cloud-sdk/platform/google_appengine/goapp',
 'serve',
 'app.yaml']
os.environ
{
 'DIRENV_DIR': '-/Users/tweeeety/gitrepos/Hoge',
 'GOPATH': '/Users/tweeeety/gitrepos/Hoge',
 'PERL_CPANM_OPT': '--mirror http://cpan.metacpan.org',
 'APPENGINE_DEV_APPSERVER': '/usr/local/Caskroom/google-cloud-sdk/latest/google-cloud-sdk/platform/google_appengine/dev_appserver.py',
 'GOROOT': '/usr/local/Caskroom/google-cloud-sdk/latest/google-cloud-sdk/platform/google_appengine/goroot-1.8',
 'DIRENV_DIFF': 'hogehogehogehogehogehogehogehogehogehogehogehogehogehogehogehogehogehogehogehogehogehogehogehoge',
 'TERM_PROGRAM_VERSION': '326',
 'SHELL': '/bin/bash',
 'LOGNAME': 'tweeeety',
 'USER': 'tweeeety',
 'HOME': '/Users/tweeeety',
 'PATH': '/usr/local/opt/imagemagick@6/bin:/Users/tweeeety/.plenv/shims:/Users/tweeeety/.plenv/bin:/Users/tweeeety/.nodebrew/current/bin:/Users/tweeeety/.mysqlenv/mysqls/5.1.69/bin:/Users/tweeeety/.mysqlenv/bin:/Users/tweeeety/.mysqlenv/shims:/Users/tweeeety/.mysqlenv/mysql-build/bin:/usr/local/sbin:/Users/tweeeety/.mysqlenv/bin:/Users/tweeeety/.mysqlenv/shims:/Users/tweeeety/.mysqlenv/mysql-build/bin:/usr/local/heroku/bin:/Users/tweeeety/.chefdk/gem/ruby/2.1.0/bin:/opt/chefdk/bin:/Users/tweeeety/.rbenv/shims:/Users/tweeeety/.rbenv/shims:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/local/go/bin:/sbin:/usr/sbin:/Users/tweeeety/local/bin:/usr/local/opt/go/libexec/bin:/Users/tweeeety/.go/bin:/usr/local/Caskroom/google-cloud-sdk/latest/google-cloud-sdk/platform/google_appengine',
 'PS1': '[\\[\\033[40;1;32m\\]\\w \\[\\033[40;2;37m\\]\\t\\[\\033[0m\\]]$ ',
 'TERM_PROGRAM': 'Apple_Terminal',
 'LANG': 'ja_JP.UTF-8',
 'TERM': 'xterm-256color',
 'Apple_PubSub_Socket_Render': '/tmp/launch-95EcNN/Render',
 'SHLVL': '1',
 'TEST_MYSQL_SOCK': '/tmp/mysql.sock',
 'SECURITYSESSIONID': 'hoge',
 'RBENV_SHELL': 'bash',
 'EDITOR': 'vim',
 'TERM_SESSION_ID': 'hogehogehogehoge',
 'SSH_AUTH_SOCK': '/tmp/launch-Z5iV4t/Listeners',
 'MODULE': 'Hoge',
 'TEST_LOCAL_MYSQL_SOCK': '/tmp/mysql.sock',
 '_': '/usr/local/Caskroom/google-cloud-sdk/latest/google-cloud-sdk/platform/google_appengine/goapp',
 'TMPDIR': '/var/folders/5x/2wrjp7td5bj9cprs4c4cwxswqmbl26/T/',
 'PERL5LIB': '/Users/tweeeety/testApp/Sample/lib/:',
 'PLENV_SHELL': 'bash',
 'OLDPWD': '/Users/tweeeety/gitrepos/Fuga',
 '__CF_USER_TEXT_ENCODING': '0x2F45CC46:1:14',
 'PWD': '/Users/tweeeety/gitrepos/Hoge',
 'DIRENV_WATCHES': 'hogehogehogehogehogehogehogehogehogehogehogehogehogehogehogehogehogehogehogehogehogehogehogehoge',
 '__CHECKFIX1436934': '1'
}

おわり

なんとなくわかった気になりました＼(＾o＾)／