Docker for AWS

Launch Stack on AWS (Costs Money!)

  1. Picked 1 Manager Node (t2.micro), Picked 1 Worker Node (t2.medium or t2.large)
    1. Or 1 Manager Node(t2.medium or t2.large), 0 Worker Nodes(doesn’t matter)
  2. Picked one of my AWS security keys
  3. Launch using Create Button (Costs Money!)
  4. Services->EC2->Docker-Manager to get Public DNS
    1. Or CloudFormation->Stacks->Docker->Outputs->Managers
      1. Click on link under “Managers->Value” and see Public DNS of Manager Node
  5. Ssh to manager node using AWS security key and docker as username
    1. Either use “Connect” in EC2 or
    2. ssh -i “mykey.pem” docker@<Public DNS>
      1. Make sure mykey.pem has correct permissions
        1. chmod 400 mykey.pem
    3. or convert .pem to .ppk and use putty
  6. Launch a docker container (choose one for now)
    1. docker service create --name dsnb -p 8888:8888 jupyter/datascience-notebook
      1. github link, dockerhub link
      2. took about 20 minutes to start up 🙁
  7. docker node ls
  8. docker node inspect <node-name>
  9. docker service ls
  10. docker service inspect <node-service-name>
    1. docker service inspect dsnb
  11. docker service ps <node-service-name>
    1. docker service ps dsnb
  12. containerid=`docker ps | grep jupyter | cut -d' ' -f1`
  13. docker exec -it $containerid bash
  14. Connect to port on Manager to access services on worker nodes
    1. CloudFormation->Stacks->Docker->Outputs->DefaultDNSTarget for web address
    2. jupyter notebook list
    3. Open <DefaultDNSTarget>:<port>?token=<token> in web browser
      1. Use 8888 for <port> in this example
  15. Make New Python 2 Notebook and then:
    1. !git clone https://github.com/TrackDR/Geospatial-Jupyter-Notebooks
    2. !git clone https://github.com/rajathkumarmp/Python-Lectures
      1. Python-Lectures/01.ipynb to start, rest are good too
    3. Open notebook out of any cloned directory
  16. CloudFormation->Stacks->Docker->Other Actions->Delete Stack when done using

Docs

  1. Deploy your app

Other Jupyter/Ipython Github Repositories to clone

  1. !git clone https://github.com/TrackDR/ModdedIpythonNotebooks
  2. !git clone https://github.com/JWarmenhoven/ISLR-python
  3. !git clone https://github.com/jdwittenauer/ipython-notebooks
    1. notebooks/ipython-notebooks/notebooks/language/Intro.ipynb to start
  4. !git clone https://github.com/masinoa/machine_learning
  5. !git clone https://github.com/jakevdp/sklearn_tutorial
  6. !git clone https://github.com/scikit-image/skimage-tutorials
    1. skimage-tutorials/lectures/00_images_are_arrays.ipynb
  7. !git clone https://github.com/spatialaudio/digital-signal-processing-lecture
    1. docs
  8. !git clone https://github.com/AllenDowney/ThinkDSP
  9. !git clone https://github.com/AllenDowney/ThinkBayes2
  10. !git clone https://github.com/AllenDowney/ThinkStats2
  11. !git clone https://github.com/AllenDowney/ThinkComplexity2
  12. !git clone https://github.com/AllenDowney/ModSimPy
  13. !git clone https://github.com/AllenDowney/CompStats
  14. !git clone https://github.com/multivac61/wave_digital_notebook
  15. !git clone https://github.com/learnml/machine-learning-specialization
  16. !git clone https://github.com/ipython/ipython-in-depth
  17. !git clone https://github.com/yhilpisch/dx
  18. !git clone https://github.com/yhilpisch/py4fi
  19. !git clone https://github.com/quantopian/research_public
  20. !git clone https://github.com/darribas/gds_scipy16
  21. https://github.com/jupyter/jupyter/wiki/A-gallery-of-interesting-Jupyter-Notebooks#introductory-tutorials

Docker images to run

  1. Jupyter
    1. jupyter/datascience-notebook

Docker images that will need to modify to run here, but looking to the future

  1. Kaggle
    1. kaggle/python
    2. kaggle/rstats
  2. Quantlib
    1. docker service create –name quantlib -p 8888:8888 lballabio/quantlib-notebook
    2. Videos at vimeo, blog w/ex, book
  3. QSTK
    1. ipython docker image
    2. ontouchstart docker image
    3. Twisted logic docker image
  4. BitQuant docker image
  5. Needs Nvidia-docker so may not work here
    1. Torch/iTorch/Ubuntu 14.04 Docker image: docker pull kaixhin/torch
    2. Torch/iTorch/CUDA 7/Ubuntu 14.04 Docker image: docker pull kaixhin/cuda-torch
  6. OSGeo list of Docker images
  7. Docker container for QGIS models (github, dockerhub)
    • can use with OSGeoLive
  8. My Dockerfiles at GitHub

Alternatives to Docker for AWS for running Jupyter Notebooks

  1. CoCalc (subscription based)
    1. Website, Pricing, Wikipedia, Chrome Extension, GitHub
    2. Or Run your own CoCalc using their Docker image
  2. Python Anywhere (website, blog, help, forums, pricing, modulestwitter, edu, edu help)
  3. Anaconda Enterprise Notebooks (Pricey! Website)

My Video Game Playlists in Japanese for Immersion

I have taken my own advice and started playing a few games in Japanese to try to learn the language better.  I record them and then upload to YouTube.  Here are a few of my current playlists.

Labyrinth of Refrain

Shining Resonance Refrain

Ys VIII

Bayonetta

Atelier Lydie and Suelle

世界樹の迷宮X – Etrian Odyssey X

魔界戦記ディスガイア Refine

God Wars: The Complete Legend

The Lost Child

ゼルダ無双 ハイラルオールスターズ DX – Hyrule Warriors Definitive Edition – Nintendo Switch

Snack World Trejarers Gold – Nintendo Switch

アライアンス・アライブ Alliance Alive – Nintendo 3DS

Radiant Historia: Perfect Chronology

Deep Strange Journey 3DS – Japanese Version

Dragon Questドラゴンクエスト (3DS Virtual Console)

Pocket Monsters: Silver Version

Yokai Watch Busters 2: Treasure Legend Banbaraya Sword

Etrian Mystery Dungeon 2

モンスターハンター ストーリーズ (Monster Hunter Stories)

Xenoblade Chronicles 2

Fire Emblem Warriors

One Piece Pirate Warriors 3

One Piece Unlimited World Red Deluxe Edition

Monster Hunter XX

Duolingo Japanese

Duolingo Hiragana 2

Duolingo Hiragana 3

I am Setsuna (Nintendo Switch) Jap/Eng

Disgaea 5 Complete (Nintendo Switch) Jap/Eng

Minna de WaiWai! Spelunker

Corpse Party 3DS Jap/Eng

Puyo Pop Fever DS

PuyoPuyo Tetris Japanese Demo

マリオ&ルイージRPG1 DX #1 (Mario & Luigi: Super Star Saga + Bowser’s Minions)

Mindsnacks Japanese

Pokémon Ultra Moon

Geospatial Python

Libraries

Ecosystems

Docker

Examples

Courses

  • GeoPython – AutoGIS 2016 at University of Helsinki (webpage)
  • Geographic Data Science with PySAL and the pydata stack Beginner | SciPy 2016 | Sergio Rey (youtube, github, page)