Run some cool GitHubs on Azure (Python)

  1. Azure sample notebooks (run on Azure)
  2. Azure Python for DS 101 (run on Azure)
  3. Azure cntk  (run on Azure)
  4. Azure Python Data Science Handbook (run on Azure)
  5. https://github.com/TrackDR/ModdedIpythonNotebooks (run on Azure)
  6. https://github.com/JWarmenhoven/ISLR-python (run on Azure)
  7. https://github.com/jdwittenauer/ipython-notebooks (run on Azure)
  8. https://github.com/masinoa/machine_learning (run on Azure)
  9. https://github.com/jakevdp/sklearn_tutorial (run on Azure)
  10. https://github.com/scikit-image/skimage-tutorials (run on Azure)
  11. https://github.com/spatialaudio/digital-signal-processing-lecture (run on Azure)
    1. docs
  12. https://github.com/AllenDowney/ThinkDSP (run on Azure)
  13. https://github.com/AllenDowney/ThinkBayes2 (run on Azure)
  14. https://github.com/AllenDowney/ThinkStats2 (run on Azure)
  15. https://github.com/AllenDowney/ThinkComplexity2 (run on Azure)
  16. https://github.com/AllenDowney/ModSimPy (run on Azure)
  17. https://github.com/AllenDowney/CompStats (run on Azure)
  18. https://github.com/multivac61/wave_digital_notebook (run on Azure)
  19. https://github.com/learnml/machine-learning-specialization (run on Azure)
  20. https://github.com/ipython/ipython-in-depth (run on Azure)
  21. https://github.com/yhilpisch/dx (run on Azure)
  22. https://github.com/yhilpisch/py4fi (run on Azure)
  23. https://github.com/quantopian/research_public (run on Azure)
  24. https://github.com/darribas/gds_scipy16 (run on Azure)
  25. https://github.com/jupyter/jupyter/wiki/A-gallery-of-interesting-Jupyter-Notebooks#introductory-tutorials
  26. Open CV notebooks on Azure
  27. Geospatial notebooks on Azure
  28. Python Lectures on Azure

Good for Python/IPython/Jupyter Beginners

  • Sample notebooks (run on Azure)
    • Introduction to Python.ipynb
  • Python-Lectures (run on Azure)
    • 01.ipynb
  • https://github.com/jdwittenauer/ipython-notebooks (run on Azure)
    1. notebooks/ipython-notebooks/notebooks/language/Intro.ipynb to start
  • Python for DS 101 (run on Azure)
    • 01.Basics.ipynb

Neat starting examples

  • Images
    • https://github.com/scikit-image/skimage-tutorials (run on Azure)
      1. skimage-tutorials/lectures/00_images_are_arrays.ipynb
    • https://github.com/TrackDR/ModdedIpythonNotebooks (run on Azure)
      • ImageStuff.ipynb
  • Finance
    • https://github.com/yhilpisch/py4fi (run on Azure)
      • ipython3/03_Introductory_Examples.ipynb
  • DSP
    • https://github.com/AllenDowney/ThinkDSP (run on Azure)
      • code/chap01.ipynb
    • https://github.com/spatialaudio/digital-signal-processing-lecture (run on Azure)
      • index.ipynb
  • Gaussian Mixture Models
    • Python Data Science Handbook (run on Azure)
      • notebooks/05.12-Gaussian-Mixtures.ipynb
    • https://github.com/jakevdp/sklearn_tutorial (run on Azure)
      • notebooks/04.3-Density-GMM.ipynb

Docker for AWS

Launch Stack on AWS (Costs Money!)

  1. Picked 1 Manager Node (t2.micro), Picked 1 Worker Node (t2.medium or t2.large)
    1. Or 1 Manager Node(t2.medium or t2.large), 0 Worker Nodes(doesn’t matter)
  2. Picked one of my AWS security keys
  3. Launch using Create Button (Costs Money!)
  4. Services->EC2->Docker-Manager to get Public DNS
    1. Or CloudFormation->Stacks->Docker->Outputs->Managers
      1. Click on link under “Managers->Value” and see Public DNS of Manager Node
  5. Ssh to manager node using AWS security key and docker as username
    1. Either use “Connect” in EC2 or
    2. ssh -i “mykey.pem” docker@<Public DNS>
      1. Make sure mykey.pem has correct permissions
        1. chmod 400 mykey.pem
    3. or convert .pem to .ppk and use putty
  6. Launch a docker container (choose one for now)
    1. docker service create --name dsnb -p 8888:8888 jupyter/datascience-notebook
      1. github link, dockerhub link
      2. took about 20 minutes to start up 🙁
  7. docker node ls
  8. docker node inspect <node-name>
  9. docker service ls
  10. docker service inspect <node-service-name>
    1. docker service inspect dsnb
  11. docker service ps <node-service-name>
    1. docker service ps dsnb
  12. containerid=`docker ps | grep jupyter | cut -d' ' -f1`
  13. docker exec -it $containerid bash
  14. Connect to port on Manager to access services on worker nodes
    1. CloudFormation->Stacks->Docker->Outputs->DefaultDNSTarget for web address
    2. jupyter notebook list
    3. Open <DefaultDNSTarget>:<port>?token=<token> in web browser
      1. Use 8888 for <port> in this example
  15. Make New Python 2 Notebook and then:
    1. !git clone https://github.com/TrackDR/Geospatial-Jupyter-Notebooks
    2. !git clone https://github.com/rajathkumarmp/Python-Lectures
      1. Python-Lectures/01.ipynb to start, rest are good too
    3. Open notebook out of any cloned directory
  16. CloudFormation->Stacks->Docker->Other Actions->Delete Stack when done using

Docs

  1. Deploy your app

Other Jupyter/Ipython Github Repositories to clone

  1. !git clone https://github.com/TrackDR/ModdedIpythonNotebooks
  2. !git clone https://github.com/JWarmenhoven/ISLR-python
  3. !git clone https://github.com/jdwittenauer/ipython-notebooks
    1. notebooks/ipython-notebooks/notebooks/language/Intro.ipynb to start
  4. !git clone https://github.com/masinoa/machine_learning
  5. !git clone https://github.com/jakevdp/sklearn_tutorial
  6. !git clone https://github.com/scikit-image/skimage-tutorials
    1. skimage-tutorials/lectures/00_images_are_arrays.ipynb
  7. !git clone https://github.com/spatialaudio/digital-signal-processing-lecture
    1. docs
  8. !git clone https://github.com/AllenDowney/ThinkDSP
  9. !git clone https://github.com/AllenDowney/ThinkBayes2
  10. !git clone https://github.com/AllenDowney/ThinkStats2
  11. !git clone https://github.com/AllenDowney/ThinkComplexity2
  12. !git clone https://github.com/AllenDowney/ModSimPy
  13. !git clone https://github.com/AllenDowney/CompStats
  14. !git clone https://github.com/multivac61/wave_digital_notebook
  15. !git clone https://github.com/learnml/machine-learning-specialization
  16. !git clone https://github.com/ipython/ipython-in-depth
  17. !git clone https://github.com/yhilpisch/dx
  18. !git clone https://github.com/yhilpisch/py4fi
  19. !git clone https://github.com/quantopian/research_public
  20. !git clone https://github.com/darribas/gds_scipy16
  21. https://github.com/jupyter/jupyter/wiki/A-gallery-of-interesting-Jupyter-Notebooks#introductory-tutorials

Docker images to run

  1. Jupyter
    1. jupyter/datascience-notebook

Docker images that will need to modify to run here, but looking to the future

  1. Kaggle
    1. kaggle/python
    2. kaggle/rstats
  2. Quantlib
    1. docker service create –name quantlib -p 8888:8888 lballabio/quantlib-notebook
    2. Videos at vimeo, blog w/ex, book
  3. QSTK
    1. ipython docker image
    2. ontouchstart docker image
    3. Twisted logic docker image
  4. BitQuant docker image
  5. Needs Nvidia-docker so may not work here
    1. Torch/iTorch/Ubuntu 14.04 Docker image: docker pull kaixhin/torch
    2. Torch/iTorch/CUDA 7/Ubuntu 14.04 Docker image: docker pull kaixhin/cuda-torch
  6. OSGeo list of Docker images
  7. Docker container for QGIS models (github, dockerhub)
    • can use with OSGeoLive
  8. My Dockerfiles at GitHub

Alternatives to Docker for AWS for running Jupyter Notebooks

  1. CoCalc (subscription based)
    1. Website, Pricing, Wikipedia, Chrome Extension, GitHub
    2. Or Run your own CoCalc using their Docker image
  2. Python Anywhere (website, blog, help, forums, pricing, modulestwitter, edu, edu help)
  3. Anaconda Enterprise Notebooks (Pricey! Website)