AI Adventures in Azure: making use of blob i/o

As part of my AI for Earth software development, I am working hard on a system that simultaneously applies a supervised classification algorithm, and inverted radiative transfer model, an albedo calculation and an energy balance model pixel-wise over satellite imagery over the Greenland Ice Sheet. This is hugely computationally expensive and very memory hungry. I … Continue reading AI Adventures in Azure: making use of blob i/o

AI Adventures in Azure: hosting and hitting REST APIs using the Azure container registry

This week I had the pleasure of attending the AI for Earth summit at Microsoft HQ where the focus was on developing codes that run locally into APIs hosted on Azure that can be accessed from anywhere. I've got to admit I'm still developing my working knowledge of this process, but I am getting there, … Continue reading AI Adventures in Azure: hosting and hitting REST APIs using the Azure container registry

AI Adventures in Azure: Blob storage

My AI for Earth project is quite memory intensive so I have been learning about ways to take the data storage off the local disk and into the cloud, while still maintaining on the fly access to crucial files on my virtual or local machine. My classification problem started off requiring just a few GB … Continue reading AI Adventures in Azure: Blob storage

AI Adventures in Azure: Ways to Program in Python on the DSVM

Having introduced the set up and configuration of a new virtual machine and the ways to interact with it, I will now show some ways to use it to start programming in Python. This post will assume that the VM is allocated and that the user is accessing the VM using a remote desktop client. … Continue reading AI Adventures in Azure: Ways to Program in Python on the DSVM

AI Adventures in Azure: Uploading data to the VM

There are many ways to transfer data from local storage to the virtual machine. Azure provides Blob storage for unstructured data managed through the user's storage account as well as specific storage options for files and tables. There is also the option to use Data Lakes. These are all useful for storing large datasets and … Continue reading AI Adventures in Azure: Uploading data to the VM

AI Adventures in Azure: Choosing VM Size

The main purpose of a VM is to accelerate scripts compared to running locally on a laptop or desktop by outsourcing the computation to a more powerful remote computer. There is an overwhelming number of options for Azure VM sizes, each of which is optimised for a particular purpose, so to get the best performance … Continue reading AI Adventures in Azure: Choosing VM Size

AI Adventures in Azure: Accessing the VM via terminal or remote desktop

Accessing the Data Science Virtual Machine Once the virtual machine is set up and started (by clicking “start” on the appropriate VM in the Azure portal) there are several ways to interface with it. The first is via the terminal (I am running Ubuntu 16.04 on both my local machine and the virtual machine). To … Continue reading AI Adventures in Azure: Accessing the VM via terminal or remote desktop

AI Adventures in Azure

A lot of my work at the moment requires quite computationally heavy geospatial analysis that stretches the processing capabilities of my laptop. I invested in a pretty powerful machine – i7-7700GHz processor, 32GB RAM – and sped things up by spreading the load across cores and threads, but it can still be locked up for … Continue reading AI Adventures in Azure