Friday, July 29, 2016

Solving Manning's equation with Python

Manning's equation is a very common formula used in hydraulic engineering. It is an empirical formula that estimates the average velocity of open channel flow, based on a roughness coefficient.



The problem of solving Manning's formula is that it is an implicit formula, for the water depth variable, that defines the R (Hydraulic Radius).

A common approach for solving this equation is using numerical methods, as the Newton-Raphson method. An implementation for the resolution of the Manning Formula for pipes, by the Newton-Raphson approximation, is shown below.
def manningTub(depth, args):
    Q = args[0]
    diamMM = args[1]
    nMann = args[2]
    slope = args[3]
    diamM = diamMM / 1000.0
    angle = 2*acos(1-2*depth/diamM)
    area = diamM*diamM/8.0*(angle-sin(angle))
    radius = diamM/4*(1-sin(angle)/angle)
    mannR = (Q*nMann/slope**0.5)-(area*radius**(2.0/3.0))
    return mannR

def solve(f, x0, h, args):
    lastX = x0
    nextX = lastX + 10* h  # "different than lastX so loop starts OK
    while (abs(lastX - nextX) > h):  # this is how you terminate the loop - note use of abs()
        newY = f(nextX,args)  # just for debug... see what happens
        print "f(", nextX, ") = ", newY     # print out progress... again just debug
        lastX = nextX
        nextX = lastX - newY / derivative(f, lastX, h, args)  # update estimate using N-R
    return nextX

def derivative(f, x, h, args):
      return (f(x+h,args) - f(x-h,args)) / (2.0*h)  # might want to return a small non-zero if ==0

#### EXAMPLE CALL

xFound = solve(manningTub, initDepth, 0.001, [flowRate,diam,nMann,enSlope])    # call the solver

Thursday, April 14, 2016

Time-Series in Python

Dealing with timeseries is a very common task in Hydrology.

One of the possibilities to process timeseries in python is to use a simple list.

For example, we can have a list of lists like this:

series1 = [ ['01/01/1900',0.0],['01/02/1900',0.1],['01/03/1900',0.3],['01/04/1900',0.4],['01/05/1900',2.2]...]

In this case, the ['01/01/1900',0.0] is composed of lists with a string representing the date, and a float number representing a value.

To properly make computations with dates, including sorting and grouping, it is necessary to interpret the string as a datetime format.

import datetime



for i in series1:

    i[0]=datetime.datetime.strptime(i[0], '%m/%d/%Y')


datetime objects accepts being sorted, making possible to sort the list based on the date, for example:

series1.sort(key=lambda x: x[0])

And we can make sums or averages based on specific months or years:

#eg. List of year 1900

lst1900 = [item for item in series1 if item[0].year==1900]



#Sum of 1900's values:

sum1900 = sum[item[1] for item in series1 if item[0].year==1900]



# avg of 1900's values

avg1900 = sum1900 / float(len(lst1900))










Thursday, March 31, 2016

Extracting GPS data from pictures

It is possible to extract EXIF data from pictures, including the latitude and longitude - if it is available - with a Python native library - PIL (Python Imaging Library) .

Below is an example code of how to extract this information.

The following code opens the image file with  PIL.Image.open , then read the info with ._getexif(), naming and extracting the proper information with the functions TAGS and GPSTAGS.

The following code also includes a simple function for converting the degres minutes seconds format to decimal degrees.



from PIL import Image
import PIL.Image
from PIL.ExifTags import TAGS, GPSTAGS

def get_exif_data(image):
    """Returns a dictionary from the exif data of an PIL Image item. Also converts the GPS Tags"""
    exif_data = {}
    info = image._getexif()
    if info:
        for tag, value in info.items():
            decoded = TAGS.get(tag, tag)
            if decoded == "GPSInfo":
                gps_data = {}
                for t in value:
                    sub_decoded = GPSTAGS.get(t, t)
                    gps_data[sub_decoded] = value[t]

                exif_data[decoded] = gps_data
            else:
                exif_data[decoded] = value

    return exif_data

def _get_if_exist(data, key):
    if key in data:
        return data[key]
  
    return None
 
def _convert_to_degress(value):
    """Helper function to convert the GPS coordinates stored in the EXIF to degress in float format"""
    d0 = value[0][0]
    d1 = value[0][1]
    d = float(d0) / float(d1)

    m0 = value[1][0]
    m1 = value[1][1]
    m = float(m0) / float(m1)

    s0 = value[2][0]
    s1 = value[2][1]
    s = float(s0) / float(s1)

    return d + (m / 60.0) + (s / 3600.0)

def get_lat_lon(exif_data):
    """Returns the latitude and longitude, if available, from the provided exif_data (obtained through get_exif_data above)"""
    lat = None
    lon = None

    if "GPSInfo" in exif_data:  
        gps_info = exif_data["GPSInfo"]

        gps_latitude = _get_if_exist(gps_info, "GPSLatitude")
        gps_latitude_ref = _get_if_exist(gps_info, 'GPSLatitudeRef')
        gps_longitude = _get_if_exist(gps_info, 'GPSLongitude')
        gps_longitude_ref = _get_if_exist(gps_info, 'GPSLongitudeRef')

        if gps_latitude and gps_latitude_ref and gps_longitude and gps_longitude_ref:
            lat = _convert_to_degress(gps_latitude)
            if gps_latitude_ref != "N":                     
                lat = 0 - lat

            lon = _convert_to_degress(gps_longitude)
            if gps_longitude_ref != "E":
                lon = 0 - lon

    return lat, lon

###EXAMPLE CODE
filename = '//path//to//image.jpg'
image = PIL.Image.open(filename)
exif_data = get_exif_data(image)
latlong = get_lat_lon(exif_data)

Wednesday, February 3, 2016

Good and Pythonic way to iterate over files in a folder



First, we get the path by some way, most of cases with an open folder dialog.

Then we make a list with all files in that folder.

This can be useful for batch analyze / read/ write files.


import os


path = 'c://temp'
onlyfiles = [ f for f in os.listdir(path) if os.path.isfile(os.path.join(path,f))]

Monday, December 14, 2015

Text string to Excel

Sometimes we just want to handle some data, from or to excel.

Here is a tip that can help in your scripts, when you want to copy data from or to an Excel spreadsheet.

Microsoft Excel recognizes the tabulation character ('\t') as cell separation, and a line break character ('\n') as a normal line break.

For example, if you copy some string data to the clipboard, as explained in previous post, you can paste it to excel cells, if the data is separated by '\t' character.

Sunday, December 6, 2015

Probability: How to do combinations without repetition


Command:
itertools.combinations(iterable, r)

Return r length subsequences of elements from the input iterable.

Combinations are emitted in lexicographic sort order. So, if the input iterable is sorted, the combination tuples will be produced in sorted order.

Elements are treated as unique based on their position, not on their value. So if the input elements are unique, there will be no repeat values in each combination.

Example:
from itertools import combinations

print list(combinations([1,2,3,4], 3))

itertools.combinations returns an iterator. An iterator is something that you can for on. So, that is why it is transformed into a list in the example.




Wednesday, December 2, 2015

SciPy minimize example - Fitting IDF Curves


SciPy (pronounced “Sigh Pie”) is an open source Python library used by scientists, analysts, and engineers doing scientific computing and technical computing.

SciPy contains modules for optimization, linear algebra, integration, interpolation, special functions, FFT, signal and image processing, ODE solvers and other tasks common in science and engineering.

In this post I will show how to use a powerful function of SciPy - minimize.

Minimize has some methods of minimizing functions. Its official documentation is shown here.

The example used is to find coefficients of a standard rainfall IDF (intensity-duration-frequency) equation. The format of the equation is as following:



We have to input the intensity data to be fitted, and the equation as a function.


import numpy as np
from scipy.optimize import minimize

# This is the IDF function, for return periods of 10 and 25 years
def func2(par, res):
    f10 =  (par[0] * 10  **par[1])/((res[0,:]+par[2])**par[3])
    f25 =  (par[0] * 25  **par[1])/((res[0,:]+par[2])**par[3])
    erroTotQ = np.sum((f10-res[1,:])**2+(f25-res[2,:])**2)
    return erroTotQ

#durations array - in minutes
d=[5,10,15,20,25,30,35,40,60,120,180,360,540,720,900,1260,1440]

# Rainfall intensities, mm/h, same lenght as minutes
r10=[236.1,174.0,145.6,128.3,116.3,107.3,100.3,94.5,79.1,48.4,34.2,18.9,13.4,10.5,8.7,6.5,5.8]
r25=[294.5,217.1,181.6,160.0,145.1,133.9,125.1,118.0,98.7,60.4,42.7,23.6,16.7,13.1,10.8,8.1,7.2]

# the following line only gather the duration and the intensities as a numpy array,
# in order to pass all of the idf constants as one single parameter, named "valid"
valid = np.vstack((d, r10, r25))

#initial guess
param1 = [5000, 0.1, 10, 0.9]

res2 = minimize(func2, param1, args=(valid,), method='Nelder-Mead')
np.set_printoptions(formatter={'float': '{: 0.3f}'.format})
print res2

Wednesday, November 25, 2015

Plotting Charts with PyQGIS

In this post, I will show an example of how to plot a simple line graph in PyQGIS, with matplotlib library. This library is already included in PyQGIS.

The example code is below.

import matplotlib.pyplot as plt



# x and y data as same length lists

radius = [1.0, 2.0, 3.0, 4.0, 5.0, 6.0]

area = [3.14159, 12.56636, 28.27431, 50.26544, 78.53975, 113.09724]



plt.plot(radius, area)

plt.xlabel('Radius')

plt.ylabel('Area')

plt.title('Area of a Circle')

# show grid lines

ax = plt.axes()

ax.grid(True)

# show plot window

plt.show()

Monday, November 23, 2015

Asking for User Input

In QGIS python (PyQGIS), the python comman raw_input() does not work as expected. And it will be not very practical to enter any input in Python Console.

One good alternative is to use "Qdialog". It is a module of PyQt (PyQt4.QtGui) library.
It can show a window asking for input, or asking to choose a file or folder.

We have to import it:

from PyQt4.QtGui import QFileDialog, QInputDialog, QMessageBox

QfileDialog - Window to select files or folders. It can be getOpenFileName - for opening files, getSaveFileName, for saving files, getExistingDirectory to select a folder. Example code for these types is shown below.

fileOpen = QFileDialog.getOpenFileName(None,'Select File:', 'c:/' , '.xls (*.xls)')

saveFileName = QtGui.QFileDialog.getSaveFileName(self, 'Dialog Title', '/path/to/default/directory', selectedFilter='*.txt')

folderSel = QFileDialog.getExistingDirectory(None,"Select Folder: ", 'c:/')


QInputDialog - customizable window. An example is shown below, showing a function that calls a customizable input window with defined width and height:

def windowSize(title, question, widthW, heightW):
    winD = QInputDialog(iface.mainWindow())
    winD.setInputMode(QInputDialog.TextInput)
    winD.setWindowTitle(title);
    winD.setLabelText(question)
    winD.resize(widthW,heightW)
    ok = winD.exec_()
    textInput = winD.textValue()
    return textInput

QMessageBox - It shows only a MessageBox With some information you want. Example:

QMessageBox.information(iface.mainWindow(), "Title", "Message")