data inspect note

December 16, 2021

Just some useful toolkit note, including api, useful tip, etc.

request data

There are two most use library in python world, acording to Google search: request and http.client. request seems a easiest and quickest way to fetch web content and api.

but http.client seems to be more oo way. so I just try to use http.client to fetch data instead.

import http.client
import json
url = 'www.twse.com.tw'
route = '/v1/exchangeReport/MI_INDEX'

# params just the parameter u want to use in GET request
params = urllib.parse.urlencode({
    'date': date,
    'response': 'json' 
    '_': str(round(time.time() * 1000) - 500)
})

# Note that don't need to add protocal (http or https) in url.
# instend, just choose to use HTTPSConnection or HttpConnection to make a request.
# also note that port is needed to be selected.
client = http.client.HTTPSConnection(url, 443, timeout=10) 
client.request('GET', f'{self._route}?{params}')
with client.getresponse() as res: 
    # use json to loads the response and parse to dict.
    data = json.loads(res.read()) 
    return data

panda

import panda as pd

load json to data farme:

use Dataframe. in my case, I use json.loads + Dataframe + pd.concat to achieve goal.

def load_to_df(filename, df):
    try:
        with open(filename, 'r') as f:
            j = json.loads(f.read())
            # concat
            date = filename.split('.')[0]
            print(f'load date: {date}')
            temp = pd.DataFrame(columns=j['fields8'], data=j['data8'], index=[f'{date}({i+1})' for i, _ in enumerate(j['data8'])])
            df = pd.concat([df, temp])
    except Exception as e:
        print('[ERROR] error happen in file: ' + filename)
        print(f'[ERROR] msg: {e}' )
        print('')
    return df

pd.concat will create a new DataFframe, that is good, in my opinion.

note that use for comprehension to create index, which is a elegent way to create, and show purpose.

get data row from DataFrame by index.

Just find two api that may be useful: df.loc() and df.ioc(). This time I just use loc() to acheive my goal.

ref from: here

def get_overall_by_date(date, df):
    return df.loc[[date + f'({i+1})' for i in range(3)]]

Data visualization.

Choose to use matplotlib, which seems a easiest solution.

import matplotlib.pyplot as plt