{"componentChunkName":"component---src-templates-blog-post-js","path":"/2021-12-16-python-data-inspect-note1/","result":{"data":{"site":{"siteMetadata":{"title":"Howard's Note","disqusShortname":""}},"markdownRemark":{"id":"f8c1759e-0eb7-5ca1-927b-2b12ec5d377b","html":"

Just some useful toolkit note, including api, useful tip, etc.

\n

request data

\n

There are two most use library in python world, acording to Google search: request and http.client. request seems a easiest and quickest way to fetch web content and api.

\n

but http.client seems to be more oo way. so I just try to use http.client to fetch data instead.

\n
import http.client\nimport json\nurl = 'www.twse.com.tw'\nroute = '/v1/exchangeReport/MI_INDEX'\n\n# params just the parameter u want to use in GET request\nparams = urllib.parse.urlencode({\n    'date': date,\n    'response': 'json' \n    '_': str(round(time.time() * 1000) - 500)\n})\n\n# Note that don't need to add protocal (http or https) in url.\n# instend, just choose to use HTTPSConnection or HttpConnection to make a request.\n# also note that port is needed to be selected.\nclient = http.client.HTTPSConnection(url, 443, timeout=10) \nclient.request('GET', f'{self._route}?{params}')\nwith client.getresponse() as res: \n    # use json to loads the response and parse to dict.\n    data = json.loads(res.read()) \n    return data
\n

panda

\n
import panda as pd
\n\n
def load_to_df(filename, df):\n    try:\n        with open(filename, 'r') as f:\n            j = json.loads(f.read())\n            # concat\n            date = filename.split('.')[0]\n            print(f'load date: {date}')\n            temp = pd.DataFrame(columns=j['fields8'], data=j['data8'], index=[f'{date}({i+1})' for i, _ in enumerate(j['data8'])])\n            df = pd.concat([df, temp])\n    except Exception as e:\n        print('[ERROR] error happen in file: ' + filename)\n        print(f'[ERROR] msg: {e}' )\n        print('')\n    return df
\n

pd.concat will create a new DataFframe, that is good, in my opinion.

\n

note that use for comprehension to create index, which is a elegent way to create, and show purpose.

\n\n
def get_overall_by_date(date, df):\n    return df.loc[[date + f'({i+1})' for i in range(3)]]
\n

Data visualization.

\n

Choose to use matplotlib, which seems a easiest solution.

\n
import matplotlib.pyplot as plt
","frontmatter":{"title":"data inspect note","date":"December 16, 2021","description":null,"tags":["python","note"],"template":"post"}}},"pageContext":{"slug":"/2021-12-16-python-data-inspect-note1/","previous":{"node":{"fields":{"slug":"/2021-12-05-python-notebook-nbextensions-issue/"},"frontmatter":{"date":"05 December, 2021","title":"jupyter notebook nbextensiions 404 issue.","tags":["python"]}}},"next":{"node":{"fields":{"slug":"/2022-01-30/"},"frontmatter":{"date":"30 January, 2022","title":"Cljure FAQ","tags":["clojure","note"]}}}}},"staticQueryHashes":["2841359383","3810545343","916993862"]}