Python - argparse module part1)

In [5]:

from IPython.core.display import display, HTML

display(HTML("<style>.container { width:140% !important; }</style>"))

View Source

Python argparse Part1)¶

usuage of argparse python module

- Attempt1. Handling argparse module on jupyter notebook 
- Attempt2. Updating parse argument from config.json
- Goal. Handling model parameters using both "argparse" and "config.json" 

Attempt 1. Handling argparse module on jupyter notebook¶

In [1]:

import argparse
import json
import os
from pprint import pprint

In [2]:

# Instantiate Parser
parser = argparse.ArgumentParser()

In [3]:

# Adding Parser Arguments

parser.add_argument('--seed', type=int, default=42, help='random seed (default: 42)')
parser.add_argument('--epochs', type=int, default=1, help='number of epochs to train (default: 1)')
parser.add_argument('--dataset', type=str, default='MaskBaseDataset', help='dataset augmentation type (default: MaskBaseDataset)')
parser.add_argument('--augmentation', type=str, default='BaseAugmentation', help='data augmentation type (default: BaseAugmentation)')
parser.add_argument("--resize", nargs="+", type=int,default=[128, 96], help='resize size for image when training')
parser.add_argument('--batch_size', type=int, default=64, help='input batch size for training (default: 64)')
parser.add_argument('--valid_batch_size', type=int, default=1000, help='input batch size for validing (default: 1000)')
parser.add_argument('--model', type=str, default='BaseModel', help='model type (default: BaseModel)')
parser.add_argument('--optimizer', type=str, default='SGD', help='optimizer type (default: SGD)')
parser.add_argument('--lr', type=float, default=1e-3, help='learning rate (default: 1e-3)')
parser.add_argument('--val_ratio', type=float, default=0.2, help='ratio for validaton (default: 0.2)')
parser.add_argument('--criterion', type=str, default='cross_entropy', help='criterion type (default: cross_entropy)')
parser.add_argument('--lr_decay_step', type=int, default=20, help='learning rate scheduler deacy step (default: 20)')
parser.add_argument('--log_interval', type=int, default=20, help='how many batches to wait before logging training status')
parser.add_argument('--name', default='exp', help='model save at {SM_MODEL_DIR}/{name}')
parser.add_argument('--config', type=str, default = './model_config_basic.json')
parser.add_argument('--data_dir', type=str, default='/opt/ml/input/data/train/images')
parser.add_argument('--model_dir', type=str, default='./model')

Out[3]:

_StoreAction(option_strings=['--model_dir'], dest='model_dir', nargs=None, const=None, default='./model', type=<class 'str'>, choices=None, help=None, metavar=None)

Terminal Case¶

Executing parser on Jupyter gives error

args = parser.parse_args()

In [4]:

args = parser.parse_args()

usage: ipykernel_launcher.py [-h] [--seed SEED] [--epochs EPOCHS]
                             [--dataset DATASET] [--augmentation AUGMENTATION]
                             [--resize RESIZE [RESIZE ...]]
                             [--batch_size BATCH_SIZE]
                             [--valid_batch_size VALID_BATCH_SIZE]
                             [--model MODEL] [--optimizer OPTIMIZER] [--lr LR]
                             [--val_ratio VAL_RATIO] [--criterion CRITERION]
                             [--lr_decay_step LR_DECAY_STEP]
                             [--log_interval LOG_INTERVAL] [--name NAME]
                             [--config CONFIG] [--data_dir DATA_DIR]
                             [--model_dir MODEL_DIR]
ipykernel_launcher.py: error: unrecognized arguments: -f /Users/philhoonoh/Library/Jupyter/runtime/kernel-067c8d6c-fd04-49eb-88e6-3614f02927d8.json

An exception has occurred, use %tb to see the full traceback.

SystemExit: 2

/Users/philhoonoh/opt/anaconda3/envs/py38_transformers/lib/python3.8/site-packages/IPython/core/interactiveshell.py:3452: UserWarning: To exit: use 'exit', 'quit', or Ctrl-D.
  warn("To exit: use 'exit', 'quit', or Ctrl-D.", stacklevel=1)

Jupyter Case¶

Insert [] or "" as an argument to parser.parse_args()

args = parser.parse_args("")
args = parser.parse_args([])

In [4]:

args = parser.parse_args("")
pprint(vars(args))

{'augmentation': 'BaseAugmentation',
 'batch_size': 64,
 'config': './model_config_basic.json',
 'criterion': 'cross_entropy',
 'data_dir': '/opt/ml/input/data/train/images',
 'dataset': 'MaskBaseDataset',
 'epochs': 1,
 'log_interval': 20,
 'lr': 0.001,
 'lr_decay_step': 20,
 'model': 'BaseModel',
 'model_dir': './model',
 'name': 'exp',
 'optimizer': 'SGD',
 'resize': [128, 96],
 'seed': 42,
 'val_ratio': 0.2,
 'valid_batch_size': 1000}

Attempt 2. Updating parse argument from config.json¶

model_config_basic.json¶

{
    "seed": 42,
    "epochs": 50,
    "dataset": "MaskSplitByProfileDataset",
    "augmentation": "BasicAugmentation2",
    "resize": [224,224],
    "batch_size": 64,
    "valid_batch_size": 128,
    "model": "EfficientNet",
    "optimizer": "Adam",
    "lr": 1e-4,
    "criterion": "cross_entropy",
    "name": "exp",
    "lr_decay_step": 20,
    "log_interval": 20,
    "data_dir": "/opt/ml/input/data/train/images",
    "model_dir": "./model"
}

In [ ]:

# reading model_config_basic.json file

def read_json(file):
    with open(file) as json_file:
        data = json.load(json_file)
    return data

config = read_json(args.config)
pprint(config)

In [ ]:

# updating args object with config

def update_argument(args, configs):
    for arg in configs:
        if arg in args:
            setattr(args, arg, configs[arg])
        else:
            raise ValueError(f"no argument {arg}")
    return args

args = update_argument(args, config)
pprint(vars(args))

Conclusion¶

• We can now using parser arguments and modifying model parameters using config.json file

Need more attention¶

• argparse versus config.json

  • Here config.json precedes over parser argument, meaning config value having same key with parser overwrites default value

  • However making config.json as a default value and overwriting default value from the command line using argparse is more common practice!

  • How can we acheive that? -> check out argparse Part2)