Compare commits
1 Commits
50c9b3f41d
...
horner_sch
| Author | SHA1 | Date | |
|---|---|---|---|
|
495b87e70e |
1
.gitignore
vendored
1
.gitignore
vendored
@@ -1,4 +1,3 @@
|
||||
*.png
|
||||
*.pkl
|
||||
*.pt
|
||||
*.zip
|
||||
|
||||
69
README.md
69
README.md
@@ -1,22 +1,10 @@
|
||||
# InteractionImitation
|
||||
Imitation Learning with the [Interaction Dataset](https://interaction-dataset.com/) via the [InteractionSimulator](https://github.com/sisl/InteractionSimulator) gym environments.
|
||||
|
||||
Code for "[SHAIL: Safety-Aware Hierarchical Adversarial Imitation Learning for Autonomous Driving in Urban Environments](https://arxiv.org/abs/2204.01922)", which appeared at the 2023 International Conference on Robotics and Automation (ICRA).
|
||||
If you find this repository useful, please cite the paper:
|
||||
|
||||
```
|
||||
@article{jamgochian2022shail,
|
||||
author = {Arec Jamgochian and Etienne Buehrle and Johannes Fischer and Mykel J. Kochenderfer},
|
||||
title = {{SHAIL}: Safety-Aware Hierarchical Adversarial Imitation Learning for Autonomous Driving in Urban Environments},
|
||||
journal = {arXiv:2204.01922 [cs]},
|
||||
year = {2022}
|
||||
}
|
||||
```
|
||||
Imitation Learning with the INTERACTION Dataset
|
||||
|
||||
## Getting started
|
||||
Clone the `InteractionSimulator` with the `shail` tag and pip install the module.
|
||||
Clone InteractionSimulator and pip install the module.
|
||||
```
|
||||
git clone --branch shail https://github.com/sisl/InteractionSimulator.git
|
||||
git clone https://github.com/sisl/InteractionSimulator.git
|
||||
cd InteractionSimulator
|
||||
pip install -e .
|
||||
cd ..
|
||||
@@ -31,28 +19,53 @@ The INTERACTION dataset contains a two folders which should be copied into a fol
|
||||
- the contents of `recorded_trackfiles` should be copied to `./InteractionSimulator/datasets/trackfiles`
|
||||
- the contents of `maps` should be copied to `./InteractionSimulator/datasets/maps`
|
||||
|
||||
## Processing and saving expert demos
|
||||
Once the repository has been set up, you need to generate two separate sets of expert demos for tracks 0-4. The first command generates true joint and individual states and actions necessary for evaluating, saving them in `expert_data/`. The second command generates trajectory rollouts according to individual agent observations, which is later used as expert data for the learning models.
|
||||
## Processing, saving, and loading expert demos
|
||||
Once the repository has been set up, you can process and save expert track demonstrations with:
|
||||
```
|
||||
python -m src.expert --locs='[DR_USA_Roundabout_FT]' --tracks='[0,1,2,3,4]'
|
||||
python -m intersimple-expert-rollout-setobs2 --tracks='[0,1,2,3,4]'
|
||||
python src/expert_data.py --loc [LOCNUM] --track [TRACKNUM]
|
||||
```
|
||||
You can (and should) process all tracks at once at location 0 with:
|
||||
```
|
||||
python src/expert_data.py --all-tracks
|
||||
```
|
||||
|
||||
You can then train a default behavior cloning policy with the following. Be sure to check help for main.py for running options.
|
||||
```
|
||||
python src/main.py --train
|
||||
```
|
||||
You can run tensorboard by running the following and opening `localhost:6006` (or alternatively port-forwarding 6006 from the remote server)
|
||||
```
|
||||
tensorboard --logdir output/
|
||||
```
|
||||
You can then test the learned policy with the following, and see the animation file in `output/`:
|
||||
```
|
||||
python src/main.py --test
|
||||
```
|
||||
|
||||
|
||||
## Tuning hyperparameters and training finalized models
|
||||
To tune models, we use `ray[tune]` grid searches. You can run see the commands we used to train in the top half of `train_models.sh`, as well as the hyperparameters we search over in `bc-experiment.py`, `gail-experiment.py`, and `shail-experiment.py`. After training the models, configurations get saved in `best_configs/` (the best SHAIL confg gets copied to a HAIL config, with the appropriate environment parameters changed for ablation). However, upon manual inspection of the training runs, we note some better performance than the automatically-set configs at earlier epochs, so we adjust the `best_configs` manually.
|
||||
|
||||
After the `best_configs/` are set, we rerun each configuration with multiple seeds. The commands to do so are in the bottom half of `train_models.sh`. This saves different learned policy files to `test_policies/`.
|
||||
|
||||
|
||||
## Evaluating models
|
||||
To evaluate the learned policies, we rerun each model in particular setting, evaluate all our metrics, and average over different trained model seeds. The commands to do so are in `evaluate_models.sh`.
|
||||
You can load the experts actions manually
|
||||
```
|
||||
from src import expert_data
|
||||
observations, actions = expert_data.load_expert_data(loc = [LOCNUM], track = [TRACKNUM])
|
||||
for (s, a) in zip (observations, actions):
|
||||
# do some imitation learning
|
||||
```
|
||||
|
||||
|
||||
## Package Structure
|
||||
```
|
||||
InteractionImitation
|
||||
|- TODO
|
||||
|- demos
|
||||
|- algorithms
|
||||
|- BC
|
||||
|- AdVIL
|
||||
|- nets
|
||||
|- Encoder
|
||||
|- DeepSet
|
||||
|- Decoder
|
||||
|- policies
|
||||
|- discriminators
|
||||
|- demo_generators
|
||||
```
|
||||
|
||||
## Type Definitions
|
||||
|
||||
218
bc-experiment.py
218
bc-experiment.py
@@ -1,218 +0,0 @@
|
||||
# %%
|
||||
import os
|
||||
|
||||
from tqdm import tqdm
|
||||
from src.core.sampling import rollout
|
||||
from src.core.gail import gail_ppo, Buffer
|
||||
from src.core.value import SetValue
|
||||
from src.core.policy import SetPolicy
|
||||
from src.core.discriminator import DeepsetDiscriminator
|
||||
import torch
|
||||
|
||||
from intersim.envs import IntersimpleLidarFlatRandom
|
||||
from intersim.envs.intersimple import speed_reward
|
||||
import functools
|
||||
from src.util.wrappers import CollisionPenaltyWrapper, TransformObservation, Setobs
|
||||
import numpy as np
|
||||
from torch.utils.tensorboard import SummaryWriter
|
||||
from ray import tune
|
||||
from datetime import datetime
|
||||
import json
|
||||
|
||||
DIR = os.path.dirname(os.path.abspath(__file__))
|
||||
activations = [torch.nn.Tanh, torch.nn.LeakyReLU]
|
||||
|
||||
obs_min = np.array([
|
||||
[-1000, -1000, 0, -np.pi, -1e-1, 0.],
|
||||
[0, -np.pi, -20, -20, -np.pi, -1e-1],
|
||||
[0, -np.pi, -20, -20, -np.pi, -1e-1],
|
||||
[0, -np.pi, -20, -20, -np.pi, -1e-1],
|
||||
[0, -np.pi, -20, -20, -np.pi, -1e-1],
|
||||
[0, -np.pi, -20, -20, -np.pi, -1e-1],
|
||||
]).reshape(-1)
|
||||
|
||||
obs_max = np.array([
|
||||
[1000, 1000, 20, np.pi, 1e-1, 0.],
|
||||
[50, np.pi, 20, 20, np.pi, 1e-1],
|
||||
[50, np.pi, 20, 20, np.pi, 1e-1],
|
||||
[50, np.pi, 20, 20, np.pi, 1e-1],
|
||||
[50, np.pi, 20, 20, np.pi, 1e-1],
|
||||
[50, np.pi, 20, 20, np.pi, 1e-1],
|
||||
]).reshape(-1)
|
||||
|
||||
def training_function(config):
|
||||
np.random.seed(config['seed'])
|
||||
torch.manual_seed(config['seed'])
|
||||
|
||||
# choose validation environment
|
||||
if config['experiment'] == 'A':
|
||||
envs = [Setobs(TransformObservation(CollisionPenaltyWrapper(
|
||||
IntersimpleLidarFlatRandom(
|
||||
n_rays=5,
|
||||
reward=functools.partial(
|
||||
speed_reward,
|
||||
collision_penalty=0
|
||||
),
|
||||
check_collisions=True,
|
||||
stop_on_collision=config['trainenv']['stop_on_collision'],
|
||||
use_idm=config['trainenv']['use_idm'],
|
||||
), collision_distance=6, collision_penalty=100),
|
||||
lambda obs: (obs - obs_min) / (obs_max - obs_min + 1e-10)
|
||||
)) for _ in range(60)]
|
||||
elif config['experiment'] == 'B':
|
||||
envs = sum([[Setobs(TransformObservation(CollisionPenaltyWrapper(
|
||||
IntersimpleLidarFlatRandom(
|
||||
n_rays=5,
|
||||
reward=functools.partial(
|
||||
speed_reward,
|
||||
collision_penalty=0
|
||||
),
|
||||
check_collisions=True,
|
||||
stop_on_collision=config['trainenv']['stop_on_collision'],
|
||||
use_idm=config['trainenv']['use_idm'],
|
||||
track=track,
|
||||
), collision_distance=6, collision_penalty=100),
|
||||
lambda obs: (obs - obs_min) / (obs_max - obs_min + 1e-10)
|
||||
)) for _ in range(15)] for track in range(4)],[])
|
||||
else:
|
||||
raise NotImplementedError
|
||||
|
||||
env_fn = lambda i: envs[i]
|
||||
|
||||
# load expert data
|
||||
|
||||
if config['experiment'] == 'A':
|
||||
expert_data = torch.load(os.path.join(DIR, 'intersimple-expert-data-setobs2-loc0-track0.pt'))
|
||||
elif config['experiment'] == 'B':
|
||||
expert_data = [
|
||||
torch.load(os.path.join(DIR, 'intersimple-expert-data-setobs2-loc0-track0.pt')),
|
||||
torch.load(os.path.join(DIR, 'intersimple-expert-data-setobs2-loc0-track1.pt')),
|
||||
torch.load(os.path.join(DIR, 'intersimple-expert-data-setobs2-loc0-track2.pt')),
|
||||
torch.load(os.path.join(DIR, 'intersimple-expert-data-setobs2-loc0-track3.pt')),
|
||||
]
|
||||
d0 = [d[0] for d in expert_data]
|
||||
d1 = [d[1] for d in expert_data]
|
||||
d2 = [d[2] for d in expert_data]
|
||||
d3 = [d[3] for d in expert_data]
|
||||
expert_data = (torch.cat(d0), torch.cat(d1), torch.cat(d2), torch.cat(d3))
|
||||
|
||||
expert_data = Buffer(*expert_data)
|
||||
|
||||
# configure and train policy
|
||||
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
|
||||
|
||||
policy = SetPolicy(expert_data.actions.shape[-1],
|
||||
n_hidden_layers=config['policy']['n_hidden_layers'],
|
||||
hidden_layer_size=config['policy']['hidden_layer_size'],
|
||||
activation=activations[config['policy']['activation']] ) # config net architecture
|
||||
policy = policy.to(device)
|
||||
|
||||
pi_opt = torch.optim.Adam(policy.parameters(), lr=config['policy']['learning_rate'])
|
||||
pi_lr_scheduler = torch.optim.lr_scheduler.ExponentialLR(pi_opt, gamma=config['policy']['learning_rate_decay'])
|
||||
|
||||
expert_states = expert_data.states[~expert_data.dones].to(device)
|
||||
expert_actions = expert_data.actions[~expert_data.dones].to(device)
|
||||
|
||||
for epoch in range(config['train_epochs']):
|
||||
pi_opt.zero_grad()
|
||||
loss = -policy.log_prob(policy(expert_states), expert_actions).mean()
|
||||
loss.backward()
|
||||
pi_opt.step()
|
||||
pi_lr_scheduler.step()
|
||||
|
||||
if epoch % 25 == 0:
|
||||
gen_states, gen_actions, gen_rewards, gen_dones, gen_collisions = rollout(env_fn, policy.cpu(), n_episodes=60, max_steps_per_episode=200)
|
||||
gen_mean_episode_length = (~gen_dones).sum() / gen_states.shape[0]
|
||||
gen_mean_reward_per_episode = gen_rewards[~gen_dones].sum() / gen_states.shape[0]
|
||||
gen_collision_rate = (1. * gen_collisions.any(-1)).mean()
|
||||
|
||||
tune.report(
|
||||
gen_mean_reward_per_episode=gen_mean_reward_per_episode.item(),
|
||||
mean_episode_length=gen_mean_episode_length.item(),
|
||||
gen_collision_rate=gen_collision_rate.item(),
|
||||
loss=loss.item(),
|
||||
)
|
||||
|
||||
# save model checkpoints
|
||||
ep = epoch + 1
|
||||
if (ep % 50 == 0):
|
||||
torch.save(policy.state_dict(), f'policy_epoch{ep}.pt')
|
||||
|
||||
# save model
|
||||
torch.save(policy.state_dict(), 'policy_final.pt')
|
||||
|
||||
if __name__ == '__main__':
|
||||
import argparse
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument('--train', choices=['A', 'B'])
|
||||
parser.add_argument('--epochs', type=int, default=500)
|
||||
parser.add_argument('--test', type=str, help='path to config file to run final training on')
|
||||
parser.add_argument('--test_seeds', type=int, default=5)
|
||||
parser.add_argument('--test_cpus', type=int, help='number of cpus available to split test seed training over')
|
||||
args = parser.parse_args()
|
||||
|
||||
assert (args.train is None) ^ (args.test is None), 'Must either train on an experiment or test with a config file'
|
||||
|
||||
# if no test config specified, train
|
||||
if args.test is None:
|
||||
print('Running Tuning for Experiment %s'%(args.train))
|
||||
analysis = tune.run(
|
||||
training_function,
|
||||
config={
|
||||
'experiment': args.train,
|
||||
'trainenv': {
|
||||
'stop_on_collision': False,
|
||||
'use_idm':True,
|
||||
},
|
||||
'policy': {
|
||||
'learning_rate': 3e-4,
|
||||
'learning_rate_decay': tune.grid_search([0.999, 1.0]),
|
||||
'hidden_layer_size': tune.grid_search([10, 20, 40]),
|
||||
'n_hidden_layers': tune.grid_search([2, 3]),
|
||||
'activation':tune.grid_search([0, 1]),
|
||||
},
|
||||
'train_epochs': args.epochs,
|
||||
'seed': 0,
|
||||
}
|
||||
# TODO resources_per_trial={'gpu': 1}
|
||||
)
|
||||
best_config = analysis.get_best_config(metric='gen_collision_rate', mode='min')
|
||||
print('Best config: ', best_config)
|
||||
|
||||
# safe best_config
|
||||
if not os.path.isdir(os.path.join(DIR, 'best_configs')):
|
||||
os.mkdir(os.path.join(DIR, 'best_configs'))
|
||||
|
||||
# save gail
|
||||
with open(os.path.join(DIR, 'best_configs',f'bc_exp{args.train}.json'), 'w', encoding='utf-8') as f:
|
||||
json.dump(best_config, f, ensure_ascii=False, indent=4)
|
||||
|
||||
# if config file specified, rerun it with appropriate number of seeds
|
||||
else:
|
||||
with open(args.test, 'rb') as f:
|
||||
config = json.load(f)
|
||||
|
||||
print(f'Retraining {args.test} with {args.test_seeds} seeds on experiment {config["experiment"]}')
|
||||
|
||||
# rerun with appropriate number of seeds
|
||||
rpt = {'cpu': int(args.test_cpus/args.test_seeds)} if (args.test_cpus is not None) else None
|
||||
config['seed'] = tune.grid_search(list(range(1,args.test_seeds+1)))
|
||||
analysis = tune.run(training_function, config=config, resources_per_trial=rpt)
|
||||
|
||||
# move final policies to appropriate directory
|
||||
split_ = os.path.basename(args.test).split('_')
|
||||
model = split_[0]
|
||||
exper = split_[-1].split('.')[0]
|
||||
savepath = os.path.join('test_policies',model,exper)
|
||||
|
||||
if not os.path.isdir(savepath):
|
||||
os.makedirs(savepath)
|
||||
|
||||
import shutil
|
||||
for i in range(args.test_seeds):
|
||||
s = analysis._checkpoints[i]['config']['seed']
|
||||
check_dir = analysis._checkpoints[i]['logdir']
|
||||
shutil.copyfile(os.path.join(check_dir,'policy_final.pt'),
|
||||
os.path.join(savepath, f'policy_seed{s}.pt'))
|
||||
shutil.copyfile(os.path.join(check_dir,'params.json'),
|
||||
os.path.join(savepath, 'config.json')) # copy config automatically
|
||||
@@ -1,16 +0,0 @@
|
||||
{
|
||||
"experiment": "A",
|
||||
"trainenv": {
|
||||
"stop_on_collision": false,
|
||||
"use_idm": true
|
||||
},
|
||||
"policy": {
|
||||
"learning_rate": 0.0003,
|
||||
"learning_rate_decay": 1.0,
|
||||
"hidden_layer_size": 40,
|
||||
"n_hidden_layers": 2,
|
||||
"activation": 0
|
||||
},
|
||||
"train_epochs": 300,
|
||||
"seed": 0
|
||||
}
|
||||
@@ -1,16 +0,0 @@
|
||||
{
|
||||
"experiment": "B",
|
||||
"trainenv": {
|
||||
"stop_on_collision": false,
|
||||
"use_idm": true
|
||||
},
|
||||
"policy": {
|
||||
"learning_rate": 0.0003,
|
||||
"learning_rate_decay": 1.0,
|
||||
"hidden_layer_size": 40,
|
||||
"n_hidden_layers": 2,
|
||||
"activation": 0
|
||||
},
|
||||
"train_epochs": 300,
|
||||
"seed": 0
|
||||
}
|
||||
@@ -1,32 +0,0 @@
|
||||
{
|
||||
"experiment": "A",
|
||||
"trainenv": {
|
||||
"stop_on_collision": false,
|
||||
"use_idm": true
|
||||
},
|
||||
"policy": {
|
||||
"learning_rate": 0.0003,
|
||||
"learning_rate_decay": 1.0,
|
||||
"clip_ratio": 0.2,
|
||||
"iterations_per_epoch": 100,
|
||||
"hidden_layer_size": 40,
|
||||
"n_hidden_layers": 2,
|
||||
"activation": 0
|
||||
},
|
||||
"value": {
|
||||
"learning_rate": 0.0001,
|
||||
"weight_decay": 0.001,
|
||||
"iterations_per_epoch": 1000
|
||||
},
|
||||
"discriminator": {
|
||||
"learning_rate": 0.001,
|
||||
"weight_decay": 0.0001,
|
||||
"iterations_per_epoch": 100,
|
||||
"n_hidden_layers_element": 4,
|
||||
"n_hidden_layers_global": 1,
|
||||
"hidden_layer_size": 10,
|
||||
"activation": 0
|
||||
},
|
||||
"train_epochs": 100,
|
||||
"seed": 0
|
||||
}
|
||||
@@ -1,32 +0,0 @@
|
||||
{
|
||||
"experiment": "B",
|
||||
"trainenv": {
|
||||
"stop_on_collision": false,
|
||||
"use_idm": true
|
||||
},
|
||||
"policy": {
|
||||
"learning_rate": 0.0003,
|
||||
"learning_rate_decay": 1.0,
|
||||
"clip_ratio": 0.2,
|
||||
"iterations_per_epoch": 100,
|
||||
"hidden_layer_size": 40,
|
||||
"n_hidden_layers": 2,
|
||||
"activation": 0
|
||||
},
|
||||
"value": {
|
||||
"learning_rate": 0.0001,
|
||||
"weight_decay": 0.001,
|
||||
"iterations_per_epoch": 1000
|
||||
},
|
||||
"discriminator": {
|
||||
"learning_rate": 0.001,
|
||||
"weight_decay": 0.0001,
|
||||
"iterations_per_epoch": 100,
|
||||
"n_hidden_layers_element": 4,
|
||||
"n_hidden_layers_global": 1,
|
||||
"hidden_layer_size": 10,
|
||||
"activation": 0
|
||||
},
|
||||
"train_epochs": 100,
|
||||
"seed": 0
|
||||
}
|
||||
@@ -1,34 +0,0 @@
|
||||
{
|
||||
"experiment": "A",
|
||||
"trainenv": {
|
||||
"stop_on_collision": false,
|
||||
"safe_actions_collision_method": null,
|
||||
"abort_unsafe_collision_method": null,
|
||||
"use_idm": true
|
||||
},
|
||||
"policy": {
|
||||
"learning_rate": 0.0003,
|
||||
"learning_rate_decay": 1.0,
|
||||
"clip_ratio": 0.2,
|
||||
"iterations_per_epoch": 100,
|
||||
"hidden_layer_size": 40,
|
||||
"n_hidden_layers": 2,
|
||||
"activation": 0,
|
||||
"option": 0
|
||||
},
|
||||
"value": {
|
||||
"learning_rate": 0.001,
|
||||
"iterations_per_epoch": 1000
|
||||
},
|
||||
"discriminator": {
|
||||
"learning_rate": 0.001,
|
||||
"weight_decay": 0.0001,
|
||||
"iterations_per_epoch": 100,
|
||||
"n_hidden_layers_element": 4,
|
||||
"n_hidden_layers_global": 1,
|
||||
"hidden_layer_size": 10,
|
||||
"activation": 0
|
||||
},
|
||||
"train_epochs": 90,
|
||||
"seed": 0
|
||||
}
|
||||
@@ -1,34 +0,0 @@
|
||||
{
|
||||
"experiment": "B",
|
||||
"trainenv": {
|
||||
"stop_on_collision": false,
|
||||
"safe_actions_collision_method": null,
|
||||
"abort_unsafe_collision_method": null,
|
||||
"use_idm": true
|
||||
},
|
||||
"policy": {
|
||||
"learning_rate": 0.0003,
|
||||
"learning_rate_decay": 1.0,
|
||||
"clip_ratio": 0.2,
|
||||
"iterations_per_epoch": 100,
|
||||
"hidden_layer_size": 20,
|
||||
"n_hidden_layers": 2,
|
||||
"activation": 0,
|
||||
"option": 0
|
||||
},
|
||||
"value": {
|
||||
"learning_rate": 0.001,
|
||||
"iterations_per_epoch": 1000
|
||||
},
|
||||
"discriminator": {
|
||||
"learning_rate": 0.001,
|
||||
"weight_decay": 0.0001,
|
||||
"iterations_per_epoch": 100,
|
||||
"n_hidden_layers_element": 4,
|
||||
"n_hidden_layers_global": 2,
|
||||
"hidden_layer_size": 10,
|
||||
"activation": 0
|
||||
},
|
||||
"train_epochs": 85,
|
||||
"seed": 0
|
||||
}
|
||||
@@ -1,34 +0,0 @@
|
||||
{
|
||||
"experiment": "A",
|
||||
"trainenv": {
|
||||
"stop_on_collision": false,
|
||||
"safe_actions_collision_method": "circle",
|
||||
"abort_unsafe_collision_method": "circle",
|
||||
"use_idm": true
|
||||
},
|
||||
"policy": {
|
||||
"learning_rate": 0.0003,
|
||||
"learning_rate_decay": 1.0,
|
||||
"clip_ratio": 0.2,
|
||||
"iterations_per_epoch": 100,
|
||||
"hidden_layer_size": 40,
|
||||
"n_hidden_layers": 2,
|
||||
"activation": 0,
|
||||
"option": 0
|
||||
},
|
||||
"value": {
|
||||
"learning_rate": 0.001,
|
||||
"iterations_per_epoch": 1000
|
||||
},
|
||||
"discriminator": {
|
||||
"learning_rate": 0.001,
|
||||
"weight_decay": 0.0001,
|
||||
"iterations_per_epoch": 100,
|
||||
"n_hidden_layers_element": 4,
|
||||
"n_hidden_layers_global": 1,
|
||||
"hidden_layer_size": 10,
|
||||
"activation": 0
|
||||
},
|
||||
"train_epochs": 90,
|
||||
"seed": 0
|
||||
}
|
||||
@@ -1,34 +0,0 @@
|
||||
{
|
||||
"experiment": "B",
|
||||
"trainenv": {
|
||||
"stop_on_collision": false,
|
||||
"safe_actions_collision_method": "circle",
|
||||
"abort_unsafe_collision_method": "circle",
|
||||
"use_idm": true
|
||||
},
|
||||
"policy": {
|
||||
"learning_rate": 0.0003,
|
||||
"learning_rate_decay": 1.0,
|
||||
"clip_ratio": 0.2,
|
||||
"iterations_per_epoch": 100,
|
||||
"hidden_layer_size": 20,
|
||||
"n_hidden_layers": 2,
|
||||
"activation": 0,
|
||||
"option": 0
|
||||
},
|
||||
"value": {
|
||||
"learning_rate": 0.001,
|
||||
"iterations_per_epoch": 1000
|
||||
},
|
||||
"discriminator": {
|
||||
"learning_rate": 0.001,
|
||||
"weight_decay": 0.0001,
|
||||
"iterations_per_epoch": 100,
|
||||
"n_hidden_layers_element": 4,
|
||||
"n_hidden_layers_global": 2,
|
||||
"hidden_layer_size": 10,
|
||||
"activation": 0
|
||||
},
|
||||
"train_epochs": 85,
|
||||
"seed": 0
|
||||
}
|
||||
BIN
checkpoints/bc-intersimple-setobs2.pt
Normal file
BIN
checkpoints/bc-intersimple-setobs2.pt
Normal file
Binary file not shown.
BIN
checkpoints/gail-intersimple-setobs2-03-02-22.pt
Normal file
BIN
checkpoints/gail-intersimple-setobs2-03-02-22.pt
Normal file
Binary file not shown.
BIN
checkpoints/gail-options-setobs2-15-02-2022.pt
Normal file
BIN
checkpoints/gail-options-setobs2-15-02-2022.pt
Normal file
Binary file not shown.
BIN
checkpoints/gail-options-setobs2-Feb15_18-49-05.pt
Normal file
BIN
checkpoints/gail-options-setobs2-Feb15_18-49-05.pt
Normal file
Binary file not shown.
BIN
checkpoints/gail-ppo-options-setobs2-Feb15_22-05-38.pt
Normal file
BIN
checkpoints/gail-ppo-options-setobs2-Feb15_22-05-38.pt
Normal file
Binary file not shown.
BIN
checkpoints/sgail-options-setobs2.pt
Normal file
BIN
checkpoints/sgail-options-setobs2.pt
Normal file
Binary file not shown.
BIN
checkpoints/sgail-ppo-options-setobs2-17-02-2022.pt
Normal file
BIN
checkpoints/sgail-ppo-options-setobs2-17-02-2022.pt
Normal file
Binary file not shown.
BIN
checkpoints/wgail-options-setobs2-Feb16_01-06-27.pt
Normal file
BIN
checkpoints/wgail-options-setobs2-Feb16_01-06-27.pt
Normal file
Binary file not shown.
BIN
checkpoints/wgail-ppo-options-setobs2-Feb16_04-02-56.pt
Normal file
BIN
checkpoints/wgail-ppo-options-setobs2-Feb16_04-02-56.pt
Normal file
Binary file not shown.
42
config/networks.json5
Normal file
42
config/networks.json5
Normal file
@@ -0,0 +1,42 @@
|
||||
{
|
||||
ego_encoder: {
|
||||
input_dim: 5, // number of state vars
|
||||
hidden_n: 0,
|
||||
hidden_dim: 5,
|
||||
output_dim: 5
|
||||
},
|
||||
deepsets: {
|
||||
input_dim: 6, // number of relative state vars for others
|
||||
phi: {
|
||||
hidden_n: 2,
|
||||
hidden_dim: 20,
|
||||
},
|
||||
latent_dim: 20,
|
||||
rho: {
|
||||
hidden_n: 2,
|
||||
hidden_dim: 10,
|
||||
},
|
||||
output_dim: 10
|
||||
},
|
||||
path_encoder: {
|
||||
input_dim: 40, // 2 * path length for (x,y) coordinates
|
||||
hidden_n: 0,
|
||||
hidden_dim: 20,
|
||||
output_dim: 10,
|
||||
},
|
||||
head: {
|
||||
input_dim: 0, // computed in policy constructor
|
||||
hidden_n: 3,
|
||||
hidden_dim: 50,
|
||||
output_dim: 1, // number of outputs e.g. number of actions, or just one
|
||||
final_activation: 'sigmoid',
|
||||
},
|
||||
optim: {
|
||||
optimizer: 'adam',
|
||||
lr: 1e-3,
|
||||
weight_decay: 0.1,
|
||||
},
|
||||
train_epochs: 200,
|
||||
train_batch_size: 32,
|
||||
loss: 'huber',
|
||||
}
|
||||
85
config/value_dice.json5
Normal file
85
config/value_dice.json5
Normal file
@@ -0,0 +1,85 @@
|
||||
{
|
||||
policy_net: {
|
||||
ego_encoder: {
|
||||
input_dim: 5, // number of state vars
|
||||
hidden_n: 0,
|
||||
hidden_dim: 5,
|
||||
output_dim: 5
|
||||
},
|
||||
deepsets: {
|
||||
input_dim: 6, // number of relative state vars for others
|
||||
phi: {
|
||||
hidden_n: 2,
|
||||
hidden_dim: 20,
|
||||
},
|
||||
latent_dim: 20,
|
||||
rho: {
|
||||
hidden_n: 2,
|
||||
hidden_dim: 10,
|
||||
},
|
||||
output_dim: 10
|
||||
},
|
||||
path_encoder: {
|
||||
input_dim: 40, // 2 * path length for (x,y) coordinates
|
||||
hidden_n: 0,
|
||||
hidden_dim: 20,
|
||||
output_dim: 10,
|
||||
},
|
||||
head: {
|
||||
input_dim: 0, // computed in policy constructor
|
||||
hidden_n: 3,
|
||||
hidden_dim: 50,
|
||||
output_dim: 1, // number of outputs e.g. number of actions, or just one
|
||||
final_activation: 'sigmoid',
|
||||
},
|
||||
},
|
||||
value_net: {
|
||||
ego_encoder: {
|
||||
input_dim: 5, // number of state vars
|
||||
hidden_n: 0,
|
||||
hidden_dim: 5,
|
||||
output_dim: 5
|
||||
},
|
||||
deepsets: {
|
||||
input_dim: 6, // number of relative state vars for others
|
||||
phi: {
|
||||
hidden_n: 2,
|
||||
hidden_dim: 20,
|
||||
},
|
||||
latent_dim: 20,
|
||||
rho: {
|
||||
hidden_n: 2,
|
||||
hidden_dim: 10,
|
||||
},
|
||||
output_dim: 10
|
||||
},
|
||||
path_encoder: {
|
||||
input_dim: 40, // 2 * path length for (x,y) coordinates
|
||||
hidden_n: 0,
|
||||
hidden_dim: 20,
|
||||
output_dim: 10,
|
||||
},
|
||||
action_dim: 1, // number of actions
|
||||
head: {
|
||||
input_dim: 0, // computed in policy constructor
|
||||
hidden_n: 3,
|
||||
hidden_dim: 50,
|
||||
output_dim: 1, // number of outputs e.g. number of actions, or just one
|
||||
final_activation: 'id',
|
||||
},
|
||||
},
|
||||
policy_optim: {
|
||||
optimizer: 'adam',
|
||||
lr: 1e-3,
|
||||
weight_decay: 0.1,
|
||||
},
|
||||
value_optim: {
|
||||
optimizer: 'adam',
|
||||
lr: 1e-3,
|
||||
weight_decay: 0.1,
|
||||
},
|
||||
train_epochs: 200,
|
||||
train_batch_size: 32,
|
||||
discount: 0.95,
|
||||
clip_grad_norm: 1.,
|
||||
}
|
||||
10
cp-videos.sh
10
cp-videos.sh
@@ -1,10 +0,0 @@
|
||||
# cp-videos videos/ videos/icra23/
|
||||
|
||||
agents=( 5 27 39 43 47 53 63 81 83 87 93 96 105 113 124 127 130 134 )
|
||||
|
||||
for a in "${agents[@]}"
|
||||
do
|
||||
cp "$1/expert_agent/loc0/track0/agent${a}_ani.mp4" "$2/t${a}expert.mp4"
|
||||
cp "$1/idm/loc0/track0/agent${a}_ani.mp4" "$2/t${a}idm.mp4"
|
||||
cp "$1/shail/loc0/track0/agent${a}_ani.mp4" "$2/t${a}shail.mp4"
|
||||
done
|
||||
@@ -1,108 +0,0 @@
|
||||
import os
|
||||
from src.eval_main import eval_main
|
||||
from src.evaluation.utils import load_and_average
|
||||
import torch
|
||||
import json
|
||||
|
||||
activations = [torch.nn.Tanh, torch.nn.LeakyReLU]
|
||||
|
||||
def main(method:str='expert', folder:str=None, locations=[(0,0)], skip_running=False, save_videos:bool=False, videos_folder:str='videos', first_seed_only:bool=False):
|
||||
|
||||
exclude_keys_from_policy_kwargs = {'learning_rate', 'learning_rate_decay', 'clip_ratio', 'iterations_per_epoch', 'option'}
|
||||
policy_kwargs = {}
|
||||
|
||||
if method in ['expert', 'expert_agent']:
|
||||
env, env_kwargs ='NRasterizedRouteIncrementingAgent', {}
|
||||
elif method in ['idm']:
|
||||
env, env_kwargs ='NRasterizedRouteIncrementingAgent', {'use_idm':True}
|
||||
elif method in ['bc','gail']:
|
||||
env='NormalizedContinuousEvalEnv'
|
||||
env_kwargs={'stop_on_collision':True, 'max_episode_steps':1000, 'use_idm':True}
|
||||
elif method in ['hail']:
|
||||
env = 'NormalizedSafeOptionsEvalEnv'
|
||||
env_kwargs={'stop_on_collision':True, 'max_episode_steps':1000, 'safe_actions_collision_method': None, 'abort_unsafe_collision_method': None, 'use_idm':True}
|
||||
elif method in ['shail']:
|
||||
env = 'NormalizedSafeOptionsEvalEnv'
|
||||
env_kwargs={'stop_on_collision':True, 'max_episode_steps':1000, 'use_idm':True}
|
||||
else:
|
||||
raise NotImplementedError
|
||||
|
||||
files = ['']
|
||||
|
||||
if folder is not None:
|
||||
files = [os.path.join(folder, f) for f in os.listdir(folder) if os.path.isfile(os.path.join(folder, f))]
|
||||
files = [f for f in files if f.endswith('.pt')]
|
||||
|
||||
if first_seed_only:
|
||||
files = files[:1]
|
||||
|
||||
with open(os.path.join(folder, 'config.json'), 'rb') as f:
|
||||
config = json.load(f)
|
||||
|
||||
print('%i policy files found in %s folder' %(len(files), folder))
|
||||
print('found policy config', config['policy'])
|
||||
|
||||
policy_config = {k: v for k, v in config['policy'].items() if k not in exclude_keys_from_policy_kwargs}
|
||||
policy_config['activation'] = activations[policy_config['activation']]
|
||||
print('final policy config', policy_config)
|
||||
|
||||
policy_kwargs.update(policy_config)
|
||||
print('final policy kwargs', policy_kwargs)
|
||||
|
||||
if not skip_running:
|
||||
for policy_file in files:
|
||||
# run metrics on that file
|
||||
outbase = eval_main(locations=locations,
|
||||
method=method,
|
||||
policy_file=policy_file,
|
||||
policy_kwargs=policy_kwargs,
|
||||
env=env,
|
||||
env_kwargs=env_kwargs,
|
||||
videos_folder=None if not save_videos else videos_folder)
|
||||
outfolder = os.path.dirname(outbase)
|
||||
else:
|
||||
locstr = 'loc_'+'_'.join([f'r{ro}t{tr}' for (ro,tr) in locations])
|
||||
if folder is None:
|
||||
outfolder = os.path.join('out',method,locstr)
|
||||
else:
|
||||
path_items = folder.split('/')
|
||||
outfolder = os.path.join('out', '/'.join(path_items[1:]), locstr)
|
||||
|
||||
# load metrics from save_path
|
||||
average_metrics = load_and_average(outfolder)
|
||||
if method in ['expert', 'idm']:
|
||||
latex_print(average_metrics, light=True)
|
||||
else:
|
||||
latex_print(average_metrics)
|
||||
|
||||
def latex_print(am, light=False):
|
||||
"""
|
||||
print latex line
|
||||
|
||||
am (Dict[str,tuple]): dict mapping metric_name to (mean, std)
|
||||
"""
|
||||
|
||||
print('success rate, distance travelled, RWSE_10, |DeltaV|, AccelJSD')
|
||||
if light:
|
||||
if 'rwse_10s' in am.keys():
|
||||
print("%2.1f& %2.1f & %2.1f & %1.2f& "
|
||||
"%0.3f \\\\" %( 100*am['success rate'][0], am['mean travel distance'][0], am['rwse_10s'][0],
|
||||
am['average absolute average velocity'][0],am['acceleration distribution divergence'][0] ))
|
||||
return
|
||||
|
||||
|
||||
print("%2.1f& %2.1f & $---$ & $---$ & "
|
||||
"$---$ \\\\" %( 100*am['success rate'][0], am['mean travel distance'][0]))
|
||||
return
|
||||
|
||||
print("%2.1f \\scriptstyle\\pm %2.1f & %2.1f \\scriptstyle\\pm %2.1f & "
|
||||
"%2.1f \\scriptstyle\\pm %1.1f & %1.2f \\scriptstyle\\pm %1.2f & "
|
||||
"%0.3f \\scriptstyle\\pm %0.3f \\\\" %( 100*am['success rate'][0], 100*am['success rate'][1],
|
||||
am['mean travel distance'][0] , am['mean travel distance'][1] ,
|
||||
am['rwse_10s'][0] , am['rwse_10s'][1] ,
|
||||
am['average absolute average velocity'][0] , am['average absolute average velocity'][1] ,
|
||||
am['acceleration distribution divergence'][0] , am['acceleration distribution divergence'][1] ))
|
||||
|
||||
if __name__=='__main__':
|
||||
import fire
|
||||
fire.Fire(main)
|
||||
@@ -1,18 +1,42 @@
|
||||
# can add --skip_running if you've already run the saved policies through the test environments and have appropriate
|
||||
# metrics in the out folder. Doing so will generate average metrics quickly.
|
||||
# eval_main inputs
|
||||
# locations: List[Tuple[int,int]]= [(0,0)],
|
||||
# method: str='expert',
|
||||
# policy_file: str='',
|
||||
# policy_kwargs: dict={},
|
||||
# env: str='NRasterizedRouteIncrementingAgent',
|
||||
# env_kwargs: dict={},
|
||||
# seed: int=0
|
||||
|
||||
# Experiment A
|
||||
python -m eval_experiments
|
||||
python -m eval_experiments --method idm
|
||||
python -m eval_experiments --method bc --folder='test_policies/bc/expA'
|
||||
python -m eval_experiments --method gail --folder='test_policies/gail/expA'
|
||||
python -m eval_experiments --method hail --folder='test_policies/hail/expA'
|
||||
python -m eval_experiments --method shail --folder='test_policies/shail/expA'
|
||||
# expert
|
||||
python -m src.eval_main
|
||||
|
||||
# Experiment B
|
||||
python -m eval_experiments --locations='[(0,4)]'
|
||||
python -m eval_experiments --method idm --locations='[(0,4)]'
|
||||
python -m eval_experiments --method bc --folder='test_policies/bc/expB' --locations='[(0,4)]'
|
||||
python -m eval_experiments --method gail --folder='test_policies/gail/expB' --locations='[(0,4)]'
|
||||
python -m eval_experiments --method hail --folder='test_policies/hail/expB' --locations='[(0,4)]'
|
||||
python -m eval_experiments --method shail --folder='test_policies/shail/expB' --locations='[(0,4)]'
|
||||
# idm
|
||||
python -m src.eval_main --method=idm
|
||||
|
||||
# behavior cloning
|
||||
python -m src.eval_main --method=bc --policy_file='checkpoints/bc-intersimple-setobs2.pt' --env='NormalizedContinuousEvalEnv' --env_kwargs='{stop_on_collision:True}' --seed=0
|
||||
python -m src.eval_main --method=bc --policy_file='checkpoints/bc-intersimple-setobs2.pt' --env='NormalizedContinuousEvalEnv' --env_kwargs='{stop_on_collision:True}' --seed=1
|
||||
python -m src.eval_main --method=bc --policy_file='checkpoints/bc-intersimple-setobs2.pt' --env='NormalizedContinuousEvalEnv' --env_kwargs='{stop_on_collision:True}' --seed=2
|
||||
python -m src.eval_main --method=bc --policy_file='checkpoints/bc-intersimple-setobs2.pt' --env='NormalizedContinuousEvalEnv' --env_kwargs='{stop_on_collision:True}' --seed=3
|
||||
python -m src.eval_main --method=bc --policy_file='checkpoints/bc-intersimple-setobs2.pt' --env='NormalizedContinuousEvalEnv' --env_kwargs='{stop_on_collision:True}' --seed=4
|
||||
python -m src.evaluation.utils load_and_average out/bc
|
||||
|
||||
# GAIL
|
||||
python -m src.eval_main --method=gail --policy_file='checkpoints/gail-intersimple-setobs2-03-02-22.pt' --env='NormalizedContinuousEvalEnv' --env_kwargs='{stop_on_collision:True}' --seed=0
|
||||
python -m src.eval_main --method=gail --policy_file='checkpoints/gail-intersimple-setobs2-03-02-22.pt' --env='NormalizedContinuousEvalEnv' --env_kwargs='{stop_on_collision:True}' --seed=1
|
||||
python -m src.eval_main --method=gail --policy_file='checkpoints/gail-intersimple-setobs2-03-02-22.pt' --env='NormalizedContinuousEvalEnv' --env_kwargs='{stop_on_collision:True}' --seed=2
|
||||
python -m src.eval_main --method=gail --policy_file='checkpoints/gail-intersimple-setobs2-03-02-22.pt' --env='NormalizedContinuousEvalEnv' --env_kwargs='{stop_on_collision:True}' --seed=3
|
||||
python -m src.eval_main --method=gail --policy_file='checkpoints/gail-intersimple-setobs2-03-02-22.pt' --env='NormalizedContinuousEvalEnv' --env_kwargs='{stop_on_collision:True}' --seed=4
|
||||
python -m src.evaluation.utils load_and_average out/gail
|
||||
|
||||
# options GAIL
|
||||
python -m src.eval_main --method=ogail --policy_file='checkpoints/gail-options-setobs2-Feb15_18-49-05.pt' --env='NormalizedOptionsEvalEnv' --env_kwargs='{stop_on_collision:True}'
|
||||
|
||||
# options GAIL-PPO
|
||||
python -m src.eval_main --method=ogail-ppo --policy_file='checkpoints/gail-ppo-options-setobs2-Feb15_22-05-38.pt' --env='NormalizedOptionsEvalEnv' --env_kwargs='{stop_on_collision:True}'
|
||||
|
||||
# SHAIL
|
||||
python -m src.eval_main --method=sgail --policy_file='checkpoints/sgail-options-setobs2.pt' --env='NormalizedSafeOptionsEvalEnv' --env_kwargs='{stop_on_collision:True,max_episode_steps:1000}'
|
||||
|
||||
# SHAIL-PPO
|
||||
python -m src.eval_main --method=sgail-ppo --policy_file='checkpoints/sgail-ppo-options-setobs2-17-02-2022.pt' --env='NormalizedSafeOptionsEvalEnv' --env_kwargs='{stop_on_collision:True,max_episode_steps:1000}'
|
||||
|
||||
203
experiments/experiment.py
Normal file
203
experiments/experiment.py
Normal file
@@ -0,0 +1,203 @@
|
||||
import json5
|
||||
from functools import partial
|
||||
import os
|
||||
opj = os.path.join
|
||||
|
||||
# set up ray tune
|
||||
import ray
|
||||
from ray import tune
|
||||
from ray.tune import Analysis, ExperimentAnalysis
|
||||
from ray.tune.schedulers import ASHAScheduler
|
||||
from hyperopt import hp
|
||||
from ray.tune.suggest.hyperopt import HyperOptSearch
|
||||
|
||||
# get graphs
|
||||
import intersim
|
||||
from intersim.graphs import ConeVisibilityGraph
|
||||
|
||||
|
||||
from src.main import basestr, main
|
||||
|
||||
def parse_args():
|
||||
"""
|
||||
Parse arguments to main
|
||||
Returns:
|
||||
kwargs: dictionary of arguments:
|
||||
train (bool): whether to run train loop
|
||||
test (bool): whether to run test loop
|
||||
method (str): the method to try for imitation
|
||||
loc (int): the location index of the roundabout
|
||||
config (str): config path
|
||||
seed (int): RNG seed
|
||||
"""
|
||||
import argparse
|
||||
parser = argparse.ArgumentParser(description='Save Expert Trajectories')
|
||||
parser.add_argument('--loc', default=0, type=int,
|
||||
help='location (default 0)')
|
||||
parser.add_argument("--train", help="train model",
|
||||
action="store_true")
|
||||
parser.add_argument("--ray", help="use ray tune to run multiple experiments",
|
||||
action="store_true")
|
||||
parser.add_argument("--test", help="test model",
|
||||
action="store_true")
|
||||
parser.add_argument("--method", help="modeling method",
|
||||
choices=['bc', 'gail', 'advil', 'vd'], default='bc')
|
||||
parser.add_argument("--config", help="config file path",
|
||||
default=None, type=str)
|
||||
parser.add_argument('--seed', default=0, type=int,
|
||||
help='seed')
|
||||
parser.add_argument('--nframes', default=500, type=int,
|
||||
help='frames for test animation')
|
||||
parser.add_argument('--nsamples', default=200, type=int,
|
||||
help='number of ray samples')
|
||||
parser.add_argument('--graph', action='store_true',
|
||||
help='whether to mask the relative states based on a ConeVisibilityGraph')
|
||||
parser.add_argument('-d', default='./expert_data', type=str,
|
||||
help='data directory')
|
||||
parser.add_argument('-o', default=None, type=str,
|
||||
help='output directory')
|
||||
args = parser.parse_args()
|
||||
kwargs = {
|
||||
'train':args.train,
|
||||
'test':args.test,
|
||||
'method':args.method,
|
||||
'loc':args.loc,
|
||||
'config_path':args.config,
|
||||
'seed':args.seed,
|
||||
'ray':args.ray,
|
||||
'nframes':args.nframes,
|
||||
'nsamples':args.nsamples,
|
||||
'datadir':os.path.abspath(args.d),
|
||||
'graph':None,
|
||||
'outdir': opj('output',args.method,'loc%02i'%(args.loc)),
|
||||
'train_tracks':[0,1,2],
|
||||
'cv_tracks':[3],
|
||||
'test_tracks':[4],
|
||||
}
|
||||
if args.o:
|
||||
kwargs['outdir'] = args.o
|
||||
if args.graph:
|
||||
kwargs['graph'] = ConeVisibilityGraph(r=20, half_angle=120)
|
||||
return kwargs
|
||||
|
||||
def get_full_config(ray_config:dict, method:str)->dict:
|
||||
"""
|
||||
Get full model configuration from ray config and method string
|
||||
Args:
|
||||
ray_config (dict): ray config
|
||||
method (str): method to get full configuration for
|
||||
"""
|
||||
if method == 'bc':
|
||||
from src.bc import bc_config
|
||||
config = bc_config(ray_config)
|
||||
elif method == 'vd':
|
||||
from src.value_dice import vd_config
|
||||
config = vd_config(ray_config)
|
||||
else:
|
||||
raise NotImplementedError
|
||||
return config
|
||||
|
||||
def get_ray_config(method:str)->dict:
|
||||
"""
|
||||
Get configuration for ray based on method.
|
||||
Args:
|
||||
method (str): method to get configuration for
|
||||
Returns:
|
||||
ray_config (dict): configuration for ray
|
||||
"""
|
||||
if method == 'bc':
|
||||
ray_config = {
|
||||
"lr": tune.loguniform(1e-5, 1e-3),
|
||||
"weight_decay": tune.choice([0, 0.1]),
|
||||
"loss": tune.choice(['huber', 'mse']),
|
||||
"train_batch_size": tune.choice([16,32,64]),
|
||||
"deepsets_phi_hidden_n": tune.randint(1,5),
|
||||
"deepsets_phi_hidden_dim": tune.lograndint(8,65),
|
||||
"deepsets_latent_dim": tune.lograndint(8,129),
|
||||
"deepsets_rho_hidden_n": tune.randint(0,3),
|
||||
"deepsets_rho_hidden_dim": tune.lograndint(8,129),
|
||||
"deepsets_output_dim": tune.lograndint(4,129),
|
||||
"head_hidden_n": tune.randint(1,6),
|
||||
"head_hidden_dim": tune.lograndint(16,257),
|
||||
"head_final_activation": tune.choice(['sigmoid', None]),
|
||||
}
|
||||
elif method == 'vd':
|
||||
ray_config = {
|
||||
"policy_lr": tune.loguniform(1e-5, 1e-3),
|
||||
"value_lr": tune.loguniform(1e-5, 1e-3),
|
||||
"policy_weight_decay": tune.choice([0, 0.1]),
|
||||
"value_weight_decay": tune.choice([0, 0.1]),
|
||||
"train_batch_size": tune.choice([16,32,64]),
|
||||
"deepsets_phi_hidden_n": tune.randint(1,5),
|
||||
"deepsets_phi_hidden_dim": tune.lograndint(8,65),
|
||||
"deepsets_latent_dim": tune.lograndint(8,129),
|
||||
"deepsets_rho_hidden_n": tune.randint(0,3),
|
||||
"deepsets_rho_hidden_dim": tune.lograndint(8,129),
|
||||
"deepsets_output_dim": tune.lograndint(4,129),
|
||||
"head_hidden_n": tune.randint(1,6),
|
||||
"head_hidden_dim": tune.lograndint(16,257),
|
||||
"head_final_activation": tune.choice(['sigmoid', None]),
|
||||
"clip_grad_norm": tune.choice([.5, 1., 5., 10.]),
|
||||
"discount": tune.choice([.95, .99])
|
||||
}
|
||||
else:
|
||||
raise NotImplementedError
|
||||
return ray_config
|
||||
|
||||
if __name__ == '__main__':
|
||||
kwargs = parse_args()
|
||||
|
||||
# make prefix of output files
|
||||
|
||||
if kwargs['config_path']:
|
||||
# load config
|
||||
with open(kwargs['config_path'], 'r') as cfg:
|
||||
config = json5.load(cfg)
|
||||
if not os.path.isdir(kwargs['outdir']):
|
||||
os.makedirs(kwargs['outdir'])
|
||||
filestr = opj(kwargs['outdir'], basestr(**kwargs))
|
||||
if kwargs['ray']:
|
||||
filestr = kwargs['config_path'].replace('_config.json','')
|
||||
main(config, filestr=filestr, **kwargs)
|
||||
|
||||
elif kwargs['ray'] and kwargs['train']:
|
||||
|
||||
ray.shutdown()
|
||||
ray.init(log_to_driver=False)
|
||||
|
||||
def ray_train(config, datadir=None):
|
||||
full_config = get_full_config(config, kwargs['method'])
|
||||
main(full_config, filestr='exp', **kwargs)
|
||||
|
||||
ray_config = get_ray_config(kwargs['method'])
|
||||
search = HyperOptSearch(ray_config, max_concurrent=8, metric='cv_loss',mode="min",)
|
||||
custom_scheduler = ASHAScheduler(metric='cv_loss', mode="min", grace_period=15)
|
||||
|
||||
analysis = tune.run(
|
||||
ray_train,
|
||||
#config=ray_config,
|
||||
search_alg=search,
|
||||
scheduler=custom_scheduler,
|
||||
local_dir=kwargs['outdir'],
|
||||
#resources_per_trial={"cpu": 2},
|
||||
time_budget_s=120*60,
|
||||
num_samples=kwargs['nsamples'],
|
||||
)
|
||||
elif kwargs['ray'] and kwargs['test']:
|
||||
analysis = Analysis(kwargs['outdir'], default_metric="cv_loss", default_mode="min")
|
||||
config = analysis.get_best_config()
|
||||
filepath = analysis.get_best_logdir()
|
||||
filestr = opj(filepath, 'exp')
|
||||
config_path = filestr+'_config.json'
|
||||
with open(config_path, 'r') as cfg:
|
||||
config = json5.load(cfg)
|
||||
print("Best ray experiment:", filepath)
|
||||
main(config, filestr=filestr, **kwargs)
|
||||
else:
|
||||
raise Exception('No valid config found')
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
9
experiments/experiments.sh
Executable file
9
experiments/experiments.sh
Executable file
@@ -0,0 +1,9 @@
|
||||
#!/bin/sh
|
||||
|
||||
python experiments/experiment.py --ray --train -d ./expert_data/base
|
||||
python experiments/experiment.py --ray --test -d ./expert_data/base --nframes 1000
|
||||
python experiments/experiment.py --ray --train -d ./expert_data/reg
|
||||
python experiments/experiment.py --ray --test -d ./expert_data/reg --nframes 1000
|
||||
python experiments/experiment.py --ray --train -d ./expert_data/reg_graph --graph
|
||||
python experiments/experiment.py --ray --test -d ./expert_data/reg_graph --graph --nframes 1000
|
||||
|
||||
5
experiments/train_vd.sh
Executable file
5
experiments/train_vd.sh
Executable file
@@ -0,0 +1,5 @@
|
||||
#!/bin/sh
|
||||
|
||||
# python experiments/experiment.py --method vd --train --ray -d expert_data/reg -o output/vd/loc00/reg --nsamples 400
|
||||
# python experiments/experiment.py --test --ray --method vd -d expert_data/normal -o output/vd/loc00/normal --nframes 1000
|
||||
python experiments/experiment.py --train --method vd --config config/value_dice.json5
|
||||
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
@@ -1,243 +0,0 @@
|
||||
# %%
|
||||
import os
|
||||
|
||||
import gym
|
||||
from src.core.gail import gail_ppo, Buffer
|
||||
from src.core.value import SetValue
|
||||
from src.core.policy import SetPolicy
|
||||
from src.core.discriminator import DeepsetDiscriminator
|
||||
import torch
|
||||
|
||||
from intersim.envs import IntersimpleLidarFlatRandom
|
||||
from intersim.envs.intersimple import speed_reward
|
||||
import functools
|
||||
from src.util.wrappers import CollisionPenaltyWrapper, TransformObservation, Setobs
|
||||
import numpy as np
|
||||
from torch.utils.tensorboard import SummaryWriter
|
||||
from ray import tune
|
||||
from datetime import datetime
|
||||
import json
|
||||
|
||||
DIR = os.path.dirname(os.path.abspath(__file__))
|
||||
activations = [torch.nn.Tanh, torch.nn.LeakyReLU]
|
||||
|
||||
obs_min = np.array([
|
||||
[-1000, -1000, 0, -np.pi, -1e-1, 0.],
|
||||
[0, -np.pi, -20, -20, -np.pi, -1e-1],
|
||||
[0, -np.pi, -20, -20, -np.pi, -1e-1],
|
||||
[0, -np.pi, -20, -20, -np.pi, -1e-1],
|
||||
[0, -np.pi, -20, -20, -np.pi, -1e-1],
|
||||
[0, -np.pi, -20, -20, -np.pi, -1e-1],
|
||||
]).reshape(-1)
|
||||
|
||||
obs_max = np.array([
|
||||
[1000, 1000, 20, np.pi, 1e-1, 0.],
|
||||
[50, np.pi, 20, 20, np.pi, 1e-1],
|
||||
[50, np.pi, 20, 20, np.pi, 1e-1],
|
||||
[50, np.pi, 20, 20, np.pi, 1e-1],
|
||||
[50, np.pi, 20, 20, np.pi, 1e-1],
|
||||
[50, np.pi, 20, 20, np.pi, 1e-1],
|
||||
]).reshape(-1)
|
||||
|
||||
def training_function(config):
|
||||
np.random.seed(config['seed'])
|
||||
torch.manual_seed(config['seed'])
|
||||
|
||||
if config['experiment'] == 'A':
|
||||
envs = [Setobs(TransformObservation(CollisionPenaltyWrapper(
|
||||
IntersimpleLidarFlatRandom(
|
||||
n_rays=5,
|
||||
reward=functools.partial(
|
||||
speed_reward,
|
||||
collision_penalty=0
|
||||
),
|
||||
check_collisions=True,
|
||||
stop_on_collision=config['trainenv']['stop_on_collision'],
|
||||
use_idm=config['trainenv']['use_idm'],
|
||||
), collision_distance=6, collision_penalty=100),
|
||||
lambda obs: (obs - obs_min) / (obs_max - obs_min + 1e-10)
|
||||
)) for _ in range(60)]
|
||||
|
||||
elif config['experiment'] == 'B':
|
||||
envs = sum([[Setobs(TransformObservation(CollisionPenaltyWrapper(
|
||||
IntersimpleLidarFlatRandom(
|
||||
n_rays=5,
|
||||
reward=functools.partial(
|
||||
speed_reward,
|
||||
collision_penalty=0
|
||||
),
|
||||
check_collisions=True,
|
||||
stop_on_collision=config['trainenv']['stop_on_collision'],
|
||||
use_idm=config['trainenv']['use_idm'],
|
||||
track=track,
|
||||
), collision_distance=6, collision_penalty=100),
|
||||
lambda obs: (obs - obs_min) / (obs_max - obs_min + 1e-10)
|
||||
)) for _ in range(15)] for track in range(4)],[])
|
||||
|
||||
else:
|
||||
raise NotImplementedError
|
||||
|
||||
env_fn = lambda i: envs[i]
|
||||
|
||||
policy = SetPolicy(env_fn(0).action_space.shape[0],
|
||||
n_hidden_layers=config['policy']['n_hidden_layers'],
|
||||
hidden_layer_size=config['policy']['hidden_layer_size'],
|
||||
activation=activations[config['policy']['activation']] ) # config net architecture
|
||||
pi_opt = torch.optim.Adam(policy.parameters(), lr=config['policy']['learning_rate'])
|
||||
pi_lr_scheduler = torch.optim.lr_scheduler.ExponentialLR(pi_opt, gamma=config['policy']['learning_rate_decay'])
|
||||
|
||||
value = SetValue() # config net architecture
|
||||
v_opt = torch.optim.Adam(value.parameters(), lr=config['value']['learning_rate'], weight_decay=config['value']['weight_decay'])
|
||||
|
||||
discriminator = DeepsetDiscriminator(
|
||||
n_hidden_layers_element=config['discriminator']['n_hidden_layers_element'],
|
||||
n_hidden_layers_global=config['discriminator']['n_hidden_layers_global'],
|
||||
hidden_layer_size=config['discriminator']['hidden_layer_size'],
|
||||
activation=activations[config['discriminator']['activation']],
|
||||
)
|
||||
disc_opt = torch.optim.Adam(discriminator.parameters(), lr=config['discriminator']['learning_rate'], weight_decay=config['discriminator']['weight_decay'])
|
||||
|
||||
if config['experiment'] == 'A':
|
||||
expert_data = torch.load(os.path.join(DIR, 'intersimple-expert-data-setobs2-loc0-track0.pt'))
|
||||
elif config['experiment'] == 'B':
|
||||
expert_data = [
|
||||
torch.load(os.path.join(DIR, 'intersimple-expert-data-setobs2-loc0-track0.pt')),
|
||||
torch.load(os.path.join(DIR, 'intersimple-expert-data-setobs2-loc0-track1.pt')),
|
||||
torch.load(os.path.join(DIR, 'intersimple-expert-data-setobs2-loc0-track2.pt')),
|
||||
torch.load(os.path.join(DIR, 'intersimple-expert-data-setobs2-loc0-track3.pt')),
|
||||
]
|
||||
d0 = [d[0] for d in expert_data]
|
||||
d1 = [d[1] for d in expert_data]
|
||||
d2 = [d[2] for d in expert_data]
|
||||
d3 = [d[3] for d in expert_data]
|
||||
expert_data = (torch.cat(d0), torch.cat(d1), torch.cat(d2), torch.cat(d3))
|
||||
|
||||
expert_data = Buffer(*expert_data)
|
||||
|
||||
def callback(info):
|
||||
tune.report(gen_mean_reward_per_episode=info['gen/mean_reward_per_episode'],
|
||||
disc_mean_reward_per_episode=info['disc/mean_reward_per_episode'],
|
||||
mean_episode_length=info['gen/mean_episode_length'],
|
||||
gen_collision_rate=info['gen/collision_rate'])
|
||||
|
||||
# save model checkpoints
|
||||
ep = info['epoch'] + 1
|
||||
if (ep % 25 == 0):
|
||||
torch.save(info['policy'].state_dict(), f'policy_epoch{ep}.pt')
|
||||
|
||||
value, policy = gail_ppo(
|
||||
env_fn=env_fn,
|
||||
expert_data=expert_data,
|
||||
discriminator=discriminator,
|
||||
disc_opt=disc_opt,
|
||||
disc_iters=config['discriminator']['iterations_per_epoch'],
|
||||
policy=policy,
|
||||
value=value,
|
||||
v_opt=v_opt,
|
||||
v_iters=config['value']['iterations_per_epoch'],
|
||||
epochs=config['train_epochs'],
|
||||
rollout_episodes=60,
|
||||
rollout_steps=200,
|
||||
gamma=0.99,
|
||||
gae_lambda=0.9,
|
||||
clip_ratio=config['policy']['clip_ratio'],
|
||||
pi_opt=pi_opt,
|
||||
pi_iters=config['policy']['iterations_per_epoch'],
|
||||
logger=SummaryWriter(comment='gail-ppo-options-setobs2'),
|
||||
callback=callback,
|
||||
lr_schedulers=[pi_lr_scheduler],
|
||||
)
|
||||
|
||||
# save model
|
||||
torch.save(policy.state_dict(), 'policy_final.pt')
|
||||
|
||||
if __name__ == '__main__':
|
||||
import argparse
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument('--train', choices=['A', 'B'])
|
||||
parser.add_argument('--epochs', type=int, default=200)
|
||||
parser.add_argument('--test', type=str, help='path to config file to run final training on')
|
||||
parser.add_argument('--test_seeds', type=int, default=5)
|
||||
parser.add_argument('--test_cpus', type=int, help='number of cpus available to split test seed training over')
|
||||
args = parser.parse_args()
|
||||
|
||||
assert (args.train is None) ^ (args.test is None), 'Must either train on an experiment or test with a config file'
|
||||
|
||||
# if no test config specified, train
|
||||
if args.test is None:
|
||||
print('Running Tuning for Experiment %s'%(args.train))
|
||||
analysis = tune.run(
|
||||
training_function,
|
||||
config={
|
||||
'experiment': args.train,
|
||||
'trainenv': {
|
||||
'stop_on_collision': False,
|
||||
'use_idm': True,
|
||||
},
|
||||
'policy': {
|
||||
'learning_rate': 3e-4,
|
||||
'learning_rate_decay': 1.0,
|
||||
'clip_ratio': 0.2,
|
||||
'iterations_per_epoch': 100,
|
||||
'hidden_layer_size': tune.grid_search([20, 40]),
|
||||
'n_hidden_layers': tune.grid_search([2, 3]),
|
||||
'activation':0,
|
||||
},
|
||||
'value': {
|
||||
'learning_rate': 1e-4,
|
||||
'weight_decay': 1e-3,
|
||||
'iterations_per_epoch': 1000,
|
||||
},
|
||||
'discriminator': {
|
||||
'learning_rate': 1e-3,
|
||||
'weight_decay': 1e-4,
|
||||
'iterations_per_epoch': 100,
|
||||
'n_hidden_layers_element': tune.grid_search([3,4]),
|
||||
'n_hidden_layers_global': tune.grid_search([1,2]),
|
||||
'hidden_layer_size': 10,
|
||||
'activation': 0,
|
||||
},
|
||||
'train_epochs': args.epochs,
|
||||
'seed': 0,
|
||||
}
|
||||
)
|
||||
best_config = analysis.get_best_config(metric='gen_collision_rate', mode='min')
|
||||
print('Best config: ', best_config)
|
||||
|
||||
# safe best_config
|
||||
if not os.path.isdir(os.path.join(DIR, 'best_configs')):
|
||||
os.mkdir(os.path.join(DIR, 'best_configs'))
|
||||
|
||||
# save gail
|
||||
with open(os.path.join(DIR, 'best_configs',f'gail_exp{args.train}.json'), 'w', encoding='utf-8') as f:
|
||||
json.dump(best_config, f, ensure_ascii=False, indent=4)
|
||||
|
||||
# if config file specified, rerun it with appropriate number of seeds
|
||||
else:
|
||||
with open(args.test, 'rb') as f:
|
||||
config = json.load(f)
|
||||
|
||||
print(f'Retraining {args.test} with {args.test_seeds} seeds on experiment {config["experiment"]}')
|
||||
|
||||
# rerun with appropriate number of seeds
|
||||
rpt = {'cpu': int(args.test_cpus/args.test_seeds)} if (args.test_cpus is not None) else None
|
||||
config['seed'] = tune.grid_search(list(range(1,args.test_seeds+1)))
|
||||
analysis = tune.run(training_function, config=config, resources_per_trial=rpt)
|
||||
|
||||
# move final policies to appropriate directory
|
||||
split_ = os.path.basename(args.test).split('_')
|
||||
model = split_[0]
|
||||
exper = split_[-1].split('.')[0]
|
||||
savepath = os.path.join('test_policies',model,exper)
|
||||
|
||||
if not os.path.isdir(savepath):
|
||||
os.makedirs(savepath)
|
||||
|
||||
import shutil
|
||||
for i in range(args.test_seeds):
|
||||
s = analysis._checkpoints[i]['config']['seed']
|
||||
check_dir = analysis._checkpoints[i]['logdir']
|
||||
shutil.copyfile(os.path.join(check_dir,'policy_final.pt'),
|
||||
os.path.join(savepath, f'policy_seed{s}.pt'))
|
||||
shutil.copyfile(os.path.join(check_dir,'params.json'),
|
||||
os.path.join(savepath, 'config.json')) # copy config automatically
|
||||
10
generate_demos.sh
Executable file
10
generate_demos.sh
Executable file
@@ -0,0 +1,10 @@
|
||||
#DEFAULT PARAMETERS:
|
||||
# locs:list=None, (default to all locations)
|
||||
# tracks:list=None, (default to all tracks)
|
||||
# env_class:str='NRasterizedIncrementingAgent',
|
||||
# env_args:dict={width:36,height:36,m_per_px:2},
|
||||
# expert_class:str='NRasterizedRouteIncrementingAgent',
|
||||
# expert_args:dict={mu:0.001}):
|
||||
|
||||
# python -m src.data.expert --locs='[DR_USA_Roundabout_FT]' --tracks='[0]'
|
||||
python -m src.data.expert --locs='[DR_USA_Roundabout_FT]' --tracks='[0]'
|
||||
@@ -1,20 +0,0 @@
|
||||
# can add --skip_running if you've already run the saved policies through the test environments and have appropriate
|
||||
# metrics in the out folder. Doing so will generate average metrics quickly.
|
||||
|
||||
# Experiment A
|
||||
python -m eval_experiments
|
||||
python -m eval_experiments --method expert_agent --save_videos --first_seed_only
|
||||
python -m eval_experiments --method idm --save_videos --first_seed_only
|
||||
python -m eval_experiments --method bc --folder='test_policies/bc/expA' --save_videos --first_seed_only
|
||||
python -m eval_experiments --method gail --folder='test_policies/gail/expA' --save_videos --first_seed_only
|
||||
python -m eval_experiments --method hail --folder='test_policies/hail/expA' --save_videos --first_seed_only
|
||||
python -m eval_experiments --method shail --folder='test_policies/shail/expA' --save_videos --first_seed_only
|
||||
|
||||
# Experiment B
|
||||
python -m eval_experiments --locations='[(0,4)]'
|
||||
python -m eval_experiments --method expert_agent --locations='[(0,4)]' --save_videos --first_seed_only
|
||||
python -m eval_experiments --method idm --locations='[(0,4)]' --save_videos --first_seed_only
|
||||
python -m eval_experiments --method bc --folder='test_policies/bc/expB' --locations='[(0,4)]' --save_videos --first_seed_only
|
||||
python -m eval_experiments --method gail --folder='test_policies/gail/expB' --locations='[(0,4)]' --save_videos --first_seed_only
|
||||
python -m eval_experiments --method hail --folder='test_policies/hail/expB' --locations='[(0,4)]' --save_videos --first_seed_only
|
||||
python -m eval_experiments --method shail --folder='test_policies/shail/expB' --locations='[(0,4)]' --save_videos --first_seed_only
|
||||
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
@@ -1,65 +0,0 @@
|
||||
import torch
|
||||
import functools
|
||||
from src.core.sampling import rollout_sb3
|
||||
from intersim.envs import IntersimpleLidarFlatIncrementingAgent
|
||||
from intersim.envs.intersimple import speed_reward
|
||||
from intersim.expert import NormalizedIntersimpleExpert
|
||||
from src.util.wrappers import CollisionPenaltyWrapper, Setobs
|
||||
import numpy as np
|
||||
from gym.wrappers import TransformObservation
|
||||
|
||||
obs_min = np.array([
|
||||
[-1000, -1000, 0, -np.pi, -1e-1, 0.],
|
||||
[0, -np.pi, -20, -20, -np.pi, -1e-1],
|
||||
[0, -np.pi, -20, -20, -np.pi, -1e-1],
|
||||
[0, -np.pi, -20, -20, -np.pi, -1e-1],
|
||||
[0, -np.pi, -20, -20, -np.pi, -1e-1],
|
||||
[0, -np.pi, -20, -20, -np.pi, -1e-1],
|
||||
]).reshape(-1)
|
||||
|
||||
obs_max = np.array([
|
||||
[1000, 1000, 20, np.pi, 1e-1, 0.],
|
||||
[50, np.pi, 20, 20, np.pi, 1e-1],
|
||||
[50, np.pi, 20, 20, np.pi, 1e-1],
|
||||
[50, np.pi, 20, 20, np.pi, 1e-1],
|
||||
[50, np.pi, 20, 20, np.pi, 1e-1],
|
||||
[50, np.pi, 20, 20, np.pi, 1e-1],
|
||||
]).reshape(-1)
|
||||
|
||||
def main(track:int, loc:int=0):
|
||||
env = IntersimpleLidarFlatIncrementingAgent(
|
||||
loc=loc,
|
||||
track=track,
|
||||
n_rays=5,
|
||||
reward=functools.partial(
|
||||
speed_reward,
|
||||
collision_penalty=0
|
||||
),
|
||||
)
|
||||
|
||||
policy = NormalizedIntersimpleExpert(env, mu=0.001)
|
||||
|
||||
env = Setobs(TransformObservation(
|
||||
CollisionPenaltyWrapper(
|
||||
env,
|
||||
collision_distance=6, collision_penalty=100
|
||||
), lambda obs: (obs - obs_min) / (obs_max - obs_min + 1e-10)
|
||||
))
|
||||
print(env.nv, 'vehicles')
|
||||
expert_data = rollout_sb3(env, policy, n_episodes=150, max_steps_per_episode=200)
|
||||
|
||||
states, actions, rewards, dones = expert_data
|
||||
print(f'Expert mean episode length {(~dones).sum() / states.shape[0]}')
|
||||
print(f'Expert mean reward per episode {rewards[~dones].sum() / states.shape[0]}')
|
||||
print(f'Observation mean', states[~dones].mean(0))
|
||||
print(f'Observation std', states[~dones].std(0))
|
||||
|
||||
torch.save(expert_data, f'intersimple-expert-data-setobs2-loc{loc}-track{track}.pt')
|
||||
|
||||
def loop(tracks:list=[0]):
|
||||
for track in tracks:
|
||||
main(track)
|
||||
|
||||
if __name__=='__main__':
|
||||
import fire
|
||||
fire.Fire(loop)
|
||||
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user