updating experiment B vecenv

README.md

@@ -1,22 +1,10 @@
 # InteractionImitation
-Imitation Learning with the [Interaction Dataset](https://interaction-dataset.com/) via the [InteractionSimulator](https://github.com/sisl/InteractionSimulator) gym environments.
+Imitation Learning with the INTERACTION Dataset
-
-Code for "[SHAIL: Safety-Aware Hierarchical Adversarial Imitation Learning for Autonomous Driving in Urban Environments](https://arxiv.org/abs/2204.01922)".
-If you find this repository useful, please cite the paper:
-
-```
-@article{jamgochian2022shail,
-  author = {Arec Jamgochian and Etienne Buehrle and Johannes Fischer and Mykel J. Kochenderfer},
-  title = {{SHAIL}: Safety-Aware Hierarchical Adversarial Imitation Learning for Autonomous Driving in Urban Environments},
-  journal = {arXiv:2204.01922 [cs]},
-  year = {2022}
-}
-```
 
 ## Getting started
-Clone the `InteractionSimulator` with the `shail` tag and pip install the module.
+Clone InteractionSimulator and pip install the module.
 ```
-git clone --branch shail https://github.com/sisl/InteractionSimulator.git
+git clone https://github.com/sisl/InteractionSimulator.git
 cd InteractionSimulator
 pip install -e .
 cd ..
@@ -31,28 +19,53 @@ The INTERACTION dataset contains a two folders which should be copied into a fol
   - the contents of `recorded_trackfiles` should be copied to `./InteractionSimulator/datasets/trackfiles`
   - the contents of `maps` should be copied to `./InteractionSimulator/datasets/maps`
 
-## Processing and saving expert demos
+## Processing, saving, and loading expert demos
-Once the repository has been set up, you need to generate two separate sets of expert demos for tracks 0-4. The first command generates true joint and individual states and actions necessary for evaluating, saving them in `expert_data/`. The second command generates trajectory rollouts according to individual agent observations, which is later used as expert data for the learning models.
+Once the repository has been set up, you can process and save expert track demonstrations with:
 ```
-python -m src.expert --locs='[DR_USA_Roundabout_FT]' --tracks='[0,1,2,3,4]'
+python src/expert_data.py --loc [LOCNUM] --track [TRACKNUM]
-python -m intersimple-expert-rollout-setobs2 --tracks='[0,1,2,3,4]'
+```
+You can (and should) process all tracks at once at location 0 with:
+```
+python src/expert_data.py --all-tracks
+```
+
+You can then train a default behavior cloning policy with the following. Be sure to check help for main.py for running options.
+```
+python src/main.py --train
+```
+You can run tensorboard by running the following and opening `localhost:6006` (or alternatively port-forwarding 6006 from the remote server)
+```
+tensorboard --logdir output/
+```
+You can then test the learned policy with the following, and see the animation file in `output/`:
+```
+python src/main.py --test
 ```
 
 
-## Tuning hyperparameters and training finalized models
+You can load the experts actions manually
-To tune models, we use `ray[tune]` grid searches. You can run see the commands we used to train in the top half of `train_models.sh`, as well as the hyperparameters we search over in `bc-experiment.py`, `gail-experiment.py`, and `shail-experiment.py`. After training the models, configurations get saved in `best_configs/` (the best SHAIL confg gets copied to a HAIL config, with the appropriate environment parameters changed for ablation). However, upon manual inspection of the training runs, we note some better performance than the automatically-set configs at earlier epochs, so we adjust the `best_configs` manually.
+```
-
+from src import expert_data
-After the `best_configs/` are set, we rerun each configuration with multiple seeds. The commands to do so are in the bottom half of `train_models.sh`. This saves different learned policy files to `test_policies/`.
+observations, actions = expert_data.load_expert_data(loc = [LOCNUM], track = [TRACKNUM])
-
+for (s, a) in zip (observations, actions):
-
+    # do some imitation learning
-## Evaluating models
+```
-To evaluate the learned policies, we rerun each model in particular setting, evaluate all our metrics, and average over different trained model seeds. The commands to do so are in `evaluate_models.sh`. 
 
 
 ## Package Structure
 ```
 InteractionImitation
-|- TODO
+|- demos
+|- algorithms
+   |- BC
+   |- AdVIL
+|- nets
+   |- Encoder
+   |- DeepSet
+   |- Decoder
+|- policies
+|- discriminators
+|- demo_generators
 ```
 
 ## Type Definitions

bc-experiment.py

@@ -55,7 +55,6 @@ def training_function(config):
                 ),
                 check_collisions=True,
                 stop_on_collision=config['trainenv']['stop_on_collision'],
-                use_idm=config['trainenv']['use_idm'],
             ), collision_distance=6, collision_penalty=100),
             lambda obs: (obs - obs_min) / (obs_max - obs_min + 1e-10)
         )) for _ in range(60)]
@@ -69,8 +68,6 @@ def training_function(config):
                 ),
                 check_collisions=True,
                 stop_on_collision=config['trainenv']['stop_on_collision'],
-                use_idm=config['trainenv']['use_idm'],
-                track=track,
             ), collision_distance=6, collision_penalty=100),
             lambda obs: (obs - obs_min) / (obs_max - obs_min + 1e-10)
         )) for _ in range(15)] for track in range(4)],[])
@@ -145,7 +142,7 @@ if __name__ == '__main__':
     import argparse
     parser = argparse.ArgumentParser()
     parser.add_argument('--train', choices=['A', 'B'])
-    parser.add_argument('--epochs', type=int, default=500)
+    parser.add_argument('--epochs', type=int, default=1000)
     parser.add_argument('--test', type=str, help='path to config file to run final training on')
     parser.add_argument('--test_seeds', type=int, default=5)
     parser.add_argument('--test_cpus', type=int, help='number of cpus available to split test seed training over')
@@ -162,14 +159,13 @@ if __name__ == '__main__':
                 'experiment': args.train,
                 'trainenv': {
                     'stop_on_collision': False, 
-                    'use_idm':True,
                 },
                 'policy': {
                     'learning_rate': 3e-4, 
-                    'learning_rate_decay': tune.grid_search([0.999, 1.0]), 
+                    'learning_rate_decay': tune.grid_search([0.001, 1.0]), 
-                    'hidden_layer_size': tune.grid_search([10, 20, 40]),
+                    'hidden_layer_size': tune.grid_search([10, 20, 40, 80]),
-                    'n_hidden_layers': tune.grid_search([2, 3]), 
+                    'n_hidden_layers': tune.grid_search([2, 3, 4]), 
-                    'activation':tune.grid_search([0, 1]),
+                    'activation':0,
                 },
                 'train_epochs': args.epochs,
                 'seed': 0,
@@ -214,5 +210,3 @@ if __name__ == '__main__':
             check_dir = analysis._checkpoints[i]['logdir']
             shutil.copyfile(os.path.join(check_dir,'policy_final.pt'), 
                 os.path.join(savepath, f'policy_seed{s}.pt'))
-            shutil.copyfile(os.path.join(check_dir,'params.json'),
-                os.path.join(savepath, 'config.json')) # copy config automatically

best_configs/bc_expA.json

@@ -1,8 +1,7 @@
 {
     "experiment": "A",
     "trainenv": {
-        "stop_on_collision": false,
+        "stop_on_collision": false
-        "use_idm": true
     },
     "policy": {
         "learning_rate": 0.0003,

best_configs/bc_expB.json

@@ -1,8 +1,7 @@
 {
     "experiment": "B",
     "trainenv": {
-        "stop_on_collision": false,
+        "stop_on_collision": false
-        "use_idm": true
     },
     "policy": {
         "learning_rate": 0.0003,

best_configs/gail-trpo_expA.json

@@ -0,0 +1,31 @@
+{
+    "experiment": "A",
+    "trainenv": {
+        "stop_on_collision": false
+    },
+    "policy": {
+        "learning_rate": 0.0003,
+        "learning_rate_decay": 1.0,
+        "delta": 0.01,
+        "iterations_per_epoch": 100,
+        "hidden_layer_size": 40,
+        "n_hidden_layers": 2,
+        "activation": 0
+    },
+    "value": {
+        "learning_rate": 0.0001,
+        "weight_decay": 0.001,
+        "iterations_per_epoch": 1000
+    },
+    "discriminator": {
+        "learning_rate": 0.001,
+        "weight_decay": 0.0001,
+        "iterations_per_epoch": 100,
+        "n_hidden_layers_element": 4,
+        "n_hidden_layers_global": 1,
+        "hidden_layer_size": 10,
+        "activation": 0
+    },
+    "train_epochs": 100,
+    "seed": 0
+}

best_configs/gail-trpo_expB.json

@@ -0,0 +1,31 @@
+{
+    "experiment": "B",
+    "trainenv": {
+        "stop_on_collision": false
+    },
+    "policy": {
+        "learning_rate": 0.0003,
+        "learning_rate_decay": 1.0,
+        "delta": 0.01,
+        "iterations_per_epoch": 100,
+        "hidden_layer_size": 40,
+        "n_hidden_layers": 2,
+        "activation": 0
+    },
+    "value": {
+        "learning_rate": 0.0001,
+        "weight_decay": 0.001,
+        "iterations_per_epoch": 1000
+    },
+    "discriminator": {
+        "learning_rate": 0.001,
+        "weight_decay": 0.0001,
+        "iterations_per_epoch": 100,
+        "n_hidden_layers_element": 4,
+        "n_hidden_layers_global": 1,
+        "hidden_layer_size": 10,
+        "activation": 0
+    },
+    "train_epochs": 100,
+    "seed": 0
+}

best_configs/gail_expA.json

@@ -1,8 +1,7 @@
 {
     "experiment": "A",
     "trainenv": {
-        "stop_on_collision": false,
+        "stop_on_collision": false
-        "use_idm": true
     },
     "policy": {
         "learning_rate": 0.0003,

best_configs/gail_expB.json

@@ -1,8 +1,7 @@
 {
     "experiment": "B",
     "trainenv": {
-        "stop_on_collision": false,
+        "stop_on_collision": false
-        "use_idm": true
     },
     "policy": {
         "learning_rate": 0.0003,

best_configs/hail-etienne2_expA.json

@@ -0,0 +1,33 @@
+{
+    "experiment": "A",
+    "trainenv": {
+        "stop_on_collision": false,
+        "safe_actions_collision_method": null,
+        "abort_unsafe_collision_method": null
+    },
+    "policy": {
+        "learning_rate": 0.0003,
+        "learning_rate_decay": 1.0,
+        "clip_ratio": 0.2,
+        "iterations_per_epoch": 100,
+        "hidden_layer_size": 10,
+        "n_hidden_layers": 3,
+        "activation": 0,
+        "option": 0
+    },
+    "value": {
+        "learning_rate": 0.001,
+        "iterations_per_epoch": 1000
+    },
+    "discriminator": {
+        "learning_rate": 0.001,
+        "weight_decay": 0.0001,
+        "iterations_per_epoch": 100,
+        "n_hidden_layers_element": 3,
+        "n_hidden_layers_global": 2,
+        "hidden_layer_size": 10,
+        "activation": 0
+    },
+    "train_epochs": 100,
+    "seed": 0
+}

best_configs/hail-etienne2_expB.json

@@ -0,0 +1,33 @@
+{
+    "experiment": "B",
+    "trainenv": {
+        "stop_on_collision": false,
+        "safe_actions_collision_method": null,
+        "abort_unsafe_collision_method": null
+    },
+    "policy": {
+        "learning_rate": 0.0003,
+        "learning_rate_decay": 1.0,
+        "clip_ratio": 0.2,
+        "iterations_per_epoch": 100,
+        "hidden_layer_size": 10,
+        "n_hidden_layers": 3,
+        "activation": 0,
+        "option": 0
+    },
+    "value": {
+        "learning_rate": 0.001,
+        "iterations_per_epoch": 1000
+    },
+    "discriminator": {
+        "learning_rate": 0.001,
+        "weight_decay": 0.0001,
+        "iterations_per_epoch": 100,
+        "n_hidden_layers_element": 3,
+        "n_hidden_layers_global": 2,
+        "hidden_layer_size": 10,
+        "activation": 0
+    },
+    "train_epochs": 100,
+    "seed": 0
+}

best_configs/hail-etienne3_expA.json

@@ -0,0 +1,33 @@
+{
+    "experiment": "A",
+    "trainenv": {
+        "stop_on_collision": false,
+        "safe_actions_collision_method": null,
+        "abort_unsafe_collision_method": null
+    },
+    "policy": {
+        "learning_rate": 0.0003,
+        "learning_rate_decay": 1.0,
+        "clip_ratio": 0.2,
+        "iterations_per_epoch": 100,
+        "hidden_layer_size": 40,
+        "n_hidden_layers": 3,
+        "activation": 0,
+        "option": 0
+    },
+    "value": {
+        "learning_rate": 0.001,
+        "iterations_per_epoch": 1000
+    },
+    "discriminator": {
+        "learning_rate": 0.001,
+        "weight_decay": 0.0001,
+        "iterations_per_epoch": 100,
+        "n_hidden_layers_element": 3,
+        "n_hidden_layers_global": 2,
+        "hidden_layer_size": 10,
+        "activation": 0
+    },
+    "train_epochs": 100,
+    "seed": 0
+}

best_configs/hail-etienne3_expB.json

@@ -0,0 +1,33 @@
+{
+    "experiment": "B",
+    "trainenv": {
+        "stop_on_collision": false,
+        "safe_actions_collision_method": null,
+        "abort_unsafe_collision_method": null
+    },
+    "policy": {
+        "learning_rate": 0.0003,
+        "learning_rate_decay": 1.0,
+        "clip_ratio": 0.2,
+        "iterations_per_epoch": 100,
+        "hidden_layer_size": 40,
+        "n_hidden_layers": 3,
+        "activation": 0,
+        "option": 0
+    },
+    "value": {
+        "learning_rate": 0.001,
+        "iterations_per_epoch": 1000
+    },
+    "discriminator": {
+        "learning_rate": 0.001,
+        "weight_decay": 0.0001,
+        "iterations_per_epoch": 100,
+        "n_hidden_layers_element": 3,
+        "n_hidden_layers_global": 2,
+        "hidden_layer_size": 10,
+        "activation": 0
+    },
+    "train_epochs": 100,
+    "seed": 0
+}

best_configs/hail-etienne_expA.json

@@ -0,0 +1,33 @@
+{
+    "experiment": "A",
+    "trainenv": {
+        "stop_on_collision": false,
+        "safe_actions_collision_method": null,
+        "abort_unsafe_collision_method": null
+    },
+    "policy": {
+        "learning_rate": 0.0003,
+        "learning_rate_decay": 1.0,
+        "clip_ratio": 0.2,
+        "iterations_per_epoch": 100,
+        "hidden_layer_size": 20,
+        "n_hidden_layers": 4,
+        "activation": 0,
+        "option": 0
+    },
+    "value": {
+        "learning_rate": 0.001,
+        "iterations_per_epoch": 1000
+    },
+    "discriminator": {
+        "learning_rate": 0.001,
+        "weight_decay": 0.0001,
+        "iterations_per_epoch": 100,
+        "n_hidden_layers_element": 3,
+        "n_hidden_layers_global": 2,
+        "hidden_layer_size": 10,
+        "activation": 0
+    },
+    "train_epochs": 100,
+    "seed": 0
+}

best_configs/hail-etienne_expB.json

@@ -0,0 +1,33 @@
+{
+    "experiment": "B",
+    "trainenv": {
+        "stop_on_collision": false,
+        "safe_actions_collision_method": null,
+        "abort_unsafe_collision_method": null
+    },
+    "policy": {
+        "learning_rate": 0.0003,
+        "learning_rate_decay": 1.0,
+        "clip_ratio": 0.2,
+        "iterations_per_epoch": 100,
+        "hidden_layer_size": 20,
+        "n_hidden_layers": 4,
+        "activation": 0,
+        "option": 0
+    },
+    "value": {
+        "learning_rate": 0.001,
+        "iterations_per_epoch": 1000
+    },
+    "discriminator": {
+        "learning_rate": 0.001,
+        "weight_decay": 0.0001,
+        "iterations_per_epoch": 100,
+        "n_hidden_layers_element": 3,
+        "n_hidden_layers_global": 2,
+        "hidden_layer_size": 10,
+        "activation": 0
+    },
+    "train_epochs": 100,
+    "seed": 0
+}

best_configs/hail_expA.json

@@ -3,8 +3,7 @@
     "trainenv": {
         "stop_on_collision": false,
         "safe_actions_collision_method": null,
-        "abort_unsafe_collision_method": null,
+        "abort_unsafe_collision_method": null
-        "use_idm": true
     },
     "policy": {
         "learning_rate": 0.0003,

best_configs/hail_expB.json

@@ -3,8 +3,7 @@
     "trainenv": {
         "stop_on_collision": false,
         "safe_actions_collision_method": null,
-        "abort_unsafe_collision_method": null,
+        "abort_unsafe_collision_method": null
-        "use_idm": true
     },
     "policy": {
         "learning_rate": 0.0003,

best_configs/shail-etienne2_expA.json

@@ -0,0 +1,33 @@
+{
+    "experiment": "A",
+    "trainenv": {
+        "stop_on_collision": false,
+        "safe_actions_collision_method": "circle",
+        "abort_unsafe_collision_method": "circle"
+    },
+    "policy": {
+        "learning_rate": 0.0003,
+        "learning_rate_decay": 1.0,
+        "clip_ratio": 0.2,
+        "iterations_per_epoch": 100,
+        "hidden_layer_size": 10,
+        "n_hidden_layers": 3,
+        "activation": 0,
+        "option": 0
+    },
+    "value": {
+        "learning_rate": 0.001,
+        "iterations_per_epoch": 1000
+    },
+    "discriminator": {
+        "learning_rate": 0.001,
+        "weight_decay": 0.0001,
+        "iterations_per_epoch": 100,
+        "n_hidden_layers_element": 3,
+        "n_hidden_layers_global": 2,
+        "hidden_layer_size": 10,
+        "activation": 0
+    },
+    "train_epochs": 100,
+    "seed": 0
+}

best_configs/shail-etienne2_expB.json

@@ -0,0 +1,33 @@
+{
+    "experiment": "B",
+    "trainenv": {
+        "stop_on_collision": false,
+        "safe_actions_collision_method": "circle",
+        "abort_unsafe_collision_method": "circle"
+    },
+    "policy": {
+        "learning_rate": 0.0003,
+        "learning_rate_decay": 1.0,
+        "clip_ratio": 0.2,
+        "iterations_per_epoch": 100,
+        "hidden_layer_size": 10,
+        "n_hidden_layers": 3,
+        "activation": 0,
+        "option": 0
+    },
+    "value": {
+        "learning_rate": 0.001,
+        "iterations_per_epoch": 1000
+    },
+    "discriminator": {
+        "learning_rate": 0.001,
+        "weight_decay": 0.0001,
+        "iterations_per_epoch": 100,
+        "n_hidden_layers_element": 3,
+        "n_hidden_layers_global": 2,
+        "hidden_layer_size": 10,
+        "activation": 0
+    },
+    "train_epochs": 100,
+    "seed": 0
+}

best_configs/shail-etienne3_expA.json

@@ -0,0 +1,33 @@
+{
+    "experiment": "A",
+    "trainenv": {
+        "stop_on_collision": false,
+        "safe_actions_collision_method": "circle",
+        "abort_unsafe_collision_method": "circle"
+    },
+    "policy": {
+        "learning_rate": 0.0003,
+        "learning_rate_decay": 1.0,
+        "clip_ratio": 0.2,
+        "iterations_per_epoch": 100,
+        "hidden_layer_size": 40,
+        "n_hidden_layers": 3,
+        "activation": 0,
+        "option": 0
+    },
+    "value": {
+        "learning_rate": 0.001,
+        "iterations_per_epoch": 1000
+    },
+    "discriminator": {
+        "learning_rate": 0.001,
+        "weight_decay": 0.0001,
+        "iterations_per_epoch": 100,
+        "n_hidden_layers_element": 3,
+        "n_hidden_layers_global": 2,
+        "hidden_layer_size": 10,
+        "activation": 0
+    },
+    "train_epochs": 100,
+    "seed": 0
+}

best_configs/shail-etienne3_expB.json

@@ -0,0 +1,33 @@
+{
+    "experiment": "B",
+    "trainenv": {
+        "stop_on_collision": false,
+        "safe_actions_collision_method": "circle",
+        "abort_unsafe_collision_method": "circle"
+    },
+    "policy": {
+        "learning_rate": 0.0003,
+        "learning_rate_decay": 1.0,
+        "clip_ratio": 0.2,
+        "iterations_per_epoch": 100,
+        "hidden_layer_size": 40,
+        "n_hidden_layers": 3,
+        "activation": 0,
+        "option": 0
+    },
+    "value": {
+        "learning_rate": 0.001,
+        "iterations_per_epoch": 1000
+    },
+    "discriminator": {
+        "learning_rate": 0.001,
+        "weight_decay": 0.0001,
+        "iterations_per_epoch": 100,
+        "n_hidden_layers_element": 3,
+        "n_hidden_layers_global": 2,
+        "hidden_layer_size": 10,
+        "activation": 0
+    },
+    "train_epochs": 100,
+    "seed": 0
+}

best_configs/shail-etienne4_expA.json

@@ -0,0 +1,33 @@
+{
+    "experiment": "A",
+    "trainenv": {
+        "stop_on_collision": false,
+        "safe_actions_collision_method": "circle",
+        "abort_unsafe_collision_method": "circle"
+    },
+    "policy": {
+        "learning_rate": 0.0003,
+        "learning_rate_decay": 1.0,
+        "clip_ratio": 0.2,
+        "iterations_per_epoch": 500,
+        "hidden_layer_size": 40,
+        "n_hidden_layers": 3,
+        "activation": 0,
+        "option": 0
+    },
+    "value": {
+        "learning_rate": 0.001,
+        "iterations_per_epoch": 1000
+    },
+    "discriminator": {
+        "learning_rate": 0.001,
+        "weight_decay": 0.0001,
+        "iterations_per_epoch": 100,
+        "n_hidden_layers_element": 3,
+        "n_hidden_layers_global": 2,
+        "hidden_layer_size": 10,
+        "activation": 0
+    },
+    "train_epochs": 150,
+    "seed": 0
+}

best_configs/shail-etienne_expA.json

@@ -0,0 +1,33 @@
+{
+    "experiment": "A",
+    "trainenv": {
+        "stop_on_collision": false,
+        "safe_actions_collision_method": "circle",
+        "abort_unsafe_collision_method": "circle"
+    },
+    "policy": {
+        "learning_rate": 0.0003,
+        "learning_rate_decay": 1.0,
+        "clip_ratio": 0.2,
+        "iterations_per_epoch": 100,
+        "hidden_layer_size": 20,
+        "n_hidden_layers": 4,
+        "activation": 0,
+        "option": 0
+    },
+    "value": {
+        "learning_rate": 0.001,
+        "iterations_per_epoch": 1000
+    },
+    "discriminator": {
+        "learning_rate": 0.001,
+        "weight_decay": 0.0001,
+        "iterations_per_epoch": 100,
+        "n_hidden_layers_element": 3,
+        "n_hidden_layers_global": 2,
+        "hidden_layer_size": 10,
+        "activation": 0
+    },
+    "train_epochs": 100,
+    "seed": 0
+}

best_configs/shail-etienne_expB.json

@@ -0,0 +1,33 @@
+{
+    "experiment": "B",
+    "trainenv": {
+        "stop_on_collision": false,
+        "safe_actions_collision_method": "circle",
+        "abort_unsafe_collision_method": "circle"
+    },
+    "policy": {
+        "learning_rate": 0.0003,
+        "learning_rate_decay": 1.0,
+        "clip_ratio": 0.2,
+        "iterations_per_epoch": 100,
+        "hidden_layer_size": 20,
+        "n_hidden_layers": 4,
+        "activation": 0,
+        "option": 0
+    },
+    "value": {
+        "learning_rate": 0.001,
+        "iterations_per_epoch": 1000
+    },
+    "discriminator": {
+        "learning_rate": 0.001,
+        "weight_decay": 0.0001,
+        "iterations_per_epoch": 100,
+        "n_hidden_layers_element": 3,
+        "n_hidden_layers_global": 2,
+        "hidden_layer_size": 10,
+        "activation": 0
+    },
+    "train_epochs": 100,
+    "seed": 0
+}

best_configs/shail_expA.json

@@ -3,8 +3,7 @@
     "trainenv": {
         "stop_on_collision": false,
         "safe_actions_collision_method": "circle",
-        "abort_unsafe_collision_method": "circle",
+        "abort_unsafe_collision_method": "circle"
-        "use_idm": true
     },
     "policy": {
         "learning_rate": 0.0003,

best_configs/shail_expB.json

@@ -3,8 +3,7 @@
     "trainenv": {
         "stop_on_collision": false,
         "safe_actions_collision_method": "circle",
-        "abort_unsafe_collision_method": "circle",
+        "abort_unsafe_collision_method": "circle"
-        "use_idm": true
     },
     "policy": {
         "learning_rate": 0.0003,

config/networks.json5

@@ -0,0 +1,42 @@
+{
+    ego_encoder: {
+        input_dim: 5, // number of state vars
+        hidden_n: 0,
+        hidden_dim: 5,
+        output_dim: 5
+    },
+    deepsets: {
+        input_dim: 6, // number of relative state vars for others
+        phi: {
+            hidden_n: 2,
+            hidden_dim: 20,
+        },
+        latent_dim: 20,
+        rho: {
+            hidden_n: 2,
+            hidden_dim: 10,
+        },
+        output_dim: 10
+    },
+    path_encoder: {
+        input_dim: 40, // 2 * path length for (x,y) coordinates
+        hidden_n: 0,
+        hidden_dim: 20,
+        output_dim: 10,
+    },
+    head: {
+        input_dim: 0, // computed in policy constructor
+        hidden_n: 3,
+        hidden_dim: 50,
+        output_dim: 1, // number of outputs e.g. number of actions, or just one
+        final_activation: 'sigmoid',
+    },
+    optim: {
+            optimizer: 'adam',
+            lr: 1e-3,
+            weight_decay: 0.1,
+    },
+    train_epochs: 200,
+    train_batch_size: 32,
+    loss: 'huber',
+}

config/value_dice.json5

@@ -0,0 +1,85 @@
+{
+    policy_net: {
+        ego_encoder: {
+            input_dim: 5, // number of state vars
+            hidden_n: 0,
+            hidden_dim: 5,
+            output_dim: 5
+        },
+        deepsets: {
+            input_dim: 6, // number of relative state vars for others
+            phi: {
+                hidden_n: 2,
+                hidden_dim: 20,
+            },
+            latent_dim: 20,
+            rho: {
+                hidden_n: 2,
+                hidden_dim: 10,
+            },
+            output_dim: 10
+        },
+        path_encoder: {
+            input_dim: 40, // 2 * path length for (x,y) coordinates
+            hidden_n: 0,
+            hidden_dim: 20,
+            output_dim: 10,
+        },
+        head: {
+            input_dim: 0, // computed in policy constructor
+            hidden_n: 3,
+            hidden_dim: 50,
+            output_dim: 1, // number of outputs e.g. number of actions, or just one
+            final_activation: 'sigmoid',
+        },
+    },
+    value_net: {
+        ego_encoder: {
+            input_dim: 5, // number of state vars
+            hidden_n: 0,
+            hidden_dim: 5,
+            output_dim: 5
+        },
+        deepsets: {
+            input_dim: 6, // number of relative state vars for others
+            phi: {
+                hidden_n: 2,
+                hidden_dim: 20,
+            },
+            latent_dim: 20,
+            rho: {
+                hidden_n: 2,
+                hidden_dim: 10,
+            },
+            output_dim: 10
+        },
+        path_encoder: {
+            input_dim: 40, // 2 * path length for (x,y) coordinates
+            hidden_n: 0,
+            hidden_dim: 20,
+            output_dim: 10,
+        },
+        action_dim: 1, // number of actions
+        head: {
+            input_dim: 0, // computed in policy constructor
+            hidden_n: 3,
+            hidden_dim: 50,
+            output_dim: 1, // number of outputs e.g. number of actions, or just one
+            final_activation: 'id',
+        },
+    },
+    policy_optim: {
+            optimizer: 'adam',
+            lr: 1e-3,
+            weight_decay: 0.1,
+    },
+    value_optim: {
+            optimizer: 'adam',
+            lr: 1e-3,
+            weight_decay: 0.1,
+    },
+    train_epochs: 200,
+    train_batch_size: 32,
+    discount: 0.95,
+    clip_grad_norm: 1.,
+}

eval_experiments.py

@@ -12,16 +12,16 @@ def main(method:str='expert', folder:str=None, locations=[(0,0)], skip_running=F
     policy_kwargs = {}
 
     if method in ['expert', 'idm']:
-        env, env_kwargs ='NRasterizedRouteIncrementingAgent', {'use_idm':True}
+        env, env_kwargs ='NRasterizedRouteIncrementingAgent', {}
     elif method in ['bc','gail']:
         env='NormalizedContinuousEvalEnv' 
-        env_kwargs={'stop_on_collision':True, 'max_episode_steps':1000, 'use_idm':True}
+        env_kwargs={'stop_on_collision':True, 'max_episode_steps':1000}
     elif method in ['hail']:
         env = 'NormalizedSafeOptionsEvalEnv'
-        env_kwargs={'stop_on_collision':True, 'max_episode_steps':1000, 'safe_actions_collision_method': None, 'abort_unsafe_collision_method': None, 'use_idm':True}
+        env_kwargs={'stop_on_collision':True, 'max_episode_steps':1000, 'safe_actions_collision_method': None, 'abort_unsafe_collision_method': None}
     elif method in ['shail']:
         env = 'NormalizedSafeOptionsEvalEnv'
-        env_kwargs={'stop_on_collision':True, 'max_episode_steps':1000, 'use_idm':True}
+        env_kwargs={'stop_on_collision':True, 'max_episode_steps':1000}
     else:
         raise NotImplementedError
 
@@ -35,12 +35,12 @@ def main(method:str='expert', folder:str=None, locations=[(0,0)], skip_running=F
         print('%i policy files found in %s folder' %(len(files), folder))
         print('found policy config', config['policy'])
 
-        policy_config = {k: v for k, v in config['policy'].items() if k not in exclude_keys_from_policy_kwargs}
+    policy_config = {k: v for k, v in config['policy'].items() if k not in exclude_keys_from_policy_kwargs}
-        policy_config['activation'] = activations[policy_config['activation']]
+    policy_config['activation'] = activations[policy_config['activation']]
-        print('final policy config', policy_config)
+    print('final policy config', policy_config)
 
-        policy_kwargs.update(policy_config)
+    policy_kwargs.update(policy_config)
-        print('final policy kwargs', policy_kwargs)
+    print('final policy kwargs', policy_kwargs)
 
     if not skip_running:
         for policy_file in files:
@@ -77,7 +77,7 @@ def latex_print(am, light=False):
     print('success rate, distance travelled, RWSE_10, |DeltaV|, AccelJSD')
     if light:
         if 'rwse_10s' in am.keys():
-            print("%2.1f& %2.1f & %2.1f & %1.2f& "
+            print("%2.1f& %2.1f & %1.2f & %2.1f& "
             "%0.3f \\\\" %( 100*am['success rate'][0], am['mean travel distance'][0], am['rwse_10s'][0],
                             am['average absolute average velocity'][0],am['acceleration distribution divergence'][0] ))
             return
@@ -88,7 +88,7 @@ def latex_print(am, light=False):
         return
     
     print("%2.1f \\scriptstyle\\pm %2.1f & %2.1f \\scriptstyle\\pm %2.1f & "
-        "%2.1f \\scriptstyle\\pm %1.1f & %1.2f \\scriptstyle\\pm %1.2f & "
+        "%1.2f \\scriptstyle\\pm %1.2f & %2.1f \\scriptstyle\\pm %1.1f & "
         "%0.3f \\scriptstyle\\pm %0.3f \\\\" %( 100*am['success rate'][0], 100*am['success rate'][1],
                am['mean travel distance'][0] , am['mean travel distance'][1]  ,
                am['rwse_10s'][0] , am['rwse_10s'][1]  ,

evaluate_models.sh

@@ -1,18 +1,19 @@
-# can add --skip_running if you've already run the saved policies through the test environments and have appropriate
+# can add --skip_running if you've run the runs before on the saved policies
-# metrics in the out folder. Doing so will generate average metrics quickly.
 
-# Experiment A
 python -m eval_experiments 
-python -m eval_experiments --method idm 
-python -m eval_experiments --method bc --folder='test_policies/bc/expA' 
-python -m eval_experiments --method gail --folder='test_policies/gail/expA' 
-python -m eval_experiments --method hail --folder='test_policies/hail/expA'
-python -m eval_experiments --method shail --folder='test_policies/shail/expA' 
-
-# Experiment B
 python -m eval_experiments --locations='[(0,4)]' 
-python -m eval_experiments --method idm --locations='[(0,4)]' --skip_running
+python -m eval_experiments --method idm 
-python -m eval_experiments --method bc --folder='test_policies/bc/expB' --locations='[(0,4)]' --skip_running
+python -m eval_experiments --method idm --locations='[(0,4)]' 
-python -m eval_experiments --method gail --folder='test_policies/gail/expB' --locations='[(0,4)]' --skip_running
+python -m eval_experiments --method bc --folder='test_policies/bc/expA' 
-python -m eval_experiments --method hail --folder='test_policies/hail/expB' --locations='[(0,4)]' --skip_running
+python -m eval_experiments --method bc --folder='test_policies/bc/expB' --locations='[(0,4)]' 
-python -m eval_experiments --method shail --folder='test_policies/shail/expB' --locations='[(0,4)]' --skip_running
+python -m eval_experiments --method gail --folder='test_policies/gail/expA' 
+python -m eval_experiments --method gail --folder='test_policies/gail/expB' --locations='[(0,4)]' 
+python -m eval_experiments --method hail --folder='test_policies/hail/expA'
+python -m eval_experiments --method hail --folder='test_policies/hail/expB' --locations='[(0,4)]'
+python -m eval_experiments --method shail --folder='test_policies/shail/expA'
+python -m eval_experiments --method shail --folder='test_policies/shail/expB' --locations='[(0,4)]'
+
+python -m eval_experiments --method hail --folder='test_policies/hail-etienne/expA'
+python -m eval_experiments --method hail --folder='test_policies/hail-etienne/expB' --locations='[(0,4)]'
+python -m eval_experiments --method shail --folder='test_policies/shail-etienne/expA'
+python -m eval_experiments --method shail --folder='test_policies/shail-etienne/expB' --locations='[(0,4)]'

experiments/experiment.py

@@ -0,0 +1,203 @@
+import json5
+from functools import partial
+import os
+opj = os.path.join
+
+# set up ray tune
+import ray
+from ray import tune
+from ray.tune import Analysis, ExperimentAnalysis
+from ray.tune.schedulers import ASHAScheduler
+from hyperopt import hp
+from ray.tune.suggest.hyperopt import HyperOptSearch
+
+# get graphs
+import intersim
+from intersim.graphs import ConeVisibilityGraph
+
+
+from src.main import basestr, main
+
+def parse_args():
+    """
+    Parse arguments to main
+    Returns:
+        kwargs: dictionary of arguments:
+            train (bool): whether to run train loop
+            test (bool): whether to run test loop
+            method (str): the method to try for imitation
+            loc (int): the location index of the roundabout
+            config (str): config path
+            seed (int): RNG seed
+    """
+    import argparse
+    parser = argparse.ArgumentParser(description='Save Expert Trajectories')
+    parser.add_argument('--loc', default=0, type=int,
+        help='location (default 0)')
+    parser.add_argument("--train", help="train model",
+        action="store_true")
+    parser.add_argument("--ray", help="use ray tune to run multiple experiments",
+        action="store_true")
+    parser.add_argument("--test", help="test model",
+        action="store_true") 
+    parser.add_argument("--method", help="modeling method",
+        choices=['bc', 'gail', 'advil', 'vd'], default='bc')
+    parser.add_argument("--config", help="config file path",
+        default=None, type=str)
+    parser.add_argument('--seed', default=0, type=int,
+        help='seed')    
+    parser.add_argument('--nframes', default=500, type=int,
+        help='frames for test animation') 
+    parser.add_argument('--nsamples', default=200, type=int,
+        help='number of ray samples') 
+    parser.add_argument('--graph', action='store_true',
+        help='whether to mask the relative states based on a ConeVisibilityGraph')
+    parser.add_argument('-d', default='./expert_data', type=str,
+                       help='data directory')
+    parser.add_argument('-o', default=None, type=str,
+                       help='output directory')                   
+    args = parser.parse_args()
+    kwargs = {
+        'train':args.train, 
+        'test':args.test, 
+        'method':args.method,
+        'loc':args.loc,
+        'config_path':args.config,
+        'seed':args.seed,
+        'ray':args.ray,
+        'nframes':args.nframes,
+        'nsamples':args.nsamples,
+        'datadir':os.path.abspath(args.d),
+        'graph':None,
+        'outdir': opj('output',args.method,'loc%02i'%(args.loc)),
+        'train_tracks':[0,1,2],
+        'cv_tracks':[3],
+        'test_tracks':[4],
+    }
+    if args.o:
+        kwargs['outdir'] = args.o
+    if args.graph:
+        kwargs['graph'] = ConeVisibilityGraph(r=20, half_angle=120)
+    return kwargs
+
+def get_full_config(ray_config:dict, method:str)->dict:
+    """
+    Get full model configuration from ray config and method string
+    Args:
+        ray_config (dict): ray config
+        method (str): method to get full configuration for
+    """
+    if method == 'bc':
+        from src.bc import bc_config
+        config = bc_config(ray_config)
+    elif method == 'vd':
+        from src.value_dice import vd_config
+        config = vd_config(ray_config)
+    else:
+        raise NotImplementedError
+    return config
+
+def get_ray_config(method:str)->dict:
+    """
+    Get configuration for ray based on method.
+    Args: 
+        method (str): method to get configuration for
+    Returns:
+        ray_config (dict): configuration for ray
+    """
+    if method == 'bc':
+        ray_config = {
+            "lr": tune.loguniform(1e-5, 1e-3),
+            "weight_decay": tune.choice([0, 0.1]),
+            "loss": tune.choice(['huber', 'mse']),
+            "train_batch_size": tune.choice([16,32,64]),
+            "deepsets_phi_hidden_n": tune.randint(1,5),
+            "deepsets_phi_hidden_dim": tune.lograndint(8,65),
+            "deepsets_latent_dim": tune.lograndint(8,129),
+            "deepsets_rho_hidden_n": tune.randint(0,3),
+            "deepsets_rho_hidden_dim": tune.lograndint(8,129),
+            "deepsets_output_dim": tune.lograndint(4,129),
+            "head_hidden_n": tune.randint(1,6),
+            "head_hidden_dim": tune.lograndint(16,257),
+            "head_final_activation": tune.choice(['sigmoid', None]),
+        }
+    elif method == 'vd':
+        ray_config = {
+            "policy_lr": tune.loguniform(1e-5, 1e-3),
+            "value_lr": tune.loguniform(1e-5, 1e-3),
+            "policy_weight_decay": tune.choice([0, 0.1]),
+            "value_weight_decay": tune.choice([0, 0.1]),
+            "train_batch_size": tune.choice([16,32,64]),
+            "deepsets_phi_hidden_n": tune.randint(1,5),
+            "deepsets_phi_hidden_dim": tune.lograndint(8,65),
+            "deepsets_latent_dim": tune.lograndint(8,129),
+            "deepsets_rho_hidden_n": tune.randint(0,3),
+            "deepsets_rho_hidden_dim": tune.lograndint(8,129),
+            "deepsets_output_dim": tune.lograndint(4,129),
+            "head_hidden_n": tune.randint(1,6),
+            "head_hidden_dim": tune.lograndint(16,257),
+            "head_final_activation": tune.choice(['sigmoid', None]),
+            "clip_grad_norm": tune.choice([.5, 1., 5., 10.]),
+            "discount": tune.choice([.95, .99])
+        }
+    else:
+        raise NotImplementedError
+    return ray_config
+
+if __name__ == '__main__':
+    kwargs = parse_args()
+    
+    # make prefix of output files
+
+    if kwargs['config_path']:
+        # load config
+        with open(kwargs['config_path'], 'r') as cfg:
+            config = json5.load(cfg)
+        if not os.path.isdir(kwargs['outdir']):
+            os.makedirs(kwargs['outdir'])   
+        filestr = opj(kwargs['outdir'], basestr(**kwargs)) 
+        if kwargs['ray']:
+            filestr = kwargs['config_path'].replace('_config.json','')
+        main(config, filestr=filestr, **kwargs)
+
+    elif kwargs['ray'] and kwargs['train']:
+
+        ray.shutdown() 
+        ray.init(log_to_driver=False)
+
+        def ray_train(config, datadir=None):
+            full_config = get_full_config(config, kwargs['method'])
+            main(full_config, filestr='exp', **kwargs)
+        
+        ray_config = get_ray_config(kwargs['method'])
+        search = HyperOptSearch(ray_config, max_concurrent=8, metric='cv_loss',mode="min",)
+        custom_scheduler = ASHAScheduler(metric='cv_loss', mode="min", grace_period=15)
+
+        analysis = tune.run(
+            ray_train, 
+            #config=ray_config,
+            search_alg=search,
+            scheduler=custom_scheduler,
+            local_dir=kwargs['outdir'],
+            #resources_per_trial={"cpu": 2},
+            time_budget_s=120*60,
+            num_samples=kwargs['nsamples'],
+        )
+    elif kwargs['ray'] and kwargs['test']:
+        analysis = Analysis(kwargs['outdir'], default_metric="cv_loss", default_mode="min")
+        config = analysis.get_best_config()
+        filepath = analysis.get_best_logdir()
+        filestr = opj(filepath, 'exp')
+        config_path = filestr+'_config.json'
+        with open(config_path, 'r') as cfg:
+            config = json5.load(cfg)
+        print("Best ray experiment:", filepath)
+        main(config, filestr=filestr, **kwargs)
+    else:
+        raise Exception('No valid config found')
+
+    
+    
+    
+
+    

experiments/experiments.sh

@@ -0,0 +1,9 @@
+#!/bin/sh
+
+python experiments/experiment.py --ray --train -d ./expert_data/base
+python experiments/experiment.py --ray --test -d ./expert_data/base --nframes 1000
+python experiments/experiment.py --ray --train -d ./expert_data/reg
+python experiments/experiment.py --ray --test -d ./expert_data/reg --nframes 1000
+python experiments/experiment.py --ray --train -d ./expert_data/reg_graph --graph
+python experiments/experiment.py --ray --test -d ./expert_data/reg_graph --graph --nframes 1000
+

experiments/train_vd.sh

@@ -0,0 +1,5 @@
+#!/bin/sh
+
+# python experiments/experiment.py --method vd --train --ray -d expert_data/reg -o output/vd/loc00/reg --nsamples 400
+# python experiments/experiment.py --test --ray --method vd -d expert_data/normal -o output/vd/loc00/normal --nframes 1000
+python experiments/experiment.py --train --method vd --config config/value_dice.json5

gail-experiment.py

@@ -53,7 +53,6 @@ def training_function(config):
                 ),
                 check_collisions=True,
                 stop_on_collision=config['trainenv']['stop_on_collision'],
-                use_idm=config['trainenv']['use_idm'],
             ), collision_distance=6, collision_penalty=100),
             lambda obs: (obs - obs_min) / (obs_max - obs_min + 1e-10)
         )) for _ in range(60)]
@@ -68,7 +67,6 @@ def training_function(config):
                 ),
                 check_collisions=True,
                 stop_on_collision=config['trainenv']['stop_on_collision'],
-                use_idm=config['trainenv']['use_idm'],
                 track=track,
             ), collision_distance=6, collision_penalty=100),
             lambda obs: (obs - obs_min) / (obs_max - obs_min + 1e-10)
@@ -172,7 +170,6 @@ if __name__ == '__main__':
                 'experiment': args.train,
                 'trainenv': {
                     'stop_on_collision': False, 
-                    'use_idm': True, 
                 },
                 'policy': {
                     'learning_rate': 3e-4, 
@@ -239,5 +236,3 @@ if __name__ == '__main__':
             check_dir = analysis._checkpoints[i]['logdir']
             shutil.copyfile(os.path.join(check_dir,'policy_final.pt'), 
                 os.path.join(savepath, f'policy_seed{s}.pt'))
-            shutil.copyfile(os.path.join(check_dir,'params.json'),
-                os.path.join(savepath, 'config.json')) # copy config automatically

generate_demos.sh

@@ -0,0 +1,10 @@
+#DEFAULT PARAMETERS:
+# locs:list=None, (default to all locations)
+# tracks:list=None, (default to all tracks)
+# env_class:str='NRasterizedIncrementingAgent',
+# env_args:dict={width:36,height:36,m_per_px:2},
+# expert_class:str='NRasterizedRouteIncrementingAgent',  
+# expert_args:dict={mu:0.001}):
+
+# python -m src.data.expert --locs='[DR_USA_Roundabout_FT]' --tracks='[0]'
+python -m src.data.expert --locs='[DR_USA_Roundabout_FT]' --tracks='[0]'

intersimple-expert-rollout-setobs2.py

@@ -1,65 +0,0 @@
-import torch
-import functools
-from src.core.sampling import rollout_sb3
-from intersim.envs import IntersimpleLidarFlatIncrementingAgent
-from intersim.envs.intersimple import speed_reward
-from intersim.expert import NormalizedIntersimpleExpert
-from src.util.wrappers import CollisionPenaltyWrapper, Setobs
-import numpy as np
-from gym.wrappers import TransformObservation
-
-obs_min = np.array([
-    [-1000, -1000, 0, -np.pi, -1e-1, 0.],
-    [0, -np.pi, -20, -20, -np.pi, -1e-1],
-    [0, -np.pi, -20, -20, -np.pi, -1e-1],
-    [0, -np.pi, -20, -20, -np.pi, -1e-1],
-    [0, -np.pi, -20, -20, -np.pi, -1e-1],
-    [0, -np.pi, -20, -20, -np.pi, -1e-1],
-]).reshape(-1)
-
-obs_max = np.array([
-    [1000, 1000, 20, np.pi, 1e-1, 0.],
-    [50, np.pi, 20, 20, np.pi, 1e-1],
-    [50, np.pi, 20, 20, np.pi, 1e-1],
-    [50, np.pi, 20, 20, np.pi, 1e-1],
-    [50, np.pi, 20, 20, np.pi, 1e-1],
-    [50, np.pi, 20, 20, np.pi, 1e-1],
-]).reshape(-1)
-
-def main(track:int, loc:int=0):
-    env = IntersimpleLidarFlatIncrementingAgent(
-        loc=loc,
-        track=track,
-        n_rays=5,
-        reward=functools.partial(
-            speed_reward,
-            collision_penalty=0
-        ),
-    )
-
-    policy = NormalizedIntersimpleExpert(env, mu=0.001)
-
-    env = Setobs(TransformObservation(
-        CollisionPenaltyWrapper(
-            env,
-            collision_distance=6, collision_penalty=100
-        ), lambda obs: (obs - obs_min) / (obs_max - obs_min + 1e-10)
-    ))
-    print(env.nv, 'vehicles')
-    expert_data = rollout_sb3(env, policy, n_episodes=150, max_steps_per_episode=200)
-
-    states, actions, rewards, dones = expert_data
-    print(f'Expert mean episode length {(~dones).sum() / states.shape[0]}')
-    print(f'Expert mean reward per episode {rewards[~dones].sum() / states.shape[0]}')
-    print(f'Observation mean', states[~dones].mean(0))
-    print(f'Observation std', states[~dones].std(0))
-
-    torch.save(expert_data, f'intersimple-expert-data-setobs2-loc{loc}-track{track}.pt')
-
-def loop(tracks:list=[0]):
-    for track in tracks:
-        main(track)
-
-if __name__=='__main__':
-    import fire
-    fire.Fire(loop)