Evaluation via available checkpoints

[asasarkaya]

Hi, thanks for the great work. I wanted to run some evaluations using the available model checkpoints. but I am having problems while following the instructions.

I downloaded kosmos_ph_oxe-pretrain.pt, and kosmos_ph_oxe-pretrain.json. And I modified eval/simpler/eval_ckpts_bridge.py file as follows:

import os

ckpt_paths = [
    (
        "~/projects/RoboVLMs/models/kosmos_ph_oxe-pretrain.pt",
        "~/projects/RoboVLMs/configs/kosmos_ph_oxe-pretrain.json",
    )
]

for i, (ckpt, config) in enumerate(ckpt_paths):
    print("evaluating checkpoint {}".format(ckpt))
    os.system("bash scripts/bridge.bash {} {}".format(ckpt, config))

It goes through eval/simpler/main_inference.py via scripts/bridge.bash. However, at line 9 when importing
from eval.simpler.model_wrapper import BaseModelInference, in eval/simpler/model_wrapper.py, it fails, because CustomModel is not defined.

I would appreciate any help.
Thanks
PS: It would be great to have at least 1 section in readme that walks through how to run an available (one of the three available) checkpoint, tbh, I overall found the readme a bit confusing

[aopolin-lv]

I encountered the same problem.

Moreover, when I tried to use the CustomModel in Calvin by adding from eval.calvin.model_wrapper import CustomModel, I ran into several errors as follows:

• In the reset(self) method of eval/simpler/model_wrapper.py, super.reset() is used, but I believe the correct syntax is super().reset().
• In the step(self, image, goal) method of eval/simpler/model_wrapper.py, super.step(obs, goal) is used.
• In the class BaseModelInference(CustomModel) of eval.simpler.model_wrapper.py, the parent class initialization method like super().__init__() is missing. I'm unsure how to use the parent methods, such as super().reset() and super().step().
• In the load_dataset_stat() method of eval.simpler.model_wrapper.py, the files are missing, and there are no relevant instructions provided.

with open(
    "configs/data/oxe_dataset_stats/dataset_statistics_google.json", "r"
) as f:
    google_info = json.load(f)
stat["fractal20220817_data"] = google_info

with open(
    "configs/data/oxe_dataset_stats/dataset_statistics_bridge.json", "r"
) as f:
      bridge_info = json.load(f)
stat["bridge_orig"] = bridge_info

• in the eval/calvin/model_wrapper.py, it has self.self.configs["act_head"], I think it may be the self.configs["act_head"].

Thank you very much for your contributions to the open-source community. However, I sincerely hope that the code can be completed and further benefit the broader robotic learning community.

[lixinghang12]

@asasarkaya @aopolin-lv Thank you for locating these issues, we have fixed the typos and bugs, and the dataset statistics files in configs/data/oxe_dataset_stats, please try the updated codebase.

[Ericonaldo]

Hi @asasarkaya , after the bug is fixed, is the README still confused (simply following the instruction at evaluation by compeleting the file path actually should work)? We really want to improve the quality of this project.

[aopolin-lv]

@asasarkaya @aopolin-lv Thank you for locating these issues, we have fixed the typos and bugs, and the dataset statistics files in configs/data/oxe_dataset_stats, please try the updated codebase.

Thank you for the quick response. I notice the commit log is not involved in the fifth problem mentioned above, i.e., "in the eval/calvin/model_wrapper.py, it has self.self.configs["act_head"], I think it may be the self.configs["act_head"]."

In addition, after loading the uploaded statistics json file, the program still raises errors. Specifically, it raise an error when executing

RoboVLMs/eval/simpler/model_wrapper.py

Line 207 in 92fe166

action = super().step(obs, goal)

The error is as follows:

Exception has occurred: KeyError       (note: full exception trace is shown but execution is paused at: _run_module_as_main)
'robot_obs'
  File "/home/projects/RoboVLMs/eval/calvin/model_wrapper.py", line 407, in step
    state = obs["robot_obs"]  # (15,)
  File "/home/projects/RoboVLMs/eval/simpler/model_wrapper.py", line 216, in step
    action = super().step(obs, goal)
  File "/home/projects/RoboVLMs/eval/simpler/maniskill2_evaluator.py", line 117, in run_maniskill2_eval_single_episode
    raw_action, action = model.step(image, task_description)
  File "/home/projects/RoboVLMs/eval/simpler/maniskill2_evaluator.py", line 237, in maniskill2_evaluator
    run_maniskill2_eval_single_episode(
  File "/home/projects/RoboVLMs/eval/simpler/main_inference.py", line 260, in <module>
    success_arr = maniskill2_evaluator(model, args)
  File "/home/miniconda3/envs/robovlms_simpler/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/home/miniconda3/envs/robovlms_simpler/lib/python3.10/runpy.py", line 196, in _run_module_as_main (Current frame)
    return _run_code(code, main_globals, None,
KeyError: 'robot_obs'

I think it is due to the obs variable does not contain the key named "robot_obs".
Expect for your response.

[Ericonaldo]

Hi @aopolin-lv Sorry for missing a line, I just forced update the commit, can you try again?

[aopolin-lv]

Hi @aopolin-lv Sorry for missing a line, I just forced update the commit, can you try again?

Still many bugs, to evaluate the SimplerEnv, my setting is:

• ckpt kosmos_ph_oxe-pretrain.pt
• config kosmos_ph_oxe-pretrain.json

The variable self.unnorm_keyis None and raises an error when running

RoboVLMs/eval/simpler/model_wrapper.py

Line 218 in f726030

action_norm_stats = self.dataset_stat[self.unnorm_key]["action"]

Moreover, the config file seems like lacking the key "robovlm_name", "vlm" and I am not sure if it should have the key "norm_action", "norm_min", "norm_max".

[lixinghang12]

Hi @aopolin-lv, sorry for uploading the old version config and eval wrapper, we have updated the config files in huggingface and the evaludation code of model wrapper, please try again.

[asasarkaya]

Thanks for the updates, soon, I will try again and let you guys know how it goes

[asasarkaya]

Hey again,

I am still having problems with config such as

    args.model_name = configs["config"].split("/")[-1].split(".")[0]
KeyError: 'config'

To test it reliably from your end, I would suggest you guys to clone your own repo into a new isolated environment, and follow the instructions there and finally run python eval/simpler/eval_ckpts_bridge.py with the provided config files and model checkpoints.

I believe the provided code is not test with the provided config before pushing them into the main branch.

[lixinghang12]

@asasarkaya Thanks for your response, sorry for uploading the wrong config file to huggingface. We have now passed the running test in SimplerEnv and updated the github codebase and config files in huggingface, please try again. FYI, here is an example logging output for a success running episode

[asasarkaya]

Thanks for updating the repo and the config. I think there is one last confusion for me. I see the followings in the config json:

"model_url": "https://huggingface.co/microsoft/kosmos-2-patch14-224",

    "model_path": ".vlms/kosmos-2-patch14-224",
    "model_config": ".vlms/kosmos-2-patch14-224/config.json",

    "tokenizer": {
        "type": "AutoProcessor",
        "pretrained_model_name_or_path": ".vlms/kosmos-2-patch14-224",
        "tokenizer_type": "kosmos",
        "max_text_len": 256,
        "additional_special_tokens": null
    },
    "vlm": {
        "type": "AutoModelForVision2Seq",
        "name": "kosmos",
        "pretrained_model_name_or_path": ".vlms/kosmos-2-patch14-224"

I thought I would not need to download/use untrained vlm backbone (kosmos-2-patch14-224). What should I provide for those paths? Does not the provided model checkpoint already include trained/fine-tuned final models/weights?

[asasarkaya]

After some more digging as well as trial/error, I was able to run the evaluation. Here are my notes that can help others to run it as well:

    "model_path": ".vlms/dummy",
    "model_config": "configs/kosmos_ph_oxe-pretrain.json",

    "tokenizer": {
        "type": "AutoProcessor",
        "pretrained_model_name_or_path": "microsoft/kosmos-2-patch14-224",
        "tokenizer_type": "kosmos",
        "max_text_len": 256,
        "additional_special_tokens": null
    },
    "vlm": {
        "type": "AutoModelForVision2Seq",
        "name": "kosmos",
        "pretrained_model_name_or_path": "microsoft/kosmos-2-patch14-224"

• microsoft/kosmos-2-patch14-224 is basically the huggingface repo_id for the VLM backbone

• I downloaded the config from this link and renamed/placed it as configs/kosmos_backbone_config.json.

• As an uggly temporary solution, at line 75 in RoboVLMs/robovlms/model/backbone/base_backbone.py, I hardcoded the config path as configs/kosmos_backbone_config.json

• RoboVLMs/blob/main/scripts/bridge.bash, at line 5, I changed python env sourcing to my own conda environment activation (such as conda activate robovlms-simpler)

• In RoboVLMs/blob/main/scripts/bridge.bash, at line 9, make sure that you have the right path to that image. You can simply set the path right such as rgb_overlay_path=SimplerEnv/ManiSkill2_real2sim/data/real_inpainting/bridge_real_eval_1.png

• Similarly, at line 40, it should be rgb_overlay_path=SimplerEnv/ManiSkill2_real2sim/data/real_inpainting/bridge_sink.png

Update: Last bit of the output of my eval run looks a bit concerning, it says Average success 0.0

[lixinghang12]

@asasarkaya The checkpoint you utilized is the one pre-trained on Open X-Embodiment, and its performance on task put eggplant in bucket is 0. We have uploaded the post trained checkpoints and configs on google robot and bridge, please try the updated checkpoints to reproduce the results.

[RunsenXu]

After some more digging as well as trial/error, I was able to run the evaluation. Here are my notes that can help others to run it as well:

    "model_path": ".vlms/dummy",
    "model_config": "configs/kosmos_ph_oxe-pretrain.json",

    "tokenizer": {
        "type": "AutoProcessor",
        "pretrained_model_name_or_path": "microsoft/kosmos-2-patch14-224",
        "tokenizer_type": "kosmos",
        "max_text_len": 256,
        "additional_special_tokens": null
    },
    "vlm": {
        "type": "AutoModelForVision2Seq",
        "name": "kosmos",
        "pretrained_model_name_or_path": "microsoft/kosmos-2-patch14-224"

• microsoft/kosmos-2-patch14-224 is basically the huggingface repo_id for the VLM backbone
• I downloaded the config from this link and renamed/placed it as configs/kosmos_backbone_config.json.
• As an uggly temporary solution, at line 75 in RoboVLMs/robovlms/model/backbone/base_backbone.py, I hardcoded the config path as configs/kosmos_backbone_config.json
• RoboVLMs/blob/main/scripts/bridge.bash, at line 5, I changed python env sourcing to my own conda environment activation (such as conda activate robovlms-simpler)
• In RoboVLMs/blob/main/scripts/bridge.bash, at line 9, make sure that you have the right path to that image. You can simply set the path right such as rgb_overlay_path=SimplerEnv/ManiSkill2_real2sim/data/real_inpainting/bridge_real_eval_1.png
• Similarly, at line 40, it should be rgb_overlay_path=SimplerEnv/ManiSkill2_real2sim/data/real_inpainting/bridge_sink.png

Update: Last bit of the output of my eval run looks a bit concerning, it says Average success 0.0

Hi,

I can evaluate the pre-trained checkpoints based on the solutions here. Thanks so much! To summarize, here are what I did for evaluating on Bridge of SimplerEnv
I am using the codebase with commit ID 0b6213a

1. Download the checkpoint and config file kosmos_ph_bridge-post-train.pt and kosmos_ph_post_train_bridge.json from the huggingface.
2. Modify the kosmos_ph_post_train_bridge.json file like above.
3. Download the kosmos_backbone_config.json config file, moved its position, and change the file path at line 75 in RoboVLMs/robovlms/model/backbone/base_backbone.py like above.
4. I run the script with command like:
   bash scripts/bridge.bash RoboVLMs/chechkpoints/kosmos_ph_bridge-post-train.pt RoboVLMs/configs/kosmos_ph_post_train_bridge.json
   It will automatically download some checkpoints and run evaluation. I got average success 0.25 with the output like this:

Namespace(policy_model='robovlm', policy_setup='widowx_bridge', ckpt_path='/ssd/xurunsen/projects/vla/RoboVLMs/chechkpoints/kosmos_ph_bridge-post-train.pt', env_name='PutCarrotOnPl
ateInScene-v0', additional_env_save_tags=None, scene_name='bridge_table_1_v1', enable_raytracing=False, robot='widowx', obs_camera_name=None, action_scale=1.0, control_freq=5, sim_
freq=500, max_episode_steps=60, rgb_overlay_path='real_inpainting/bridge_real_eval_1.png', robot_init_x_range=[0.147, 0.147, 1.0], robot_init_y_range=[0.028, 0.028, 1.0], robot_ini
t_rot_quat_center=[0.0, 0.0, 0.0, 1.0], robot_init_rot_rpy_range=[0.0, 0.0, 1.0, 0.0, 0.0, 1.0, 0.0, 0.0, 1.0], obj_variation_mode='episode', obj_episode_range=[0, 24], obj_init_x_
range=[-0.35, -0.12, 5], obj_init_y_range=[-0.02, 0.42, 5], additional_env_build_kwargs=None, logging_dir='results_v3', tf_memory_limit=3072, octo_init_rng=0, config_path='/ssd/xur
unsen/projects/vla/RoboVLMs/configs/kosmos_ph_post_train_bridge.json', ckpt_dir='', no_cache=False, double_step=False, robot_init_xs=array([0.147]), robot_init_ys=array([0.028]), r
obot_init_quats=[array([0., 0., 0., 1.], dtype=float32)], model_name='finetune_kosmos_cont-lstm-post_full-ft_text_vision_wd=0_hist=16_act=10_use-hand_aug-shift_act-norm_lr-1e-5_bs-
128_warm-0_oxe-pre_bridge_17-09')                                                                                                                                                   
           Average success 0.25 

[lixinghang12]

Hi @RunsenXu , the reported average success is of a single task (put carrot on plate i guess). The rollout videos are saved in results_v3. You can run eval/simpler/get_results.py to get the results for all tasks.