GitHub - facebookresearch/droidlet: A modular embodied agent architecture and pl...
source link: https://github.com/facebookresearch/droidlet
Go to the source link to view the article. You can view the picture content, updated content and better typesetting reading experience. If the link is broken, please click the button below to view the snapshot at that time.
This repository, corresponding tutorials and docs are still being refined (and not ready yet).
droidlet
helps you rapidly build agents (real or virtual) that perform a wide variety of tasks specified by humans. The agents can use natural language, memory and humans in the loop.
droidlet
is an early research project for AI researchers to explore ideas around grounded dialogue, interactive learning and human-computer interfaces.
droidlet
is in active development and is fairly unstable in design, API, performance and correctness. It is not meant for any production use.
Reach out to us at [email protected], to discuss your use case or just share your thoughts!
Getting Started
You want to do one of three things:
- Robots: Reproduce and extend the PyRobot based agent on real robots such as LocoBot or photo-realistic simulators such as AIHabitat.
- Minecraft: Reproduced and extend the
minecraft
based game agent - New Agent: write your own agent from scratch, starting from our
base_agent
abstraction
git clone --recursive https://github.com/facebookresearch/droidlet.git cd droidlet
- Linux - Python 3 (Anaconda recommended) - NVIDIA GPU (8GB+) - PyRobot-compatible robot or sim - Habitat-sim instructions below
- Linux - Python 3 (Anaconda recommended) - NVIDIA GPU (4GB+) - Minecraft - more instructions below
- Linux - Python 3 (Anaconda recommended)
conda create -n droidlet_env python=3.7 \ pytorch==1.7.1 torchvision==0.8.2 \ cudatoolkit=11.0 -c pytorch conda activate droidlet_env pip install -r \ agents/locobot/requirements.txt python setup.py develop
pip install -r \ agents/craftassist/requirements.txt
pip install -r requirements.txt
The tutorials introduce the `base_agent` architecture and take you through the 4 components of an Agent
6. API Documentation 7. Agent-specific API Documentation Locobot agent API CraftAssist agent API Not ApplicableDocumentation, Tutorials and Papers
Two papers cover the design of droidlet:
- droidlet: modular, heterogenous, multi-modal agents covers the overall design of
droidlet
as an embodied AI platform that is extensible to physical robots and simulators. - CraftAssist: A Framework for Dialogue-enabled Interactive Agents covers the design of the dialogue parser and the task system of an earlier version of
droidlet
that is specific to the game Minecraft
Citation
If you use droidlet in your work, please cite our arXiv paper:
@misc{pratik2021droidlet,
title={droidlet: modular, heterogenous, multi-modal agents},
author={Anurag Pratik and Soumith Chintala and Kavya Srinet and Dhiraj Gandhi and Rebecca Qian and Yuxuan Sun and Ryan Drew and Sara Elkafrawy and Anoushka Tiwari and Tucker Hart and Mary Williamson and Abhinav Gupta and Arthur Szlam},
year={2021},
eprint={2101.10384},
archivePrefix={arXiv},
primaryClass={cs.RO}
}
License
droidlet is MIT licensed.
Other Links
Datasets
Download links to the datasets described in section 6 of Technical Whitepaper are provided here:
In the root of each tarball is a README that details the file structure contained within.
Recommend
About Joyk
Aggregate valuable and interesting links.
Joyk means Joy of geeK