YouTube

Dream with Apple Vision Pro 🪄 Text-to-3D in VR - Scale AI Generative Hackathon Demo🧑‍💻

2023-07-16 · 04:32 UTC ·Sigil Wen ·1 min read

Brief

Dream with Apple Vision Pro (demo) showcases a text-to-3D workflow built during the Scale AI Generative Hackathon: user text processed via Scale AI Spellbook and GPT-4, Shap-E generates 3D assets on an A100 GPU (via Replicate), Modal hosts Whisper/endpoints, and the result is streamed using the Apple VisionOS SDK to a Meta Quest 2 for immersive preview.

Why it matters

Built by Molly Cantillon, Rahul Chalamala, Jonathan Lei, and Sigil Wen at the Scale AI Hackathon (SF); video published 2023-07-16 and demo titled “Dream with Apple Vision Pro”

Key details

  • Pipeline combines Scale AI’s Spellbook, OpenAI GPT-4 for text understanding, and Shap-E for text-to-3D generation, with Modal (Whisper and hosted endpoints) and Replicate (Ben Firshman provided an A100 GPU for Shap-E)
  • Demo implemented with the Apple VisionOS SDK but streamed to a Meta Quest 2 for testing (hardware borrowed from Mehran Jalali); source code available at https://github.com/Sigil-Wen/Dream-with-Vision-Pro
Source evidence

Built Dream with Vision Pro, a lucid text-to-3D tool built with the Apple VisionOS SDK. Powered by Scale AI's Spellbook, OpenAI's GPT-4 and Shap-E, Modal, Replicate, and the Meta Quest 2, we empower you to transform your imagination into stunning immersive experiences.

Built by Molly Cantillon, Rahul Chalamala, Jonathan Lei, and Sigil Wen at the @Scale_AI Hackathon in SF :D

Thank you Scale AI Spellbook team for the credits and ease of use, Ben Firshman of Replicate for the dedicated A100 GPU we run Shap-E on, Erik Bernhardsson of Modal for dedicated Whisper and hosted endpoints, and especially Mehran Jalali for letting us borrow the Meta Quest 2 for testing ❤️

Excited for when the Apple Vision Pro comes out to build it on a real apple headset instead of streaming it to the Occulus 🥲

Github: https://github.com/Sigil-Wen/Dream-with-Vision-Pro

Twitter post: https://twitter.com/0xSigil/status/1680400417321287680?s=20

Apple Vision Pro Developer Discord: https://discord.gg/QQRTVxmSRb

hackathon #scaleai #generativeai #ai

Channel: Sigil Wen
Published: 2023-07-16
Video URL: https://www.youtube.com/watch?v=apjxMbotr6E