Shap-E by OpenAI is a free text to image generation model but instead of generating plain images, it can create 3D objects. Alternatively, it can also convert an existing image to 3D as well. It is open source, and you can run it locally if you want to. Additionally, there is an online hosted version of this tool available that I will mention later in this post.
AI has now come a long way. Interesting tools and products are being launched every day and thus the open-source community is not staying behind. After FreedomGPT and Alpaca, we now have a remarkable open-source text to 3D objects converter by OpenAI. Shap-E after taking a text prompt from you, generates a GLB file which is a 3D file format to save and view 3D shapes.
Both the tools; text to 3D and image to 3D are available in the same Shap-E model. It is even offering Jupyter notebooks for the same as well that you can easily run in Jupyter Lab or Google Colab. Furthermore, if you’d like to explore the code then you are free to do that.
Using Shap-E by OpenAI to Convert Text to 3D Objetcs:
As I have mentioned already then you can run Shap-E locally. But that will require you to have high end PC with high GPU and CPU requirements. If you don’t have that luxury, then you can simply try the model on Hugging Face here.
Hugging Face is a freemium platform where people can deploy their AI models. The hosted interface of Shap-E looks like this.
Now, you just enter some text to create a 3D object. Keep the prompt short and simple. After that, start the tool and then wait for a few seconds.
A few seconds later, you will see that it will generate a 3D object for you. You can fully interact with the 3D object design that it generated for you. You can zoom and view it from any angle.
To save it on your PC, you can use the download button in the top right corner. A GLB will download and then you can use that further editing in a 3D editing software that supports GLB format.
There are ready-made prompts that you can use to quick 3D render. These are just here to show you an idea of what it is capable of.
Using Shap-E by OpenAI to Convert Images to 3D:
Another thing you can do is convert a 2D object image into 3D. This part of Shap-E is a bit crude and does not return very satisfactory results. But nevertheless, you can still try it and maybe later edit the 3D object file to fix the colors and other irregularities.
Switch to the Image to 3D tab and then upload an image there. Make sure that the image you are going to upload has transparent background and the 2D object is clearly visible.
Click Run and then wait for the output. After a few seconds, you will see that it will generate the 3D version of the image that you submitted. The output is the same as above. You can view and inspect the 3D object and download the result in a GLB file.
In this way, you can use the image to 3D tool of Shap-E to create 3D objects from text and images. Right now, it is in development, and I hope that it will be improved in the later updates.
Closing thoughts:
Shap-E offers a whole new way of AI image generation. The 3D objects that it generates can be used for various purposes. I liked the overall concept, and it kind of works great as well for generating simple objects. But still there is room for improvement, and I hope the developers and maintainers refine the model to generate more accurate results that require minimum post editing.
