In this post, I will talk about an Instagram crawler to get posts of any user with comments, likes. This is actually a Python script, that with the help of browser automation can crawl an Instagram profile. It creates a JSON file and adds all information there that you can analyze later. In the JSON file, you can see all the post details with their captions, number of likes, number of comments, media URL, post URL, user details like bio, number of followers, number of following, and much more. It saves all this data in JSON and creates other files to store comments on posts.
Here this script uses Instagram automation to get the user details by taking their Instagram handle. It uses Selenium to automate your Chrome browser and crawl though the profile of the target users. With default settings, it can crawl profiles of public users only but you can make it crawl information from private accounts as well. To do that, you will have to specify your login credentials in its settings file. You can also specify multiple usernames in it to crawl their profiles and get the data.
Using this Instagram Profile Crawler to get Posts of any User with Comments, Likes:
The name of this tool is Instagram Profilecrawl and you can setup this in your PC very easily. You only need Python and Chrome driver in system PATH in order to make it work. There are some dependencies that you need to install before using it and I will mention how to do that.
Firstly, download Chromedriver and install Python and then clone this repository. After getting ZIP file of Instagram Profilecrawl, simply extract it and then open command prompt or terminal in the same folder. Now, execute the following command and then you will be ready to use it. Also, keep in mind that you have to paste the binary of Chromedriver that you download in the system PATH.
pip install -r requirements.txt
Now, its time to use the script. You just have to run it with the username of the target user whose profile you want to crawl. Simply run the following command in the specified syntax and then it will start doing its work. It will open Chrome browser automatically, will reach the user’s IG profile and start getting posts. When it is done, it will close the browser. You can see log of operations in the CLI and then analyze them.
python crawl_profile.py IG_Handle
In the repository folder, it will save the extracted data in “profiles” folder. It saves JSON file with the name of the user. You can see that in the screenshot below. All the information about different posts is in the JSON file. And if you want then you can run this script with multiple handles to crawl their profiles.
In this way, you can run and use this script to crawl Instagram profile of any user. The above method will be able to crawl profiles of any public Instagram user. But if you want to crawl profile of a private user then you have to specify your login credentials in the “util/Settings.py” file. And there are other options that you can configure. For example, if you only want to get certain number of posts from target Instagram account then you can do that.
Final thoughts
If you want to crawl basic information of any Instagram user along with posts details then this article will help you. If you have some technical skills then you will easily be able to use this method. You just run the script and then let it do the heavy lifting on its own. However, the script sometimes skips some posts from crawling I don’t know why. But can get you most of the user posts.
