These scrapes were designed to be used with JSON only. scraper.py -c URL N_RESULTS –FILE_FORMAT The file names will follow this format: “u-USERNAME DATE.” Post Comments
#Universal xml scraper virtualmachine code
I have not implemented code to allow users to specify different numbers of results returned for individual attributes. NOTE: The number of results returned will be applied to all attributes. NOTE: If you are not allowed to access a Redditor’s lists, PRAW will raise a 403 HTTP Forbidden exception and the program will just append a “FORBIDDEN” underneath that section in the exported file. Of these Redditor attributes, the following will include additional attributes: Submissions, Hot, New, Controversial, Top, Upvoted, Downvoted, Gilded, Gildings, Hidden, and Saved Comments Title Date Created Date Created Score Upvotes Text Upvote Ratio Parent ID ID Link ID NSFW? Edited? Text Stickied? Replying to (title of post or comment) In Subreddit (Subreddit name) You can also scrape Redditor profiles and specify how many results are returned. scraper.py -u USER N_RESULTS –FILE_FORMAT If you have searched for keywords in a Subreddit, file names are formatted as such: “r-SUBREDDIT-Search-‘KEYWORDS’ DATE.” Redditors The file names will follow this format: “r-SUBREDDIT-POST_CATEGORY DATE.”
Once you configure the settings for the scrape, the program will save the results to either a. NOTE: All results are returned if you search for something within a Subreddit, so you will not be able to specify how many results to keep. I have also added a search option where you can search for keyword(s) within a Subreddit and the scraper will get all posts that are returned from the search. You can specify Subreddits, which category of posts, and how many results are returned from each scrape. scraper.py -r SUBREDDIT N_RESULTS_OR_KEYWORDS –FILE_FORMAT * Includes additional attributes see Redditors section for more information Subreddits
Subreddits Redditors Post Comments Title Name Parent ID Flair Fullname Comment ID Date Created ID Author Upvotes Date Created Date Created Upvote Ratio Comment Karma Upvotes ID Link Karma Text Is Locked? Is Employee? Edited? NSFW? Is Friend? Is Submitter? Is Spoiler? Is Mod? Stickied? Stickied? Is Gold? URL Submissions* Comment Count Comments* Text Hot* New* Controversial* Top* Upvoted* (may be forbidden) Downvoted* (may be forbidden) Gilded* Gildings* (may be forbidden) Hidden* (may be forbidden) Saved* (may be forbidden) These attributes will be included in each scrape. Scrape speeds will be determined by the speed of your internet connection.Ī Table of All Subreddit, Redditor, and Post Comments Attributes This is a universal Reddit scraper that can scrape Subreddits, Redditors, and comments on posts.
0 kommentar(er)
