Copied to clipboard!

SerpApi Release Notes

History

  • 2019-11-12
  • Extract total_results on Baidu
  • Update inline products screenshots in documentation
  • Extract thumbnails in compact inline products
  • 2019-11-11
  • Extract compact inline products
  • Extract regular inline products
  • Move inline_products extraction to own module
  • Extract extended inline_products
  • 2019-11-08
  • Update public JSON sample
  • 2019-11-07
  • Add pagination.previous and fix pagination.next on Baidu
  • 2019-11-06
  • Fix Google video results parsing on mobile devices
  • Fix tests
  • Clarify code comment for extract_js_image_by
  • Unit test performance-optimized JSUtils.unescape
  • Rename `place_results` hash to `local_results (tbm=lcl)
  • 2019-11-05
  • WIP update Google Scholar validator
  • Decode HTML entities with regular expressions instead of loops
  • Add support to Google Scholar API for `empty results state`
  • 2019-11-04
  • Test saved HTML with different designs of Google Images results
  • Extract original image URL from JS
  • Remove unused code
  • 2019-11-03
  • Fix and improve places_results parser
  • 2019-11-01
  • Parse Google Images from both HTML and JS
  • 2019-10-31
  • Start images extraction
  • Change spelling in the documentation
  • 2019-10-30
  • Add top_stories_link
  • 2019-10-29
  • Fix tests
  • Move top_stories extraction to own file
  • Parse top_stories defensively
  • Support list format of top stories
  • Support carousel format of top stories
  • Fix mobile ads parsing without description
  • 2019-10-28
  • Add support for results state to all Google API endpoints
  • Improve places reuslts parser
  • 2019-10-25
  • Exclude 'rich_snippet.bottom' for organic results with list format for sitelinks
  • Parse list format for sitelinks
  • Update bing related questions parser
  • Add related questions parser
  • Document Direct Answer Box snippet selector usage in organic_results
  • Code improvements for places results parser
  • 2019-10-24
  • Fix parsing of hotel bookings
  • Exclude incomplete answer_box duplicates in organic_results
  • Use new gps_coordinates method
  • 2019-10-23
  • Improve reliability of geting a place type
  • Enable `organic_results_state` for `tbm` results
  • Add places results parser
  • Rename file to `places_results` and move under proper conditional
  • Use both of new and old answer box CSS selectors
  • 2019-10-22
  • Test Bing pagination with high number of results
  • Handle pagination without 'other_pages'
  • Fix GPS coordinates extraction in local results
  • Fix 'knowledge_graph.people_also_search_for' parsing
  • Update selectors to parse dictionary answer box of Google results
  • WIP tbm local results parser
  • 2019-10-21
  • Move pagination-specific methods to correct modules
  • 2019-10-19
  • Fix for product results
  • Fix for local pack results
  • 2019-10-18
  • Add support for editorial_reviews to knowledge_graph and update description
  • Test pagination in Google Scholar results
  • Small fix
  • 2019-10-17
  • Baidu integration test
  • Update `local_pack` parser
  • Bing pagination
  • Edit `get_valid_link` method to support `data:image` links
  • Update product_results parser
  • 2019-10-16
  • Add serpapi_pagination to Baidu
  • Unify pagination in Google Scholar
  • Unify pagination and serpapi_pagination in google's organic results
  • Improve gps and valid link methods
  • 2019-10-14
  • WIP local_pack parser
  • Update `get_valid_link` method
  • Extract pagination to separate method
  • Don't set organic_results_state when no results are found
  • Rework `extract_gps_coordinates` to enable `js` extraction
  • Additional case for 'Empty showing fixed spelling results' state
  • Code review fixes
  • Add the `get_valid_link` method
  • Be sure title is returned for `product_result`
  • 2019-10-11
  • Remove TODO
  • Remove method for setting organic results state
  • Extract spelling fix type
  • Guard agains nonexisting nodes in inline videos
  • 2019-10-10
  • Fix Airbrake errors for `product_result`
  • Fix bed links and comment out `lsig` parameter
  • Fix NoMethodError for inline video title parsing
  • 2019-10-09
  • Move set_organic_results_state to parser
  • Test empty results in Arabic and German locale
  • Fix for when labels have space in them
  • Fix syntax issue
  • WIP - local map for specific address
  • Add support for the new `lsig` parameter
  • Fix some syntax issue
  • Handle inline videos without 'channel' in HTML
  • Tests for quotes spell fix
  • 2019-10-08
  • Improve method arguments naming
  • Empty search result states implementation
  • WIP - Add support for `Mobile v2` results
  • Rename to `product_result` and other small improvements
  • 2019-10-07
  • Google product results parser
  • 2019-10-04
  • Test inline videos in three different languages
  • Support results_for_exact_spelling and fully_empty states
  • 2019-10-03
  • Describe challenges of parsing :video_thumbnail
  • 2019-10-02
  • Document inline videos in organic results
  • Base64-encode thumbnails
  • Convert indentation to spaces
  • Fix Google and Google Scholar parser
  • 2019-10-01
  • Improve organic results parser
  • Support inline video results
  • 2019-09-30
  • Remove Google Search specific validators from Google Scholar
  • Improve organic results
  • Fix syntax
  • Show error if Google hasn't returned any results
  • 2019-09-25
  • Remove parameters that don't work with Google Scholar
  • Add support for Google Scholar `cites` and `cluster` parameters
  • 2019-09-24
  • WIP Google Scholar organic results
  • WIP Google Scholar parser
  • 2019-09-21
  • Add support for stats to Google Scholar results
  • Reduce some whitespace
  • Add support for related searches to Google Scholar results
  • 2019-09-20
  • Add support for Google Scholar specific parameters
  • 2019-09-17
  • WIP Add basic skeleton to support Google Scholar API
  • 2019-09-10
  • WIP Support Bing organic search in mobile
  • 2019-09-04
  • Fix #234
  • Fix #228
  • 2019-09-03
  • Fix issue with new Google design
  • 2019-08-26
  • Fix Baidu issues
  • 2019-08-22
  • Fix issue with rich snippet questions not being supported and messing with direct answer
  • 2019-08-20
  • Fix issue with related searches including link with blank hrefs
  • Fix data extraction for Google Shopping API
  • Fix product headers extraction
  • Fix product reviews, ratings, and number of reviews extraction
  • Don't transform a Float product rating into an Integer
  • Fix extensions not being correctly extracted
  • Snakecase keys to avoid JSON parsing issue for Google Product API
  • Fix issue with extracting related searches on mobile
  • Fix issue on mobile with organic results and ad results getting mixed up
  • Extract more rich data form organic results extensions
  • Stop removing direct answer from document
  • 2019-08-19
  • Add Google News API support for people also search for blocks
  • Fix issue with knowledge graph when key already exists
  • Add `raw_hours` key for people relying on a string instead of an object
  • Fix rating and review extraction for knowledge graph API
  • Fix position issue with Google Images Results API
  • Add support for extra layout variation
  • 2019-08-15
  • Fix Bing search ads
  • Fix issues with stopping recreating hash dynamically
  • 2019-08-13
  • Fix Google Images pagination end infinite loop
  • 2019-08-11
  • Fix issue with knowledge graph
  • Fix issues with direct answer box and regular hash
  • Use regular hash instead of recursive created hash
  • 2019-08-09
  • Fix price, extension, and snippet for shopping results #210
  • Actually avoid parsing parent knowledge graph section
  • Ensure `reviews_from_the_web` has results
  • Don't parse results from parent knowledge graph section for `see_more_about1`
  • 2019-08-08
  • Only create hash for `see_more_about` if it exist
  • Avoid setting empty reviews
  • Add support for `See more about` section
  • 2019-08-07
  • Fix issue with valid searches about Captcha being blocked
  • 2019-08-06
  • Don't include missing hours in local results #93
  • 2019-08-05
  • Update rating to float
  • Extract number from `review_count`
  • Add support for businesses' weekly schedule and reviews from the web
  • 2019-08-02
  • Add support for custom location by latitude and longitude input
  • Fix naming of hours, address and description in local pack results
  • Fix missing ratings in local results box #93
  • Fix links and title links for Binb organic results
  • WIP Rewrite final_hash and spec logic
  • 2019-07-30
  • Display JSON and HTML endpoints for Baidu search
  • Allow guest to reach to their JSON and HTML endpoints via tokens
  • 2019-07-26
  • Add Google Spell Check API
  • Fix commits and PR to streamline tokens
  • 2019-07-25
  • Fix typo
  • Avoid parsing answer divs when they are not there
  • WIP Use shorter tokens that look more user friendly
  • 2019-07-23
  • Add `location_used` to search json when using Bing #183
  • Fix thumbnail and images
  • 2019-07-19
  • Fix missing parameters error with validators
  • Add support for setting of precise location #181
  • Early preparation, add cookie setting
  • 2019-07-18
  • Fix mobile title for desktop news
  • Fix mobile search
  • 2019-07-16
  • Fix Google News API
  • 2019-07-14
  • Fix extraction of ratings from reviews
  • 2019-07-09
  • Remove conditional pagination (trouble on mobile)
  • 2019-07-05
  • Add product id to shopping results
  • 2019-07-02
  • Update knowledge graph to support parsing of business' operational status #116
  • 2019-06-27
  • Fix issue when knowledge graph answer type is missing
  • Remove raw_submodules from final hash
  • chore(app/*) whitespace changes and newlines at EOF
  • 2019-06-23
  • merge organic_results and video_results for desktop fix video_results parser for mobile
  • 2019-06-21
  • Fix temporary live total results empty error
  • Handle edge cases of Google pagination
  • 2019-06-19
  • Fix Base64 Errors
  • 2019-06-17
  • update pagination parameters with Search#clone_with_new_parameters
  • remove `then`'s from Bing knowledge graph results
  • 2019-06-16
  • convert bing knowledge graph into a hash vs. list of hashes
  • add `formatted` field to parseable bing submodules
  • Resolve conflicts
  • Actually fix bugs
  • 2019-06-15
  • Local map bug fixes
  • 2019-06-13
  • remove unique top-level type parsers & fallback to generics
  • remove unused include for bing ads results
  • 2019-06-11
  • Beta operational Baidu parsing
  • 2019-06-10
  • Fix emtpy hash being evuated as true issue #167
  • Add new serpapi search links for `place_id` #167
  • Rename location_id to place_id to match Google own documentation #167
  • Use `api_key` instead of `user_id` to encrypt in order to allow re-generation
  • Use only "Success" status instead of both "Cached" and "Success" for successful searches
  • Use tokens to authentifcate users tied to one specific request
  • stub out unimplemented Bing KG submodule type
  • filter out JavaScript links in bing KG link parser
  • add bing 'people_also_search' submodule formatter
  • add written_works, timeline, and interesting_stories bing formatters
  • add generic bing header submodule format to food & famous people
  • use Ruby's splat operator to get head & tail of Bing submodules
  • add generic format logic for Bing's knowledge graph header submodule
  • add 'Organization SocialActive' to Bing Organization types
  • 2019-06-07
  • Extract `ludocid` (location_id) from local pack
  • 2019-06-06
  • Add local map support and GPS coordinates for knowledge graph results
  • Support for dictionary answer box
  • Add parsing support for "Related news" and "rich snippet" results
  • Update parser with dynamic module loading based on required element
  • [squash] start on organization-specific Bing KG fields
  • add Bing 'flatmodule' knowledge graph parsing
  • add Bing 'slide' submodule parser
  • 2019-06-05
  • parse multiple Bing knowledge graph lists when they exist
  • replace Bing 'answers' with 'knowledge_graph'
  • 2019-06-03
  • add framework that captures typical Bing answer types
  • 2019-06-02
  • refactor bing ads pull helper fns from organic results module
  • move bing organic results into their own parser module
  • move bing ads results into their own parser module
  • 2019-05-31
  • Fail-safes for missing elements in local pack mobile
  • Add pagination, source icon for Baidu
  • 2019-05-29
  • Add parsing of address on specific style for local pack mobile
  • Extract phone number from local pack mobile
  • Add support for local packs with mobile device #109
  • Initial commit
  • 2019-05-26
  • Fix media parsing in `product_results`
  • 2019-05-25
  • Add fail safes for when elements don't exist or output is nil
  • 2019-05-24
  • add title field to Bing ad results
  • fixup tracking_link (vs. link) field names for Bing ad sitelinks
  • 2019-05-23
  • Ensure values aren't null, actually add parsing of related_search_boxes
  • Remove `log` in validator
  • add basic serpapi_pagination logic
  • Modify validation of successul request in product page
  • Finalize renaming `product` paramter to `product_id`
  • Strip quotes at start and end of review summary
  • Avoid parsing unrelated sections for `related_search_boxes`
  • 2019-05-22
  • Rename`related_search_box` to `related_search_boxes`
  • Add documentation page, navbar menu item, and icon for related searches
  • Routine cleanup
  • 2019-05-21
  • Update `specs_results` for alternate product page
  • Update `related_products_results` for alternate product page
  • Fix removal of text for expand description
  • 2019-05-20
  • Update `sellers_results` to work alongside alternate product page
  • Update `reviews_results` to work alongside alternate product page
  • Update `product_results` to work alongside alternate product page
  • Use parameters `specs` and `reviews` as booleans
  • Refactor get_related_search_box
  • 2019-05-19
  • Add additional selectors for parsing results on alternate product page #105
  • Check if `title` tag contains `Product not found` text
  • Checking if the url contains `/shopping/product` isn't really necessary
  • Use `if` `end` block if two or more conditions are used
  • 2019-05-18
  • Initial write-up for related search box #129
  • Bug fixes and adjustments
  • 2019-05-17
  • add Bing-specific query parameters
  • 2019-05-15
  • Support knowledge graph type extraction for business listing
  • 2019-05-14
  • replace the link with a tracking_link url
  • Use a more precise selector for knowledge graph type extractor
  • Fix knowledge graph type support
  • Changes per code review
  • 2019-05-13
  • add bing ad parsing logic
  • 2019-05-12
  • add in `form=QBRE` to trigger Bing ad results
  • move bing organic parsing into it's own fn
  • only return valid Bing organic results
  • 2019-05-10
  • Support float and integer number extraction for all locales
  • Massive overhaul
  • add basic Bing organic result parsing logic
  • 2019-05-03
  • Rename store/seller to name in sellers results, more adjustments to rating/reviews parsing in reviews_results
  • 2019-05-02
  • Fix stars and total reviews regex for international languages (hopefully), begin documentation for subparts of product API
  • 2019-05-01
  • Fix google_domain setting for engines "google" and "google_product", pagination fix for sellers and revies
  • 2019-04-30
  • Fix parsing of Google calculator direct answer
  • Add local stores parsing, rating and reviews to integer, fix param name on Google Prod API documentation,
  • 2019-04-28
  • Add some documentation for global api, reviews results. Adjust sellers results for online and local, refine filtering/sorting inline with documentation,
  • Fix reviews, adjust price extraction for multilingual
  • Add filtering, sorting, pagination, full results for reviews and specs/details
  • 2019-04-26
  • Use `unless` in replace of `if` blocks
  • Add feature to iterate through shopping results and fetch product page results (Not completed)
  • Restructuring
  • Naming, ensure elements exist before creating hash and iteration
  • There are ads but needs seperate parsing
  • Initial Google Product parsing integration
  • 2019-04-24
  • Support ludocid parameter (Google My Business Listing CID)
  • Structuring adjustments
  • Migrating to respective branch for development
  • 2019-04-23
  • Add parsing for specification details section
  • Add parsing for additional review data
  • Fix skeleton for other engine support
  • Initial product page parsing
  • 2019-04-18
  • Fix conflicting html attribute
  • Seperate search domain and search engine parameters
  • 2019-04-05
  • Support basic Bing search WIP