Copied to clipboard!

SerpApi Release Notes


History

2020-01-21
- Fix pagination

2020-01-16
- Update hash names

2020-01-15
- Update menu_pages.rb
- Update knowledge_graph.rb
- Update top_stories parser
- Fix news parser
- Extrack `stick` parameter
- Update parser
- Remove top_carousel endpoint
- Add aditional info to Google's KG popular_times
- Update validator

2020-01-14
- Fix google_product validator
- Update regex

2020-01-13
- Enforce indentation cop on all .rb files

2020-01-10
- Fix product_results test by making rating a float number
- Fix rating extraction in google product results
- Add comment about music block in Baidu organic results

2020-01-09
- Exclude music block from organic search
- Fix Baidu parser
- #431: Replaced URI with Addressable. Removed unused code.
- #237: Changed default query form coffee to barista for jobs engine in playgroud. Fixed typo in query_ordered.
- #237: Added defensive guard clauses.
- #237: Addressed comments for documentation and tests.
- Update local_pack parser
- Rename PATTERNS to IMAGE_URL_PREFIXES

2020-01-08
- Disable all Rubocop cops except indentation-related ones
- Fix video_results extraction
- Fix inline_images parsing from Google SERPs
- Fix google_flight parsing
- Check source attribute prioritization of get_valid_link
- Revert changes in Bing local parser

2019-12-30
- Undo unescaped link error changes to validator.
- Fix unescaped urls raising InvalidURIError.
- Add missing saved HTMLs for sports results
- Improve get_valid_link
- Check if place is verified
- Extract gps on desktop
- Don't show `top_carousel` if `stick` parameter is used
- #237: Added parser, validator, parameters modules for jobs engine. #237: Scrape basic jobs data. #237: Added tests for jobs scraping. #237: Added chips scraping. #237: Added playground page for jobs engine. #237: Added documentation page for jobs. #237: Changed tests to use automatic test file generation.

2019-12-27
- Fix ruby style
- Don't use to_sym in Bing local results
- Update parser
- Add yahoo local pack parser
- Add device support to Yahoo

2019-12-26
- Update ads_results.rb
- Update ads_results.rb
- Update organic results for mobile support
- Update related_searches.rb
- Add device support to Yahoo
- Add tests
- Add rich_snippet parser to Yahoo organic results
- Update parser.rb

2019-12-25
- Fix inline_shopping_results_spec
- Update parsers
- Fix possible top stories exceptions
- Add yahoo related_searches parser
- Fix Google's top_stories Airbrake error
- Add Yahoo related_questions parser
- Add device support to Yahoo
- Add get_valid_link method to yahoo
- Escape url before parsing.

2019-12-24
- get_valid_link in all engines
- Update knowledge_graph.rb
- Update top_carousel parser
- Fix people_also_search_for carousel
- Add top_carousel parser
- Add profiles parser

2019-12-23
- Fix boston_red_sox_spec
- Fix shipping_results spec again
- Fix lionel messi stats spec
- Fix local_pack_for_specific_address_desktop_spec
- Fix tests in refiny_by spec
- Fix tests in refine_by

2019-12-20
- Fix Bing local pack
- #250: Fixed doc example.
- #250: Handled ads links.
- Fix thumbnail and source extraction in inline top stories
- Update sports_results.rb

2019-12-18
- #250: Added extensions. Fixed incorrect price extraction.

2019-12-17
- Re-create .rubocop_todo.yml

2019-12-16
- #250: extract_numbers -> extract_number.
- Move top_stories_inline to mobile only method

2019-12-13
- #250: fixed mobile reviews parsing.
- Add more classes for mobile
- Update top_stories parser
- Added guard clause.
- Added refine_by key fallback. Added test for Japanese search.

2019-12-12
- Add hotel pricing parser
- #260: Fixed mobile parsing.
- Fix top stories links
- Fix popular_times parser
- Revert "Revert "260 add refine by""

2019-12-11
- Revert "260 add refine by"
- Revert "Google kg popular_times results"

2019-12-10
- #250: Added mobile inline shopping results.
- #260: Convert refine by text to symbol.
- Update Google shopping_results

2019-12-09
- Add popular_times parser
- Update sports_results.rb
- Fix and improve Baidu answer box results
- Fix Google sports parser

2019-12-06
- Add "missing" and "must_include" annotations parser
- Rename review_summary to user_reviews
- Add critic_reviews and reviews_summary to Google's kg
- Update Google's get_valid_link method
- Show original description and saved HTML file path in RSpec output
- Add another selector for related_questions

2019-12-04
- #260: Addressed review comments.
- Update local_results.rb
- Fix parser, add more options to extract gps

2019-12-03
- Update bing local pack
- revert changes made to clone_with_new_parameters method
- Update Bing local pack parser

2019-12-02
- #260: Added 'Refine by' extraction.
- Add Bing local pack parser
- Update error message
- Update bing validator
- Update some ad results to include title

2019-11-30
- Fix #367

2019-11-29
- #101: Added inline images for google organic search.
- Revert "Revert "Bing related searches and inline ads""

2019-11-27
- Revert "Bing related searches and inline ads"
- Lint Ruby code on pre-commit hook
- Remove debug.
- Use extract_integer.
- Added total_results for bing search.
- Improve top and bottom ads results
- Improve Bing parsers
- Add support for inline ads
- Add related searches parser to Bing
- Made fixes after code review.
- Be more defensive when extracting video duration
- Add "unsupported device" error message
- Fix typo

2019-11-26
- Add support for mobile sports parser
- Parse favicon in Yahoo! ads results
- Extract thumbnail urls for organic search from js.
- Fix maps API parser
- Move Yahoo! parsers to the separate modules

2019-11-25
- Parse Yahoo! search results on different locales
- Test Yahoo! on different languages
- Small fix
- yahoo_domain for Yahoo!
- Knowledge graph fields added: - See photos - See outside
- Update and improve sports parser

2019-11-22
- Update parser
- Remove 'n' parameter from Yahoo! search API
- Yahoo! search parameters on playground
- Add documentation
- Update parser and add support for baseball

2019-11-21
- Add parser for formula and moto gp
- Add parser for tennis athletes
- Handle Yahoo! web search parameters
- Add parser for basketball and american football athletes
- Improve `get_valid_link` method

2019-11-20
- Update team sport and add individual athlete (soccer) parser
- Yahoo! pagination
- Update results_state_and_spelling_fix.rb
- Improve parser error handling
- Re-use `dig`
- Update maps validator and tests
- Yahoo! query_displayed and total_results
- Button in ads_results for Yahoo! Search API

2019-11-19
- Yahoo! ads results with sitelinks
- WIP sports_results parser
- Yahoo! ads results with list sitelinks
- Replace blank? with present?
- Re-using `dig` in maps parser
- Add `google_domain` support to Maps
- Test Yahoo! agains real queries

2019-11-18
- Parse Yahoo! organic results
- Improve Maps parser
- Make sure place and search results are not loaded at the same time

2019-11-17
- Fix for when html shows wrong type of results
- Separate results state and spelling fix to different file

2019-11-15
- Update and improve `Maps` parser
- Add validator test
- Add `position` to some place results and rename hash

2019-11-12
- Extract total_results on Baidu
- Improve and update parser
- Revert "Improve and update parser"
- Add ReCaptcha validator
- Improve and update parser
- Fix Google Maps URL constructor
- Update inline products screenshots in documentation
- Extract thumbnails in compact inline products

2019-11-11
- Extract compact inline products
- Extract regular inline products
- Move inline_products extraction to own module
- Extract extended inline_products
- Add support for countries cities and provinces
- Add support for hotel ads

2019-11-08
- Improve `Maps API` parser
- Rename state from `place` to `local`
- Update public JSON sample

2019-11-07
- Add pagination.previous and fix pagination.next on Baidu

2019-11-06
- Update `Maps API` parser
- Fix Google video results parsing on mobile devices
- Fix tests
- Clarify code comment for extract_js_image_by
- Unit test performance-optimized JSUtils.unescape
- Rename `place_results` hash to `local_results (tbm=lcl)

2019-11-05
- WIP update Google Scholar validator
- Decode HTML entities with regular expressions instead of loops
- Add support to Google Scholar API for `empty results state`

2019-11-04
- WIP Add Google Maps API parser
- Add support for `type` parameter
- Test saved HTML with different designs of Google Images results
- Extract original image URL from JS
- Remove unused code

2019-11-03
- Fix and improve places_results parser

2019-11-01
- WIP Initial files for `Google Maps API`
- Parse Google Images from both HTML and JS

2019-10-31
- Start images extraction
- Change spelling in the documentation

2019-10-30
- Add top_stories_link

2019-10-29
- Fix tests
- Move top_stories extraction to own file
- Parse top_stories defensively
- Support list format of top stories
- Support carousel format of top stories
- Fix mobile ads parsing without description

2019-10-28
- Add support for results state to all Google API endpoints
- Improve places reuslts parser

2019-10-25
- Exclude 'rich_snippet.bottom' for organic results with list format for sitelinks
- Parse list format for sitelinks
- Update bing related questions parser
- Add related questions parser
- Document Direct Answer Box snippet selector usage in organic_results
- Code improvements for places results parser

2019-10-24
- Fix parsing of hotel bookings
- Exclude incomplete answer_box duplicates in organic_results
- Use new gps_coordinates method

2019-10-23
- Improve reliability of geting a place type
- Enable `organic_results_state` for `tbm` results
- Add places results parser
- Rename file to `places_results` and move under proper conditional
- Use both of new and old answer box CSS selectors

2019-10-22
- Test Bing pagination with high number of results
- Handle pagination without 'other_pages'
- Fix GPS coordinates extraction in local results
- Fix 'knowledge_graph.people_also_search_for' parsing
- Update selectors to parse dictionary answer box of Google results
- WIP tbm local results parser

2019-10-21
- Move pagination-specific methods to correct modules

2019-10-19
- Fix for product results
- Fix for local pack results

2019-10-18
- Add support for editorial_reviews to knowledge_graph and update description
- Test pagination in Google Scholar results
- Small fix

2019-10-17
- Baidu integration test
- Update `local_pack` parser
- Bing pagination
- Edit `get_valid_link` method to support `data:image` links
- Update product_results parser

2019-10-16
- Add serpapi_pagination to Baidu
- Unify pagination in Google Scholar
- Unify pagination and serpapi_pagination in google's organic results
- Improve gps and valid link methods

2019-10-14
- WIP local_pack parser
- Update `get_valid_link` method
- Extract pagination to separate method
- Don't set organic_results_state when no results are found
- Rework `extract_gps_coordinates` to enable `js` extraction
- Additional case for 'Empty showing fixed spelling results' state
- Code review fixes
- Add the `get_valid_link` method
- Be sure title is returned for `product_result`

2019-10-11
- Remove TODO
- Remove method for setting organic results state
- Extract spelling fix type
- Guard agains nonexisting nodes in inline videos

2019-10-10
- Fix Airbrake errors for `product_result`
- Fix bed links and comment out `lsig` parameter
- Fix NoMethodError for inline video title parsing

2019-10-09
- Move set_organic_results_state to parser
- Test empty results in Arabic and German locale
- Fix for when labels have space in them
- Fix syntax issue
- WIP - local map for specific address
- Add support for the new `lsig` parameter
- Fix some syntax issue
- Handle inline videos without 'channel' in HTML
- Tests for quotes spell fix

2019-10-08
- Improve method arguments naming
- Empty search result states implementation
- WIP - Add support for `Mobile v2` results
- Rename to `product_result` and other small improvements

2019-10-07
- Google product results parser

2019-10-04
- Test inline videos in three different languages
- Support results_for_exact_spelling and fully_empty states

2019-10-03
- Describe challenges of parsing :video_thumbnail

2019-10-02
- Document inline videos in organic results
- Base64-encode thumbnails
- Convert indentation to spaces
- Fix Google and Google Scholar parser

2019-10-01
- Improve organic results parser
- Support inline video results

2019-09-30
- Remove Google Search specific validators from Google Scholar
- Improve organic results
- Fix syntax
- Show error if Google hasn't returned any results

2019-09-25
- Remove parameters that don't work with Google Scholar
- Add support for Google Scholar `cites` and `cluster` parameters

2019-09-24
- WIP Google Scholar organic results
- WIP Google Scholar parser

2019-09-21
- Add support for stats to Google Scholar results
- Reduce some whitespace
- Add support for related searches to Google Scholar results

2019-09-20
- Add support for Google Scholar specific parameters

2019-09-17
- WIP Add basic skeleton to support Google Scholar API

2019-09-10
- WIP Support Bing organic search in mobile

2019-09-04
- Fix #234
- Fix #228

2019-09-03
- Fix issue with new Google design

2019-08-26
- Fix Baidu issues

2019-08-22
- Fix issue with rich snippet questions not being supported and messing with direct answer

2019-08-20
- Fix issue with related searches including link with blank hrefs
- Fix data extraction for Google Shopping API
- Fix product headers extraction
- Fix product reviews, ratings, and number of reviews extraction
- Don't transform a Float product rating into an Integer
- Fix extensions not being correctly extracted
- Snakecase keys to avoid JSON parsing issue for Google Product API
- Fix issue with extracting related searches on mobile
- Fix issue on mobile with organic results and ad results getting mixed up
- Extract more rich data form organic results extensions
- Stop removing direct answer from document

2019-08-19
- Add Google News API support for people also search for blocks
- Fix issue with knowledge graph when key already exists
- Add `raw_hours` key for people relying on a string instead of an object
- Fix rating and review extraction for knowledge graph API
- Fix position issue with Google Images Results API
- Add support for extra layout variation

2019-08-15
- Fix Bing search ads
- Fix issues with stopping recreating hash dynamically

2019-08-13
- Fix Google Images pagination end infinite loop

2019-08-11
- Fix issue with knowledge graph
- Fix issues with direct answer box and regular hash
- Use regular hash instead of recursive created hash

2019-08-09
- Fix price, extension, and snippet for shopping results #210
- Actually avoid parsing parent knowledge graph section
- Ensure `reviews_from_the_web` has results
- Don't parse results from parent knowledge graph section for `see_more_about1`

2019-08-08
- Only create hash for `see_more_about` if it exist
- Avoid setting empty reviews
- Add support for `See more about` section

2019-08-07
- Fix issue with valid searches about Captcha being blocked

2019-08-06
- Don't include missing hours in local results #93

2019-08-05
- Update rating to float
- Extract number from `review_count`
- Add support for businesses' weekly schedule and reviews from the web

2019-08-02
- Add support for custom location by latitude and longitude input
- Fix naming of hours, address and description in local pack results
- Fix missing ratings in local results box #93
- Fix links and title links for Binb organic results
- WIP Rewrite final_hash and spec logic

2019-07-30
- Display JSON and HTML endpoints for Baidu search
- Allow guest to reach to their JSON and HTML endpoints via tokens

2019-07-26
- Add Google Spell Check API
- Fix commits and PR to streamline tokens

2019-07-25
- Fix typo
- Avoid parsing answer divs when they are not there
- WIP Use shorter tokens that look more user friendly

2019-07-23
- Add `location_used` to search json when using Bing #183
- Fix thumbnail and images

2019-07-19
- Fix missing parameters error with validators
- Add support for setting of precise location #181
- Early preparation, add cookie setting

2019-07-18
- Fix mobile title for desktop news
- Fix mobile search

2019-07-16
- Fix Google News API

2019-07-14
- Fix extraction of ratings from reviews

2019-07-09
- Remove conditional pagination (trouble on mobile)

2019-07-05
- Add product id to shopping results

2019-07-02
- Update knowledge graph to support parsing of business' operational status #116

2019-06-27
- Fix issue when knowledge graph answer type is missing
- Remove raw_submodules from final hash
- chore(app/*) whitespace changes and newlines at EOF

2019-06-23
- merge organic_results and video_results for desktop fix video_results parser for mobile

2019-06-21
- Fix temporary live total results empty error
- Handle edge cases of Google pagination

2019-06-19
- Fix Base64 Errors

2019-06-17
- update pagination parameters with Search#clone_with_new_parameters
- remove `then`'s from Bing knowledge graph results

2019-06-16
- convert bing knowledge graph into a hash vs. list of hashes
- add `formatted` field to parseable bing submodules
- Resolve conflicts
- Actually fix bugs

2019-06-15
- Local map bug fixes

2019-06-13
- remove unique top-level type parsers & fallback to generics
- remove unused include for bing ads results

2019-06-11
- Beta operational Baidu parsing

2019-06-10
- Fix emtpy hash being evuated as true issue #167
- Add new serpapi search links for `place_id` #167
- Rename location_id to place_id to match Google own documentation #167
- Use `api_key` instead of `user_id` to encrypt in order to allow re-generation
- Use only "Success" status instead of both "Cached" and "Success" for successful searches
- Use tokens to authentifcate users tied to one specific request
- stub out unimplemented Bing KG submodule type
- filter out JavaScript links in bing KG link parser
- add bing 'people_also_search' submodule formatter
- add written_works, timeline, and interesting_stories bing formatters
- add generic bing header submodule format to food & famous people
- use Ruby's splat operator to get head & tail of Bing submodules
- add generic format logic for Bing's knowledge graph header submodule
- add 'Organization SocialActive' to Bing Organization types

2019-06-07
- Extract `ludocid` (location_id) from local pack

2019-06-06
- Add local map support and GPS coordinates for knowledge graph results
- Support for dictionary answer box
- Add parsing support for "Related news" and "rich snippet" results
- Update parser with dynamic module loading based on required element
- [squash] start on organization-specific Bing KG fields
- add Bing 'flatmodule' knowledge graph parsing
- add Bing 'slide' submodule parser

2019-06-05
- parse multiple Bing knowledge graph lists when they exist
- replace Bing 'answers' with 'knowledge_graph'

2019-06-03
- add framework that captures typical Bing answer types

2019-06-02
- refactor bing ads pull helper fns from organic results module
- move bing organic results into their own parser module
- move bing ads results into their own parser module

2019-05-31
- Fail-safes for missing elements in local pack mobile
- Add pagination, source icon for Baidu

2019-05-29
- Add parsing of address on specific style for local pack mobile
- Extract phone number from local pack mobile
- Add support for local packs with mobile device #109
- Initial commit

2019-05-26
- Fix media parsing in `product_results`

2019-05-25
- Add fail safes for when elements don't exist or output is nil

2019-05-24
- add title field to Bing ad results
- fixup tracking_link (vs. link) field names for Bing ad sitelinks

2019-05-23
- Ensure values aren't null, actually add parsing of related_search_boxes
- Remove `log` in validator
- add basic serpapi_pagination logic
- Modify validation of successul request in product page
- Finalize renaming `product` paramter to `product_id`
- Strip quotes at start and end of review summary
- Avoid parsing unrelated sections for `related_search_boxes`

2019-05-22
- Rename`related_search_box` to `related_search_boxes`
- Add documentation page, navbar menu item, and icon for related searches
- Routine cleanup

2019-05-21
- Update `specs_results` for alternate product page
- Update `related_products_results` for alternate product page
- Fix removal of text for expand description

2019-05-20
- Update `sellers_results` to work alongside alternate product page
- Update `reviews_results` to work alongside alternate product page
- Update `product_results` to work alongside alternate product page
- Use parameters `specs` and `reviews` as booleans
- Refactor get_related_search_box

2019-05-19
- Add additional selectors for parsing results on alternate product page #105
- Check if `title` tag contains `Product not found` text
- Checking if the url contains `/shopping/product` isn't really necessary
- Use `if` `end` block if two or more conditions are used

2019-05-18
- Initial write-up for related search box #129
- Bug fixes and adjustments

2019-05-17
- add Bing-specific query parameters

2019-05-15
- Support knowledge graph type extraction for business listing

2019-05-14
- replace the link with a tracking_link url
- Use a more precise selector for knowledge graph type extractor
- Fix knowledge graph type support
- Changes per code review

2019-05-13
- add bing ad parsing logic

2019-05-12
- add in `form=QBRE` to trigger Bing ad results
- move bing organic parsing into it's own fn
- only return valid Bing organic results

2019-05-10
- Support float and integer number extraction for all locales
- Massive overhaul
- add basic Bing organic result parsing logic

2019-05-03
- Rename store/seller to name in sellers results, more adjustments to rating/reviews parsing in reviews_results

2019-05-02
- Fix stars and total reviews regex for international languages (hopefully), begin documentation for subparts of product API

2019-05-01
- Fix google_domain setting for engines "google" and "google_product", pagination fix for sellers and revies

2019-04-30
- Fix parsing of Google calculator direct answer
- Add local stores parsing, rating and reviews to integer, fix param name on Google Prod API documentation,

2019-04-28
- Add some documentation for global api, reviews results. Adjust sellers results for online and local, refine filtering/sorting inline with documentation,
- Fix reviews, adjust price extraction for multilingual
- Add filtering, sorting, pagination, full results for reviews and specs/details

2019-04-26
- Use `unless` in replace of `if` blocks
- Add feature to iterate through shopping results and fetch product page results (Not completed)
- Restructuring
- Naming, ensure elements exist before creating hash and iteration
- There are ads but needs seperate parsing
- Initial Google Product parsing integration

2019-04-24
- Support ludocid parameter (Google My Business Listing CID)
- Structuring adjustments
- Migrating to respective branch for development

2019-04-23
- Add parsing for specification details section
- Add parsing for additional review data
- Fix skeleton for other engine support
- Initial product page parsing

2019-04-18
- Fix conflicting html attribute
- Seperate search domain and search engine parameters

2019-04-05
- Support basic Bing search WIP