SerpApi Release Notes

History

2022-01-22

  • • fix(home_depot): spelling fix error on empty result for some searches
  • • styles(bing): rename recipe to recipes for all the naming
  • • fix(google): inline videos video URL is parsed incorrectly

2022-01-21

  • • Update parser

2022-01-20

  • • New class
  • • Revert "Update app/models/search/google/parser.rb"
  • • Revert "Update app/models/search/google/parser.rb"
  • • Update app/models/search/google/parser.rb
  • • Update app/models/search/google/parser.rb
  • • Update app/models/search/google/parser.rb
  • • Update app/models/search/google/parser.rb
  • • Update app/models/search/google/parser.rb
  • • Update app/models/search/google/parser.rb
  • • fix(bing-recipe): make sure image and link are present before adding
  • • fix(bing-recipe): prevent empty array
  • • refactor(bing-recipe): remove X-Ray from link
  • • refactor(bing-recipe): make sure link is present
  • • refactor(bing-recipe): make sure image is present
  • • style(bing-recipe): remove comment
  • • refactor(bing-recipe): rename module name to RecipesResults

2022-01-19

  • • Fix thumbnails not being properly selected
  • • Update parser
  • • Add x-ray
  • • Stop showing empty results error when there are results
  • • Fix minor typo errors

2022-01-18

  • • Remove first page from other pages and update spec
  • • Simplify the checking.
  • • fix(google): add source link to inline images
  • • fix(google): inline images link not showing full path
  • • feat(bing): parse recipe results for grid and mobile

2022-01-17

  • • Update parser and add new spec
  • • feat(bing): add recipe results to bing

2022-01-14

  • • Update video_results.rb
  • • Update places_results.rb
  • • Update parser

2022-01-13

  • • Combine mobile results with desktop, and add spec for it
  • • Fix desktop video blocks
  • • Fix parser

2022-01-11

  • • Fix the term parameter to accept spaces, and added spec test.
  • • fix(bing): add X-Ray to entire related question block

2022-01-10

  • • fix(bing): add X-Ray to related questions
  • • fix(google): proper result showing no result in search information
  • • refactor: use && instead of and

2022-01-07

  • • fix(bing-knowledge-graph): wrongly select another element

2022-01-06

  • • Update parser and spec
  • • refactor: simplify spec

2022-01-05

  • • Fix typo errors and change first regex match to regex match in some methods
  • • refactor(bing-knowledge-graph): improve code readability

2022-01-04

  • • Update parser and playground demo JS file
  • • Update parser
  • • fix(bing-knowledge-graph): remove empty result in country facts
  • • refactor(bing-knowledge-graph): rename variables in country facts

2022-01-03

  • • Update parser and documentation
  • • Add `serpapi_more_results_link` to Google Answer Box `organic_result` type `carousel`
  • • Remove unnecessary check for presence
  • • fix(bing): knowledge graph error on some searches

2021-12-31

  • • Fix `apple_app_store` `num` parameter not working - Remove `num` variable manipulation in parameters - Add `num` parameter manipulation in `search.rb`

2021-12-30

  • • Hide the pagination for empty results
  • • Parser update
  • • Fix failing specs
  • • Support Google asnwer box flights updated legacy version - Update css targets for updated legacy version - Extract deeper information from the updated legacy version on desktop - Fix `flight_info` with`-` and `.` to be shown as one value ( .....s, ------s ) - Fix organic results to avoid conflict with answer box flights on mobile - Update parser to extract links from deeper info - Update parser to extract search_information for updated legacy version - Update parser to extract icon of airline company - Add specs - Add test HTMLs
  • • First part of addressing changes
  • • Update app/models/search/naver/parser/image_results.rb
  • • Update app/models/search/naver/parser/image_results.rb
  • • Update app/models/search/naver/parser/image_results.rb
  • • Update app/models/search/naver/parser/custom_pagination.rb
  • • Update app/models/search/naver/parameters.rb
  • • Update app/models/search/naver/parameters.rb
  • • Update app/models/search/naver/parameters.rb
  • • Update app/models/search/xray.rb
  • • Update app/models/search/metadata_and_parameters.rb
  • • Update parser

2021-12-29

  • • Remove commented lines in parser
  • • Remove unnecessary conditions and safe guards in parser - Remove unncecessary consitions and safe guards in parser - Add spacing in reviews spec - Update variable names with more understandable ones in sort_spec and add spacing - Add spacing in validator_spec
  • • Update Google Answer Box Flights deeper info - Update extraction of deeper infor for Google Answer Box Flights - Add new key `ad` - Fix misparse of `ad` as flight - Move all the images in Google Answer Box Documentation for faster load time - Add new image to Google Answer Box Documentation Flights - Update Google Answer Box Flights Documentation with `First Class, Non-Stop, Round-Trip Flights from Austin to Orlando` to showcase the precision of the update - Add new specs with different kinds for destktop and mobile - Add new test files
  • • Simplify parser
  • • Update parser

2021-12-28

  • • Fix parser for duplicated result and add spec tests.
  • • Update Google answer_box `about_this_result` - Extract `about_this_result` and `about_page_link` for answer_box type:`organic_result`
  • • Update parser

2021-12-27

  • • Update parser
  • • refactor(local-ads): parse missing attribute
  • • Naver Image API spec test

2021-12-26

  • • Update parser
  • • Pretty HTML
  • • Add custom pagination

2021-12-25

  • • Fix conflicting css with older translate answer boxes
  • • Update Google related_questions parser - Add new types of related_questions - Add list key to be parsed
  • • Update safe navigation for Google inline_people_also_search_for
  • • Refactor the new elements to be answer_box with organic_result type
  • • Revert changes in Google organic_results

2021-12-23

  • • Add safe_query to the search
  • • Update local_ads with serpapi link
  • • Add tests
  • • fix(local-ads): nil in selecting ads node
  • • Update Google `inline_people_also_search_for` with new safe navigations
  • • Fix Google `related_questions` not present for some results
  • • Fix Google `organic_results` not parsing some elements
  • • fix(bing): parse nutrition facts error

2021-12-22

  • • Add validator
  • • Add prettified HTML support
  • • Add parameters support
  • • Add parser
  • • Fix typo mistake in parser
  • • test(inline-video): fix spec failing
  • • Add fundamentals of the pretty HTML Part 1

2021-12-21

  • • Add X-ray and validations
  • • fix(inline-video-carousel): conflicted with answer box elements

2021-12-20

  • • fundamentals for Naver Image Api.
  • • fix(inline-people-search-for): error thrown when clone link doesn't have query params
  • • fix(organic-results): rich snippet parse wrongly in mobile

2021-12-18

  • • Fix the css class, to remove the duplicated result

2021-12-17

  • • Fix Google Knowledge Graph `audience_reviews` not parsing all reviews - Fix parser - Add spec file - Add test file
  • • Fix where key moments include is
  • • Update parser
  • • fix(map-place): not checking longtitude
  • • Add safe navigation

2021-12-16

  • • Fix typor error on parser
  • • Fix minor mistake in parsing websites
  • • Fix minor mathematical statement for pagination
  • • Add safety to parsing websites in local_results
  • • Remove unnecessary safe navigations
  • • Fix Google Knowledge Graph not parsing `hours` for some results
  • • Change mistaken key `years_in_bussiness` to `years_in_business`
  • • Add safe guard
  • • nit(google): use && instead of and

2021-12-15

  • • Fix missing big top stories
  • • Fix local_results parsed as organic_results in some searches
  • • Fix new kinds of local_results to be parsed - Add recent local_results that are not parsed - Fix keys that are not parsed for desktop - Fix keys that are not parsed for mobile - Fix misparses by introducing a method - Fix maps not being picked up for some results in desktop - Fix maps not being picked up for some results in mobile - Fix dummy coordinate values to be avoided (0,0) for some searches
  • • Update parser
  • • Use get_valid_link to retrieve URL
  • • Fix date showing in snippet
  • • Add safe guards
  • • Handle potential error on missing lat long in place result

2021-12-14

  • • Remove redundant css lookup
  • • Fix Google not parsing `local_ads` for some searches - Update parser with new class names - Update parser with new fields for tablet - Update xray markings and setting - Update spec with new examples covering all new aspects - Add new test files
  • • Fix answer_box parsing the wrong box
  • • Add X-Ray to answer box directions train result
  • • Fix answer box destination not parsing to and from correctly for some locale
  • • Fix error when gps_coordinates not available
  • • Fix direct answer box table error on certain table format
  • • Fix organic result containing video not showing in mobile
  • • Trim down code
  • • Fix typo
  • • Fix Naver pagination for non-ASCII queries

2021-12-13

  • • Add support for key_moments in organic_results
  • • Fix parser and add spec test.
  • • Fix indentation
  • • Fix src images not loaded on mobile for key_moments
  • • Add extracting for key moments that are not on page
  • • Fix Google `local_ads` not appearing for some searches
  • • Update inline people also search for parser for mobile
  • • Update default `page` parameter - Update playground and documentation explanation for `page` - Update default of `page` from 0 to 1 - Update handling of parameters - Update catching the error for `page=0` - Update examples on documentation for `page` change - Update pagination spec values with new ones for `page` change - Update reviews spec with a clearer order
  • • Update tablet extraction
  • • Update parser
  • • Update parser and spec - Refactor `about_this_result` extraction to be used by both desktop and mobile - Update spec output order - Update spec to give types as output - Fix minor indentation style in parser
  • • Add squish text
  • • Use text method directly
  • • Add directions answer box parser

2021-12-10

  • • Use index instead of length
  • • Add parsing for link
  • • Fix the CSS class
  • • Remove unneeded escaping
  • • Ensure thumbnail exists before adding xray
  • • Add xray to moment
  • • Update comma to last index of Hash
  • • Parse variant swatches in Product Result in Walmart Product engine (#2270)

2021-12-09

  • • Don't write gps_coordinate if nil
  • • Add parser for key moments in videos

2021-12-08

  • • Fix unaccepted page values to return error - Fix negative numbers giving error - Fix string containing values to give error
  • • Update app/models/search/google/parser/direct_answer_box.rb
  • • update node parser names
  • • Update parser

2021-12-07

  • • Fix long text not using alt text for some people also search for
  • • Update direct_answer_box.rb
  • • Update app/models/search/google/parser/direct_answer_box.rb
  • • Update app/models/search/google/parser/direct_answer_box.rb
  • • Update app/models/search/google/parser/direct_answer_box.rb
  • • Add parsing for url and title in answer hover box
  • • Update parser
  • • Fix error on direct answer box for some searches
  • • Fix Walmart `page` parameter to only support values between 1 and 100
  • • Fix rich snippet rating parse decimal to whole number
  • • Refactor code
  • • Refactor google local pack reviews parser
  • • Remove unnecessary spaces and declaration
  • • Fix spec error
  • • Refactor home depot check empty result

2021-12-06

  • • Update parser add the missing travel
  • • Fix Google Search `cached_page_link` key appearing when there isn't a cached web page - Fix Google Search `cached_page_link` key appearing when there isn't a cached web page - Fix `answer_box` and `organic_results` have same item. - Add `about_page_link`
  • • Update hash keys, percentage and chart
  • • Handle home depot empty results
  • • Fix bing finance when volume data not present
  • • Fix google local pack reviews more than 1000 not parsed correctly
  • • Remove unnecessary declaration

2021-12-03

  • • Make sure extracted_gps_parameters isn't nil
  • • Fix mobile product result getting extra producs
  • • Fix manufacturer link missing on mobile
  • • Fix infinite loop when page is set to less then 1
  • • Fix date extraction
  • • Fix parser
  • • Revert "Revert "Add Google Top Sights API""
  • • Revert "Add Google Top Sights API"
  • • Imrpove sport results spec and code
  • • Imrpove sport results thumbnail parser
  • • Fix Xray apply wrongly
  • • Fix google sport results sets on parsing

2021-12-02

  • • Change key names, to not have any conflicts.
  • • Change time to UTC timezone
  • • Check if details is not empty
  • • Fix Apple App Store `organic_results' invalid `links`
  • • Update validator
  • • Fix race condition in Walmart's `validator_spec`
  • • Add X-Ray to google sport results
  • • Update parser to support more tennis olympics games

2021-12-01

  • • Fix ratings and reviews extraction
  • • Update price and currency css class
  • • Add link
  • • Add xray to product_result and fix pricing not being parsed
  • • Update parser
  • • Fix time in the chart
  • • Add X-Ray
  • • Fix google olymipics tennis sport results
  • • Refactor Google Search `top_stories` `link`
  • • Refactor encrypt json path
  • • Refactor extract_ludocid_and_lsig in google local_results

2021-11-30

  • • Fix answer box, and add the currency chart
  • • Add `store_id` parameter to Walmart Product
  • • Fix Google Search `top_stories` duplicate items issue - Fix `live` key appearing breaking down visuals in `mobile` - Fix duplication of items inside keys - Remove unnecessary lines of code for link parsing - Check for duplication within tests for `mobile` and `desktop`
  • • Refactor Apple Reviews `review_date` key from Hash to String
  • • Fix merge conflict for Apple Reviews API
  • • Parse top sights map
  • • Fix undefined method 'scan' in local pack

2021-11-29

  • • Fix thumbnails and added extension for video or the number of photos in the blog post
  • • Fix parser not extracting lyrics, and fix the play_link
  • • Update parser
  • • Add documentation
  • • Add top sights parser
  • • Fix Apple App Store giving 500 error
  • • Update parser
  • • Fix Google Search Top Stories `link` not present in some results
  • • Fix local result type return true

2021-11-28

  • • Fix Google Search vertical top stories duplicated keys(not final)
  • • Add Google Search Top Stories `live` key for desktop and mobile - Update parser for `live` badge - Make `live` badge appear in HTML - Update parser for new kind of top stories (horizontal array) - Update top stories desktop and mobile specs - Move images - Expand documentation for live key

2021-11-26

  • • Add presence check to more css selectors in baidu organic results
  • • Add popular destinations parser
  • • Fix google local pack mobile type is parsing incorrectly
  • • Add years_in_business for Google Search Local Pack mobile
  • • Revert "Add years_in_business to Google Search Local Pack mobile"
  • • Revert "Update yeras_in_business regex to suppor other languages"
  • • Revert "Squish rating text in Google local results"
  • • Fix Google Map search parsing error (#2301)

2021-11-25

  • • Fix #2346 nil error
  • • Check any instead of just if nil
  • • Add xray for hover info
  • • Compact hash
  • • Update nexearch_results.rb
  • • Update nexearch - add music_results
  • • nexearch update - apps_results spec tests
  • • Update nexearch - add apps_results
  • • Update nexearch - add academic_results
  • • Parse hover info and remove from snippet in answer box
  • • Revert `rating` parser changes

2021-11-24

  • • Use unescape_js in JSDivExtractor
  • • Update app/models/search/naver/parser/ads_results.rb
  • • Remove the dot from the total_time
  • • Update nexearch - add influencer_results
  • • Update nexearch - Travel spec tests.
  • • Add unclaimed_listing check for Google Maps
  • • Add video author parser on mobile
  • • Update nexearch - add travel to knowledge_graph
  • • Check for presence in Google `local_ads`
  • • Add multiple options state and product page url id on Walmart engine

2021-11-23

  • • Update nexearch - add knowledge_ results spec tests.
  • • update nexearch - add knowledge_results
  • • Update mobile parser
  • • Update parser
  • • Revert "Update parser"
  • • Revert "Update parser logic"
  • • Update parser logic
  • • Update parser
  • • Update parser
  • • Don't show empty text field in content rating
  • • Add video duration support on desktop
  • • Fix json path can't decrypt non ascii characters
  • • Update to new method name
  • • Fix the cause of fail for older spec files in Google 'discover_more_places'
  • • Fix Google Search `local_ads` mobile not showing results anymore - Fix mobile not showing up in recent results - Add new test files to spec - Expand `hours` element to be scraped in different results

2021-11-22

  • • Fix missing keys for mobile, desktop and improved the test
  • • extract float using utils
  • • Update empty video results detection
  • • Update mobile video results parser
  • • Update desktop video results parser
  • • Update validator logic
  • • Squish rating text in Google local results

2021-11-21

  • • Improve js object parsing

2021-11-19

  • • update the parser & add tests
  • • Apply suggestions from code review
  • • Extract ad results, add tests and fix conflicts with web organic results
  • • Refactor tieba parser for baidu engine

2021-11-18

  • • Add Walmart Stores List
  • • Add parser for hotels answer box

2021-11-17

  • • Fix duration
  • • Update Google Search `top_stories` in mobile - Add `live` key - Refactor mobile specs - Add a test file
  • • Update yeras_in_business regex to suppor other languages
  • • Update regex for years_in_business to support general languages
  • • Update years_in_business regex to support general languages
  • • Fix yeras_in_business regex not working for other language

2021-11-16

  • • Pass location cookie to Walmart Search
  • • Add years_in_business to Google Search Local Pack mobile
  • • Add X-Ray to local_results in Google Search Local Pack mobile
  • • Change engine name to `apple_app_store`
  • • Add years_in_business to Google Search Local Pack desktop
  • • Parse number of years in business on Google local results correctly
  • • Add X-Ray to Google Knowledge Graph name and extensions

2021-11-15

  • • Fix Google Knowledge duplicated results
  • • Fix parser & add spec
  • • Fix extensions in Google Knowledge Graph not present for some searches
  • • Fix Google Knowledge duplicated results
  • • Update business open status to select single css with at_css

2021-11-12

  • • Fix parser
  • • Fix direct answer box error when when row is merged (#2284)
  • • Fix parser
  • • Fix direct answer box error when when row is merged (#2284)

2021-11-11

  • • Make come back fo fulfillment_badge in Walmart Search
  • • Update movie_showtimes.rb
  • • Refactor images extraction for mobile
  • • Minor change in parser
  • • Update movie_showtimes.rb
  • • Add new api business open status to the knowledge graph doc
  • • Parse business future date and recently open in Google Search knowledge graph
  • • Make come back fo fulfillment_badge in Walmart Search

2021-11-10

  • • Update app/models/search/google/parser/direct_answer_box.rb
  • • Fix images not extrated from google top carousel
  • • Fix & add spec test file
  • • Update app/models/search/google/parser/direct_answer_box.rb
  • • Fix images not extrated from google top carousel
  • • Fix some image links appearing as protocol relative
  • • Fix & add spec test file

2021-11-09

  • • Add dates results
  • • Add parsing for popular times and live data on google maps
  • • Parse location in Walmart Product
  • • Update parser
  • • Add parsing for popular times and live data on google maps
  • • Refactor `text` key to `snippet` - Resize images in documentation - Make images appear on playground - Simplify desktop image extraction - Refactor `text` key to `snippet` key - Add more examples in documentation - Add new spec files - Seperate mobile and desktop tests - Add a seperate test for optional snippet value
  • • Implement X-Ray to location in Walmart Product
  • • Parse location in Walmart Product
  • • Add mobile template to Walmart Product playground
  • • Update parser
  • • Add dates results
  • • Parse variant swatches for Walmart Product Result

2021-11-08

  • • Revert "Fix parser and add spec test."
  • • Fix parser and add spec test.
  • • Fix parser extract query
  • • Add user_nickname and customer_type to Walmart Product top reviews
  • • Update parser to fix extracting the reviews
  • • Fix Walmart search spec failing
  • • Revert "Add location with own defined constant key"
  • • Add location with own defined constant key
  • • Update comment
  • • Update parser
  • • Fix typo
  • • Update parser to fix extracting the reviews
  • • Revert "Fix parser and add spec test."
  • • Fix parser and add spec test.
  • • Fix parser extract query
  • • Update documnetation - Fix a mistake in documentation - Remove `see_more_link` - Update documentation
  • • Add user_nickname and customer_type to Walmart Product top reviews
  • • Update parser
  • • Fix Walmart search spec failing
  • • Revert "Add location with own defined constant key"
  • • Fix typo
  • • Update comment
  • • Add location with own defined constant key

2021-11-05

  • • Revert "Update Gems (#2086)"
  • • Update Gems (#2086)
  • • Add parser
  • • Revert "Update Gems (#2086)"
  • • Add parser
  • • Update Gems (#2086)
  • • Add X-Ray to Walmart Product desktop
  • • Fix minPrice is not parse in Walmart search
  • • Responsibe Walmart search view

2021-11-04

  • • Add swatch image to Walmart search variant swatches
  • • Parse Walmart search location
  • • Rename fulfillment_badge to fulfillment_badges as now it takes array
  • • Fix Walmart search fulfillment_badge key change in Walmart data
  • • Parse sponsored product in Walmart search
  • • Fix Walmart search organic results consist null item
  • • Revert "Fix Walmart search has empty organic result item"
  • • Fix Walmart search has empty organic result item
  • • merge ratings fix
  • • fix ratings
  • • fix thumbnails
  • • tests pass, updated parser for non english
  • • Parse Walmart search location
  • • Parse sponsored product in Walmart search
  • • Add swatch image to Walmart search variant swatches
  • • Rename fulfillment_badge to fulfillment_badges as now it takes array
  • • Fix Walmart search fulfillment_badge key change in Walmart data
  • • Fix Walmart search organic results consist null item
  • • Revert "Fix Walmart search has empty organic result item"
  • • Fix Walmart search has empty organic result item

2021-11-03

  • • add spec, update to catch more user reviews
  • • Safe navigation
  • • add spec for olympic tennis results
  • • Use hl for cookie language
  • • add spec, update to catch more user reviews
  • • forgot ?
  • • fix up hours logic
  • • fix double count
  • • add more specs
  • • Refactor key names - Refactor `service_availability` to `hours` - Refactor `phone_number` to `phone` - Fix wrong tests in dashboard menu - Fix typo error in menu name - Update specs in accordance with key names change - change unless statements to if - Update documentation accordingly
  • • Fix `images` not appearing in mobile
  • • Support Google Search `discover_more_places` - Fix issue with `related_search_boxes` appearing in irrelevant places. - Fix issue with `related_search_boxes` using `discover_more_places` as xray - Add parser - Add tests - Add documentation
  • • Add X-Ray to Walmart search
  • • Add a validator
  • • Add conditional variable for ltr pages

2021-11-02

  • • better hours handling
  • • better hours handling
  • • better hours handeling
  • • mark more fore xray
  • • fix hours, and double listing

2021-11-01

  • • fix undefined method size causing airbrake
  • • fix google news on mobile (#2186)
  • • add google as_* params (#2170)
  • • use both classes
  • • fix tennis players parsing
  • • fix undefined method size causing airbrake
  • • fix google news on mobile (#2186)
  • • fix phone parsing
  • • add google as_* params (#2170)
  • • Fix empty results error
  • • Support Google Search Local Ads - Add Parser - Add Specs - Add Documentation

2021-10-29

  • • Fix Parser
  • • add test
  • • add tieba result parsing
  • • Update App Store - Update `mobile` default value and push it on playground (only select parameter is not working for engine=app_store) - Update engine and documentation definitions - Update route /app_store to /app-store - Update new titles for Documentation - Update `mobile` default parameter in backend - Update prettified html structure - Add Mobile Layout - Add Tablet Layout - Refactor `release_date` and `size` key value structres - Refactor relevant tests
  • • Check proxy country_code not the ll country_code
  • • Fix Parser

2021-10-28

  • • Fix the parser (Bing knowledge graph)
  • • Remove redundant code
  • • Add spec
  • • Fix the parser (Bing knowledge graph)
  • • Fix google KG and add direct answer box on RTL language results (#2208)
  • • Adjustments
  • • Dynamic geo proxy url

2021-10-27

  • • Don't return nil values
  • • Correct the node name
  • • Don't return nil values
  • • Final changes and fixes.
  • • Update `id` of review to be string
  • • Update parameter fallback values - Remove `country`, `id`, `sort` fallback values - Update `page` value and remove fallback value of `page` ( Default is declared in search parameters )

2021-10-26

  • • extract without style
  • • Fix baidu page navigation, and fix issues with updated baidu tests
  • • fix google local results places on mobile
  • • fix airbrake #6074 (#2201)
  • • Fix baidu page navigation, and fix issues with updated baidu tests
  • • extract without style

2021-10-25

  • • Update parser to add support for additional serpapi links
  • • Update google play parser to support serpapi links to play product pages

2021-10-24

  • • Fix issues
  • • Improve knowledge_graph part 5 (Game results) with spec tests
  • • Knowledge_graph part 5 (Game results) with spec tests
  • • Knowledge_graph part 5 (TV Show results) with spec tests

2021-10-23

  • • Improve knowledge_graph (Movie results) with spec tests
  • • Retry Google Maps using a country-targeted proxy
  • • Knowledge_graph part 4 (Movie results) with spec tests

2021-10-22

  • • Remove binding.pry
  • • Resolve one of the missing cached links to be shown
  • • Add customer review badges - Add `customer_type` to customer reviews - Add `customer_type` to spec tests
  • • Revert nil? to blank?
  • • add new class & spec

2021-10-21

  • • Resolve walmart customer username and date not appearing
  • • add google lyrics (#2149)
  • • Knowledge_graph part 3 (Person full results) with spec tests
  • • fix again for ratings (#2182)
  • • google: fix local results rating (#2172)
  • • Adjust Google Reverse Image Search error handling

2021-10-20

  • • country knwoledge_graph and specs

2021-10-19

  • • Update Prettified HTML - Remove loading icon in prettified HTML - Remove extra space with iTunes Layout in prettified HTML - Remove unnecessary safe navigations and conditional hashes
  • • Update key `reviews_state` to `reviews_results_state` - Update `reviews_state` - Update simply parser - Update reorder pagination keys
  • • Update simplify parameters
  • • Update parameters - Update validation concern for product_id - Remove unnecessary safe_query for `product_id`
  • • Add Errors in returning final hash
  • • Update parameters - Update validator for `term` with a different validator - Remove unnecessary parameter value assignment for `page` - Remove unnecessart check for `term`
  • • Update Parser - Include Error type in returning final hash. - Simplify pagination hash.
  • • Fixing title X-ray
  • • add summary and some fixes

2021-10-18

  • • Update app/models/search/bing/parser/finance_answer.rb
  • • Update app/models/search/bing/parser/finance_answer.rb
  • • Update app/models/search/bing/parser/finance_answer.rb
  • • Update app/models/search/bing/parser/finance_answer.rb
  • • Knowledge_graph Part 1
  • • Update Apple Regions Documentation - Update Apple Regions in line with new design - Update default country from UAE to US - Fix typo error in Apple Constants
  • • Fix parsing developers with artist links

2021-10-17

  • • Naver changes.

2021-10-15

  • • move custom header method to validator.rb
  • • Slight code changes

2021-10-14

  • • Update validator
  • • Update pagination - Update parser with 'current' key in pagination - Update pagination spec with 'current' key
  • • Update prettified HTML - Refactor prettified HTML entirely - pass query term to prettified html - Change Documentation Images

2021-10-13

  • • use nil coalescing here (#2150)
  • • Stock infromation with specs
  • • Don't escape "<" and ">"

2021-10-12

  • • Use regex instead of nokogiri for faster about_this_result parsing
  • • Update tests
  • • Update parser

2021-10-11

  • • Remove reused parameters - Remove reused parameters - Update some statements with ruby style - Remove extra spaces

2021-10-09

  • • Update validator.rb
  • • Fix typo
  • • Check if place is from a different country than ll

2021-10-08

  • • fix x-ray selector and use TextUtils.convert_to_snakecase_sym for keys for bing knowledge graph (#2125)
  • • Using Xpath and commenting the X-ray for title
  • • Update app/models/search/yahoo/parser/organic_results.rb
  • • Add documentation

2021-10-07

  • • Rename symbol to currency
  • • Fix google inline shopping result ratings on mobile (#2101)
  • • add correct "when" key for first google event (#2105)
  • • Change from shipping to delivery

2021-10-06

  • • remove redundant titles in keys
  • • Use "price" and "symbol" keys
  • • Update gemfile
  • • Fixes and specs
  • • Fix `about_this_result` parsing

2021-10-05

  • • Remove escaped control characters
  • • Fix Yahoo snippet
  • • Remove escaped control characters

2021-10-04

  • • fix getting image when js extraction fails (#2094)
  • • Bing knowledge graph updates (#2064)
  • • Don't include empty arrays in about_this_result
  • • Update dynamic block parser with info comments
  • • Add support for reviews responses
  • • Update reviews parser
  • • Use money-heuristics to determine if keys contain a currency symbol
  • • Fix for playground HTML not loading JS
  • • Apply various fixes and updates to the parser

2021-10-03

  • • Fixing and include tests

2021-10-01

  • • Fix home depot airbreak #2090
  • • update google local_results addresses (#2079)
  • • fix airbreak (#2085)
  • • Add safe navigation in walmart parser

2021-09-30

  • • Fix indentation
  • • Remove more control codes
  • • Fix html UI
  • • Fix indentation
  • • Remove more control codes

2021-09-29

  • • Squish text
  • • Don't show empty links
  • • Get links from knowledge graph items instead of list of strings
  • • Fix issues from review comments

2021-09-28

  • • Parse related_questions and related_results
  • • Update app/models/search/walmart_product/parser.rb
  • • Add support for Google Play Reviews
  • • Update app/models/search/google/parser/shopping_results.rb

2021-09-27

  • • Update app/models/search/walmart_product/parser.rb
  • • item_nil? is a better name
  • • fix various nil errors in the walmart parser
  • • add number of pictures to google mobile knowledge graph
  • • Change page_size and num to ps
  • • Update app/models/search/home_depot/parser/pagination.rb
  • • Update app/models/search/home_depot/parser/pagination.rb
  • • Update app/models/search/home_depot/parser/pagination.rb
  • • Update app/models/search/home_depot/parser/pagination.rb
  • • Update app/models/search/home_depot/parameters.rb
  • • Update app/models/search/home_depot/parameters.rb

2021-09-24

  • • parse related keywords and fix issue with changing indexes
  • • Move displayed query to search_information

2021-09-23

  • • Add initial about this result parsing
  • • Add support for tv shows and episodes

2021-09-22

  • • Update shopping_results.rb
  • • Parse search query for reverse image search
  • • Revert "Fix Google missing sport results"
  • • Revert "Fix Google missing sport results"
  • • Update pagination and pagination spec - 'current' key in pagination
  • • Update field names

2021-09-21

  • • use compact with blank check
  • • add optional chaining for idml and reviews
  • • use compact
  • • merge new and old walmart product parsing
  • • game fix and add new spec
  • • merge!
  • • Fix how we extract individual items on page
  • • Exclude posts from parsing as knowledge graph attributes
  • • Update parser
  • • Add parser

2021-09-20

  • • Parse mobile customer reviews
  • • Escape `.` and `$` in error names as Mongo disallows them in keys
  • • merge source if exists
  • • rethink source: nil for now
  • • fix
  • • cleanup some nil keys
  • • merge master
  • • Fix sport results, Desktop and mobile.
  • • fix knowledge_graph title for tv-shows
  • • update google related questions parser
  • • Update pagination method name
  • • Fix parser
  • • Use query_scheme_and_domain
  • • Show "0" for `extracted_price` when `price` is "free"
  • • Remove `InclusionValidator` class

2021-09-17

  • • Fix multiline JSON parsing
  • • YAML not working on "End-of-text" (\u0003) unicode
  • • Adjust regex
  • • Rescue when date parsing fails
  • • Fix tests in `google/shopping_results/inline_shopping_results_spec.rb`
  • • Fixing
  • • add tag to google shopping results
  • • Fix offer_type wording and convert time to ms
  • • Update related_questions.rb
  • • Add offer_type and arrival_date and fix tests
  • • Update Spec Files and Documentation - Add Apple Reviews API Documentation - Update Apple Search API Documentation - Add Documentation Images - Fix Menu Names - Fix pagination not showing for some results - Fix results_stats showing wrong value for some results - Spec tests for pagination and sort - Spec files for pagination and sort
  • • Fix multiline JSON parsing
  • • YAML not working on "End-of-text" (\u0003) unicode
  • • Adjust regex

2021-09-16

  • • Initial impl for new walmart product page
  • • Revert "Initial impl for new walmart product page"
  • • Initial impl for new walmart product page
  • • Fix parsing the correct link on google related questions
  • • Conflicts fixes
  • • Ad publisher to google info
  • • Update validator.rb
  • • Make sure, pagination parameter is a string
  • • Fix `disallow_explicit` parameter - Update Documentation - Update Menu - Update Organic Results (more compact solution) - Update specs to work with `should` - Fix `disallow_explicit` in initialization

2021-09-15

  • • Fix some lists appearing as strings
  • • Fix people also searched for picking up extra data
  • • Stock_information spec
  • • Remove rescue
  • • Update frontend and backend to minimal working conditions - Fix Playground Layout - Fix Proxy Preference for Apple Reviews - Fix Pagination - Fix Results Stats - Fix Parameters - Add Validator - Fix Parser not showing rating - Update individual review prettified item - Update xray markings - Add Empty Results spec test and files - Add Reviews spec files - Add Validator spec (to be updated) and files
  • • Update validator.rb
  • • Update validator.rb
  • • Update validator.rb
  • • Update parser and add x-ray
  • • Fix 'organic_results' parsed from a seperate file - Fix 'organic_results' parsed from a seperate file - Add 'POST' in validator - Fix 'disallow_explicit' in parameters

2021-09-14

  • • Support map "People Also Search For" links
  • • Update organic_results.rb
  • • Update organic_results.rb
  • • Add Apple Reviews API -Initial Commit
  • • Fix 'explicit' parameter in Playground Remove 'explicit' parameter Add 'disallow_explicit' boolean parameter Add checkbox for 'disallow_explicit' in Playground Add number boxes for 'num' and 'page' parameters in Playground Fix 'empty results' giving the error 'Malformed Request'
  • • Update parser
  • • Stock information and inline videos for nxearch

2021-09-13

  • • Don't parse CSS in Google Job chip types
  • • Add App Store API - Add Localization Headers - Add Parameters - Add Parser - Add Spec Tests - Define Proxy Strategy - Add Documentation - Add Prettified HTML File (to be renewed later) - Add Playground Page - Support Handling Empty Result and Bad Requests in Validator and Parser - Add Apple Regions and Apple Languages Lists
  • • Fix parser

2021-09-10

  • • Add initial support for Google Play Product

2021-09-09

  • • Apply code style changes
  • • Update validators

2021-09-08

  • • Display actual ZIP code on the prettified Home Depot page
  • • Correctly generate search ID for the Home Depot when ZIP code and Store ID change

2021-09-07

  • • Remove dupe code
  • • rubocop:disable Layout/HashAlignment
  • • Add offers (mobile)
  • • Add desktop Local Posts for Google Maps Place
  • • Add knowledge graph Local Posts and Products

2021-09-06

  • • Refactor to work with pretty HTML

2021-09-03

  • • Fix X-Ray for Google `shopping_results[].extensions`
  • • Filter out empty `lists` and `paragraphs` from Bing Knowledge Graph
  • • Fix error in Bing Knowledge Graph parser
  • • Add custom headers support
  • • Fix code typo

2021-09-02

  • • Update direct_answer_box.rb
  • • Extract price to float
  • • Fix extensions missing for some searches
  • • Fix 500 error on Google Shopping
  • • Fixing the issues and improving the answer box

2021-09-01

  • • Revert "Google Events not scraping multiple days event's time"

2021-08-31

  • • Inconsistent indentation fix
  • • Specs & mobile & doc

2021-08-30

  • • Update app/models/search/google/parser/direct_answer_box.rb
  • • Update app/models/search/google/parser/direct_answer_box.rb
  • • Update app/models/search/google/parser/direct_answer_box.rb
  • • Update app/models/search/google/parser/direct_answer_box.rb
  • • Update app/models/search/google/parser/direct_answer_box.rb
  • • Update app/models/search/google/parser/direct_answer_box.rb
  • • Update app/models/search/google/parser/direct_answer_box.rb
  • • Update app/models/search/google/parser/direct_answer_box.rb
  • • Changes & improvements
  • • Parser update
  • • Add prettified html support for items highlights
  • • Add items highlight parser

2021-08-29

  • • Update events_results.rb

2021-08-28

  • • Add translation box & specs

2021-08-27

  • • Fix some empty space issues

2021-08-26

  • • Add tests
  • • Fix link extraction

2021-08-25

  • • Add more parameter validations
  • • Add support for price parameter
  • • Show `see_more_link` for pagination results
  • • Extract `serpapi_pagination` to separate method
  • • Home depot finish

2021-08-24

  • • Allow raw HTML data to be parsed without status to fix tests
  • • Display `error_breakdown` in our JSON only in development environment for now
  • • Rename `proxy_errors` to `error_breakdown`
  • • Add "403 - Forbidden" validation
  • • Update pagination to be multilingual
  • • Don't show empty pagination hash
  • • Support pagination for different html structures, when using pagination

2021-08-23

  • • Return early hash with metadata if search is queued, processing, or errored
  • • Handle JSON errors raised by `SearchSplitter` at the same spot

2021-08-22

  • • Pagination, specs and improvements

2021-08-21

  • • Improvement
  • • Pagination & Improvement
  • • Spelling fix remove and pagination template add
  • • Improvements
  • • Refactor home depot parser part 2

2021-08-20

  • • Support Google Knowledge Graph Movies Add 'knowledge_graph.movies' Add 'knowledge_graph.movies' spec files Add 'knowledge_graph.movies' documentation
  • • Fix parser naming
  • • Update prettified HTML
  • • Remove topic_id parameter support
  • • Add parser for app screenshots
  • • Refactor parser

2021-08-19

  • • Don't treat ReCaptcha text in snippet as an actual ReCaptcha page
  • • Support Google Knowledge Graph Episodes Support Google Knowledge Graph Episodes Add related spec files Add related documentation Fix minor documentation problem in Google Knowledge Graph Notable Moments

2021-08-18

  • • Changes

2021-08-17

  • • Home depot changes
  • • test specs update & mobile selector
  • • Test specs update
  • • Google Shopping Product Fixes
  • • Fix Google Inline Videos

2021-08-14

  • • link correction
  • • Support new Google Knowledge Graph Layout Fix horizontal extra bar layout Fix vertical extra bar layout Fix mobile horizontal extra bar layout Fix and Update knowledge_graph.cast Fix and Update knowledge_graph.related_media Fix and Update knowledge_graph.people_also_search_for Fix and Update knowledge_graph.see_results_about Add x-ray markings for all fixed and added blocks and their elements Add spec file and html files for horizontal, vertical and mobile-horizontal extra bar layouts Fix spec file mislabeled as knowledge_graph.header_images to knowledge_graph.related_media Add old class names for new layout on mobile Add x-ray markings for 'header_images' Add documentation for new blocks and example for different layout Add knowledge_graph.editorial_reviews Add knowledge_graph.clip Add knowledge_graph.audience_reviews Add knowledge_graph.user_statistics Add knowledge_graph.watch_now
  • • Requested changes on Yahoo! Shopping
  • • Improvements

2021-08-13

  • • fix problems and improvement

2021-08-12

  • • New link selector
  • • Update organic_results_videos_mobile.rb
  • • Naver changes

2021-08-10

  • • Improvments
  • • Fix missing keys, Google_jobs and new specs
  • • Fix null value and spec
  • • Fix missing job_id

2021-08-09

  • • Filtering, Specs and Screenshots quality improve.
  • • Update selectors

2021-08-06

  • • Improve and fix some problem

2021-08-05

  • • Fix extraction of images results with products

2021-08-04

  • • User agent update
  • • Unstuck HomeDepot tests on GitHub Actions
  • • Headers by bart

2021-08-03

  • • Improvements
  • • Add prettify HTML support for Google Play API
  • • Update parser
  • • Fix some errors and add test
  • • Support Google Knowledge Graph Notable Moments Add Google Knowledge Graph 'notable_moments' Add spec files for Google Knowledge Graph 'notable_moments' Add documentation for Google Knowledge Graph 'notable_moments'

2021-08-02

  • • Support Google Search Results Short Videos Add Google 'short_videos' Add spec files for Google 'short_videos' Add documentation for Google 'short_videos'
  • • Support Google Knowledge Graph 'ordering_options' -Add Google Knowledge Graph 'ordering_options' -Add Google Knowledge Graph 'ordering_options' x-ray markings -Support Google Knowledge Graph 'service_options' text version -Add Google Knowledge Graph 'service_options' text version x-ray markings -Support Google Knowledge Graph 'health_and_safety' -Add Google Knowledge Graph 'health_and_safety' x-ray markings -Fix Google Knowledge Graph 'reviews_from_the_web' not parsing on some results -Add Google Knowledge Graph 'reviews_from_the_web' x-ray markings -Add spec file for 'ordering_options', 'service_options', 'health_and_safety', 'reviews_from_the_web' -Extend Google Knowledge Graph Documentation

2021-07-31

  • • Remove spaces

2021-07-30

  • • Update `graph_ql` method
  • • Refactor product API
  • • Fix Google Search Results 'inline_people_also_search_for'
  • • Add parser for additional html blocks
  • • Add x-ray for first pages

2021-07-29

  • • Update validator.rb
  • • Update parse_html.rb
  • • Update parse_html.rb
  • • shopping_yahoo_gql, safe_get, and parse_html mix multiple responsibilities.
  • • Remove num parameter
  • • Update parser
  • • Update Google Play pagination

2021-07-28

  • • Changes
  • • Update parse_html.rb
  • • Update parser.rb

2021-07-27

  • • Add pagination parser
  • • Fix POST request to Google Play Store

2021-07-26

  • • Update parser

2021-07-23

  • • Add initial parser
  • • Add initial Google Play support

2021-07-16

  • • Check Fix odd layout
  • • Fix odd layout (top-level filter)
  • • Improvements and changes
  • • Home Depot product parser

2021-07-15

  • • Fix thumbnail extraction for Google inline people also search for block

2021-07-14

  • • Pagination and product wip
  • • Fix Google related_questions on mobile
  • • Add merchant_description to Google knowledge_graph

2021-07-13

  • • Add `place_id` parser for Googles knowledge graph
  • • Simplifying and fixes
  • • Pagination & parser fixes (Home depot)

2021-07-12

  • • Fix merge conflict
  • • Update app/models/search/duckduckgo/parser/inline_videos.rb
  • • Update parser

2021-07-11

  • • Parser refactor (GraphQL)

2021-07-10

  • • Fix Knowledge Graph not parsing full menu url
  • • Fix snippet class for expanded sitelinks

2021-07-09

  • • menu_array to menu_links
  • • //WIP

2021-07-08

  • • Revert "All problems fixed under a new pr."
  • • All problems fixed under a new pr.
  • • Correct date extraction from organic results on mobile too
  • • Fix to latest failing spec
  • • Update parser
  • • Restore missing # AUTO_ADD_HTMLS
  • • Revert "Changes to last accidental commit and new additions"
  • • Fix x-ray html scrolling on some engines
  • • Final changes before the push Missing Auto add HTMLS Indentation Differences
  • • parser and util refactor

2021-07-07

  • • Simplify date extraction from Google organic results
  • • Changes to last accidental commit and new additions
  • • X-Ray for `rich_snippet_table` and `rich_snippet_list`
  • • Fix X-Ray for Text nodes
  • • Revert accidental commits

2021-07-06

  • • Support for older versions of knowledge graph
  • • Use single `at_css` call instead of multiple `search`es
  • • New spec file, changes on spec files, new class names, old snippet class names
  • • Fix extraction of Google Images when Ads present on page
  • • Fix finance results price movement extraction
  • • Update direct_answer_box.rb
  • • Fix parser
  • • Fix edge case of `original` image extraction on Google Images
  • • Update direct_answer_box.rb
  • • Fix extraction of `original` image on Google Images

2021-07-05

  • • Fix rating extraction in regular Google search
  • • Fix rating and reviews extraction in local_results
  • • Creating the parser
  • • Spec fail fix and name change
  • • Improvements and updates
  • • Update parser
  • • Fixed according to failing specs
  • • Mobile parser fix
  • • Update parser to scrape sources
  • • Add parser

2021-07-03

  • • Changes made according to reviews by Zil
  • • Duplicate definitions for date merged Reviews addressed Co-author : Ilya Zub <ilya@serpapi.com>

2021-07-02

  • • Update parser
  • • Update Google Maps
  • • Remove events localization parameters
  • • Update documentation icon
  • • Remove "gl" support
  • • Fix parser
  • • Update prettify html

2021-07-01

  • • Extract other options in Google Shopping
  • • Exclude links to other google_product variants
  • • Update validator
  • • Add prettify HTMLs
  • • More changes made from reviews
  • • Changes committed regarding rewiews by Ilya
  • • Add parser for open_state
  • • Miscellaneous fixes

2021-06-30

  • • Update facts extraction
  • • Revert four accidential commits to master

2021-06-29

  • • Resolves #1804
  • • Resolves #1804
  • • Changes to spec and better parsing
  • • Fixes Description not parsing
  • • Revert change in file permissions
  • • Add parser
  • • Changes to spec and better parsing
  • • Update documentation
  • • Update parser and add x-ray
  • • Fixes Description not parsing

2021-06-28

  • • Add playground support
  • • Add initial Google Maps Photos engine support
  • • Remove duplicate
  • • Add support for sitelinks expanded and inline

2021-06-26

  • • Fix title, adding sitelinks and xrays
  • • Profiles (old knowledge graph)

2021-06-25

  • • Possible fix to last spec fail
  • • Revert changes to file permissions
  • • Little changes
  • • Mark with X-Ray Baidu Related News
  • • Last Push
  • • new knowledge graph finish

2021-06-24

  • • Documentation for all changes and missing parts Simplified related_videos parsing
  • • Just before the documentation Spec file fixed, sparatic parsing failure fixed, other_area_names unique now
  • • Add finance answer box parser

2021-06-23

  • • Fix thumbnail links
  • • Fix duration in mobile videos
  • • Resolves #1799 along with requested changes

2021-06-22

  • • get valid link method
  • • get valid link method
  • • Update app/models/search/bing/parser/knowledge_graph_results.rb
  • • Removing position from knowledge graph
  • • get valid link method
  • • Removing position from knowledge graph
  • • Removing position from knowledge graph
  • • Removing position from knowledge graph
  • • Removing position from knowledge graph
  • • Improve field extraction
  • • Fix parser
  • • Parse weather answer box
  • • Resolves #1788 alongside related spec files and test.

2021-06-21

  • • Update pricing (offers) selectors
  • • Support for Israeli pages and spec files
  • • Add results state for `Not many results contain`

2021-06-20

  • • Hardened with safe navigation
  • • Related videos missing alerted by spec
  • • Fixes related news, false snippet taking Fixes related news, age and source mistakes

2021-06-18

  • • Update app/models/search/google/parser/product_result.rb
  • • Revert "Squashed commit of the following:"
  • • Squashed commit of the following:
  • • Update parser
  • • Add pagination
  • • Add duckduckgo main documentation
  • • Update knowledge_graph.rb
  • • Remove shopping results parser
  • • Update parser to escape HTML
  • • Rename region parameter to `kl`
  • • Update organic_results.rb
  • • Update app/models/search/baidu/parser/organic_results.rb
  • • Update app/models/search/baidu/parser/organic_results.rb
  • • Update organic_results.rb
  • • Resolves #1701
  • • Update app/models/search/baidu/parser/organic_results.rb
  • • Update app/models/search/baidu/parser/organic_results.rb
  • • Update app/models/search/baidu/parser/organic_results.rb

2021-06-17

  • • Fix thumbnail extraction
  • • Update some block extractions
  • • Update app/models/search/baidu/parser/organic_results.rb
  • • Rename Google Maps Reviews `place_id` parameter to `data_id`
  • • Resolves #1012 New test htmls for #1012 Resolves partial description issue for #1776

2021-06-16

  • • Fix request url
  • • Update parser
  • • Add DuckDuckGo prettify support
  • • Update parser
  • • Add support for place_id
  • • Fixes #1780, Partially fixes #1776, retracts #1609

2021-06-15

  • • Tests for (places,view,books) and more improvements.
  • • Add new lines at the end of the files
  • • Test upadtes and improvements
  • • Extract place_id

2021-06-14

  • • Fixes #1013 , #1609
  • • Update app/models/search/naver/parser/organic_results_nexearch.rb
  • • Update app/models/search/naver/parser/organic_results_nexearch.rb
  • • Update app/models/search/naver/parser/organic_results_nexearch.rb
  • • Update app/models/search/naver/parser/organic_results_nexearch.rb
  • • Update app/models/search/naver/parser/organic_results_videos.rb
  • • Update app/models/search/naver/parser/organic_results_videos.rb
  • • Update app/models/search/naver/parser/organic_results_videos.rb
  • • Update app/models/search/naver/parser/organic_results_videos.rb
  • • Update app/models/search/naver/parser/organic_results_videos.rb
  • • Update app/models/search/naver/parser/organic_results_videos.rb
  • • Update app/models/search/naver/parser/organic_results_videos.rb
  • • Update app/models/search/naver/parser/organic_results_videos_mobile.rb
  • • Update app/models/search/naver/parser/organic_results_videos.rb
  • • Update app/models/search/naver/parser/organic_results_videos_mobile.rb
  • • Update app/models/search/naver/parser/organic_results_videos_mobile.rb
  • • Update app/models/search/naver/parser/organic_results_videos_mobile.rb
  • • Update app/models/search/naver/parser/organic_results_videos_mobile.rb
  • • Update app/models/search/naver/parser/organic_results_videos_mobile.rb
  • • Fix video pagination and ducomentation highlights
  • • Update related_searches.rb
  • • Update app/models/search/naver/parser/related_searches.rb
  • • Update app/models/search/naver/parser/related_searches.rb
  • • Update app/models/search/naver/parser/organic_results_nexearch.rb
  • • Update app/models/search/naver/parser/organic_results_nexearch.rb
  • • Update app/models/search/naver/parser/organic_results_nexearch.rb
  • • Update app/models/search/naver/parser/organic_results_nexearch.rb

2021-06-13

  • • Update app/models/search/naver/parser/organic_results_news.rb
  • • Update app/models/search/naver/parser/organic_results_news.rb
  • • Update app/models/search/naver/parser/organic_results_news.rb
  • • Update app/models/search/naver/parser/organic_results_news.rb
  • • Adding squish to the organic result (link)
  • • Removing x-ray. One shadows another.
  • • Update app/models/search/naver/parser/organic_results.rb
  • • Update app/models/search/naver/parser/organic_results.rb
  • • Update app/models/search/naver/parameters.rb
  • • Update app/models/search/naver/parameters.rb
  • • Update app/models/search/naver/parameters.rb
  • • Update app/models/search/naver/parameters.rb
  • • Update knowledge_graph_results.rb

2021-06-12

  • • Update organic_results.rb

2021-06-11

  • • Update product_result mobile
  • • Update product_result desktop
  • • Retrieve search token in the background and use it to make the request
  • • Fix failing tests for Walmart parser
  • • Add spelling_fix parser
  • • Add knowledge_graph parser
  • • Add shopping_results parser
  • • Add related_searches parser
  • • Add organic_results parser
  • • Add news_results parser
  • • Add inline_videos parser
  • • Add inline_images parser
  • • Add ads parser
  • • Add playground support
  • • Playrgound, documentation, fixes, tests (Naver)
  • • Fix snippet and link
  • • Fix snippet (Yandex)

2021-06-10

  • • Extract table `parse_organic_results` method
  • • Update parser
  • • New bing knowledge graph
  • • Fix Incorrect Yandex search results

2021-06-09

  • • Don't fail Walmart tests when optional data is missing
  • • Parse pagination on Walmart beta website

2021-06-08

  • • Parse filters on Walmart beta website
  • • Extract rating and reviews from Walmart beta website
  • • Parse search results on the Walmart beta website
  • • Naver Tests finished

2021-06-07

  • • Rename answer box hash with multiple boxes

2021-06-06

  • • Remove comments
  • • Page parameter & general fixes

2021-06-04

  • • Add initial parser commit
  • • Finishing mobile & tablet and updates on the playground

2021-06-03

  • • Fix Google Product seller results parsing with the Arabic language
  • • Fix extraction of `duration` of `inline_video_carousels` on tablet
  • • Rename `inline_videos_groups` to `inline_video_carousels`
  • • Add TODO comment for unused conditional branch in Google Product reviews content
  • • Add `link`, `position` to `inline_videos_groups`

2021-06-02

  • • Correct spelling on the Inline Videos documentation page
  • • Fix parser

2021-06-01

  • • Rename selling_points to highlights
  • • Remove unneeded comment
  • • Extract selling_points and features
  • • Adding mark for xray and fix the hash issue.
  • • Finish the Validator.
  • • Fix parser
  • • Fix parser
  • • add the shopping to nexearch
  • • Update parameters.rb
  • • Update validator.rb
  • • Playground updates

2021-05-31

  • • Extract full description of Google Product review content
  • • Fix features extraction
  • • Replace old features with new features block
  • • Extract rating from stars if reviews not present
  • • Remove old HTML files of `inline_videos`

2021-05-28

  • • Add comment about google product review content length
  • • Fix reviews pagination
  • • Add x-ray to reviews
  • • Fix rating extraction in Google Product
  • • WIP: Add tests for inline_videos_groups
  • • Add x-ray
  • • Add thumbnail parser

2021-05-27

  • • Fix Google Product description extraction

2021-05-26

  • • Add Naver API
  • • Fix parser
  • • Fix Bing x-ray node

2021-05-25

  • • Add x-ray markup to Google Product API
  • • Fixing missing `next` & `previous`

2021-05-24

  • • Fix extraction of `top_stories` on mobile

2021-05-21

  • • Update app/models/search/yandex/parser/pagination.rb

2021-05-20

  • • Mark Google Flights Answer Box with X-Ray
  • • Fix Google Flights parsing in `answer_box`

2021-05-18

  • • Fix related questions extraction
  • • Update xray.rb

2021-05-17

  • • Capture more elements in related searches block
  • • Edit file name from Pagination to pagination

2021-05-14

  • • Fix the missing eBay pagination from JSON

2021-05-12

  • • Fix Google Related Searches parsing on mobile
  • • Add X-Ray for Google `inline_videos`

2021-05-11

  • • Extract `platform` for horizontal Inline Videos on mobile

2021-05-10

  • • Adding yandex pagination
  • • Adding Yandex pagination to xray
  • • Create Pagination.rb
  • • Update parser.rb

2021-05-06

  • • Refactor Twitter Results parser

2021-05-05

  • • Implement Twitter results parser on desktop
  • • Fix parser
  • • Update parser
  • • Update parser
  • • Validate product_id
  • • Validate product_id
  • • Validate hd_filter_tokens
  • • Validation for product_id

2021-05-04

  • • Update parser
  • • Clear `start` and `num` parameters when generating `google_scholar` links
  • • Remove x-ray for citation table

2021-04-30

  • • Fix failing test
  • • Rename some parameters
  • • Update validator
  • • Update parser

2021-04-29

  • • Fix validator
  • • Add test for 302
  • • Exclude related searches from profiles

2021-04-28

  • • Add new element - review
  • • Add x-ray markup for discover_more_places
  • • Mark rating with x-ray

2021-04-27

  • • Support `position` field on `inline_videos`
  • • Fix Ads recognition and add x-ray markup

2021-04-26

  • • Improve variable naming
  • • Parse next div if price is present
  • • Fix Google local results address and hours extraction
  • • Fix parser

2021-04-23

  • • Update parser

2021-04-22

  • • Rename `user_id` to `author_id`
  • • Update parser
  • • Exclude Related Questions from Google Organic Results

2021-04-21

  • • Add pagination
  • • Extract SerpApi author link
  • • Update `get_valid_link` method
  • • Add prettify html for co author section
  • • Add engine parser
  • • Add parameter support

2021-04-19

  • • Add profiles parser for main Google Scholar organic results section

2021-04-16

  • • Fix parsers empty field
  • • Update parser
  • • Extract user ID
  • • Add Google Scholar Profiles parser

2021-04-14

  • • Update parser to fix position issue

2021-04-13

  • • Update documentation
  • • Fix block_position parser
  • • Fix parser

2021-04-09

  • • offers param is the same type as specs and reviews
  • • Make "offers" consistent with "specs", "reviews"
  • • Set offers=true for sellers=online parameters

2021-04-08

  • • Add xray markup for business operational status
  • • Remove unused code
  • • Add parser for inline shopping results
  • • Differentiate between temporarily and permanently closed

2021-04-07

  • • Fix Google Knowledge Graph business operational status

2021-04-06

  • • Update parser
  • • Improve Google Ads description selector in mobile and tablet
  • • Update parser
  • • Improve Google Ads description selector in desktop
  • • Add displayed_link and description x-ray markup to Google desktop ads
  • • Add x-ray markup for Google mobile Ads
  • • Fix description extraction in Google Ads desktop and mobile
  • • Capture new Yandex captcha page

2021-04-02

  • • Improve Google Autocomplete engine Youtube client regex
  • • Fix service_options extraction and JSON result key name
  • • Fix service options extraction in Google Local results
  • • Fix service options extraction in Google Maps Places
  • • Fix service options extraction in Google Search

2021-04-01

  • • Updated google products tests.

2021-03-30

  • • Updated doc pages removing local_results. Removed num parameter from google product.
  • • Replaced sellers with boolean offers.
  • • Removed nodes traversing. Added xrath text lookup.
  • • Updated google shopping pagination params. Changed local sellers doc page pointing deprecation.

2021-03-26

  • • Disable LinkedIn APIs
  • • Handle local_pack case with wrong address/missing phone.
  • • Fix exceptions in direct_answer_box_table

2021-03-25

  • • Fix NoMethodError in Google Local Results mobile
  • • Fix typo. Added guard clause.
  • • Fixed local pack: - title beign polluted with 'Why this ad?' - address being polluted with JS. - missing phone extraction.
  • • Update all engines to use the same playground wrapper

2021-03-24

  • • Fix Google Inline People Also Search For extraction
  • • Updated selectors and tests for related searches.

2021-03-23

  • • Fix knowledge graph hours extraction. Extract hours break.
  • • Fixed profiles data slipping into people_also_search_for in Google knowledge graph.
  • • "0" specs, reviews parameters handling

2021-03-20

  • • Add BooleanConflictWithOtherParametersValidator

2021-03-16

  • • Fix Google Organic results missing snippet, date, resolution and displayed_results

2021-03-15

  • • Fix 'streetview' extraction for Google Local Pack

2021-03-12

  • • Update app/models/search/google_product/parser/sellers_results.rb
  • • Update app/models/search/google_product/parser/sellers_results.rb
  • • Update app/models/search/google_product/parser/sellers_results.rb
  • • Fix online_sellers

2021-03-10

  • • Fix Google Maps parser

2021-03-08

  • • Fix Google Images without auto-corrected results
  • • Update parser

2021-03-04

  • • Hardcode baidu_domain
  • • Fix SSRF in yandex_domain

2021-03-03

  • • Update parser
  • • Don't extract original images as thumbnails
  • • Don't symbolize_keys in JSON.parse
  • • Fix Google Images Results extraction

2021-03-02

  • • Test Google Maps validation for incorrect symbols
  • • Require comma between coordinates in `ll` parameter
  • • Changed svg path selectors to classes of parent elements.
  • • Do not parse Google's related_questions as answer_box
  • • Fix extraction of an answer in the knowledge graph
  • • Fix parser

2021-03-01

  • • Validate `ll` parameter for `google_maps` engine
  • • Update code
  • • Update parser
  • • Add documentation

2021-02-27

  • • Fix parser

2021-02-26

  • • Remove comments
  • • Extract related search boxes from Javascript

2021-02-25

  • • Extract dictionary result of Answer Box from JS
  • • Add `reviews_link` to Google Maps API
  • • Remove false duplicates of answer_box in organic results
  • • Fix parser
  • • Fix mobile parser
  • • Update parser

2021-02-24

  • • Fix Google Answer Box tests
  • • Update parser
  • • Rename data_id to place_id
  • • Add more parameters support
  • • Update parser
  • • Fix answer_box extraction
  • • Fix table answer box extraction
  • • Fix parser
  • • Extract full title without domain name.
  • • Revert "Revert "Scrape answer box content""

2021-02-23

  • • Revert "Fix wrong query_ordered when `q` is missing"

2021-02-22

  • • Add parser
  • • Add Google Maps Reviews engine
  • • Fix wrong query_ordered when `q` is missing
  • • Require `q` parameter unless Google Reverse Image
  • • Update google_maps_search_pagination? method

2021-02-19

  • • Require `q` parameter for Google engine
  • • Move Google Maps search pagination condition to separate method

2021-02-18

  • • Add more explicit message for missing `ll` parameter when using pagination

2021-02-17

  • • Extract local_results extraction to parse_search_results_by_node
  • • Remove `alt` parameter from SerpApi link
  • • Fix empty results
  • • Add pagination to Google Maps

2021-02-16

  • • Improve readability of query_generic method

2021-02-12

  • • Fixed google knowledge graph fields extraction.

2021-02-10

  • • Fix parser

2021-02-09

  • • Reuse latitude and longitude from `ll` parameter
  • • Remove Google Maps Search engine
  • • Small fixes
  • • Use google_maps engine for pagination too

2021-02-08

  • • Changed htilrad to lrad for radius refinement.
  • • Include JSDivExtractor module in reverse image API
  • • Prettify Google Scholar Cite HTML response

2021-02-05

  • • Fix parser
  • • Revert "Fix parser"
  • • Fix parser
  • • Update parser

2021-02-04

  • • Codestyle changes.
  • • Fix Google Jobs htidocid extraction

2021-02-03

  • • Extract structured extensions as detected_extensions.
  • • Use calculate_md5 for hashing instead of query_ordered
  • • Fix ratings extraction in Google Jobs Listing engine
  • • Fix Google Jobs Listing API caching issue
  • • Make constants use moduls

2021-02-02

  • • Update parser
  • • Fix parser

2021-02-01

  • • Update parser

2021-01-29

  • • Use clone_with_new_url_parameters to build serpapi link
  • • Restored compatibility with older html examples.
  • • Fix parser
  • • Update parser
  • • Update parser
  • • Update parser to include title in Google's kg mobile

2021-01-28

  • • Use unpack1 instead of unpack for performance
  • • Fixed selector to obtain full-length title.
  • • Rename constants

2021-01-27

  • • Added htilrad for Google Jobs.
  • • Update searches_controller
  • • Update parsers
  • • Add pretty html for Google Maps Search

2021-01-26

  • • Parse see more link in Google inline people also search for block
  • • Add serpapi_link inline people_also_search_for items
  • • Refactor Google parameters
  • • Scrape Google inline people also search for section
  • • Updated LinkedIn Profile: changed naming in json response. updated tests and documentation. fixed typos in xray marking. added xray markings for thumbnails.
  • • Add support for Google Maps Search engine
  • • Extract whole title in Google Jobs API.

2021-01-21

  • • Remove empty string info elements in graph_results
  • • Add X-Ray markings for popular times
  • • Rename typical_time_to_spend to typipal_time_spent
  • • Prefer && instead of and operator
  • • Scrape undisplayed images. Undisplayed images are marked as "The rest of the results might not be what you're looking for".

2021-01-19

  • • Update documentation
  • • Handle one more edge case
  • • Move main get_knowledge_graph method to the top
  • • Rewrite popular times knowledge graph parser and update test htmls

2021-01-18

  • • Fix desktop ad phone selector

2021-01-15

  • • Do not use encrypted_start_of_api_key
  • • Fix Google images thumbnail extraction

2021-01-12

  • • Added validation of profile_id.
  • • Added xray for LinkedIn Profile.
  • • Update parser

2021-01-11

  • • Style code to be more consistent

2021-01-08

  • • Add initial encryption / decryption of image JSON path

2021-01-06

  • • Fix job listing job ID validation error message
  • • Stop polluting base class with validations

2021-01-05

  • • Added newlines.

2021-01-03

  • • Fixed typo in documentation. Fixed empty result parsing. Added test.

2020-12-30

  • • Added uule param to be passed when used directly, not as a location.
  • • Removed code duplication.
  • • Changed YAML #load -> #safe_load. Removed unnecessary navigation operator.

2020-12-29

  • • Added params validation for linkedin.

2020-12-28

  • • Added xray marking for LinkedIn public search.
  • • Updated documentation.
  • • Fix condition ordering to fix query_displayed value
  • • Refactor inclusion_with_value_modifier to inclusion_case_insensitive validator
  • • Replace if presence checks with allow_blank option in InclusionWithValueModifierValidator
  • • Use allow_blank: true instead of explicit presence check
  • • Fix Google Scholar link extraction edge case

2020-12-25

  • • Updated title extraction for yandex tablet search.
  • • Fix google jobs params extraction. Prevent google job listing fail if param is missing.

2020-12-23

  • • Revert "Scrape answer box content"
  • • Do not display query_displayed for Google Scholar if it is empty

2020-12-21

  • • Add explicit cites_id and cluster_id to final json response
  • • Usage of cites + q triggers a search within cited articles

2020-12-18

  • • Support cites parameter and fix serpapi URL building

2020-12-17

  • • Add uncommited change.
  • • Fixed yandex mobile search.

2020-12-16

  • • Fix date, displayed_results and image resolution extraction
  • • Fix Rubocop linting
  • • Make q optional if tbs is present for google engine
  • • Add rich snippet `reviews` support on mobile

2020-12-15

  • • Added documentation for LinkedIn profile. Added tests for LinkedIn profile. Covered missing sections from profile.
  • • Fix X-Ray markings for Google Mobile Search
  • • Fix Google reverse image search returning 500
  • • Fix displayed results, snippet and date
  • • Fix parser
  • • Update parameters.rb

2020-12-14

  • • Add `lr` parameter and update code
  • • Update parser

2020-12-11

  • • Add back markings for Google top stories
  • • Add X-Ray markings for Yahoo! engine
  • • Update parser
  • • Test unsuccessful searche statuses for google_reverse_image validator
  • • Update link handling
  • • Update validator

2020-12-10

  • • Make sure we are talking about the same `self.doc`
  • • Refactor regex for extracting body tag
  • • Remove duplicate line (typo)
  • • Remove typo
  • • Support X-Ray markings for Google Autocomplete pages

2020-12-09

  • • Fix issue with Google Maps being purely JSON and JS
  • • Fix parser
  • • Revert "Fix parser"
  • • Fix parser
  • • Add another condition to distinguish between kg and product result blocks

2020-12-08

  • • Extract selector for inline products to a variable

2020-12-07

  • • Fix rating selector for Google Shopping
  • • Update parser
  • • Fix Google gps coord extraction
  • • Improve Google lcl parser code

2020-12-04

  • • Changed json response scheme. Renamed 'description' -> 'occupation' Added 'about' Transformed 'meta' into 'last_job' and 'last_education'
  • • Revert "Revert "Fix address extraction from local results""

2020-12-02

  • • Removed outdated test examples.

2020-11-30

  • • Update parser
  • • Stop showing empty results
  • • Update parser
  • • Update parser
  • • Fix Google top stories empty results
  • • Removed stripping of a RLM and hypen minus in organic results date extraction. Code style changes. Changed test descriptions.
  • • Remove left-to-right marks.

2020-11-27

  • • Fix Bing JS images extraction
  • • Remove unneeded kg_block_header_name assignment

2020-11-26

  • • Handle empty thumbnails.
  • • Added bot user agents that work for LinkedIn scraping. Modified LinkedIn playground to display first and last name inputs in the top. Updated documentation, used Jimmi Neutron as a name for search. Fixed error when search gives a single result redirecting to profile page. Updated tests.
  • • Code style fix.
  • • Fix regex to include optional white space
  • • Catch first occurrence
  • • Update Youtube parsing Regex
  • • Reuse selectors in kg for items and headers

2020-11-25

  • • Rename LocationValidator to LocationPresenceValidator
  • • Add q parameter validation to google_scholar_cite engine
  • • Update parser to not mute airbrake notification
  • • Fix parser

2020-11-24

  • • Improve X-Ray markings for inline products
  • • Group regexes for JS image extraction again
  • • Update parser
  • • Update parser
  • • Update parser

2020-11-23

  • • Linkedin profile API initial commit.
  • • Fix Bing location cookie value for desktop
  • • Add Location validator for Bing search engine
  • • Add AtLeastOnePresentValidator and improve google scholar and walmart validation
  • • Double check and fix all validations based on parameters used
  • • Add custom conflicting parameters validator
  • • Move google_listing query encoding validation to engine module
  • • Move Bing, Yahoo, eBay validation to search engine modules
  • • Move hl and gl validation to all google search engine modules
  • • Add google domain validation to all google search engines
  • • Move query params validations to engine modules

2020-11-19

  • • Update app/models/search/xray.rb
  • • Add X-Ray markings for Google inline products
  • • Refactor parameter validations into `#{engine}/parameters.rb` module
  • • Don't group JS_IMAGE_REGEXES due to no performance improvement
  • • Speed up Google Images extraction

2020-11-18

  • • Handle Walmart Product 520 responses
  • • Added missing selector for rich_snippet on desktop.
  • • Added questions extraction for questions in mobile rich snippet.
  • • Update parser and tests

2020-11-17

  • • Update parser
  • • Get extracted JS thumbnails in O(1) time

2020-11-16

  • • Fix thumbnail extraction for Google Shopping
  • • Update and fix google product result parser
  • • Fix Yandex query_randomized
  • • Fix yandex_domain not being displayed in search parameters
  • • Mobile organic results missing rich snippet.

2020-11-13

  • • Fix Google product review results on mobile
  • • Revert "Fix address extraction from local results"
  • • Enable usage of lr parameter in Yandex search engine

2020-11-12

  • • Fix thumbnail extraction for cases when several thumbnail IDs point to the same image
  • • Fix another parser bug with treating local_results phone as address

2020-11-11

  • • Inject <head> if not present and switch to Nokogiri injection for X-Ray assets
  • • Expand X-Ray marking to more fields
  • • Defer responsability of setting xrayHtml classes to Ruby instead of JS
  • • Fix Google shopping missing title, link and remove empty extensions
  • • Remove shopping results in ads

2020-11-09

  • • Add linkedin public search API.
  • • Normalize all X-Ray markings to one style
  • • Clean up original prototyping code
  • • Expand X-Ray markings to more parsers
  • • Handle cases with unavailable item
  • • Capture Walmart product engine ReCaptcha

2020-11-06

  • • Fix Home Depot Product Search engine

2020-11-04

  • • Expand X-Ray markers to more data
  • • Get the parent Nokogiri node when getting an array of them
  • • Display better errors when not getting a Nokogiri node for X-Ray
  • • Move prototyped Nokogiri fixes from Google HTML beautifuler to X-Ray
  • • Mark snippet with `inner_wrap` to make it more readable
  • • Inject X-Ray markings via Nokogiri instead of JS
  • • Improve pagination
  • • Refactor code to get next page token

2020-11-03

  • • Correctly parse Youtube next page token

2020-11-02

  • • Changed online sellers retrieving by appending /offers to url. Updated documentation for Google Product API.

2020-11-01

  • • Fix parser

2020-10-31

  • • Fix empty results
  • • Update parser
  • • Update documentation
  • • Add results state
  • • Add spelling fix
  • • Fix and update ads parser

2020-10-30

  • • Added displayed_results/date for mobile results.
  • • Fix parser when horizontal ads appear
  • • Add support for Horizontal Ads
  • • Update youtube parser

2020-10-29

  • • Adderessed comments.
  • • WIP fixing edge cases
  • • Prototype injection of X-Ray classes using Nokogiri on the Ruby side instead of JS
  • • Changed name extraction to exclude redundant text.

2020-10-28

  • • Fix google jobs chips encoding
  • • Expand X-Ray markings to more JSON fields
  • • Scan HTML for thumbnails only once Related #948.
  • • Separate 'address' and 'phone' on Serbian local_results again

2020-10-27

  • • Use Addressable::URI for encoding and unencoding
  • • Fixed missing google product online sellers.
  • • Extend X-Ray markings to more engines and more JSON stuctures
  • • Fix Google Images having a <base> tag breaking relative paths
  • • Remove pagination parameter

2020-10-26

  • • Fix tests by preserving conditional mobile parsing order
  • • Fix mobile video results
  • • Fix Google inline videos desktop parsing
  • • Add video results parser
  • • Add support for new YouTube layout

2020-10-23

  • • Changed selector for google mobile related results.
  • • Fixed sitelinks extraction in google desktop. Skipped spec for local results direction link.
  • • Fixed google mobile results missing some organic results.

2020-10-22

  • • Added displayed_results extraction. Added tests for different localisations.
  • • Mark X-Ray for Google related searches
  • • Corrected selector for spelling fix.
  • • Encode filters parameter in Bing search
  • • Revert "Revert "Added spell check 'including results for' case.""

2020-10-21

  • • Mark for X-Ray a sample of JSON paths and Nokogiri nodes
  • • Add X-Ray JSON to CSS paths generations and injection to X-Ray view
  • • Re-use get_local_map for desktop and mobile Google local results
  • • Make sure 'address' and 'type' differ
  • • Fix phone extraction on French local results
  • • Fix phone extraction on non-English locale

2020-10-20

  • • Update validator.rb
  • • Update validator.rb
  • • Update Youtube validator

2020-10-19

  • • Revert "Added spell check 'including results for' case."

2020-10-16

  • • Extract resolution_image. Unified date extraction from organic results.
  • • Renamed thumbnail_link to thumbnail_destination_url.

2020-10-14

  • • Added spell check 'including results for' case.
  • • Fix address extraction on Serbian local results
  • • Fix description extraction on mobile Google local_results
  • • Added missing case for extract_js_image. Removed redundant condition.

2020-10-13

  • • Fixed incorrect extraction of a thumbnail. Added tests for image extractor.

2020-10-12

  • • Bind X-Ray files to homepage, playground, and inspect pages
  • • Update parser

2020-10-09

  • • Do not include rich snippets as date in mobile results
  • • Fix snippet extraction in Google mobile results
  • • Extract date and fix snippet in mobile organic results
  • • Fix snippet extraction from Google organic results

2020-10-08

  • • Add a date representation to Google organic results
  • • Make non-failing tests fail for Google local_parser

2020-10-07

  • • Remove empty lines and fix code style
  • • Fix phone extraction for some Google local results on mobile
  • • Handle empty sitelink element.

2020-10-06

  • • Fix 'address' extraction for desktop and mobile Google local results

2020-10-05

  • • Fix Bing links
  • • Fix Google organic results carousel extraction
  • • Get full snippet contents.

2020-10-02

  • • Added extra selector for answer box list extraction.
  • • Added new case for answer box table extraction.
  • • Force JSON output to use stricly 2 decimals for certain Floats
  • • Fix error 500 on blocked and recaptcha responses from Scraper API
  • • Add ads documentation
  • • Upgrade parser and tests

2020-10-01

  • • Added tests for different localisations. Fixed rating extraction for arabic language.
  • • Added new selectors for desktop snippet and inline sitelinks.
  • • Fix ads position
  • • Update and fix parsers
  • • Add support for mobile ads
  • • Update ads parser
  • • Update inline_shopping_results

2020-09-30

  • • Added serpapi_link for products in Home Depot. Updated documentation.
  • • Fix Yandex failures
  • • Update parser
  • • Update parser

2020-09-29

  • • Add support for Bing filters parameter
  • • Fix links

2020-09-25

  • • Added raw table array to result.
  • • Update parser
  • • Fix inline yandex images

2020-09-24

  • • Added fix to handle mobile expanded sitelinks cases.

2020-09-23

  • • Revert "Revert "Added extraction for mobile organic results.""
  • • Revert "Added extraction for mobile organic results."

2020-09-22

  • • Add conditional engine check for regex
  • • Fix getting thumbnails on Bing mobile
  • • Added new selector for rich snippet.

2020-09-21

  • • Moved collection assignment to conditional.
  • • Remove old Bing knowledge graph test
  • • Parse header images in Bing knowledge graph
  • • Fix bug of returning blank subtitle
  • • Add missing link and title to Bing mobile organic search results
  • • Extract description for retail store

2020-09-18

  • • Use get_valid_link method.
  • • Do not include empty thumbnails in results.

2020-09-17

  • • Added thumbnail_link for organic results.
  • • Added guard clauses for mobile sitelinks extraction.

2020-09-10

  • • Fix Google Knowledge Graph missing right image

2020-09-04

  • • Improve Bing parsing performance and fix ads parsinng
  • • Improve link selector in Google organic results

2020-09-02

  • • Remove duplication in extensions.
  • • Use get_valid_link to grab header image link in knowledge graph

2020-09-01

  • • Added fix for rating/votes in rich_snippet.
  • • Remove modular transformer and remove google redirection inline

2020-08-31

  • • Added more examples for answer box extraction.

2020-08-28

  • • Added extraction for mobile organic results. Added thumbnails. Added favicons. Added rich_snippet. Added sitelinks.
  • • Fix spec/search/youtube/video_results/star_wars_spec.rb
  • • Added price to knowledge_graph.

2020-08-26

  • • Improve knowledge graph answer selector
  • • Fix Google knowledge graph answer selector
  • • Fix spec/search/google/inline_videos/inline_videos_mobile_spec.rb
  • • Fix spec/search/ebay/spell_check/spell_check_spec.rb
  • • Fix and test knowledge graph header images extraction on mobile
  • • Add new parameter to Youtube engine to force old layout
  • • Add validator test

2020-08-25

  • • Add header images to knowledge graph

2020-08-24

  • • Fix spec/search/yahoo/validator_spec.rb
  • • Fix spec/search/walmart/featured_item/featured_item_spec.rb

2020-08-21

  • • Fix spec/search/google/sports_results/new_york_mets_spec.rb

2020-08-20

  • • Remove a level of abstraction from transformers
  • • Fix spec/search/google/sports_results/boston_red_sox_spec.rb
  • • Added fix for apply_options should be an array.
  • • Fix spec/search/google_reverse_image/reverse_image_devito_spec.rb
  • • Added first apply option from jobs api html to jobs listing api through job_id.
  • • Update app/models/search/google/parser/js_image_extractor.rb

2020-08-19

  • • Updated documentation.
  • • Style changes to direct box table parser. Modified tests to ease html refreshing.

2020-08-18

  • • Aligned image result for Home Depot product. Updated documentation.
  • • Refreshed tests and updated parser for HomeDepot.
  • • Remove location parameter

2020-08-17

  • • Updated Home Depot Product parser.
  • • Implement Google final hash transformer to modify parsed content
  • • Update validator and add empty results test

2020-08-14

  • • Add support for similar results on mobile
  • • Update parser

2020-08-13

  • • Add date parameter and remove location parameters

2020-08-12

  • • Fix spec/search/google/product_result/product_result_for_gpu_mobile_spec.rb
  • • Fix spec/search/google/local_results/local_results_for_products_spec.rb
  • • Add trending results parser
  • • Add organic results parser
  • • Update `get_thumbnail` method to accept custom id attributes
  • • Update js_image_extractor.rb

2020-08-11

  • • Fix spec/search/google/local_pack/local_pack_for_stores_spec.rb
  • • Set initial Bing News engine

2020-08-10

  • • Addressed comments.

2020-08-06

  • • Renamed variable.

2020-08-05

  • • Extract answer box table contents as a meaningful structure.
  • • Removed '/v' option from link construction.
  • • Fix inline product extraction.
  • • Partially fix Google ads

2020-08-03

  • • Fix spec/search/google/local_pack/local_pack_for_products_spec.rb
  • • Fix spec/search/google/knowledge_graph/top_carousel_spec.rb
  • • Revert "Revert "Fix parser""
  • • Update knowledge_graph.rb

2020-08-01

  • • Update HTML's, update and fix specs
  • • Fix massage_link() nil if link starts with http
  • • "//" to "https://"
  • • Fix Phone parsing for updated HTML's

2020-07-31

  • • Add parsing of Photos
  • • Fix spec/search/google/knowledge_graph/popular_times_spec.rb
  • • Fix spec/search/google/knowledge_graph/critic_reviews_user_reviews_spec.rb
  • • Remove unused code.
  • • Utilize text utils for salary extraction. Fix node extractor spec.

2020-07-30

  • • Trim whitespace for total_results
  • • Add impressions_per_month
  • • Changed specs collection matching to a style used across the project.
  • • Revert "Fix Google dictionary results parsing"
  • • Fix Google dictionary results parsing
  • • Fix eBay organic results spec
  • • Fix Google images results related searches selector

2020-07-29

  • • Fix google places title in address
  • • Fix mobile organic results extraction from Google results

2020-07-27

  • • Fix some deprecation warnings
  • • Fix 'inline_videos' extraction in Google results
  • • Fix 'related_searches' extraction on mobile Google searches
  • • Prepend URL origin to the links of Google mobile results

2020-07-24

  • • Fix typo in 'botttom' word

2020-07-23

  • • Add sourceid=chrome-mobile parameter
  • • Fix ie=UTF-8 parameter (was e=UTF-8)
  • • Fix old Google News design parsing for French language
  • • Treat Google Scholar 302 as 302 instead of success

2020-07-22

  • • Removed extraction of answer box as knowledge graph.

2020-07-21

  • • Added guard clause for local results address extraction.
  • • Fixed spelling for quotes type.

2020-07-20

  • • Extract serpapi_link to inline_images for reverse image search.

2020-07-17

  • • Added extraction of links for amp cached pages.

2020-07-16

  • • Fixed selector for answer box dynamic result.

2020-07-13

  • • Update validator.rb
  • • Fix captcha detection when following redirect

2020-07-10

  • • Extract block_position for shopping_results.

2020-07-08

  • • Fix for knowledge graph top snippet extraction.
  • • Revert "Revert "Updated knowledge graph""
  • • Revert "Updated knowledge graph"
  • • Adjusted parsing to consider i18n in responses.

2020-07-07

  • • Added amp to organic_results. Added amp to carousel in organic_result.
  • • Added top_snippet extraction for knowledge_graph. Added revenue extraction as a detail_node from top block.

2020-07-03

  • • Updated knowledge_graph see_more_about extraction. Added test for see_more_about section.

2020-06-29

  • • Updated documentation for google jobs.

2020-06-26

  • • Added image extraction. Fixed documentation image path.

2020-06-25

  • • Added google jobs listing api.
  • • Revert "Fix parser"
  • • Add service_options parser to Google kg
  • • Fix parser

2020-06-23

  • • Remove `disable_polymer` from URL

2020-06-22

  • • Fix parser

2020-06-19

  • • Update app/models/search/google/parser/knowledge_graph.rb

2020-06-17

  • • Fix parser
  • • Update parsers and rename to `amenities` `service_options`

2020-06-16

  • • Update parsers

2020-06-11

  • • Update parser

2020-06-09

  • • Handle other empty product case.

2020-06-08

  • • Removed commented unnecessary code.
  • • Use direct proxy for home depot. Fixed price extraction for home depot product.
  • • Fix and normalize authentification via API key

2020-06-04

  • • Update app/models/search/google/parser/js_image_extractor.rb
  • • Update app/models/search/google/parser/js_image_extractor.rb

2020-06-03

  • • Update js_image_extractor.rb
  • • Update sports_results.rb
  • • Update js_image_extractor.rb
  • • Fix parser

2020-06-02

  • • Split Walmart tests into required and opional

2020-05-29

  • • Added Home Depot product scraping.

2020-05-28

  • • Added autocomplete changes. Added browser mimicking for playground.

2020-05-22

  • • Exclude Walmart filters and include optionally to the response

2020-05-15

  • • Update product_result parser

2020-05-14

  • • Update selectors
  • • Update events_results.rb
  • • Update product_result.rb
  • • Update refine_by.rb
  • • Update news_results.rb
  • • Add amenities to tbm=lcl results
  • • Update local_results.rb
  • • Update knowledge_graph.rb

2020-05-13

  • • Update inline_shopping_results.rb
  • • Update inline_shopping_results.rb
  • • Update extract_js_image_by method
  • • Update images_results.rb
  • • Update direct_answer_box.rb
  • • Update Bing local results parser
  • • Update events_results.rb
  • • Update Google Events results parser

2020-05-12

  • • Added toolbar and youtube clients.
  • • Fix YouTube spec
  • • Document Walmart Product
  • • Document Walmart Search API

2020-05-11

  • • Add soft_sort and grid parameters
  • • Update parser
  • • Update parser

2020-05-08

  • • Add parser

2020-05-06

  • • Added prettifier for autocomplete result. Changed autocomplete to use netnut. Updated doc. Added test for multiple request ends up as 403 forbidden.
  • • Handle the case when params is nil
  • • Fix regular searches being misclassify as Google Image searches as `params["ijn"]` is always an array`
  • • compact to compact!
  • • Fix getting dupe :text and :carousel top_stories

2020-05-01

  • • Add Walmart search parameters to playground
  • • Add 'grid' parameter to Walmart search
  • • Test Walmart Product reviews
  • • Test Walmart pagination

2020-04-30

  • • Fixed emapty results message for HomeDepot.
  • • Changed rating_count to reviews. Changed HomeDepot to The Home Depot in documentation.
  • • Test Walmart product_result
  • • Fix desktop top_stories selectors
  • • Fix Google Product parsing error on empty non-English results
  • • Update parser

2020-04-29

  • • Fixed typo. Changed tests for node_extractor.
  • • Removed redundant price and rating fields from response for HomeDepot. Updated documentation.
  • • Added missing icon for autocomplete. Changed client attribute default to chrome for Search model.
  • • Fix phone parser
  • • Update app/models/search/google/parser/local_results.rb
  • • Update parser

2020-04-28

  • • Changed raw price to have dot delimiter. Changed rating to display number of stars. Added rating_raw to display rating percent. Updated documentation.
  • • Updated documentation.
  • • Added psy-ab as a client option for autocomplete.
  • • Test Walmart Product product results
  • • Revert "Fix parser"
  • • Fix lsig extraction
  • • Update gps extraction
  • • Fix parser
  • • Revert "Update parser"
  • • Update parser

2020-04-27

  • • Fixed thumnail extraction for HomeDepot.
  • • Move old price and price save into price group in response.
  • • Changed price and rating to numeric values. Changed price response to have raw, extracted and currency values. Changed tests.
  • • Fixed price extraction. Added link extraction. Added product_id extraction. Added badge extraction. Fixed documentation link to sorting/ordering.
  • • Add hl support to Google Maps

2020-04-24

  • • Add google autocomplete. Add autocomplete documentation.
  • • Extract Walmart spelling fix
  • • Update organic_results.rb
  • • Revert "Update organic_results.rb"
  • • Update organic_results.rb
  • • Update parser
  • • Update parser
  • • Pass all Walmart parameters to url
  • • Add available_on parser
  • • Add Walmart search pagination

2020-04-21

  • • Test Walmart filters and related queries

2020-04-16

  • • Integration test for Walmart organic results and Walmart Product result

2020-04-15

  • • Add walmart_product engine
  • • Escape chips for google_jobs.

2020-04-14

  • • Update thumbnail node
  • • Update inline_videos.rb
  • • Update top_stories source

2020-04-13

  • • Fix thumbnail extraction. Removed device parameter.
  • • Added pagination extraction for HomeDepot.
  • • Update inline_videos.rb
  • • Update inline_videos.rb
  • • Update news results
  • • Update parser
  • • Update parser

2020-04-10

  • • Extract related queries and filters from Walmart search
  • • Extract featured item on Walmart
  • • Stop showing event results when top_carousel is present
  • • Update parser

2020-04-09

  • • Added highlighting for HomeDepot in playground dropdown menu. Used paragraphs for documentation_examples. Used get_valid_link and get_thumbnail.
  • • Update parser

2020-04-08

  • • Update top carousel

2020-04-07

  • • Updated documentation. Changed images to jpg in order to reduce size.
  • • Added spelling check for HomeDepot.
  • • Parse Walmart organic results

2020-04-03

  • • Added filtering for HomeDepot. Added documentation for filtering, price bounds, sorting.
  • • Safe navigation for array and and if it's a string
  • • Update Google Jobs validator
  • • Fix phone extraction for Google local pack
  • • Update yahoo organic results tests
  • • Remove unused code from Google parser
  • • Update top_stories parser and tests
  • • Fix boston red sox sports results
  • • Update Google product result parser and tests
  • • Update popular products parser
  • • Fix specific address local pack parser
  • • Fix Google local pack gps coordinates

2020-04-02

  • • Implement changes
  • • Update app/models/search/google/parser.rb
  • • Update node_extractor.rb
  • • Update Bing local_results parser

2020-04-01

  • • Added HomeDepot scraping.
  • • Update app/models/search/google/parser.rb
  • • Update app/models/search/google/parser.rb
  • • Update app/models/search/google/parser.rb
  • • Update app/models/search/google/parser/local_results.rb
  • • Update parser

2020-03-31

  • • Fix places results gps for some searches
  • • Exclude related searches widget from images_results

2020-03-30

  • • Add parser

2020-03-28

  • • Fix lsig extraction

2020-03-27

  • • Fix parser

2020-03-26

  • • Update rich_snippet parser
  • • Fix and update popular_times parser
  • • Add question mark at the end of method name to indicate boolean

2020-03-25

  • • Added new selector for top stories title.
  • • Update parsers
  • • Restructure parser logic

2020-03-24

  • • Fix displayed_brand and displayed_link
  • • Fix people_also_search_for parser
  • • Fix phone parser across apis
  • • Add constraints on thumbnail extraction as an option.
  • • Fix parameter encoding
  • • Rename hashes

2020-03-23

  • • Update spec
  • • Fix thumbnail extraction from inline top stories

2020-03-20

  • • Clean up the checks for empty organic results
  • • Look for preceding g-tray-header to define group name.
  • • Add validator
  • • Remove json parser
  • • Sepparate main `organic` results parser to its own file
  • • Update documentation
  • • Changed tests for mobile top stories.
  • • Remove unnecessary whitespace

2020-03-19

  • • Fix inline top stories for mobile.
  • • Update parser
  • • Revert "Fix Google product pagination"
  • • Fix Google Product star rating extraction with no reviews

2020-03-18

  • • Implement changes
  • • Update parser to support live videos
  • • Add hl parameter
  • • Finish parser

2020-03-17

  • • Add request retrying to Baidu for empty results

2020-03-14

  • • Update selectors

2020-03-13

  • • Fix selector
  • • Unit-test get_thumbnail
  • • Fix google_reverse_image_spec
  • • Fix design_alt_spec
  • • Fix local_results_for_products_spec

2020-03-12

  • • Use get_thumbnail everywhere
  • • Normalize Google Scholar Cite
  • • Normalize Google Product

2020-03-11

  • • Update Google Product "Product not found" selector
  • • Normalize Google Jobs
  • • Normalize Google Events
  • • Normalize Google
  • • Fix Yandex infinite loop on empty results response
  • • Normalize Yandex
  • • Normalize Yahoo!
  • • Normalize Bing
  • • Normalize Baidu

2020-03-10

  • • Fixed missing inline videos for mobile. Added carousel parsing for organic results for mobile. Fixed related searches for mobile.
  • • Fix spec/search/google/images_results/images_results_spec.rb
  • • Fix spec/search/yandex/yandex_organic_results_coffee_spec.rb
  • • Add parser for old youtube layout

2020-03-09

  • • Extract JS image for Google Local results thumbnails
  • • Fix events_results_mobile_spec tests: added selector for table layout of events
  • • #611: Get additional inline_top_stories from js.

2020-03-06

  • • Extract query_displayed from script.
  • • Added comment on query_displayed for spelling fix result.
  • • Added spelling fix for yandex search.
  • • Convert location_requested into location for and api request.

2020-03-05

  • • Fix token authentication for Guests
  • • Add new saved HTMLs

2020-03-04

  • • Fixed misspelling for multiword query.
  • • Fix Baidu captcha not getting detected
  • • Check for num parameter value and set to 21 if value is 20
  • • Fix 302 loop
  • • Disable youtube polymer

2020-03-03

  • • Added search_information for Yandex engine.
  • • Add device and location to the cypher key to make it more unique (search md5 uses user_id which is not know in token auth)
  • • Update parser
  • • Implement the changes
  • • Remove device from query
  • • Fix Ebay encoding

2020-03-02

  • • Fix api pagination bug with escaping location.
  • • Remove debugging statements
  • • Convert to binary encoding to store more hex chars in the same space
  • • Fix Yandex encoding

2020-02-28

  • • Fixed yandex loop issues.
  • • Add more parsers
  • • Add support for more parameters

2020-02-27

  • • Add video_results parser
  • • Add initial parser support

2020-02-26

  • • Fixed selectors for total_results and time_taken_displayed.
  • • Update sports results parser
  • • Update Google results_state parser

2020-02-25

  • • Fix spelling_fix selector.
  • • Fix Bing local pack gps extraction
  • • Fix Bing organic results thumbnail extraction
  • • Add and update js_image_extractor for Bing
  • • Revert "Revert "Fix spelling result for image searches.""
  • • Update Google product API results parser

2020-02-24

  • • Update Google product result parser
  • • Update google flights parser
  • • Update images_results.rb
  • • Fix Google direct answer box parsing against new design

2020-02-21

  • • Revert "Fix spelling result for image searches."

2020-02-20

  • • Fix and update Google lcl parser
  • • Fix and update Google local pack mobile parser
  • • Fixe spelling result for image searches.
  • • Make parameters escaping explicit.

2020-02-19

  • • Update sports results parser

2020-02-18

  • • #554: Fixed uule for reverse image search.
  • • #554: Fixed uule escaping for google jobs and events. Added location to events playground. Added location and uule to jobs playground. Added tests to ensure uule is encoded correctly.
  • • Transformed inline images tests. Fixed case for js image extraction for non-base64 images.

2020-02-14

  • • Added new selector for google total results.
  • • #198: Removed present_with_type helper. Moved yandex languages into a separate page Changed validator to compare against html elements.

2020-02-13

  • • Fixed yahoo local pack reviews for mobile.
  • • Fixed reverse images devito spec. Transformed reverse images witcher spec and made non-strict.
  • • Requested fixes and changes
  • • Changes and fixes
  • • Transformed yahoo organic tests.

2020-02-12

  • • Update parsers
  • • Return key as "unknown" if value empty

2020-02-11

  • • #198: Addressed PR comments.
  • • Make sure key is never blank to ensure JSON.load work in every languages
  • • Update parsers
  • • Add TextUtils.snakecase method
  • • Update ebay tests
  • • Update Google events

2020-02-10

  • • Remove comments.

2020-02-07

  • • #198: Addressed pr comments.
  • • Fix empty hashes by replacing parameterize method
  • • Update parser
  • • Update parser

2020-02-06

  • • Implement changes
  • • Transform test for samsung tv `tbm=shop` into # AUTO_ADD_SAVED_HTMLS
  • • Revert "Fix typo"
  • • Add documentation
  • • Update validator
  • • Restore old selectors for google jobs backwards compability.
  • • Update validator
  • • Add support for events_results_state
  • • Fixed google jobs parsing after updated jobs page design.

2020-02-05

  • • #198: removed unused code.
  • • #198: Added yandex search api. Added tests for yandex search parser organic search and knowledge graph. Added parser for organic search and knowledge graph. Added language and page parameters. Added playground. Added domains and languages. Added parser for inline images and inline videos. Added documentation. Added tests for inline images and inline videos. Fixed bug in organic results parsing. Fixed bug in inline images parsing. Made correction to documentation.
  • • Implement changes
  • • Extract anchor tags instead of list items and remove if link = item.at_css('a') check
  • • Use serpapi.com for all urls except development
  • • Add parser
  • • Remove location support from Google events
  • • Move extract_trending_searches_by_node
  • • Rename rank to position
  • • Fix typo
  • • Update extract_js_image method
  • • Add initial support for Google events engine API

2020-02-04

  • • Update app/models/search/ebay/parser/organic_results.rb
  • • Remove safe navigation as nil key for pagination should break anyway
  • • Implement the TrendingSearches parser
  • • Fix link and title parsing when tbs=shop
  • • Remove unnecessary check for existence of anchor tag in refine_by_item
  • • Add parser
  • • Fix 'serpapi_link' when in 'suggested_searches' of Google Images results when chips query is separated by spaces or '+' signs

2020-02-03

  • • Only use `serpapi.com` for production URLs
  • • Revert the location_requested backup
  • • Add parser

2020-01-31

  • • Move SearchSplitter Bing cookie logic to Validator
  • • Update links extraction to use get_valid_link method
  • • Update Ebay search API
  • • Add Google shopping results filters parser

2020-01-30

  • • Update Google product result mobile parser

2020-01-29

  • • Update parameters.rb
  • • Add price parameters
  • • Fix links in Google product results

2020-01-28

  • • Support new layout format that uses `span` instead of `cite` for inline top stories
  • • Update Ebay engine with proper parameter names
  • • Update parser.rb
  • • Update parser

2020-01-27

  • • Check if the result is sponsored
  • • Add raw and extracted price
  • • Fix related_searches parser when empty results
  • • Add pagination
  • • #160: Renamed organic_results to image_results for reverse image search.

2020-01-24

  • • #160: Prevent knowledge_graph image to appear null.
  • • #160: Removed unused headers from utils return value. Changed error message.
  • • Fix Google flight parser issue
  • • Fix typo
  • • Revert the check for "$$$"
  • • Update app/models/search/google_maps/parser/maps_search_results.rb
  • • Update parsers
  • • Add spelling fix

2020-01-23

  • • Addressed PR comments.
  • • Add comments
  • • Fix price parsing on restaurants and hotels
  • • Update organic_results to support results from foreign domains
  • • Update validator
  • • Update validator
  • • Add new parameters (pgn, ipg, ebay_domain)
  • • Add related_searches parser
  • • Add related_items parser
  • • Update get_valid_link method

2020-01-22

  • • #160: Fixed knowledge graph image.
  • • #160: Fixed knowledge graph image.
  • • Add results_state parser
  • • Add organic_results parser
  • • Add initial support for ebay engine

2020-01-21

  • • #160: Added playground. #160: Fixed storing reverse image result url.
  • • Prioritize `data-iurl` actual image urls over `data:image` ones
  • • Fix pagination
  • • Update parameters.rb

2020-01-20

  • • Update parameters.rb
  • • Return empty error if no results for a query
  • • Update validator

2020-01-17

  • • Fix rating extraction for Yahoo searches
  • • Add playground support for google scholar cite engine
  • • Add parser
  • • Add Google scholar cite engine support
  • • Generate google scholar cite link

2020-01-16

  • • #160: Added parameters, parser and validator. #160: Added extra request to get link to images. #160: Added specs. #160: Reuse google parser modules.
  • • Update hash names

2020-01-15

  • • Update menu_pages.rb
  • • Update knowledge_graph.rb
  • • Update top_stories parser
  • • Fix news parser
  • • Extrack `stick` parameter
  • • Update parser
  • • Remove top_carousel endpoint
  • • Add aditional info to Google's KG popular_times
  • • Update validator

2020-01-14

  • • Fix google_product validator
  • • Update regex

2020-01-13

  • • Enforce indentation cop on all .rb files

2020-01-10

  • • Fix product_results test by making rating a float number
  • • Fix rating extraction in google product results
  • • Add comment about music block in Baidu organic results

2020-01-09

  • • Exclude music block from organic search
  • • Fix Baidu parser
  • • #431: Replaced URI with Addressable. Removed unused code.
  • • #237: Changed default query form coffee to barista for jobs engine in playgroud. Fixed typo in query_ordered.
  • • #237: Added defensive guard clauses.
  • • #237: Addressed comments for documentation and tests.
  • • Update local_pack parser
  • • Rename PATTERNS to IMAGE_URL_PREFIXES

2020-01-08

  • • Disable all Rubocop cops except indentation-related ones
  • • Fix video_results extraction
  • • Fix inline_images parsing from Google SERPs
  • • Fix google_flight parsing
  • • Check source attribute prioritization of get_valid_link
  • • Revert changes in Bing local parser

2019-12-30

  • • Undo unescaped link error changes to validator.
  • • Fix unescaped urls raising InvalidURIError.
  • • Add missing saved HTMLs for sports results
  • • Improve get_valid_link
  • • Check if place is verified
  • • Extract gps on desktop
  • • Don't show `top_carousel` if `stick` parameter is used
  • • #237: Added parser, validator, parameters modules for jobs engine. #237: Scrape basic jobs data. #237: Added tests for jobs scraping. #237: Added chips scraping. #237: Added playground page for jobs engine. #237: Added documentation page for jobs. #237: Changed tests to use automatic test file generation.

2019-12-27

  • • Fix ruby style
  • • Don't use to_sym in Bing local results
  • • Update parser
  • • Add yahoo local pack parser
  • • Add device support to Yahoo

2019-12-26

  • • Update ads_results.rb
  • • Update ads_results.rb
  • • Update organic results for mobile support
  • • Update related_searches.rb
  • • Add device support to Yahoo
  • • Add tests
  • • Add rich_snippet parser to Yahoo organic results
  • • Update parser.rb

2019-12-25

  • • Fix inline_shopping_results_spec
  • • Update parsers
  • • Fix possible top stories exceptions
  • • Add yahoo related_searches parser
  • • Fix Google's top_stories Airbrake error
  • • Add Yahoo related_questions parser
  • • Add device support to Yahoo
  • • Add get_valid_link method to yahoo
  • • Escape url before parsing.

2019-12-24

  • • get_valid_link in all engines
  • • Update knowledge_graph.rb
  • • Update top_carousel parser
  • • Fix people_also_search_for carousel
  • • Add top_carousel parser
  • • Add profiles parser

2019-12-23

  • • Fix boston_red_sox_spec
  • • Fix shipping_results spec again
  • • Fix lionel messi stats spec
  • • Fix local_pack_for_specific_address_desktop_spec
  • • Fix tests in refiny_by spec
  • • Fix tests in refine_by

2019-12-20

  • • Fix Bing local pack
  • • #250: Fixed doc example.
  • • #250: Handled ads links.
  • • Fix thumbnail and source extraction in inline top stories
  • • Update sports_results.rb

2019-12-18

  • • #250: Added extensions. Fixed incorrect price extraction.

2019-12-17

  • • Re-create .rubocop_todo.yml

2019-12-16

  • • #250: extract_numbers -> extract_number.
  • • Move top_stories_inline to mobile only method

2019-12-13

  • • #250: fixed mobile reviews parsing.
  • • Add more classes for mobile
  • • Update top_stories parser
  • • Added guard clause.
  • • Added refine_by key fallback. Added test for Japanese search.

2019-12-12

  • • Add hotel pricing parser
  • • #260: Fixed mobile parsing.
  • • Fix top stories links
  • • Fix popular_times parser
  • • Revert "Revert "260 add refine by""

2019-12-11

  • • Revert "260 add refine by"
  • • Revert "Google kg popular_times results"

2019-12-10

  • • #250: Added mobile inline shopping results.
  • • #260: Convert refine by text to symbol.
  • • Update Google shopping_results

2019-12-09

  • • Add popular_times parser
  • • Update sports_results.rb
  • • Fix and improve Baidu answer box results
  • • Fix Google sports parser

2019-12-06

  • • Add "missing" and "must_include" annotations parser
  • • Rename review_summary to user_reviews
  • • Add critic_reviews and reviews_summary to Google's kg
  • • Update Google's get_valid_link method
  • • Show original description and saved HTML file path in RSpec output
  • • Add another selector for related_questions

2019-12-04

  • • #260: Addressed review comments.
  • • Update local_results.rb
  • • Fix parser, add more options to extract gps

2019-12-03

  • • Update bing local pack
  • • revert changes made to clone_with_new_parameters method
  • • Update Bing local pack parser

2019-12-02

  • • #260: Added 'Refine by' extraction.
  • • Add Bing local pack parser
  • • Update error message
  • • Update bing validator
  • • Update some ad results to include title

2019-11-30

  • • Fix #367

2019-11-29

  • • #101: Added inline images for google organic search.
  • • Revert "Revert "Bing related searches and inline ads""

2019-11-27

  • • Revert "Bing related searches and inline ads"
  • • Lint Ruby code on pre-commit hook
  • • Remove debug.
  • • Use extract_integer.
  • • Added total_results for bing search.
  • • Improve top and bottom ads results
  • • Improve Bing parsers
  • • Add support for inline ads
  • • Add related searches parser to Bing
  • • Made fixes after code review.
  • • Be more defensive when extracting video duration
  • • Add "unsupported device" error message
  • • Fix typo

2019-11-26

  • • Add support for mobile sports parser
  • • Parse favicon in Yahoo! ads results
  • • Extract thumbnail urls for organic search from js.
  • • Fix maps API parser
  • • Move Yahoo! parsers to the separate modules

2019-11-25

  • • Parse Yahoo! search results on different locales
  • • Test Yahoo! on different languages
  • • Small fix
  • • yahoo_domain for Yahoo!
  • • Knowledge graph fields added: - See photos - See outside
  • • Update and improve sports parser

2019-11-22

  • • Update parser
  • • Remove 'n' parameter from Yahoo! search API
  • • Yahoo! search parameters on playground
  • • Add documentation
  • • Update parser and add support for baseball

2019-11-21

  • • Add parser for formula and moto gp
  • • Add parser for tennis athletes
  • • Handle Yahoo! web search parameters
  • • Add parser for basketball and american football athletes
  • • Improve `get_valid_link` method

2019-11-20

  • • Update team sport and add individual athlete (soccer) parser
  • • Yahoo! pagination
  • • Update results_state_and_spelling_fix.rb
  • • Improve parser error handling
  • • Re-use `dig`
  • • Update maps validator and tests
  • • Yahoo! query_displayed and total_results
  • • Button in ads_results for Yahoo! Search API

2019-11-19

  • • Yahoo! ads results with sitelinks
  • • WIP sports_results parser
  • • Yahoo! ads results with list sitelinks
  • • Replace blank? with present?
  • • Re-using `dig` in maps parser
  • • Add `google_domain` support to Maps
  • • Test Yahoo! agains real queries

2019-11-18

  • • Parse Yahoo! organic results
  • • Improve Maps parser
  • • Make sure place and search results are not loaded at the same time

2019-11-17

  • • Fix for when html shows wrong type of results
  • • Separate results state and spelling fix to different file

2019-11-15

  • • Update and improve `Maps` parser
  • • Add validator test
  • • Add `position` to some place results and rename hash

2019-11-12

  • • Extract total_results on Baidu
  • • Improve and update parser
  • • Revert "Improve and update parser"
  • • Add ReCaptcha validator
  • • Improve and update parser
  • • Fix Google Maps URL constructor
  • • Update inline products screenshots in documentation
  • • Extract thumbnails in compact inline products

2019-11-11

  • • Extract compact inline products
  • • Extract regular inline products
  • • Move inline_products extraction to own module
  • • Extract extended inline_products
  • • Add support for countries cities and provinces
  • • Add support for hotel ads

2019-11-08

  • • Improve `Maps API` parser
  • • Rename state from `place` to `local`
  • • Update public JSON sample

2019-11-07

  • • Add pagination.previous and fix pagination.next on Baidu

2019-11-06

  • • Update `Maps API` parser
  • • Fix Google video results parsing on mobile devices
  • • Fix tests
  • • Clarify code comment for extract_js_image_by
  • • Unit test performance-optimized JSUtils.unescape
  • • Rename `place_results` hash to `local_results (tbm=lcl)

2019-11-05

  • • WIP update Google Scholar validator
  • • Decode HTML entities with regular expressions instead of loops
  • • Add support to Google Scholar API for `empty results state`

2019-11-04

  • • WIP Add Google Maps API parser
  • • Add support for `type` parameter
  • • Test saved HTML with different designs of Google Images results
  • • Extract original image URL from JS
  • • Remove unused code

2019-11-03

  • • Fix and improve places_results parser

2019-11-01

  • • WIP Initial files for `Google Maps API`
  • • Parse Google Images from both HTML and JS

2019-10-31

  • • Start images extraction
  • • Change spelling in the documentation

2019-10-30

  • • Add top_stories_link

2019-10-29

  • • Fix tests
  • • Move top_stories extraction to own file
  • • Parse top_stories defensively
  • • Support list format of top stories
  • • Support carousel format of top stories
  • • Fix mobile ads parsing without description

2019-10-28

  • • Add support for results state to all Google API endpoints
  • • Improve places reuslts parser

2019-10-25

  • • Exclude 'rich_snippet.bottom' for organic results with list format for sitelinks
  • • Parse list format for sitelinks
  • • Update bing related questions parser
  • • Add related questions parser
  • • Document Direct Answer Box snippet selector usage in organic_results
  • • Code improvements for places results parser

2019-10-24

  • • Fix parsing of hotel bookings
  • • Exclude incomplete answer_box duplicates in organic_results
  • • Use new gps_coordinates method

2019-10-23

  • • Improve reliability of geting a place type
  • • Enable `organic_results_state` for `tbm` results
  • • Add places results parser
  • • Rename file to `places_results` and move under proper conditional
  • • Use both of new and old answer box CSS selectors

2019-10-22

  • • Test Bing pagination with high number of results
  • • Handle pagination without 'other_pages'
  • • Fix GPS coordinates extraction in local results
  • • Fix 'knowledge_graph.people_also_search_for' parsing
  • • Update selectors to parse dictionary answer box of Google results
  • • WIP tbm local results parser

2019-10-21

  • • Move pagination-specific methods to correct modules

2019-10-19

  • • Fix for product results
  • • Fix for local pack results

2019-10-18

  • • Add support for editorial_reviews to knowledge_graph and update description
  • • Test pagination in Google Scholar results
  • • Small fix

2019-10-17

  • • Baidu integration test
  • • Update `local_pack` parser
  • • Bing pagination
  • • Edit `get_valid_link` method to support `data:image` links
  • • Update product_results parser

2019-10-16

  • • Add serpapi_pagination to Baidu
  • • Unify pagination in Google Scholar
  • • Unify pagination and serpapi_pagination in google's organic results
  • • Improve gps and valid link methods

2019-10-14

  • • WIP local_pack parser
  • • Update `get_valid_link` method
  • • Extract pagination to separate method
  • • Don't set organic_results_state when no results are found
  • • Rework `extract_gps_coordinates` to enable `js` extraction
  • • Additional case for 'Empty showing fixed spelling results' state
  • • Code review fixes
  • • Add the `get_valid_link` method
  • • Be sure title is returned for `product_result`

2019-10-11

  • • Remove TODO
  • • Remove method for setting organic results state
  • • Extract spelling fix type
  • • Guard agains nonexisting nodes in inline videos

2019-10-10

  • • Fix Airbrake errors for `product_result`
  • • Fix bed links and comment out `lsig` parameter
  • • Fix NoMethodError for inline video title parsing

2019-10-09

  • • Move set_organic_results_state to parser
  • • Test empty results in Arabic and German locale
  • • Fix for when labels have space in them
  • • Fix syntax issue
  • • WIP - local map for specific address
  • • Add support for the new `lsig` parameter
  • • Fix some syntax issue
  • • Handle inline videos without 'channel' in HTML
  • • Tests for quotes spell fix

2019-10-08

  • • Improve method arguments naming
  • • Empty search result states implementation
  • • WIP - Add support for `Mobile v2` results
  • • Rename to `product_result` and other small improvements

2019-10-07

  • • Google product results parser

2019-10-04

  • • Test inline videos in three different languages
  • • Support results_for_exact_spelling and fully_empty states

2019-10-03

  • • Describe challenges of parsing :video_thumbnail

2019-10-02

  • • Document inline videos in organic results
  • • Base64-encode thumbnails
  • • Convert indentation to spaces
  • • Fix Google and Google Scholar parser

2019-10-01

  • • Improve organic results parser
  • • Support inline video results

2019-09-30

  • • Remove Google Search specific validators from Google Scholar
  • • Improve organic results
  • • Fix syntax
  • • Show error if Google hasn't returned any results

2019-09-25

  • • Remove parameters that don't work with Google Scholar
  • • Add support for Google Scholar `cites` and `cluster` parameters

2019-09-24

  • • WIP Google Scholar organic results
  • • WIP Google Scholar parser

2019-09-21

  • • Add support for stats to Google Scholar results
  • • Reduce some whitespace
  • • Add support for related searches to Google Scholar results

2019-09-20

  • • Add support for Google Scholar specific parameters

2019-09-17

  • • WIP Add basic skeleton to support Google Scholar API

2019-09-10

  • • WIP Support Bing organic search in mobile

2019-09-04

  • • Fix #234
  • • Fix #228

2019-09-03

  • • Fix issue with new Google design

2019-08-26

  • • Fix Baidu issues

2019-08-22

  • • Fix issue with rich snippet questions not being supported and messing with direct answer

2019-08-20

  • • Fix issue with related searches including link with blank hrefs
  • • Fix data extraction for Google Shopping API
  • • Fix product headers extraction
  • • Fix product reviews, ratings, and number of reviews extraction
  • • Don't transform a Float product rating into an Integer
  • • Fix extensions not being correctly extracted
  • • Snakecase keys to avoid JSON parsing issue for Google Product API
  • • Fix issue with extracting related searches on mobile
  • • Fix issue on mobile with organic results and ad results getting mixed up
  • • Extract more rich data form organic results extensions
  • • Stop removing direct answer from document

2019-08-19

  • • Add Google News API support for people also search for blocks
  • • Fix issue with knowledge graph when key already exists
  • • Add `raw_hours` key for people relying on a string instead of an object
  • • Fix rating and review extraction for knowledge graph API
  • • Fix position issue with Google Images Results API
  • • Add support for extra layout variation

2019-08-15

  • • Fix Bing search ads
  • • Fix issues with stopping recreating hash dynamically

2019-08-13

  • • Fix Google Images pagination end infinite loop

2019-08-11

  • • Fix issue with knowledge graph
  • • Fix issues with direct answer box and regular hash
  • • Use regular hash instead of recursive created hash

2019-08-09

  • • Fix price, extension, and snippet for shopping results #210
  • • Actually avoid parsing parent knowledge graph section
  • • Ensure `reviews_from_the_web` has results
  • • Don't parse results from parent knowledge graph section for `see_more_about1`

2019-08-08

  • • Only create hash for `see_more_about` if it exist
  • • Avoid setting empty reviews
  • • Add support for `See more about` section

2019-08-07

  • • Fix issue with valid searches about Captcha being blocked

2019-08-06

  • • Don't include missing hours in local results #93

2019-08-05

  • • Update rating to float
  • • Extract number from `review_count`
  • • Add support for businesses' weekly schedule and reviews from the web

2019-08-02

  • • Add support for custom location by latitude and longitude input
  • • Fix naming of hours, address and description in local pack results
  • • Fix missing ratings in local results box #93
  • • Fix links and title links for Binb organic results
  • • WIP Rewrite final_hash and spec logic

2019-07-30

  • • Display JSON and HTML endpoints for Baidu search
  • • Allow guest to reach to their JSON and HTML endpoints via tokens

2019-07-26

  • • Add Google Spell Check API
  • • Fix commits and PR to streamline tokens

2019-07-25

  • • Fix typo
  • • Avoid parsing answer divs when they are not there
  • • WIP Use shorter tokens that look more user friendly

2019-07-23

  • • Add `location_used` to search json when using Bing #183
  • • Fix thumbnail and images

2019-07-19

  • • Fix missing parameters error with validators
  • • Add support for setting of precise location #181
  • • Early preparation, add cookie setting

2019-07-18

  • • Fix mobile title for desktop news
  • • Fix mobile search

2019-07-16

  • • Fix Google News API

2019-07-14

  • • Fix extraction of ratings from reviews

2019-07-09

  • • Remove conditional pagination (trouble on mobile)

2019-07-05

  • • Add product id to shopping results

2019-07-02

  • • Update knowledge graph to support parsing of business' operational status #116

2019-06-27

  • • Fix issue when knowledge graph answer type is missing
  • • Remove raw_submodules from final hash
  • • chore(app/*) whitespace changes and newlines at EOF

2019-06-23

  • • merge organic_results and video_results for desktop fix video_results parser for mobile

2019-06-21

  • • Fix temporary live total results empty error
  • • Handle edge cases of Google pagination

2019-06-19

  • • Fix Base64 Errors

2019-06-17

  • • update pagination parameters with Search#clone_with_new_parameters
  • • remove `then`'s from Bing knowledge graph results

2019-06-16

  • • convert bing knowledge graph into a hash vs. list of hashes
  • • add `formatted` field to parseable bing submodules
  • • Resolve conflicts
  • • Actually fix bugs

2019-06-15

  • • Local map bug fixes

2019-06-13

  • • remove unique top-level type parsers & fallback to generics
  • • remove unused include for bing ads results

2019-06-11

  • • Beta operational Baidu parsing

2019-06-10

  • • Fix emtpy hash being evuated as true issue #167
  • • Add new serpapi search links for `place_id` #167
  • • Rename location_id to place_id to match Google own documentation #167
  • • Use `api_key` instead of `user_id` to encrypt in order to allow re-generation
  • • Use only "Success" status instead of both "Cached" and "Success" for successful searches
  • • Use tokens to authentifcate users tied to one specific request
  • • stub out unimplemented Bing KG submodule type
  • • filter out JavaScript links in bing KG link parser
  • • add bing 'people_also_search' submodule formatter
  • • add written_works, timeline, and interesting_stories bing formatters
  • • add generic bing header submodule format to food & famous people
  • • use Ruby's splat operator to get head & tail of Bing submodules
  • • add generic format logic for Bing's knowledge graph header submodule
  • • add 'Organization SocialActive' to Bing Organization types

2019-06-07

  • • Extract `ludocid` (location_id) from local pack

2019-06-06

  • • Add local map support and GPS coordinates for knowledge graph results
  • • Support for dictionary answer box
  • • Add parsing support for "Related news" and "rich snippet" results
  • • Update parser with dynamic module loading based on required element
  • • [squash] start on organization-specific Bing KG fields
  • • add Bing 'flatmodule' knowledge graph parsing
  • • add Bing 'slide' submodule parser

2019-06-05

  • • parse multiple Bing knowledge graph lists when they exist
  • • replace Bing 'answers' with 'knowledge_graph'

2019-06-03

  • • add framework that captures typical Bing answer types

2019-06-02

  • • refactor bing ads pull helper fns from organic results module
  • • move bing organic results into their own parser module
  • • move bing ads results into their own parser module

2019-05-31

  • • Fail-safes for missing elements in local pack mobile
  • • Add pagination, source icon for Baidu

2019-05-29

  • • Add parsing of address on specific style for local pack mobile
  • • Extract phone number from local pack mobile
  • • Add support for local packs with mobile device #109
  • • Initial commit

2019-05-26

  • • Fix media parsing in `product_results`

2019-05-25

  • • Add fail safes for when elements don't exist or output is nil

2019-05-24

  • • add title field to Bing ad results
  • • fixup tracking_link (vs. link) field names for Bing ad sitelinks

2019-05-23

  • • Ensure values aren't null, actually add parsing of related_search_boxes
  • • Remove `log` in validator
  • • add basic serpapi_pagination logic
  • • Modify validation of successul request in product page
  • • Finalize renaming `product` paramter to `product_id`
  • • Strip quotes at start and end of review summary
  • • Avoid parsing unrelated sections for `related_search_boxes`

2019-05-22

  • • Rename`related_search_box` to `related_search_boxes`
  • • Add documentation page, navbar menu item, and icon for related searches
  • • Routine cleanup

2019-05-21

  • • Update `specs_results` for alternate product page
  • • Update `related_products_results` for alternate product page
  • • Fix removal of text for expand description

2019-05-20

  • • Update `sellers_results` to work alongside alternate product page
  • • Update `reviews_results` to work alongside alternate product page
  • • Update `product_results` to work alongside alternate product page
  • • Use parameters `specs` and `reviews` as booleans
  • • Refactor get_related_search_box

2019-05-19

  • • Add additional selectors for parsing results on alternate product page #105
  • • Check if `title` tag contains `Product not found` text
  • • Checking if the url contains `/shopping/product` isn't really necessary
  • • Use `if` `end` block if two or more conditions are used

2019-05-18

  • • Initial write-up for related search box #129
  • • Bug fixes and adjustments

2019-05-17

  • • add Bing-specific query parameters

2019-05-15

  • • Support knowledge graph type extraction for business listing

2019-05-14

  • • replace the link with a tracking_link url
  • • Use a more precise selector for knowledge graph type extractor
  • • Fix knowledge graph type support
  • • Changes per code review

2019-05-13

  • • add bing ad parsing logic

2019-05-12

  • • add in `form=QBRE` to trigger Bing ad results
  • • move bing organic parsing into it's own fn
  • • only return valid Bing organic results

2019-05-10

  • • Support float and integer number extraction for all locales
  • • Massive overhaul
  • • add basic Bing organic result parsing logic

2019-05-03

  • • Rename store/seller to name in sellers results, more adjustments to rating/reviews parsing in reviews_results

2019-05-02

  • • Fix stars and total reviews regex for international languages (hopefully), begin documentation for subparts of product API

2019-05-01

  • • Fix google_domain setting for engines "google" and "google_product", pagination fix for sellers and revies

2019-04-30

  • • Fix parsing of Google calculator direct answer
  • • Add local stores parsing, rating and reviews to integer, fix param name on Google Prod API documentation,

2019-04-28

  • • Add some documentation for global api, reviews results. Adjust sellers results for online and local, refine filtering/sorting inline with documentation,
  • • Fix reviews, adjust price extraction for multilingual
  • • Add filtering, sorting, pagination, full results for reviews and specs/details

2019-04-26

  • • Use `unless` in replace of `if` blocks
  • • Add feature to iterate through shopping results and fetch product page results (Not completed)
  • • Restructuring
  • • Naming, ensure elements exist before creating hash and iteration
  • • There are ads but needs seperate parsing
  • • Initial Google Product parsing integration

2019-04-24

  • • Support ludocid parameter (Google My Business Listing CID)
  • • Structuring adjustments
  • • Migrating to respective branch for development

2019-04-23

  • • Add parsing for specification details section
  • • Add parsing for additional review data
  • • Fix skeleton for other engine support
  • • Initial product page parsing

2019-04-18

  • • Fix conflicting html attribute
  • • Seperate search domain and search engine parameters

2019-04-05

  • • Support basic Bing search WIP