SerpApi Release Notes


History

2021-07-15
- Fix thumbnail extraction for Google inline people also search for block

2021-07-14
- Fix Google related_questions on mobile
- Add merchant_description to Google knowledge_graph

2021-07-12
- Fix merge conflict
- Update app/models/search/duckduckgo/parser/inline_videos.rb
- Update parser

2021-07-10
- Fix Knowledge Graph not parsing full menu url
- Fix snippet class for expanded sitelinks

2021-07-09
- menu_array to menu_links
- //WIP

2021-07-08
- Revert "All problems fixed under a new pr."
- All problems fixed under a new pr.
- Correct date extraction from organic results on mobile too
- Fix to latest failing spec
- Update parser
- Restore missing # AUTO_ADD_HTMLS
- Revert "Changes to last accidental commit and new additions"
- Fix x-ray html scrolling on some engines
- Final changes before the push Missing Auto add HTMLS Indentation Differences

2021-07-07
- Simplify date extraction from Google organic results
- Changes to last accidental commit and new additions
- X-Ray for `rich_snippet_table` and `rich_snippet_list`
- Fix X-Ray for Text nodes
- Revert accidental commits

2021-07-06
- Support for older versions of knowledge graph
- Use single `at_css` call instead of multiple `search`es
- New spec file, changes on spec files, new class names, old snippet class names
- Fix finance results price movement extraction
- Update direct_answer_box.rb
- Fix parser
- Fix edge case of `original` image extraction on Google Images
- Update direct_answer_box.rb
- Fix extraction of `original` image on Google Images

2021-07-05
- Fix rating extraction in regular Google search
- Fix rating and reviews extraction in local_results
- Spec fail fix and name change
- Improvements and updates
- Update parser
- Fixed according to failing specs
- Mobile parser fix
- Update parser to scrape sources
- Add parser

2021-07-03
- Changes made according to reviews by Zil
- Duplicate definitions for date merged Reviews addressed Co-author : Ilya Zub <ilya@serpapi.com>

2021-07-02
- Update parser
- Update Google Maps
- Remove events localization parameters
- Update documentation icon
- Remove "gl" support
- Fix parser
- Update prettify html

2021-07-01
- Extract other options in Google Shopping
- Exclude links to other google_product variants
- Update validator
- Add prettify HTMLs
- More changes made from reviews
- Changes committed regarding rewiews by Ilya
- Add parser for open_state
- Miscellaneous fixes

2021-06-30
- Update facts extraction
- Revert four accidential commits to master

2021-06-29
- Resolves #1804
- Resolves #1804
- Changes to spec and better parsing
- Fixes Description not parsing
- Revert change in file permissions
- Add parser
- Changes to spec and better parsing
- Update documentation
- Update parser and add x-ray
- Fixes Description not parsing

2021-06-28
- Add playground support
- Add initial Google Maps Photos engine support
- Remove duplicate
- Add support for sitelinks expanded and inline

2021-06-26
- Fix title, adding sitelinks and xrays
- Profiles (old knowledge graph)

2021-06-25
- Possible fix to last spec fail
- Revert changes to file permissions
- Little changes
- Mark with X-Ray Baidu Related News
- Last Push
- new knowledge graph finish

2021-06-24
- Documentation for all changes and missing parts Simplified related_videos parsing
- Just before the documentation Spec file fixed, sparatic parsing failure fixed, other_area_names unique now
- Add finance answer box parser

2021-06-23
- Fix thumbnail links
- Resolves #1799 along with requested changes

2021-06-22
- get valid link method
- get valid link method
- Update app/models/search/bing/parser/knowledge_graph_results.rb
- Removing position from knowledge graph
- get valid link method
- Removing position from knowledge graph
- Removing position from knowledge graph
- Removing position from knowledge graph
- Removing position from knowledge graph
- Improve field extraction
- Fix parser
- Parse weather answer box
- Resolves #1788 alongside related spec files and test.

2021-06-21
- Update pricing (offers) selectors
- Support for Israeli pages and spec files
- Add results state for `Not many results contain`

2021-06-20
- Hardened with safe navigation
- Related videos missing alerted by spec
- Fixes related news, false snippet taking Fixes related news, age and source mistakes

2021-06-18
- Update app/models/search/google/parser/product_result.rb
- Revert "Squashed commit of the following:"
- Squashed commit of the following:
- Update parser
- Add pagination
- Add duckduckgo main documentation
- Update knowledge_graph.rb
- Remove shopping results parser
- Update parser to escape HTML
- Rename region parameter to `kl`
- Update organic_results.rb
- Update app/models/search/baidu/parser/organic_results.rb
- Update app/models/search/baidu/parser/organic_results.rb
- Update organic_results.rb
- Resolves #1701
- Update app/models/search/baidu/parser/organic_results.rb
- Update app/models/search/baidu/parser/organic_results.rb
- Update app/models/search/baidu/parser/organic_results.rb

2021-06-17
- Fix thumbnail extraction
- Update some block extractions
- Update app/models/search/baidu/parser/organic_results.rb
- Rename Google Maps Reviews `place_id` parameter to `data_id`
- Resolves #1012 New test htmls for #1012 Resolves partial description issue for #1776

2021-06-16
- Fix request url
- Update parser
- Add DuckDuckGo prettify support
- Update parser
- Add support for place_id
- Fixes #1780, Partially fixes #1776, retracts #1609

2021-06-15
- Extract place_id

2021-06-14
- Fixes #1013 , #1609

2021-06-13
- Update knowledge_graph_results.rb

2021-06-12
- Update organic_results.rb

2021-06-11
- Update product_result mobile
- Update product_result desktop
- Retrieve search token in the background and use it to make the request
- Fix failing tests for Walmart parser
- Add spelling_fix parser
- Add knowledge_graph parser
- Add shopping_results parser
- Add related_searches parser
- Add organic_results parser
- Add news_results parser
- Add inline_videos parser
- Add inline_images parser
- Add ads parser
- Add playground support
- Fix snippet and link
- Fix snippet (Yandex)

2021-06-10
- Extract table `parse_organic_results` method
- Update parser
- New bing knowledge graph
- Fix Incorrect Yandex search results

2021-06-09
- Don't fail Walmart tests when optional data is missing
- Parse pagination on Walmart beta website

2021-06-08
- Parse filters on Walmart beta website
- Extract rating and reviews from Walmart beta website
- Parse search results on the Walmart beta website

2021-06-07
- Rename answer box hash with multiple boxes

2021-06-04
- Add initial parser commit

2021-06-03
- Fix Google Product seller results parsing with the Arabic language
- Fix extraction of `duration` of `inline_video_carousels` on tablet
- Rename `inline_videos_groups` to `inline_video_carousels`
- Add TODO comment for unused conditional branch in Google Product reviews content
- Add `link`, `position` to `inline_videos_groups`

2021-06-02
- Correct spelling on the Inline Videos documentation page
- Fix parser

2021-06-01
- Rename selling_points to highlights
- Remove unneeded comment
- Extract selling_points and features
- Fix parser
- Fix parser

2021-05-31
- Extract full description of Google Product review content
- Fix features extraction
- Replace old features with new features block
- Extract rating from stars if reviews not present
- Remove old HTML files of `inline_videos`

2021-05-28
- Add comment about google product review content length
- Fix reviews pagination
- Add x-ray to reviews
- Fix rating extraction in Google Product
- WIP: Add tests for inline_videos_groups
- Add x-ray
- Add thumbnail parser

2021-05-27
- Fix Google Product description extraction

2021-05-26
- Fix parser
- Fix Bing x-ray node

2021-05-25
- Add x-ray markup to Google Product API
- Fixing missing `next` & `previous`

2021-05-24
- Fix extraction of `top_stories` on mobile

2021-05-21
- Update app/models/search/yandex/parser/pagination.rb

2021-05-20
- Mark Google Flights Answer Box with X-Ray
- Fix Google Flights parsing in `answer_box`

2021-05-18
- Fix related questions extraction
- Update xray.rb

2021-05-17
- Capture more elements in related searches block
- Edit file name from Pagination to pagination

2021-05-14
- Fix the missing eBay pagination from JSON

2021-05-12
- Fix Google Related Searches parsing on mobile
- Add X-Ray for Google `inline_videos`

2021-05-11
- Extract `platform` for horizontal Inline Videos on mobile

2021-05-10
- Adding yandex pagination
- Adding Yandex pagination to xray
- Create Pagination.rb
- Update parser.rb

2021-05-06
- Refactor Twitter Results parser

2021-05-05
- Implement Twitter results parser on desktop
- Fix parser
- Update parser
- Update parser
- Validate product_id
- Validate product_id
- Validate hd_filter_tokens
- Validation for product_id

2021-05-04
- Update parser
- Clear `start` and `num` parameters when generating `google_scholar` links
- Remove x-ray for citation table

2021-04-30
- Fix failing test
- Rename some parameters
- Update validator
- Update parser

2021-04-29
- Fix validator
- Add test for 302
- Exclude related searches from profiles

2021-04-28
- Add new element - review
- Add x-ray markup for discover_more_places
- Mark rating with x-ray

2021-04-27
- Support `position` field on `inline_videos`
- Fix Ads recognition and add x-ray markup

2021-04-26
- Improve variable naming
- Parse next div if price is present
- Fix Google local results address and hours extraction
- Fix parser

2021-04-23
- Update parser

2021-04-22
- Rename `user_id` to `author_id`
- Update parser
- Exclude Related Questions from Google Organic Results

2021-04-21
- Add pagination
- Extract SerpApi author link
- Update `get_valid_link` method
- Add prettify html for co author section
- Add engine parser
- Add parameter support

2021-04-19
- Add profiles parser for main Google Scholar organic results section

2021-04-16
- Fix parsers empty field
- Update parser
- Extract user ID
- Add Google Scholar Profiles parser

2021-04-14
- Update parser to fix position issue

2021-04-13
- Update documentation
- Fix block_position parser
- Fix parser

2021-04-09
- offers param is the same type as specs and reviews
- Make "offers" consistent with "specs", "reviews"
- Set offers=true for sellers=online parameters

2021-04-08
- Add xray markup for business operational status
- Remove unused code
- Add parser for inline shopping results
- Differentiate between temporarily and permanently closed

2021-04-07
- Fix Google Knowledge Graph business operational status

2021-04-06
- Update parser
- Improve Google Ads description selector in mobile and tablet
- Update parser
- Improve Google Ads description selector in desktop
- Add displayed_link and description x-ray markup to Google desktop ads
- Add x-ray markup for Google mobile Ads
- Fix description extraction in Google Ads desktop and mobile
- Capture new Yandex captcha page

2021-04-02
- Improve Google Autocomplete engine Youtube client regex
- Fix service_options extraction and JSON result key name
- Fix service options extraction in Google Local results
- Fix service options extraction in Google Maps Places
- Fix service options extraction in Google Search

2021-04-01
- Updated google products tests.

2021-03-30
- Updated doc pages removing local_results. Removed num parameter from google product.
- Replaced sellers with boolean offers.
- Removed nodes traversing. Added xrath text lookup.
- Updated google shopping pagination params. Changed local sellers doc page pointing deprecation.

2021-03-26
- Disable LinkedIn APIs
- Handle local_pack case with wrong address/missing phone.
- Fix exceptions in direct_answer_box_table

2021-03-25
- Fix NoMethodError in Google Local Results mobile
- Fix typo. Added guard clause.
- Fixed local pack: - title beign polluted with 'Why this ad?' - address being polluted with JS. - missing phone extraction.

2021-03-24
- Fix Google Inline People Also Search For extraction
- Updated selectors and tests for related searches.

2021-03-23
- Fix knowledge graph hours extraction. Extract hours break.
- Fixed profiles data slipping into people_also_search_for in Google knowledge graph.
- "0" specs, reviews parameters handling

2021-03-20
- Add BooleanConflictWithOtherParametersValidator

2021-03-16
- Fix Google Organic results missing snippet, date, resolution and displayed_results

2021-03-15
- Fix 'streetview' extraction for Google Local Pack

2021-03-12
- Update app/models/search/google_product/parser/sellers_results.rb
- Update app/models/search/google_product/parser/sellers_results.rb
- Update app/models/search/google_product/parser/sellers_results.rb
- Fix online_sellers

2021-03-10
- Fix Google Maps parser

2021-03-08
- Fix Google Images without auto-corrected results
- Update parser

2021-03-04
- Hardcode baidu_domain
- Fix SSRF in yandex_domain

2021-03-03
- Update parser
- Don't extract original images as thumbnails
- Don't symbolize_keys in JSON.parse
- Fix Google Images Results extraction

2021-03-02
- Test Google Maps validation for incorrect symbols
- Require comma between coordinates in `ll` parameter
- Changed svg path selectors to classes of parent elements.
- Do not parse Google's related_questions as answer_box
- Fix extraction of an answer in the knowledge graph
- Fix parser

2021-03-01
- Validate `ll` parameter for `google_maps` engine
- Update code
- Update parser
- Add documentation

2021-02-27
- Fix parser

2021-02-26
- Remove comments
- Extract related search boxes from Javascript

2021-02-25
- Extract dictionary result of Answer Box from JS
- Add `reviews_link` to Google Maps API
- Remove false duplicates of answer_box in organic results
- Fix parser
- Fix mobile parser
- Update parser

2021-02-24
- Fix Google Answer Box tests
- Update parser
- Rename data_id to place_id
- Add more parameters support
- Update parser
- Fix answer_box extraction
- Fix table answer box extraction
- Fix parser
- Extract full title without domain name.
- Revert "Revert "Scrape answer box content""

2021-02-23
- Revert "Fix wrong query_ordered when `q` is missing"

2021-02-22
- Add parser
- Add Google Maps Reviews engine
- Fix wrong query_ordered when `q` is missing
- Require `q` parameter unless Google Reverse Image
- Update google_maps_search_pagination? method

2021-02-19
- Require `q` parameter for Google engine
- Move Google Maps search pagination condition to separate method

2021-02-18
- Add more explicit message for missing `ll` parameter when using pagination

2021-02-17
- Extract local_results extraction to parse_search_results_by_node
- Remove `alt` parameter from SerpApi link
- Fix empty results
- Add pagination to Google Maps

2021-02-16
- Improve readability of query_generic method

2021-02-12
- Fixed google knowledge graph fields extraction.

2021-02-10
- Fix parser

2021-02-09
- Reuse latitude and longitude from `ll` parameter
- Remove Google Maps Search engine
- Small fixes
- Use google_maps engine for pagination too

2021-02-08
- Changed htilrad to lrad for radius refinement.
- Include JSDivExtractor module in reverse image API
- Prettify Google Scholar Cite HTML response

2021-02-05
- Fix parser
- Revert "Fix parser"
- Fix parser
- Update parser

2021-02-04
- Codestyle changes.
- Fix Google Jobs htidocid extraction

2021-02-03
- Extract structured extensions as detected_extensions.
- Use calculate_md5 for hashing instead of query_ordered
- Fix ratings extraction in Google Jobs Listing engine
- Fix Google Jobs Listing API caching issue
- Make constants use moduls

2021-02-02
- Update parser
- Fix parser

2021-02-01
- Update parser

2021-01-29
- Use clone_with_new_url_parameters to build serpapi link
- Restored compatibility with older html examples.
- Fix parser
- Update parser
- Update parser
- Update parser to include title in Google's kg mobile

2021-01-28
- Use unpack1 instead of unpack for performance
- Fixed selector to obtain full-length title.
- Rename constants

2021-01-27
- Added htilrad for Google Jobs.
- Update searches_controller
- Update parsers
- Add pretty html for Google Maps Search

2021-01-26
- Parse see more link in Google inline people also search for block
- Add serpapi_link inline people_also_search_for items
- Refactor Google parameters
- Scrape Google inline people also search for section
- Updated LinkedIn Profile: changed naming in json response. updated tests and documentation. fixed typos in xray marking. added xray markings for thumbnails.
- Add support for Google Maps Search engine
- Extract whole title in Google Jobs API.

2021-01-21
- Remove empty string info elements in graph_results
- Add X-Ray markings for popular times
- Rename typical_time_to_spend to typipal_time_spent
- Prefer && instead of and operator
- Scrape undisplayed images. Undisplayed images are marked as "The rest of the results might not be what you're looking for".

2021-01-19
- Update documentation
- Handle one more edge case
- Move main get_knowledge_graph method to the top
- Rewrite popular times knowledge graph parser and update test htmls

2021-01-18
- Fix desktop ad phone selector

2021-01-15
- Do not use encrypted_start_of_api_key
- Fix Google images thumbnail extraction

2021-01-12
- Added validation of profile_id.
- Added xray for LinkedIn Profile.
- Update parser

2021-01-11
- Style code to be more consistent

2021-01-08
- Add initial encryption / decryption of image JSON path

2021-01-06
- Fix job listing job ID validation error message
- Stop polluting base class with validations

2021-01-05
- Added newlines.

2021-01-03
- Fixed typo in documentation. Fixed empty result parsing. Added test.

2020-12-30
- Added uule param to be passed when used directly, not as a location.
- Removed code duplication.
- Changed YAML #load -> #safe_load. Removed unnecessary navigation operator.

2020-12-29
- Added params validation for linkedin.

2020-12-28
- Added xray marking for LinkedIn public search.
- Updated documentation.
- Fix condition ordering to fix query_displayed value
- Refactor inclusion_with_value_modifier to inclusion_case_insensitive validator
- Replace if presence checks with allow_blank option in InclusionWithValueModifierValidator
- Use allow_blank: true instead of explicit presence check
- Fix Google Scholar link extraction edge case

2020-12-25
- Updated title extraction for yandex tablet search.
- Fix google jobs params extraction. Prevent google job listing fail if param is missing.

2020-12-23
- Revert "Scrape answer box content"
- Do not display query_displayed for Google Scholar if it is empty

2020-12-21
- Add explicit cites_id and cluster_id to final json response
- Usage of cites + q triggers a search within cited articles

2020-12-18
- Support cites parameter and fix serpapi URL building

2020-12-17
- Add uncommited change.
- Fixed yandex mobile search.

2020-12-16
- Fix date, displayed_results and image resolution extraction
- Fix Rubocop linting
- Make q optional if tbs is present for google engine
- Add rich snippet `reviews` support on mobile

2020-12-15
- Added documentation for LinkedIn profile. Added tests for LinkedIn profile. Covered missing sections from profile.
- Fix X-Ray markings for Google Mobile Search
- Fix Google reverse image search returning 500
- Fix displayed results, snippet and date
- Fix parser
- Update parameters.rb

2020-12-14
- Add `lr` parameter and update code
- Update parser

2020-12-11
- Add back markings for Google top stories
- Add X-Ray markings for Yahoo! engine
- Update parser
- Test unsuccessful searche statuses for google_reverse_image validator
- Update link handling
- Update validator

2020-12-10
- Make sure we are talking about the same `self.doc`
- Refactor regex for extracting body tag
- Remove duplicate line (typo)
- Remove typo
- Support X-Ray markings for Google Autocomplete pages

2020-12-09
- Fix issue with Google Maps being purely JSON and JS
- Fix parser
- Revert "Fix parser"
- Fix parser
- Add another condition to distinguish between kg and product result blocks

2020-12-08
- Extract selector for inline products to a variable

2020-12-07
- Fix rating selector for Google Shopping
- Update parser
- Fix Google gps coord extraction
- Improve Google lcl parser code

2020-12-04
- Changed json response scheme. Renamed 'description' -> 'occupation' Added 'about' Transformed 'meta' into 'last_job' and 'last_education'
- Revert "Revert "Fix address extraction from local results""

2020-12-02
- Removed outdated test examples.

2020-11-30
- Update parser
- Stop showing empty results
- Update parser
- Update parser
- Fix Google top stories empty results
- Removed stripping of a RLM and hypen minus in organic results date extraction. Code style changes. Changed test descriptions.
- Remove left-to-right marks.

2020-11-27
- Fix Bing JS images extraction
- Remove unneeded kg_block_header_name assignment

2020-11-26
- Handle empty thumbnails.
- Added bot user agents that work for LinkedIn scraping. Modified LinkedIn playground to display first and last name inputs in the top. Updated documentation, used Jimmi Neutron as a name for search. Fixed error when search gives a single result redirecting to profile page. Updated tests.
- Code style fix.
- Fix regex to include optional white space
- Catch first occurrence
- Update Youtube parsing Regex
- Reuse selectors in kg for items and headers

2020-11-25
- Rename LocationValidator to LocationPresenceValidator
- Add q parameter validation to google_scholar_cite engine
- Update parser to not mute airbrake notification
- Fix parser

2020-11-24
- Improve X-Ray markings for inline products
- Group regexes for JS image extraction again
- Update parser
- Update parser
- Update parser

2020-11-23
- Linkedin profile API initial commit.
- Fix Bing location cookie value for desktop
- Add Location validator for Bing search engine
- Add AtLeastOnePresentValidator and improve google scholar and walmart validation
- Double check and fix all validations based on parameters used
- Add custom conflicting parameters validator
- Move google_listing query encoding validation to engine module
- Move Bing, Yahoo, eBay validation to search engine modules
- Move hl and gl validation to all google search engine modules
- Add google domain validation to all google search engines
- Move query params validations to engine modules

2020-11-19
- Update app/models/search/xray.rb
- Add X-Ray markings for Google inline products
- Refactor parameter validations into `#{engine}/parameters.rb` module
- Don't group JS_IMAGE_REGEXES due to no performance improvement
- Speed up Google Images extraction

2020-11-18
- Handle Walmart Product 520 responses
- Added missing selector for rich_snippet on desktop.
- Added questions extraction for questions in mobile rich snippet.
- Update parser and tests

2020-11-17
- Update parser
- Get extracted JS thumbnails in O(1) time

2020-11-16
- Fix thumbnail extraction for Google Shopping
- Update and fix google product result parser
- Fix Yandex query_randomized
- Fix yandex_domain not being displayed in search parameters
- Mobile organic results missing rich snippet.

2020-11-13
- Fix Google product review results on mobile
- Revert "Fix address extraction from local results"
- Enable usage of lr parameter in Yandex search engine

2020-11-12
- Fix thumbnail extraction for cases when several thumbnail IDs point to the same image
- Fix another parser bug with treating local_results phone as address

2020-11-11
- Inject <head> if not present and switch to Nokogiri injection for X-Ray assets
- Expand X-Ray marking to more fields
- Defer responsability of setting xrayHtml classes to Ruby instead of JS
- Fix Google shopping missing title, link and remove empty extensions
- Remove shopping results in ads

2020-11-09
- Add linkedin public search API.
- Normalize all X-Ray markings to one style
- Clean up original prototyping code
- Expand X-Ray markings to more parsers
- Handle cases with unavailable item
- Capture Walmart product engine ReCaptcha

2020-11-06
- Fix Home Depot Product Search engine

2020-11-04
- Expand X-Ray markers to more data
- Get the parent Nokogiri node when getting an array of them
- Display better errors when not getting a Nokogiri node for X-Ray
- Move prototyped Nokogiri fixes from Google HTML beautifuler to X-Ray
- Mark snippet with `inner_wrap` to make it more readable
- Inject X-Ray markings via Nokogiri instead of JS
- Improve pagination
- Refactor code to get next page token

2020-11-03
- Correctly parse Youtube next page token

2020-11-02
- Changed online sellers retrieving by appending /offers to url. Updated documentation for Google Product API.

2020-11-01
- Fix parser

2020-10-31
- Fix empty results
- Update parser
- Update documentation
- Add results state
- Add spelling fix
- Fix and update ads parser

2020-10-30
- Added displayed_results/date for mobile results.
- Fix parser when horizontal ads appear
- Add support for Horizontal Ads
- Update youtube parser

2020-10-29
- Adderessed comments.
- WIP fixing edge cases
- Prototype injection of X-Ray classes using Nokogiri on the Ruby side instead of JS
- Changed name extraction to exclude redundant text.

2020-10-28
- Fix google jobs chips encoding
- Expand X-Ray markings to more JSON fields
- Scan HTML for thumbnails only once Related #948.
- Separate 'address' and 'phone' on Serbian local_results again

2020-10-27
- Use Addressable::URI for encoding and unencoding
- Fixed missing google product online sellers.
- Extend X-Ray markings to more engines and more JSON stuctures
- Fix Google Images having a <base> tag breaking relative paths
- Remove pagination parameter

2020-10-26
- Fix tests by preserving conditional mobile parsing order
- Fix mobile video results
- Fix Google inline videos desktop parsing
- Add video results parser
- Add support for new YouTube layout

2020-10-23
- Changed selector for google mobile related results.
- Fixed sitelinks extraction in google desktop. Skipped spec for local results direction link.
- Fixed google mobile results missing some organic results.

2020-10-22
- Added displayed_results extraction. Added tests for different localisations.
- Mark X-Ray for Google related searches
- Corrected selector for spelling fix.
- Encode filters parameter in Bing search
- Revert "Revert "Added spell check 'including results for' case.""

2020-10-21
- Mark for X-Ray a sample of JSON paths and Nokogiri nodes
- Add X-Ray JSON to CSS paths generations and injection to X-Ray view
- Re-use get_local_map for desktop and mobile Google local results
- Make sure 'address' and 'type' differ
- Fix phone extraction on French local results
- Fix phone extraction on non-English locale

2020-10-20
- Update validator.rb
- Update validator.rb
- Update Youtube validator

2020-10-19
- Revert "Added spell check 'including results for' case."

2020-10-16
- Extract resolution_image. Unified date extraction from organic results.
- Renamed thumbnail_link to thumbnail_destination_url.

2020-10-14
- Added spell check 'including results for' case.
- Fix address extraction on Serbian local results
- Fix description extraction on mobile Google local_results
- Added missing case for extract_js_image. Removed redundant condition.

2020-10-13
- Fixed incorrect extraction of a thumbnail. Added tests for image extractor.

2020-10-12
- Bind X-Ray files to homepage, playground, and inspect pages
- Update parser

2020-10-09
- Do not include rich snippets as date in mobile results
- Fix snippet extraction in Google mobile results
- Extract date and fix snippet in mobile organic results
- Fix snippet extraction from Google organic results

2020-10-08
- Add a date representation to Google organic results
- Make non-failing tests fail for Google local_parser

2020-10-07
- Remove empty lines and fix code style
- Fix phone extraction for some Google local results on mobile
- Handle empty sitelink element.

2020-10-06
- Fix 'address' extraction for desktop and mobile Google local results

2020-10-05
- Fix Bing links
- Fix Google organic results carousel extraction
- Get full snippet contents.

2020-10-02
- Added extra selector for answer box list extraction.
- Added new case for answer box table extraction.
- Force JSON output to use stricly 2 decimals for certain Floats
- Fix error 500 on blocked and recaptcha responses from Scraper API
- Add ads documentation
- Upgrade parser and tests

2020-10-01
- Added tests for different localisations. Fixed rating extraction for arabic language.
- Added new selectors for desktop snippet and inline sitelinks.
- Fix ads position
- Update and fix parsers
- Add support for mobile ads
- Update ads parser
- Update inline_shopping_results

2020-09-30
- Added serpapi_link for products in Home Depot. Updated documentation.
- Fix Yandex failures
- Update parser
- Update parser

2020-09-29
- Add support for Bing filters parameter
- Fix links

2020-09-25
- Added raw table array to result.
- Update parser
- Fix inline yandex images

2020-09-24
- Added fix to handle mobile expanded sitelinks cases.

2020-09-23
- Revert "Revert "Added extraction for mobile organic results.""
- Revert "Added extraction for mobile organic results."

2020-09-22
- Add conditional engine check for regex
- Fix getting thumbnails on Bing mobile
- Added new selector for rich snippet.

2020-09-21
- Moved collection assignment to conditional.
- Remove old Bing knowledge graph test
- Parse header images in Bing knowledge graph
- Fix bug of returning blank subtitle
- Add missing link and title to Bing mobile organic search results
- Extract description for retail store

2020-09-18
- Use get_valid_link method.
- Do not include empty thumbnails in results.

2020-09-17
- Added thumbnail_link for organic results.
- Added guard clauses for mobile sitelinks extraction.

2020-09-10
- Fix Google Knowledge Graph missing right image

2020-09-04
- Improve Bing parsing performance and fix ads parsinng
- Improve link selector in Google organic results

2020-09-02
- Remove duplication in extensions.
- Use get_valid_link to grab header image link in knowledge graph

2020-09-01
- Added fix for rating/votes in rich_snippet.
- Remove modular transformer and remove google redirection inline

2020-08-31
- Added more examples for answer box extraction.

2020-08-28
- Added extraction for mobile organic results. Added thumbnails. Added favicons. Added rich_snippet. Added sitelinks.
- Fix spec/search/youtube/video_results/star_wars_spec.rb
- Added price to knowledge_graph.

2020-08-26
- Improve knowledge graph answer selector
- Fix Google knowledge graph answer selector
- Fix spec/search/google/inline_videos/inline_videos_mobile_spec.rb
- Fix spec/search/ebay/spell_check/spell_check_spec.rb
- Fix and test knowledge graph header images extraction on mobile
- Add new parameter to Youtube engine to force old layout
- Add validator test

2020-08-25
- Add header images to knowledge graph

2020-08-24
- Fix spec/search/yahoo/validator_spec.rb
- Fix spec/search/walmart/featured_item/featured_item_spec.rb

2020-08-21
- Fix spec/search/google/sports_results/new_york_mets_spec.rb

2020-08-20
- Remove a level of abstraction from transformers
- Fix spec/search/google/sports_results/boston_red_sox_spec.rb
- Added fix for apply_options should be an array.
- Fix spec/search/google_reverse_image/reverse_image_devito_spec.rb
- Added first apply option from jobs api html to jobs listing api through job_id.
- Update app/models/search/google/parser/js_image_extractor.rb

2020-08-19
- Updated documentation.
- Style changes to direct box table parser. Modified tests to ease html refreshing.

2020-08-18
- Aligned image result for Home Depot product. Updated documentation.
- Refreshed tests and updated parser for HomeDepot.
- Remove location parameter

2020-08-17
- Updated Home Depot Product parser.
- Implement Google final hash transformer to modify parsed content
- Update validator and add empty results test

2020-08-14
- Add support for similar results on mobile
- Update parser

2020-08-13
- Add date parameter and remove location parameters

2020-08-12
- Fix spec/search/google/product_result/product_result_for_gpu_mobile_spec.rb
- Fix spec/search/google/local_results/local_results_for_products_spec.rb
- Add trending results parser
- Add organic results parser
- Update `get_thumbnail` method to accept custom id attributes
- Update js_image_extractor.rb

2020-08-11
- Fix spec/search/google/local_pack/local_pack_for_stores_spec.rb
- Set initial Bing News engine

2020-08-10
- Addressed comments.

2020-08-06
- Renamed variable.

2020-08-05
- Extract answer box table contents as a meaningful structure.
- Removed '/v' option from link construction.
- Fix inline product extraction.
- Partially fix Google ads

2020-08-03
- Fix spec/search/google/local_pack/local_pack_for_products_spec.rb
- Fix spec/search/google/knowledge_graph/top_carousel_spec.rb
- Revert "Revert "Fix parser""
- Update knowledge_graph.rb

2020-08-01
- Update HTML's, update and fix specs
- Fix massage_link() nil if link starts with http
- "//" to "https://"
- Fix Phone parsing for updated HTML's

2020-07-31
- Add parsing of Photos
- Fix spec/search/google/knowledge_graph/popular_times_spec.rb
- Fix spec/search/google/knowledge_graph/critic_reviews_user_reviews_spec.rb
- Remove unused code.
- Utilize text utils for salary extraction. Fix node extractor spec.

2020-07-30
- Trim whitespace for total_results
- Add impressions_per_month
- Changed specs collection matching to a style used across the project.
- Revert "Fix Google dictionary results parsing"
- Fix Google dictionary results parsing
- Fix eBay organic results spec
- Fix Google images results related searches selector

2020-07-29
- Fix google places title in address
- Fix mobile organic results extraction from Google results

2020-07-27
- Fix some deprecation warnings
- Fix 'inline_videos' extraction in Google results
- Fix 'related_searches' extraction on mobile Google searches
- Prepend URL origin to the links of Google mobile results

2020-07-24
- Fix typo in 'botttom' word

2020-07-23
- Add sourceid=chrome-mobile parameter
- Fix ie=UTF-8 parameter (was e=UTF-8)
- Fix old Google News design parsing for French language
- Treat Google Scholar 302 as 302 instead of success

2020-07-22
- Removed extraction of answer box as knowledge graph.

2020-07-21
- Added guard clause for local results address extraction.
- Fixed spelling for quotes type.

2020-07-20
- Extract serpapi_link to inline_images for reverse image search.

2020-07-17
- Added extraction of links for amp cached pages.

2020-07-16
- Fixed selector for answer box dynamic result.

2020-07-13
- Update validator.rb
- Fix captcha detection when following redirect

2020-07-10
- Extract block_position for shopping_results.

2020-07-08
- Fix for knowledge graph top snippet extraction.
- Revert "Revert "Updated knowledge graph""
- Revert "Updated knowledge graph"
- Adjusted parsing to consider i18n in responses.

2020-07-07
- Added amp to organic_results. Added amp to carousel in organic_result.
- Added top_snippet extraction for knowledge_graph. Added revenue extraction as a detail_node from top block.

2020-07-03
- Updated knowledge_graph see_more_about extraction. Added test for see_more_about section.

2020-06-29
- Updated documentation for google jobs.

2020-06-26
- Added image extraction. Fixed documentation image path.

2020-06-25
- Added google jobs listing api.
- Revert "Fix parser"
- Add service_options parser to Google kg
- Fix parser

2020-06-23
- Remove `disable_polymer` from URL

2020-06-22
- Fix parser

2020-06-19
- Update app/models/search/google/parser/knowledge_graph.rb

2020-06-17
- Fix parser
- Update parsers and rename to `amenities` `service_options`

2020-06-16
- Update parsers

2020-06-11
- Update parser

2020-06-09
- Handle other empty product case.

2020-06-08
- Removed commented unnecessary code.
- Use direct proxy for home depot. Fixed price extraction for home depot product.
- Fix and normalize authentification via API key

2020-06-04
- Update app/models/search/google/parser/js_image_extractor.rb
- Update app/models/search/google/parser/js_image_extractor.rb

2020-06-03
- Update js_image_extractor.rb
- Update sports_results.rb
- Update js_image_extractor.rb
- Fix parser

2020-06-02
- Split Walmart tests into required and opional

2020-05-29
- Added Home Depot product scraping.

2020-05-28
- Added autocomplete changes. Added browser mimicking for playground.

2020-05-22
- Exclude Walmart filters and include optionally to the response

2020-05-15
- Update product_result parser

2020-05-14
- Update selectors
- Update events_results.rb
- Update product_result.rb
- Update refine_by.rb
- Update news_results.rb
- Add amenities to tbm=lcl results
- Update local_results.rb
- Update knowledge_graph.rb

2020-05-13
- Update inline_shopping_results.rb
- Update inline_shopping_results.rb
- Update extract_js_image_by method
- Update images_results.rb
- Update direct_answer_box.rb
- Update Bing local results parser
- Update events_results.rb
- Update Google Events results parser

2020-05-12
- Added toolbar and youtube clients.
- Fix YouTube spec
- Document Walmart Product
- Document Walmart Search API

2020-05-11
- Add soft_sort and grid parameters
- Update parser
- Update parser

2020-05-08
- Add parser

2020-05-06
- Added prettifier for autocomplete result. Changed autocomplete to use netnut. Updated doc. Added test for multiple request ends up as 403 forbidden.
- Handle the case when params is nil
- Fix regular searches being misclassify as Google Image searches as `params["ijn"]` is always an array`
- compact to compact!
- Fix getting dupe :text and :carousel top_stories

2020-05-01
- Add Walmart search parameters to playground
- Add 'grid' parameter to Walmart search
- Test Walmart Product reviews
- Test Walmart pagination

2020-04-30
- Fixed emapty results message for HomeDepot.
- Changed rating_count to reviews. Changed HomeDepot to The Home Depot in documentation.
- Test Walmart product_result
- Fix desktop top_stories selectors
- Fix Google Product parsing error on empty non-English results
- Update parser

2020-04-29
- Fixed typo. Changed tests for node_extractor.
- Removed redundant price and rating fields from response for HomeDepot. Updated documentation.
- Added missing icon for autocomplete. Changed client attribute default to chrome for Search model.
- Fix phone parser
- Update app/models/search/google/parser/local_results.rb
- Update parser

2020-04-28
- Changed raw price to have dot delimiter. Changed rating to display number of stars. Added rating_raw to display rating percent. Updated documentation.
- Updated documentation.
- Added psy-ab as a client option for autocomplete.
- Test Walmart Product product results
- Revert "Fix parser"
- Fix lsig extraction
- Update gps extraction
- Fix parser
- Revert "Update parser"
- Update parser

2020-04-27
- Fixed thumnail extraction for HomeDepot.
- Move old price and price save into price group in response.
- Changed price and rating to numeric values. Changed price response to have raw, extracted and currency values. Changed tests.
- Fixed price extraction. Added link extraction. Added product_id extraction. Added badge extraction. Fixed documentation link to sorting/ordering.
- Add hl support to Google Maps

2020-04-24
- Add google autocomplete. Add autocomplete documentation.
- Extract Walmart spelling fix
- Update organic_results.rb
- Revert "Update organic_results.rb"
- Update organic_results.rb
- Update parser
- Update parser
- Pass all Walmart parameters to url
- Add available_on parser
- Add Walmart search pagination

2020-04-21
- Test Walmart filters and related queries

2020-04-16
- Integration test for Walmart organic results and Walmart Product result

2020-04-15
- Add walmart_product engine
- Escape chips for google_jobs.

2020-04-14
- Update thumbnail node
- Update inline_videos.rb
- Update top_stories source

2020-04-13
- Fix thumbnail extraction. Removed device parameter.
- Added pagination extraction for HomeDepot.
- Update inline_videos.rb
- Update inline_videos.rb
- Update news results
- Update parser
- Update parser

2020-04-10
- Extract related queries and filters from Walmart search
- Extract featured item on Walmart
- Stop showing event results when top_carousel is present
- Update parser

2020-04-09
- Added highlighting for HomeDepot in playground dropdown menu. Used paragraphs for documentation_examples. Used get_valid_link and get_thumbnail.
- Update parser

2020-04-08
- Update top carousel

2020-04-07
- Updated documentation. Changed images to jpg in order to reduce size.
- Added spelling check for HomeDepot.
- Parse Walmart organic results

2020-04-03
- Added filtering for HomeDepot. Added documentation for filtering, price bounds, sorting.
- Safe navigation for array and and if it's a string
- Update Google Jobs validator
- Fix phone extraction for Google local pack
- Update yahoo organic results tests
- Remove unused code from Google parser
- Update top_stories parser and tests
- Fix boston red sox sports results
- Update Google product result parser and tests
- Update popular products parser
- Fix specific address local pack parser
- Fix Google local pack gps coordinates

2020-04-02
- Implement changes
- Update app/models/search/google/parser.rb
- Update node_extractor.rb
- Update Bing local_results parser

2020-04-01
- Added HomeDepot scraping.
- Update app/models/search/google/parser.rb
- Update app/models/search/google/parser.rb
- Update app/models/search/google/parser.rb
- Update app/models/search/google/parser/local_results.rb
- Update parser

2020-03-31
- Fix places results gps for some searches
- Exclude related searches widget from images_results

2020-03-30
- Add parser

2020-03-28
- Fix lsig extraction

2020-03-27
- Fix parser

2020-03-26
- Update rich_snippet parser
- Fix and update popular_times parser
- Add question mark at the end of method name to indicate boolean

2020-03-25
- Added new selector for top stories title.
- Update parsers
- Restructure parser logic

2020-03-24
- Fix displayed_brand and displayed_link
- Fix people_also_search_for parser
- Fix phone parser across apis
- Add constraints on thumbnail extraction as an option.
- Fix parameter encoding
- Rename hashes

2020-03-23
- Update spec
- Fix thumbnail extraction from inline top stories

2020-03-20
- Clean up the checks for empty organic results
- Look for preceding g-tray-header to define group name.
- Add validator
- Remove json parser
- Sepparate main `organic` results parser to its own file
- Update documentation
- Changed tests for mobile top stories.
- Remove unnecessary whitespace

2020-03-19
- Fix inline top stories for mobile.
- Update parser
- Revert "Fix Google product pagination"
- Fix Google Product star rating extraction with no reviews

2020-03-18
- Implement changes
- Update parser to support live videos
- Add hl parameter
- Finish parser

2020-03-17
- Add request retrying to Baidu for empty results

2020-03-14
- Update selectors

2020-03-13
- Fix selector
- Unit-test get_thumbnail
- Fix google_reverse_image_spec
- Fix design_alt_spec
- Fix local_results_for_products_spec

2020-03-12
- Use get_thumbnail everywhere
- Normalize Google Scholar Cite
- Normalize Google Product

2020-03-11
- Update Google Product "Product not found" selector
- Normalize Google Jobs
- Normalize Google Events
- Normalize Google
- Fix Yandex infinite loop on empty results response
- Normalize Yandex
- Normalize Yahoo!
- Normalize Bing
- Normalize Baidu

2020-03-10
- Fixed missing inline videos for mobile. Added carousel parsing for organic results for mobile. Fixed related searches for mobile.
- Fix spec/search/google/images_results/images_results_spec.rb
- Fix spec/search/yandex/yandex_organic_results_coffee_spec.rb
- Add parser for old youtube layout

2020-03-09
- Extract JS image for Google Local results thumbnails
- Fix events_results_mobile_spec tests: added selector for table layout of events
- #611: Get additional inline_top_stories from js.

2020-03-06
- Extract query_displayed from script.
- Added comment on query_displayed for spelling fix result.
- Added spelling fix for yandex search.
- Convert location_requested into location for and api request.

2020-03-05
- Fix token authentication for Guests
- Add new saved HTMLs

2020-03-04
- Fixed misspelling for multiword query.
- Fix Baidu captcha not getting detected
- Check for num parameter value and set to 21 if value is 20
- Fix 302 loop
- Disable youtube polymer

2020-03-03
- Added search_information for Yandex engine.
- Add device and location to the cypher key to make it more unique (search md5 uses user_id which is not know in token auth)
- Update parser
- Implement the changes
- Remove device from query
- Fix Ebay encoding

2020-03-02
- Fix api pagination bug with escaping location.
- Remove debugging statements
- Convert to binary encoding to store more hex chars in the same space
- Fix Yandex encoding

2020-02-28
- Fixed yandex loop issues.
- Add more parsers
- Add support for more parameters

2020-02-27
- Add video_results parser
- Add initial parser support

2020-02-26
- Fixed selectors for total_results and time_taken_displayed.
- Update sports results parser
- Update Google results_state parser

2020-02-25
- Fix spelling_fix selector.
- Fix Bing local pack gps extraction
- Fix Bing organic results thumbnail extraction
- Add and update js_image_extractor for Bing
- Revert "Revert "Fix spelling result for image searches.""
- Update Google product API results parser

2020-02-24
- Update Google product result parser
- Update google flights parser
- Update images_results.rb
- Fix Google direct answer box parsing against new design

2020-02-21
- Revert "Fix spelling result for image searches."

2020-02-20
- Fix and update Google lcl parser
- Fix and update Google local pack mobile parser
- Fixe spelling result for image searches.
- Make parameters escaping explicit.

2020-02-19
- Update sports results parser

2020-02-18
- #554: Fixed uule for reverse image search.
- #554: Fixed uule escaping for google jobs and events. Added location to events playground. Added location and uule to jobs playground. Added tests to ensure uule is encoded correctly.
- Transformed inline images tests. Fixed case for js image extraction for non-base64 images.

2020-02-14
- Added new selector for google total results.
- #198: Removed present_with_type helper. Moved yandex languages into a separate page Changed validator to compare against html elements.

2020-02-13
- Fixed yahoo local pack reviews for mobile.
- Fixed reverse images devito spec. Transformed reverse images witcher spec and made non-strict.
- Requested fixes and changes
- Changes and fixes
- Transformed yahoo organic tests.

2020-02-12
- Update parsers
- Return key as "unknown" if value empty

2020-02-11
- #198: Addressed PR comments.
- Make sure key is never blank to ensure JSON.load work in every languages
- Update parsers
- Add TextUtils.snakecase method
- Update ebay tests
- Update Google events

2020-02-10
- Remove comments.

2020-02-07
- #198: Addressed pr comments.
- Fix empty hashes by replacing parameterize method
- Update parser
- Update parser

2020-02-06
- Implement changes
- Transform test for samsung tv `tbm=shop` into # AUTO_ADD_SAVED_HTMLS
- Revert "Fix typo"
- Add documentation
- Update validator
- Restore old selectors for google jobs backwards compability.
- Update validator
- Add support for events_results_state
- Fixed google jobs parsing after updated jobs page design.

2020-02-05
- #198: removed unused code.
- #198: Added yandex search api. Added tests for yandex search parser organic search and knowledge graph. Added parser for organic search and knowledge graph. Added language and page parameters. Added playground. Added domains and languages. Added parser for inline images and inline videos. Added documentation. Added tests for inline images and inline videos. Fixed bug in organic results parsing. Fixed bug in inline images parsing. Made correction to documentation.
- Implement changes
- Extract anchor tags instead of list items and remove if link = item.at_css('a') check
- Use serpapi.com for all urls except development
- Add parser
- Remove location support from Google events
- Move extract_trending_searches_by_node
- Rename rank to position
- Fix typo
- Update extract_js_image method
- Add initial support for Google events engine API

2020-02-04
- Update app/models/search/ebay/parser/organic_results.rb
- Remove safe navigation as nil key for pagination should break anyway
- Implement the TrendingSearches parser
- Fix link and title parsing when tbs=shop
- Remove unnecessary check for existence of anchor tag in refine_by_item
- Add parser
- Fix 'serpapi_link' when in 'suggested_searches' of Google Images results when chips query is separated by spaces or '+' signs

2020-02-03
- Only use `serpapi.com` for production URLs
- Revert the location_requested backup
- Add parser

2020-01-31
- Move SearchSplitter Bing cookie logic to Validator
- Update links extraction to use get_valid_link method
- Update Ebay search API
- Add Google shopping results filters parser

2020-01-30
- Update Google product result mobile parser

2020-01-29
- Update parameters.rb
- Add price parameters
- Fix links in Google product results

2020-01-28
- Support new layout format that uses `span` instead of `cite` for inline top stories
- Update Ebay engine with proper parameter names
- Update parser.rb
- Update parser

2020-01-27
- Check if the result is sponsored
- Add raw and extracted price
- Fix related_searches parser when empty results
- Add pagination
- #160: Renamed organic_results to image_results for reverse image search.

2020-01-24
- #160: Prevent knowledge_graph image to appear null.
- #160: Removed unused headers from utils return value. Changed error message.
- Fix Google flight parser issue
- Fix typo
- Revert the check for "$$$"
- Update app/models/search/google_maps/parser/maps_search_results.rb
- Update parsers
- Add spelling fix

2020-01-23
- Addressed PR comments.
- Add comments
- Fix price parsing on restaurants and hotels
- Update organic_results to support results from foreign domains
- Update validator
- Update validator
- Add new parameters (pgn, ipg, ebay_domain)
- Add related_searches parser
- Add related_items parser
- Update get_valid_link method

2020-01-22
- #160: Fixed knowledge graph image.
- #160: Fixed knowledge graph image.
- Add results_state parser
- Add organic_results parser
- Add initial support for ebay engine

2020-01-21
- #160: Added playground. #160: Fixed storing reverse image result url.
- Prioritize `data-iurl` actual image urls over `data:image` ones
- Fix pagination
- Update parameters.rb

2020-01-20
- Update parameters.rb
- Return empty error if no results for a query
- Update validator

2020-01-17
- Fix rating extraction for Yahoo searches
- Add playground support for google scholar cite engine
- Add parser
- Add Google scholar cite engine support
- Generate google scholar cite link

2020-01-16
- #160: Added parameters, parser and validator. #160: Added extra request to get link to images. #160: Added specs. #160: Reuse google parser modules.
- Update hash names

2020-01-15
- Update menu_pages.rb
- Update knowledge_graph.rb
- Update top_stories parser
- Fix news parser
- Extrack `stick` parameter
- Update parser
- Remove top_carousel endpoint
- Add aditional info to Google's KG popular_times
- Update validator

2020-01-14
- Fix google_product validator
- Update regex

2020-01-13
- Enforce indentation cop on all .rb files

2020-01-10
- Fix product_results test by making rating a float number
- Fix rating extraction in google product results
- Add comment about music block in Baidu organic results

2020-01-09
- Exclude music block from organic search
- Fix Baidu parser
- #431: Replaced URI with Addressable. Removed unused code.
- #237: Changed default query form coffee to barista for jobs engine in playgroud. Fixed typo in query_ordered.
- #237: Added defensive guard clauses.
- #237: Addressed comments for documentation and tests.
- Update local_pack parser
- Rename PATTERNS to IMAGE_URL_PREFIXES

2020-01-08
- Disable all Rubocop cops except indentation-related ones
- Fix video_results extraction
- Fix inline_images parsing from Google SERPs
- Fix google_flight parsing
- Check source attribute prioritization of get_valid_link
- Revert changes in Bing local parser

2019-12-30
- Undo unescaped link error changes to validator.
- Fix unescaped urls raising InvalidURIError.
- Add missing saved HTMLs for sports results
- Improve get_valid_link
- Check if place is verified
- Extract gps on desktop
- Don't show `top_carousel` if `stick` parameter is used
- #237: Added parser, validator, parameters modules for jobs engine. #237: Scrape basic jobs data. #237: Added tests for jobs scraping. #237: Added chips scraping. #237: Added playground page for jobs engine. #237: Added documentation page for jobs. #237: Changed tests to use automatic test file generation.

2019-12-27
- Fix ruby style
- Don't use to_sym in Bing local results
- Update parser
- Add yahoo local pack parser
- Add device support to Yahoo

2019-12-26
- Update ads_results.rb
- Update ads_results.rb
- Update organic results for mobile support
- Update related_searches.rb
- Add device support to Yahoo
- Add tests
- Add rich_snippet parser to Yahoo organic results
- Update parser.rb

2019-12-25
- Fix inline_shopping_results_spec
- Update parsers
- Fix possible top stories exceptions
- Add yahoo related_searches parser
- Fix Google's top_stories Airbrake error
- Add Yahoo related_questions parser
- Add device support to Yahoo
- Add get_valid_link method to yahoo
- Escape url before parsing.

2019-12-24
- get_valid_link in all engines
- Update knowledge_graph.rb
- Update top_carousel parser
- Fix people_also_search_for carousel
- Add top_carousel parser
- Add profiles parser

2019-12-23
- Fix boston_red_sox_spec
- Fix shipping_results spec again
- Fix lionel messi stats spec
- Fix local_pack_for_specific_address_desktop_spec
- Fix tests in refiny_by spec
- Fix tests in refine_by

2019-12-20
- Fix Bing local pack
- #250: Fixed doc example.
- #250: Handled ads links.
- Fix thumbnail and source extraction in inline top stories
- Update sports_results.rb

2019-12-18
- #250: Added extensions. Fixed incorrect price extraction.

2019-12-17
- Re-create .rubocop_todo.yml

2019-12-16
- #250: extract_numbers -> extract_number.
- Move top_stories_inline to mobile only method

2019-12-13
- #250: fixed mobile reviews parsing.
- Add more classes for mobile
- Update top_stories parser
- Added guard clause.
- Added refine_by key fallback. Added test for Japanese search.

2019-12-12
- Add hotel pricing parser
- #260: Fixed mobile parsing.
- Fix top stories links
- Fix popular_times parser
- Revert "Revert "260 add refine by""

2019-12-11
- Revert "260 add refine by"
- Revert "Google kg popular_times results"

2019-12-10
- #250: Added mobile inline shopping results.
- #260: Convert refine by text to symbol.
- Update Google shopping_results

2019-12-09
- Add popular_times parser
- Update sports_results.rb
- Fix and improve Baidu answer box results
- Fix Google sports parser

2019-12-06
- Add "missing" and "must_include" annotations parser
- Rename review_summary to user_reviews
- Add critic_reviews and reviews_summary to Google's kg
- Update Google's get_valid_link method
- Show original description and saved HTML file path in RSpec output
- Add another selector for related_questions

2019-12-04
- #260: Addressed review comments.
- Update local_results.rb
- Fix parser, add more options to extract gps

2019-12-03
- Update bing local pack
- revert changes made to clone_with_new_parameters method
- Update Bing local pack parser

2019-12-02
- #260: Added 'Refine by' extraction.
- Add Bing local pack parser
- Update error message
- Update bing validator
- Update some ad results to include title

2019-11-30
- Fix #367

2019-11-29
- #101: Added inline images for google organic search.
- Revert "Revert "Bing related searches and inline ads""

2019-11-27
- Revert "Bing related searches and inline ads"
- Lint Ruby code on pre-commit hook
- Remove debug.
- Use extract_integer.
- Added total_results for bing search.
- Improve top and bottom ads results
- Improve Bing parsers
- Add support for inline ads
- Add related searches parser to Bing
- Made fixes after code review.
- Be more defensive when extracting video duration
- Add "unsupported device" error message
- Fix typo

2019-11-26
- Add support for mobile sports parser
- Parse favicon in Yahoo! ads results
- Extract thumbnail urls for organic search from js.
- Fix maps API parser
- Move Yahoo! parsers to the separate modules

2019-11-25
- Parse Yahoo! search results on different locales
- Test Yahoo! on different languages
- Small fix
- yahoo_domain for Yahoo!
- Knowledge graph fields added: - See photos - See outside
- Update and improve sports parser

2019-11-22
- Update parser
- Remove 'n' parameter from Yahoo! search API
- Yahoo! search parameters on playground
- Add documentation
- Update parser and add support for baseball

2019-11-21
- Add parser for formula and moto gp
- Add parser for tennis athletes
- Handle Yahoo! web search parameters
- Add parser for basketball and american football athletes
- Improve `get_valid_link` method

2019-11-20
- Update team sport and add individual athlete (soccer) parser
- Yahoo! pagination
- Update results_state_and_spelling_fix.rb
- Improve parser error handling
- Re-use `dig`
- Update maps validator and tests
- Yahoo! query_displayed and total_results
- Button in ads_results for Yahoo! Search API

2019-11-19
- Yahoo! ads results with sitelinks
- WIP sports_results parser
- Yahoo! ads results with list sitelinks
- Replace blank? with present?
- Re-using `dig` in maps parser
- Add `google_domain` support to Maps
- Test Yahoo! agains real queries

2019-11-18
- Parse Yahoo! organic results
- Improve Maps parser
- Make sure place and search results are not loaded at the same time

2019-11-17
- Fix for when html shows wrong type of results
- Separate results state and spelling fix to different file

2019-11-15
- Update and improve `Maps` parser
- Add validator test
- Add `position` to some place results and rename hash

2019-11-12
- Extract total_results on Baidu
- Improve and update parser
- Revert "Improve and update parser"
- Add ReCaptcha validator
- Improve and update parser
- Fix Google Maps URL constructor
- Update inline products screenshots in documentation
- Extract thumbnails in compact inline products

2019-11-11
- Extract compact inline products
- Extract regular inline products
- Move inline_products extraction to own module
- Extract extended inline_products
- Add support for countries cities and provinces
- Add support for hotel ads

2019-11-08
- Improve `Maps API` parser
- Rename state from `place` to `local`
- Update public JSON sample

2019-11-07
- Add pagination.previous and fix pagination.next on Baidu

2019-11-06
- Update `Maps API` parser
- Fix Google video results parsing on mobile devices
- Fix tests
- Clarify code comment for extract_js_image_by
- Unit test performance-optimized JSUtils.unescape
- Rename `place_results` hash to `local_results (tbm=lcl)

2019-11-05
- WIP update Google Scholar validator
- Decode HTML entities with regular expressions instead of loops
- Add support to Google Scholar API for `empty results state`

2019-11-04
- WIP Add Google Maps API parser
- Add support for `type` parameter
- Test saved HTML with different designs of Google Images results
- Extract original image URL from JS
- Remove unused code

2019-11-03
- Fix and improve places_results parser

2019-11-01
- WIP Initial files for `Google Maps API`
- Parse Google Images from both HTML and JS

2019-10-31
- Start images extraction
- Change spelling in the documentation

2019-10-30
- Add top_stories_link

2019-10-29
- Fix tests
- Move top_stories extraction to own file
- Parse top_stories defensively
- Support list format of top stories
- Support carousel format of top stories
- Fix mobile ads parsing without description

2019-10-28
- Add support for results state to all Google API endpoints
- Improve places reuslts parser

2019-10-25
- Exclude 'rich_snippet.bottom' for organic results with list format for sitelinks
- Parse list format for sitelinks
- Update bing related questions parser
- Add related questions parser
- Document Direct Answer Box snippet selector usage in organic_results
- Code improvements for places results parser

2019-10-24
- Fix parsing of hotel bookings
- Exclude incomplete answer_box duplicates in organic_results
- Use new gps_coordinates method

2019-10-23
- Improve reliability of geting a place type
- Enable `organic_results_state` for `tbm` results
- Add places results parser
- Rename file to `places_results` and move under proper conditional
- Use both of new and old answer box CSS selectors

2019-10-22
- Test Bing pagination with high number of results
- Handle pagination without 'other_pages'
- Fix GPS coordinates extraction in local results
- Fix 'knowledge_graph.people_also_search_for' parsing
- Update selectors to parse dictionary answer box of Google results
- WIP tbm local results parser

2019-10-21
- Move pagination-specific methods to correct modules

2019-10-19
- Fix for product results
- Fix for local pack results

2019-10-18
- Add support for editorial_reviews to knowledge_graph and update description
- Test pagination in Google Scholar results
- Small fix

2019-10-17
- Baidu integration test
- Update `local_pack` parser
- Bing pagination
- Edit `get_valid_link` method to support `data:image` links
- Update product_results parser

2019-10-16
- Add serpapi_pagination to Baidu
- Unify pagination in Google Scholar
- Unify pagination and serpapi_pagination in google's organic results
- Improve gps and valid link methods

2019-10-14
- WIP local_pack parser
- Update `get_valid_link` method
- Extract pagination to separate method
- Don't set organic_results_state when no results are found
- Rework `extract_gps_coordinates` to enable `js` extraction
- Additional case for 'Empty showing fixed spelling results' state
- Code review fixes
- Add the `get_valid_link` method
- Be sure title is returned for `product_result`

2019-10-11
- Remove TODO
- Remove method for setting organic results state
- Extract spelling fix type
- Guard agains nonexisting nodes in inline videos

2019-10-10
- Fix Airbrake errors for `product_result`
- Fix bed links and comment out `lsig` parameter
- Fix NoMethodError for inline video title parsing

2019-10-09
- Move set_organic_results_state to parser
- Test empty results in Arabic and German locale
- Fix for when labels have space in them
- Fix syntax issue
- WIP - local map for specific address
- Add support for the new `lsig` parameter
- Fix some syntax issue
- Handle inline videos without 'channel' in HTML
- Tests for quotes spell fix

2019-10-08
- Improve method arguments naming
- Empty search result states implementation
- WIP - Add support for `Mobile v2` results
- Rename to `product_result` and other small improvements

2019-10-07
- Google product results parser

2019-10-04
- Test inline videos in three different languages
- Support results_for_exact_spelling and fully_empty states

2019-10-03
- Describe challenges of parsing :video_thumbnail

2019-10-02
- Document inline videos in organic results
- Base64-encode thumbnails
- Convert indentation to spaces
- Fix Google and Google Scholar parser

2019-10-01
- Improve organic results parser
- Support inline video results

2019-09-30
- Remove Google Search specific validators from Google Scholar
- Improve organic results
- Fix syntax
- Show error if Google hasn't returned any results

2019-09-25
- Remove parameters that don't work with Google Scholar
- Add support for Google Scholar `cites` and `cluster` parameters

2019-09-24
- WIP Google Scholar organic results
- WIP Google Scholar parser

2019-09-21
- Add support for stats to Google Scholar results
- Reduce some whitespace
- Add support for related searches to Google Scholar results

2019-09-20
- Add support for Google Scholar specific parameters

2019-09-17
- WIP Add basic skeleton to support Google Scholar API

2019-09-10
- WIP Support Bing organic search in mobile

2019-09-04
- Fix #234
- Fix #228

2019-09-03
- Fix issue with new Google design

2019-08-26
- Fix Baidu issues

2019-08-22
- Fix issue with rich snippet questions not being supported and messing with direct answer

2019-08-20
- Fix issue with related searches including link with blank hrefs
- Fix data extraction for Google Shopping API
- Fix product headers extraction
- Fix product reviews, ratings, and number of reviews extraction
- Don't transform a Float product rating into an Integer
- Fix extensions not being correctly extracted
- Snakecase keys to avoid JSON parsing issue for Google Product API
- Fix issue with extracting related searches on mobile
- Fix issue on mobile with organic results and ad results getting mixed up
- Extract more rich data form organic results extensions
- Stop removing direct answer from document

2019-08-19
- Add Google News API support for people also search for blocks
- Fix issue with knowledge graph when key already exists
- Add `raw_hours` key for people relying on a string instead of an object
- Fix rating and review extraction for knowledge graph API
- Fix position issue with Google Images Results API
- Add support for extra layout variation

2019-08-15
- Fix Bing search ads
- Fix issues with stopping recreating hash dynamically

2019-08-13
- Fix Google Images pagination end infinite loop

2019-08-11
- Fix issue with knowledge graph
- Fix issues with direct answer box and regular hash
- Use regular hash instead of recursive created hash

2019-08-09
- Fix price, extension, and snippet for shopping results #210
- Actually avoid parsing parent knowledge graph section
- Ensure `reviews_from_the_web` has results
- Don't parse results from parent knowledge graph section for `see_more_about1`

2019-08-08
- Only create hash for `see_more_about` if it exist
- Avoid setting empty reviews
- Add support for `See more about` section

2019-08-07
- Fix issue with valid searches about Captcha being blocked

2019-08-06
- Don't include missing hours in local results #93

2019-08-05
- Update rating to float
- Extract number from `review_count`
- Add support for businesses' weekly schedule and reviews from the web

2019-08-02
- Add support for custom location by latitude and longitude input
- Fix naming of hours, address and description in local pack results
- Fix missing ratings in local results box #93
- Fix links and title links for Binb organic results
- WIP Rewrite final_hash and spec logic

2019-07-30
- Display JSON and HTML endpoints for Baidu search
- Allow guest to reach to their JSON and HTML endpoints via tokens

2019-07-26
- Add Google Spell Check API
- Fix commits and PR to streamline tokens

2019-07-25
- Fix typo
- Avoid parsing answer divs when they are not there
- WIP Use shorter tokens that look more user friendly

2019-07-23
- Add `location_used` to search json when using Bing #183
- Fix thumbnail and images

2019-07-19
- Fix missing parameters error with validators
- Add support for setting of precise location #181
- Early preparation, add cookie setting

2019-07-18
- Fix mobile title for desktop news
- Fix mobile search

2019-07-16
- Fix Google News API

2019-07-14
- Fix extraction of ratings from reviews

2019-07-09
- Remove conditional pagination (trouble on mobile)

2019-07-05
- Add product id to shopping results

2019-07-02
- Update knowledge graph to support parsing of business' operational status #116

2019-06-27
- Fix issue when knowledge graph answer type is missing
- Remove raw_submodules from final hash
- chore(app/*) whitespace changes and newlines at EOF

2019-06-23
- merge organic_results and video_results for desktop fix video_results parser for mobile

2019-06-21
- Fix temporary live total results empty error
- Handle edge cases of Google pagination

2019-06-19
- Fix Base64 Errors

2019-06-17
- update pagination parameters with Search#clone_with_new_parameters
- remove `then`'s from Bing knowledge graph results

2019-06-16
- convert bing knowledge graph into a hash vs. list of hashes
- add `formatted` field to parseable bing submodules
- Resolve conflicts
- Actually fix bugs

2019-06-15
- Local map bug fixes

2019-06-13
- remove unique top-level type parsers & fallback to generics
- remove unused include for bing ads results

2019-06-11
- Beta operational Baidu parsing

2019-06-10
- Fix emtpy hash being evuated as true issue #167
- Add new serpapi search links for `place_id` #167
- Rename location_id to place_id to match Google own documentation #167
- Use `api_key` instead of `user_id` to encrypt in order to allow re-generation
- Use only "Success" status instead of both "Cached" and "Success" for successful searches
- Use tokens to authentifcate users tied to one specific request
- stub out unimplemented Bing KG submodule type
- filter out JavaScript links in bing KG link parser
- add bing 'people_also_search' submodule formatter
- add written_works, timeline, and interesting_stories bing formatters
- add generic bing header submodule format to food & famous people
- use Ruby's splat operator to get head & tail of Bing submodules
- add generic format logic for Bing's knowledge graph header submodule
- add 'Organization SocialActive' to Bing Organization types

2019-06-07
- Extract `ludocid` (location_id) from local pack

2019-06-06
- Add local map support and GPS coordinates for knowledge graph results
- Support for dictionary answer box
- Add parsing support for "Related news" and "rich snippet" results
- Update parser with dynamic module loading based on required element
- [squash] start on organization-specific Bing KG fields
- add Bing 'flatmodule' knowledge graph parsing
- add Bing 'slide' submodule parser

2019-06-05
- parse multiple Bing knowledge graph lists when they exist
- replace Bing 'answers' with 'knowledge_graph'

2019-06-03
- add framework that captures typical Bing answer types

2019-06-02
- refactor bing ads pull helper fns from organic results module
- move bing organic results into their own parser module
- move bing ads results into their own parser module

2019-05-31
- Fail-safes for missing elements in local pack mobile
- Add pagination, source icon for Baidu

2019-05-29
- Add parsing of address on specific style for local pack mobile
- Extract phone number from local pack mobile
- Add support for local packs with mobile device #109
- Initial commit

2019-05-26
- Fix media parsing in `product_results`

2019-05-25
- Add fail safes for when elements don't exist or output is nil

2019-05-24
- add title field to Bing ad results
- fixup tracking_link (vs. link) field names for Bing ad sitelinks

2019-05-23
- Ensure values aren't null, actually add parsing of related_search_boxes
- Remove `log` in validator
- add basic serpapi_pagination logic
- Modify validation of successul request in product page
- Finalize renaming `product` paramter to `product_id`
- Strip quotes at start and end of review summary
- Avoid parsing unrelated sections for `related_search_boxes`

2019-05-22
- Rename`related_search_box` to `related_search_boxes`
- Add documentation page, navbar menu item, and icon for related searches
- Routine cleanup

2019-05-21
- Update `specs_results` for alternate product page
- Update `related_products_results` for alternate product page
- Fix removal of text for expand description

2019-05-20
- Update `sellers_results` to work alongside alternate product page
- Update `reviews_results` to work alongside alternate product page
- Update `product_results` to work alongside alternate product page
- Use parameters `specs` and `reviews` as booleans
- Refactor get_related_search_box

2019-05-19
- Add additional selectors for parsing results on alternate product page #105
- Check if `title` tag contains `Product not found` text
- Checking if the url contains `/shopping/product` isn't really necessary
- Use `if` `end` block if two or more conditions are used

2019-05-18
- Initial write-up for related search box #129
- Bug fixes and adjustments

2019-05-17
- add Bing-specific query parameters

2019-05-15
- Support knowledge graph type extraction for business listing

2019-05-14
- replace the link with a tracking_link url
- Use a more precise selector for knowledge graph type extractor
- Fix knowledge graph type support
- Changes per code review

2019-05-13
- add bing ad parsing logic

2019-05-12
- add in `form=QBRE` to trigger Bing ad results
- move bing organic parsing into it's own fn
- only return valid Bing organic results

2019-05-10
- Support float and integer number extraction for all locales
- Massive overhaul
- add basic Bing organic result parsing logic

2019-05-03
- Rename store/seller to name in sellers results, more adjustments to rating/reviews parsing in reviews_results

2019-05-02
- Fix stars and total reviews regex for international languages (hopefully), begin documentation for subparts of product API

2019-05-01
- Fix google_domain setting for engines "google" and "google_product", pagination fix for sellers and revies

2019-04-30
- Fix parsing of Google calculator direct answer
- Add local stores parsing, rating and reviews to integer, fix param name on Google Prod API documentation,

2019-04-28
- Add some documentation for global api, reviews results. Adjust sellers results for online and local, refine filtering/sorting inline with documentation,
- Fix reviews, adjust price extraction for multilingual
- Add filtering, sorting, pagination, full results for reviews and specs/details

2019-04-26
- Use `unless` in replace of `if` blocks
- Add feature to iterate through shopping results and fetch product page results (Not completed)
- Restructuring
- Naming, ensure elements exist before creating hash and iteration
- There are ads but needs seperate parsing
- Initial Google Product parsing integration

2019-04-24
- Support ludocid parameter (Google My Business Listing CID)
- Structuring adjustments
- Migrating to respective branch for development

2019-04-23
- Add parsing for specification details section
- Add parsing for additional review data
- Fix skeleton for other engine support
- Initial product page parsing

2019-04-18
- Fix conflicting html attribute
- Seperate search domain and search engine parameters

2019-04-05
- Support basic Bing search WIP