SerpApi Release Notes

History

2023-06-09

• Revert json structure change

2023-06-08

• Create place searches for maps_serpapi_link where possible

2023-06-07

• Use different width height for tablet and mobile
• Apple Reviews: refactor validator condition for Item Not Available
• Include cookie to avoid empty shopping results
• Refine empty state
• Add maps_serpapi_link, photos_serpapi_link, data_id
• Add type attribute
• Add reviews_serpapi_link
• Better break down of pb parameter

2023-06-06

• instalment -> installment
• Deal with md5 calculation
• Add tests
• Check ll parameter presence
• Apple reviews: fix stuck proxies

2023-06-05

• Remove <b> tag and add value_highlight
• Add google maps autocomplete template
• Support google maps autocomplete

2023-06-02

• Add Google Local autocomplete template

2023-06-01

• Refactor duckduckgo date
• Refactor checking country
• Scrape date and date_raw for duckduckgo organic results
• Support autocomplete for Google Local

2023-05-31

• Simplify getting author data

2023-05-29

• Fix error results for some special countries

2023-05-28

• Scrape types for local results
• Scrape author info with new structure

2023-05-26

• Scrape date of organic results for duckduckgo

2023-05-24

• Google Maps: fix pagination result return empty results
• Get full monthly prices for bz shopping results

2023-05-22

• Remove 429 error check from validator
• Support as_rr parameter

2023-05-19

• Scrape google product sizes
• Clean up code

2023-05-18

• Fix failing spec
• Parse more attributes of weather

2023-05-17

• Fix weather location
• Scrape long options for Google Shopping filter

2023-05-15

• Scrape detected location

2023-05-12

• Prevent further retry when 429 status
• Remove proxy on cookie attempt

2023-05-11

• Try new `vqd` format first to speed things up
• Support new `vqd` format

2023-05-10

• Fix unicode encoded value is showing in value for arabic

2023-05-09

• Update parser
• Fix mark xray miss the first element
• Bing Local Pack: Add categories
• Bing Search API: parse new local pack places layout

2023-05-08

• Prevent empty hash in array
• Parse new weather answer box layout for mobile
• Prevent empty results and safe guarding

2023-05-06

• Add original_image
• Scrape more extensions
• Parse direct link
• Update empty result condition
• Reduce occurrence of empty results pages
• Fix Bing location cookie

2023-05-05

• Fix Google News new date selector
• Google Search API: add h,p,ws to daily forecast
• Google Search API: Extract hourly weather forecast

2023-05-04

• Turn rating into Numeric
• Parse shopping results

2023-05-03

• Parse refine this search
• Update parser
• Fix how we pass safe query
• Remove unnecessary code
• Add safe_query
• Handle google local services failed and remove serpapi proxy

2023-05-02

• Format code
• Update wording
• Parse another case for inline_ads_results
• Fix typo

2023-05-01

• Correct view full coverage link
• Google Answer Box: scrape thumbnail for finance result in desktop
• Google Related Questions: error when thumbnail content is present
• Refactor code

2023-04-30

• Scrape view full coverage link
• Scrape mobile results and sources

2023-04-28

• Remove redundant code
• Rename parameter
• Remove unnecessary logic
• Remove page parameter for playground
• Remove wrong selector for shopping_results
• Fix price_drop text for languages other than english
• Add tests
• Fix POST body request
• Add product results
• Update selector to support legacy html
• Scrape company logo for google jobs

2023-04-27

• Fix link generation
• Update POST request
• Add prettified HTML
• Update Google Images parser
• Add parser
• Add initial engine support
• Google Search API: update parser for displayed link mobile
• Google Ads API: displayed link not showing
• Google Search API: add thumbnail to ads result

2023-04-26

• Support a new layout
• Remove safeSearch
• Fix not all shopping results are parsed
• Fix invalid link
• Fix array index
• Fix shopping results selector

2023-04-25

• test(google_maps_api): Hotels pagination
• Parse source for organic_results and ads
• Support d parameter and add serpapi_link

2023-04-24

• Remove top_stories_serpapi_link and simplify doc
• Use travel-reviews selector
• Update structure
• Retry on empty response

2023-04-22

• Scrape misisng topsights fields

2023-04-21

• fix(google_maps_api): fix pagination
• Use at_css to have faster performance instead of match
• Parse related topics
• Apple App Product: parse extra app info version history
• Backward compatibility

2023-04-20

• Support top_stories on mobile
• Parse more link
• Support errorHasStatus callback result
• Parse multiple answer box for organic results

2023-04-19

• Support top_stories
• Rename watcher_count to watchers and extracted_watchers
• Remove unused check
• Fix scraping google lens results callback
• Refine xray

2023-04-18

• Fix old price not parsing
• Fix xray for thumbnail on desktop
• Add xray to attributes
• Fix safeguarding
• Add related_questions to organic_results
• Improve first page organic results
• Fix xray for google_plays engine
• Fix thumbnail not parsed

2023-04-17

• Scrape feature image for apps & games

2023-04-15

• Clean no-ops and add a request retrying spec

2023-04-14

• Fix Google Trends returning always empty results

2023-04-13

• Add inline results to ebay
• Remove empty previous pagination from the result in Ebay

2023-04-12

• Remove empty places check
• Ebay API: Update organic results with latest html
• Fix menu_items.serpapi_link

2023-04-11

• Add xray to quantity sold
• Fix query_displayed
• Remove irrelevant code

2023-04-10

• Fix duration for tablet and mobile
• Ebay API: Add quantity sold
• Fix duration not parsed
• Google Trends - prevent output error html when csv is requested
• Add proxy to google trends cookie request

2023-04-09

• Scrape kgmid without share link or tabs

2023-04-07

• Beautify and safeguard code

2023-04-06

• Add short_videos parser
• Refine code and doc
• Fixes

2023-04-05

• Adjust json structure
• Add chips parameter to playground
• Extract hidden items
• Fix xray
• Capitalize menu text
• Refactor parse_menu_items

2023-04-04

• Refine bing coupons docs
• Fix comment
• Add request retrying to Yahoo for empty results

2023-04-03

• Fix Validator literal conditions and update validator test
• Prevent unnecessary operation
• Change variable declaration to self value access
• Update supported parameter list
• Update parser
• Remove reference to tbm=isch
• Add initial support for Google Images engine
• Move images results parser to Google Images
• Fix typo and refactor parser

2023-04-02

• Refine code
• Remove tbm=vid from google parser
• Split Google Videos into a new engine

2023-03-30

• Re-use local pack parser
• Fix failing tests
• Improve code safe guard
• Fix missing to parse annual cost and after aid
• Fix organic_results.rich_snippet.bottom.questions

2023-03-29

• Extract reviews as number
• Extract all top sights
• extract_js_div: Get the last matching snippet instead of the first

2023-03-28

• Add filters to parameters
• Remove handling logic completely
• Fix hl parameter getting ignored

2023-03-27

• Safe chomp text
• Update xray for source info
• Scrape bing coupons results for desktop
• Scrape Bing coupons results

2023-03-25

• Exclude wrong image result

2023-03-24

• Refactor code style

2023-03-23

• Add tests
• Rename fields for ads
• Fix array index
• Update parser
• Get thumbnail and source log for every results
• Air quality parser - add spec
• Add full stations table data
• Air quality parse map link
• extract_js_div accept custom id name

2023-03-22

• Support bing shopping api
• Revert small changes to code style
• Add air quality answer box parser

2023-03-21

• Move institution stats function into institution stats module
• Wrapping regex into conditions
• Fix `hours` detection
• Scrape source logo
• Add new selector to Google Maps Reviews API reviews rating

2023-03-20

• Fix Mixed GTIN Searches Failing Abruptly
• Fix missing related question link and potential exception
• Change top shopping results json data structure
• Scrape Bing top shopping results
• Handle thumbnail_src nil value case
• Use parameterize to convert to snake case
• Update sports results spec to avoid false positive
• Thumbnail URL should always has complete protocol

2023-03-18

• Fix pagination issue
• Fix parser

2023-03-17

• Expand the solution for GTIN Searches to all searches

2023-03-16

• Add doc and change title to query
• Support refine_this_search
• Show digits in search_metadata after the search is successful

2023-03-15

• Reintroduce changes without mistakes
• Revert last commit
• Serve normal google_shopping_url search_metadata
• FIx Google Shopping Returning Invalid Locale
• Google Shopping API: refactor feature remarks array to dynamic digging

2023-03-14

• perf: improve performance of Google and Google Shopping validators
• Fix Google Shopping API GTIN Searches Failing
• Update home depot parser
• Update json structure
• Parse web results from things to know, and distinguish them from knowledge graph

2023-03-13

• Fix potential issue with Date.parse
• Fix exception when dates are nil
• Squish all date and time text
• Scrape all hours list
• Google Sports Results: add video highlight carousel to spotlight
• Fix typo
• Backward compatibility
• Combine organic_results and answer_table results
• Google Sports Results: miss parsing seeding as team name

2023-03-11

• fix(google): reviews in rich snippet in organic result not correct
• Parse things_to_know

2023-03-10

• Add real google_local_services_url
• Google Search API: fix weather answer box not returning precipitation and wind result

2023-03-09

• Fix Yahoo Local Pack thumbnail missing
• Fix Yahoo Dynamic links not parsed
• Fix old specs and add more specs for events
• Remove duplicated events results and scrape more fields
• Simplify code
• Google Search API: update knowledge graph parser for institution table data
• Refactor other modifications
• Fix yandex image html load error
• add safeguard

2023-03-08

• Fix Walmart Search API residue blank values
• Add fixes
• Revert tbm=lcl related changes
• Remove unused validator
• Add support for Google Local engine
• Extract GPS coordinates to separate module
• Add serpapi link
• Google Shopping API: some remark in featured shopping result is not parse
• Google Search API: add multiple kind of institutions results to knowledge graph mobile
• Google Search API: add multiple kind of institutions results to knowledge graph
• Parse vote_count
• Refactor chomp more link text

2023-03-07

• Fix Bing Local Pack not parsing website link
• Google Search API: Knowledge graph instituion stats parser update
• Scrape open hours and local hours results for places
• Google Shopping API: add store rating and reviews to documentation
• Google Shopping API: add store rating and reviews
• Google Shopping API: add features section results

2023-03-06

• Fix Yahoo Search API `ratings`, `reviews`, `more_locations_link` of `local_results` not parsed
• Fix Bing Organic Results `title` containing residue
• Parse answers for rich_snippet.bottom.extensions

2023-03-05

• Exclude text link in fact text

2023-03-03

• Fix Google Reverse Image API correct thumbnail not parsed
• Fix Ebay organic_results thumbnail not parsed
• Rename alternative price
• Update shopping results alternative prices
• Google Search Api: twitter results mobile include title from tweets

2023-03-02

• Update parser
• Fix parser
• Update parser
• Parse ads thumbnail
• Fix carousel
• Parse organic result answers
• Fix parsing inline sitelinks
• fix: attributes name change for twitter result
• Refactor delivery_options into an array
• Fix a minor no-op
• Prevent lr parameter for yandex.com
• Updates to lr and domain preferences

2023-03-01

• Support region specific cookie collection
• fix(google-search-api): parse tweets result with different author
• Definite fix (hopefully) for 302 errors in unit tests
• Handle 302 unit test errors and add a safegaurding
• Fix double :country key in parameters for display

2023-02-28

• Make related searches common parser more elegant
• Bring back an old method to parse `People Also Search For Blocks`
• Support stock_information in items and more fixes
• fix(google-search-api): parse knowledge graph institution stats
• Fix wrong cl mapping

2023-02-27

• Add parser
• Support Google Shopping Categories
• Fix protocol
• Update placement of ct parameter

2023-02-26

• Scrape price in other currency
• Fix date not parsed
• Fix key_moments not parsed

2023-02-24

• Fix organic_results with videos not parsed
• Fix more conflicts
• Support Google Shopping People Also Search For
• Add `extracted_price` and update tests
• Fix typo
• Exclude inline images from knowledge graph
• Update mobile parser

2023-02-23

• Update parser
• Fix missing domain for local map image
• Improve image related searches results
• Fix organic_result questions not parsed
• Update Related Searches extraction from JS Comments

2023-02-22

• Add Unit Tests
• Update parser
• Fix xray marking and change `directions_link` to `directions`
• Fix a mistake in the last commit
• Fix partial link on Nearby Shopping Results Map

2023-02-21

• Add Validation Concern for Local Availability Filter and Location
• Add Google Shopping Nearby Shopping Results Parser
• Add parser

2023-02-20

• Fix a duplicated match and typo
• Refactor comparison link again
• Update image size message
• Safety check empty message
• Scrape related searches for Google Images results
• Refactor checking for first ancestor
• Simplify code as only 'expanded' sitelinks were found in organic results

2023-02-17

• Remove unnecessary validation
• Update validator
• Add deprecating validator when using `place_id`
• Afterfix for second hand condition
• Create a regex for `number_of_comparisons`
• Scrape no image sizes
• Fix missing sitelinks
• Rename parser to company reviews
• Support Google Shopping Related Searches

2023-02-16

• Support Google Shopping Results Number of Comparisons
• Rename fields and scrape more typical salary info from JSON
• Support for cases in 2020
• Make all second_hand_condition lowercase

2023-02-15

• Simplify code
• Support Google Shopping API `old_price` and `extracted_old_price` for `inline_shopping_results`
• Add rescue nil to `dig`s
• Bypass baidu desktop captcha
• Fix shopping results parsed as inline_images
• Support Google Shopping API Second Hand Conditions
• Fix type test
• fix(google-jobs): change string of items to array in job highlights

2023-02-14

• Update document for new knowledge graph result
• Add validator for job type
• Make sure rating is Float
• Fix error message showing when not empty
• Rename parameters
• Scraping working at info in knowledge graph
• fix(google-jobs): job highlights with no structural
• fix(google-jobs): some job highlights content is merge

2023-02-13

• Fix Google Maps Reviews Parser residue in `snippet`
• Allow place ID parameters to be empty
• Update validator
• Add parameter validators and add POST body
• Add local place prettified HTML
• Add local place parser
• Rename date to tag and update document
• Safeguard for string
• Fix first organic result not parsed on mobile
• Fix bold text is doubled
• Fix wrong about_page_link

2023-02-12

• Include the new class into old conditions

2023-02-11

• Scrape date for google image result

2023-02-10

• Fix Google Organic Results `images` not parsed for some mobile searches
• Fix Unnecessary safeguardings
• Update local_ads parser
• Include "people also search for results" in related searches
• Update parser
• Update new regex to scrape raw data
• Fix link
• Fix regex
• Fix broken screenshot key
• Fix Google Short Videos not parsed

2023-02-09

• Fix js parsing of google inline_images
• Separate extensions
• Fix Google Events API wrong thumbnails parsed

2023-02-08

• Fix Google Events wrong event link parsed
• Fix events_results not parsed
• Fix google organic results thumbnail not parsed
• Fix google knowledge graph carousel
• spec: serpapi link q query param
• Fix Google `organic_results` `rich_snippet` -> `bottom` not parsed

2023-02-07

• Scrape job results on mobile device
• fix(google-search-api): make q optional when image search type

2023-02-06

• Fix Yandex Pagination
• Fix missing job results
• Replace optional parameter with a check for engine
• Don't re-parsing W_jd when it's empty

2023-02-05

• Scrape header images for knowledge graph
• Handle exception for some W_jd format
• Extract title and original with new html result

2023-02-03

• Make the code more elegant
• Fix a mistake on css selector
• Limit the scope of new target
• Fix Google Related Questions API not returning results
• Google Immersive Product: duplicated products
• Google Immersive Product: parse category
• Google Immersive Product: duplicated extensions
• Google Immersive Product: optimise for mobile

2023-02-01

• Update parser
• Add `squish` to title
• Add `squish` to address
• Google Immersive Product: desktop search not parse
• Walmart Product Reviews: request always fail

2023-01-31

• Add parser
• Update parser to exclude extra results
• Google Search API: knowledge graph user statistic missing

2023-01-30

• Support another case
• Fix parser
• Update necessary elements
• Fix Google Knowledge Graph `see_results_about` parsed incorrectly
• Add pagination for mobile and tablet results
• Fix Google Organic Results `displayed_results` parsed as `date` for English Results
• Fix Organic Results contaminated with KG Buttons

2023-01-26

• Improve condition usage

2023-01-25

• Add results state mobile support
• Don't use old local pack parser on tbm=lcl
• Fix Yandex Captcha appearing
• Update parsr

2023-01-24

• Update parser
• update parser
• Remove binding.pry
• Update parser
• Update validator
• fix(google-answer-box): other results in local time type

2023-01-23

• Fix Parser
• Fix parser
• Refactor regular expressions with easy-to-understand Rails functions
• Improve parser
• Fix place_id extraction
• Fix typos
• Fix an extra line

2023-01-20

• Update local results `tbm=lcl` parser
• Update local pack parser
• Fix Unnecessary Redirection Links to Direct Links in Google Shopping API

2023-01-19

• Remove duplicated parser
• Update graph regex
• Add Support for Related Shopping Results
• Remove a validator

2023-01-18

• Address another off-case for KGMID and SI Extraction

2023-01-17

• Update getting more locations link
• Change tab_links regex to cover more potential cases
• Only scrape local map and more locations link when places are present
• fix(google-answer-box): parse local time search

2023-01-13

• Revert "[Local Pack] Improve Local Pack parser"
• Support all legacy location links
• Exclude more locations link for no local pack results
• Fix some knowledge graph tabs not appearing

2023-01-12

• Remove unnecessary conditions and support a new kind in menu_items

2023-01-11

• Delete residue shopping parser
• Change parser back to Nokogiri
• Remove unnecessary condition from validator
• Remove unnecessary parameters
• Revert duplicated tag in extensions logic
• Fix a typo
• Update parser to prepare for mobile support
• Add support for tbm=lcl mobile
• Upgrade Nokogiri to Nokolexbor
• Add a new case missing address

2023-01-10

• Revert other_pages_list logic

2023-01-09

• Refactor local pack v2 and scrape place_id with new attr
• Update documentation, remove unnecessary params, and add new pagination
• Google local result start accepts multiple of 20
• Remove unnecessary Shopping Results Declaration in Parser
• Update google_finance for Nokolexbor

2023-01-06

• Exclude lsig in place_id_search link

2023-01-05

• Scrape type for device mobile
• Improve local pack parser implementation
• Revert "Generic naming"
• Revert "Extract Nokolexbor::HTML"

2023-01-04

• Update parser
• Simplify domain declaration
• Remove unnecessary presence checkers
• Check for presence of nodeset
• Simplify Results State Logic
• Update condition

2023-01-03

• Update app/models/search/google_lens/parser/knowledge_graph.rb
• Update status with "Success" response
• Add safeguard
• fix: parse new title block
• Remove duplicated local map selector and add legacy local map selector
• Support old html result and refactor code
• Refactor local result v2 result

2023-01-02

• Add x-ray for markets
• Fix validator for `type`

2022-12-30

• Restructure validator
• Use `place_id` as a default for `place_id_search` link when available

2022-12-29

• Update parser
• Update parser
• Add support for Google domain

2022-12-28

• Check for `num` presence
• Update parser
• Update parser
• Reuse `extract_price_method`
• Check text presence

2022-12-26

• Update parser
• Improve how we show UTC in graph
• New parsers cover many old specs

2022-12-25

• Remove script in address result
• New local result parser passed all old store specs
• New local result parse passed all 2022 products specs
• Detect not case 1 and exclude address extension
• Rename salary_periodicity field
• Update google jobs highlights

2022-12-24

• Re-use more code and restructure Finance parsers
• Re-use news_results from GF to GFM
• Add xray for markets

2022-12-23

• Update Google Finance answer box
• Update GFM parser
• Extract graph results
• Add reviews_original field
• Add missing search_id in log
• Fix checking case 1 with missing hours
• Remove empty local results node
• Refine google job list results and document

2022-12-22

• Fix `title` mismatched with `snippet`
• Fix HTML rendering
• Extract financials data
• Extract summary data
• Extract Nokolexbor::HTML
• Generic naming
• Refactor Apple Reviews Engine to give more than 10 pages

2022-12-21

• Nokogiri -> Nokolexbor
• Fix Google Related Questions Engine not returning any results for some next_page_token

2022-12-20

• Refactor hidden items JS extraction for better readability and safeguarding

2022-12-19

• Accept four letter `hl` parameters
• Fix X-Ray markings
• Fix title of `hover_info` picked as `answer_box` `title`

2022-12-17

• Scrape label
• Scrape year in business
• Fix thumbnail and add more specs

2022-12-16

• Try multiple strategies to scrape correct values
• Scrape order link and fix hour, address field
• Correct xray for v2 parser
• Only send airbrake issue when fallback to legacy local results parser
• Use local pack v2

2022-12-15

• Fix 500 Error on fallback parsed data
• Fix apply options results for old searches
• Scrape job highlights and related links
• add spec to inline video should not include in organic result
• refactor code

2022-12-14

• Update `Discover more` parser
• Add futures chain parser
• Add news results parser
• Add knowledge graph parser
• Add main parser
• Add initial support for Google Finance API
• Add xray for salary comparison
• fix: organic result parse content from inline video
• fix: subscribers not parsing in youtube
• fix: prevent answer box show in organic result
• fix: mobile flight duration not parsing

2022-12-13

• Fix a typo
• Make a line more elegant
• Always serve Google Product API desktop links
• Scrape essential job details
• fix: make sure answer box is present

2022-12-12

• Update parser, unit test, and add documentation

2022-12-09

• Update parser for different kinds of results
• fix: spec failing
• feat: add website to video ads

2022-12-08

• Update parser, playground, parameter definitions
• feat(google-answer-box): add flight duration
• fix: some ads results not parsing

2022-12-07

• Update parser
• Refactor key names

2022-12-06

• Support Google Knowledge Graph Buttons Results

2022-12-05

• Fix interim elements not to be parsed as price
• Fix preventing Google Local Results 0,0 coordinates from being served
• Fix Google Related Products not parsing `reviews` and `rating`
• Add parser
• Fix Ebay Search API `price` not parsed
• Update app/models/search/google_maps/parser/maps_place_results.rb
• Update app/models/search/google_maps/parser/maps_place_results.rb
• Update app/models/search/google_maps/parser/maps_place_results.rb
• Update parser
• fix: error when extension not is blank
• test: optional extensions
• fix: xray is not mark for dynamic block
• support htiltype as alternative to ltype

2022-12-03

• Update google jobs extensions

2022-12-02

• Fix css target confusion
• Add parser
• Check for elements from start to end
• Fix Google Product error when `mobile` and `tablet` devices
• Add support for Mobile Google Hotels Results Answer Box
• Scrape more local pack result cases
• Scrape no popular places
• Fix Google Knowledge Graph not parsing rest of the `web_results`

2022-12-01

• Update parser and unit tests
• Add more images link to KG
• Scrape bank local packs with website and directions

2022-11-30

• Update parser
• Add serpapi link for knowledge graph
• Better handle error in the HTML
• Fix typo
• Update serpapi_link for google job details
• fix failing spec

2022-11-29

• Don't show error if text results or related content are present
• Fix `String does not have #dig method` error
• Extract rating and reviews from KG
• Update text parser
• Update checking title content
• Fix missing title in bing images
• Update link extraction
• Fix hour div and span node both presents
• Squish extracted parts
• Update label for google places results
• Scrape google jobs link and serpapi link

2022-11-28

• Extract text results
• Extract related content
• Fix Google Organic Results with Rich Snippet List
• Safeguard old code
• Fix weird result from google autocomplete
• Fix missing phone
• Pass similar jobs to pretty template
• Remove weird result from google jobs
• Scrape similar jobs on google job listing

2022-11-26

• fix(google-answer-box): add new local time dynamic box
• fix(knowledge-graph): parse new layout for artworks

2022-11-25

• Fix Google Answer Box Hotels `rating` and `reviews`
• Fix Google Inline Shopping Results `reviews` not parsed on mobile
• Fix Google Recipes Results `rating` and `reviews`
• Fix images not parsed
• Divide parser into few files and clean unnecessary code
• Add controller and models specs
• Remove `serpapi_link`
• Fix Google Answer Box 'about_this_result' and related keys
• Handle empty results error
• Add xray support
• Update parser
• Support type and address same line
• Exclude review node without class and has reviews

2022-11-24

• Fix corrupted data in `answer_box` when `hover_info` present
• Support multiple thumbnails for places
• Scrape local pack for mobile

2022-11-23

• Fix Google Related Questions next_page_token not present for some results
• fix(google-knowledge-graph): parse more query on new header layout
• fix(google-jobs): add working from home param
• Safeguard code
• fix(walmart): search with only cat_id without query

2022-11-22

• Add a fix for a new layout and update unit test examples
• Fix Google Knowledge Graph Cast is not returned
• Spec to cover common case

2022-11-21

• Fix Organic Results Expanded and Inline Sitelinks in same result
• Squish title for expanded sitelinks
• Handle missing hours
• Add local map, price....

2022-11-20

• Scrape all fields of local results
• Scrape place_id
• Fix duplicated watch now item
• Support common result HTML: coffee, mcdonalds

2022-11-18

• Update parser
• fix(google-answer-box): replace with calculator_result
• fix: safely check result is present
• fix: remove unnecessary nil safe guard

2022-11-17

• Refactor
• Revert "Fix spec"
• Update bing images parser with reviews and rating as integer
• Add parser
• Add initial engine support
• feat(google-answer-box): add new generic type
• Ensure extracted texts are present before adding to hash
• Remove extraction of top-level link

2022-11-16

• Fix spec
• Fix typo
• Add comment
• Update parser with new date selector
• feat(knowledge-graph: add new web results and parse header images
• Add test case for when title is within a span
• Ensure `.text` is only called if platform_node has a text element
• Ensure title exists before calling text.squish
• Refactor Bing images code and specs

2022-11-15

• Fix Google Reverse Image API
• Extract `reviews_original`
• Update parser
• Support Bing Images country

2022-11-14

• Update parser to extract date
• Monitor category field on every google news search
• Add title for mobile
• Add title, link and displayed_link for desktop
• Remove unnecessary checks by relying on '/text()'
• Remove incorrect `.text.squish`
• Refactor to rely on direct accessing of attribute (2)
• Refactor to rely on direct accessing of attribute
• Refactor to rely on map
• Add mobile spec for bing images result
• Fix wordings and refactor code
• Add serpapi_link

2022-11-12

• Parse related_video for organic result
• Classify some of the organic results as medical results

2022-11-11

• Code structure for new local pack v2
• Remove unnecessary whitespace
• Remove unnecessary whitespace
• Remove unnecessary whitespace

2022-11-10

• Fix Google Play API Duplicated Organic Results Hashes
• Refactor condition to be less verbose
• Update parser
• Refactor cleaning of title
• Ensure YouTube key moments have a YouTube link
• Fix getting inline videos for tablet
• Fix issue where links start with "//"
• Revert change to existing selector
• Use in-built text() to extract text node

2022-11-09

• Fix badge containing css
• Support Menu Items for Google Shopping
• Fix issue where titles include chars with no visual representation
• Partially Fix Trust Badges
• Change tag to market
• Fix Rating and Reviews not extracted for some results
• Refactor
• New result for related searches in Bing images
• call num from self
• Fix missing serpapi link for mobile result
• Simple selector for google news
• Fix xray marking
• Carry out recent parallel development
• Fix merge conflicts
• Make Changes to Backend

2022-11-08

• Add a method to extract price movement from HTML
• Improve parser logic
• Use address position
• Remove duplicated first result for new html layout
• Update comments
• Refactor methods to add top-level attr
• Support videocard

2022-11-07

• Add `num` parameter validator
• fix spec failing
• Fix duplicated first google news result
• Better code style for hour address position

2022-11-05

• Update title node selector
• Fix test to use safe navigation operator
• Refactor to use single quotes

2022-11-04

• Use `match?` and give an example in a comment
• Fix failing test
• Fix mobile-like pagination on desktop
• Fix bing apple.com test
• Fix title and reviews for google shopping results

2022-11-03

• Fix missing organic result with video carousel
• Fix typo in conditional for Google Trends engine (still seem to work fine)
• Fix `rich_snippet` is parsed in `date` - Add a check for elements with `·` character - Expand the regex to include Arabic and Persian digits - Add a unit test - Add two unit test examples
• Fix Ads Results returning part of `snippet` in `displayed_link` in mobile - Update parser - Add a unit test - Add a unit test example
• Support Duration and Video Quality - Update parser - Separate the unit test example to make a new unit test - Update uni test example
• Fix last news item not parsed - Update css target in parser - Update unit test to check for full nine elements
• Safeguard code
• Refine phonetic parsing
• Parse link

2022-11-02

• Fix captcha detection
• Fix Yahoo Search API `displayed_link` is parsed in `snippet` for `organic_results` in `mobile` - Update parser - Add a unit test - Add a unit test example
• Fix Yandex `thumbnail` not parsed in `organic_results` - Update parser - Add a unit test example
• Fix Google News API not parsing `news_results` - Update parser - Add a unit test example
• Fix Google News API mobile pagination - Update parser - Add a unit test - Add a unit test example
• Update real search spec for places
• Parse consultation_link
• Support desktop parsing

2022-11-01

• Support answer_box - medical_result on Baidu mobile

2022-10-31

• Fix Parser
• Fix Google Play `organic_results` and `pagination` not parsed
• Fix new layout of local results
• Ensure Google Local Ads results work on mobile
• Support answer box organic_result mobile

2022-10-29

• Support answer_box - calculator_result on Baidu mobile
• Support answer_box - dictionary_result on Baidu mobile

2022-10-28

• Fix multiple problems with `sitelinks` `expanded` - Fix parser for `snippet` not extracted in `vertical expanded sitelinks` - Fix parser for `snippet` not extracted in main `organic_result` when `vertical expanded sitelinks` are present - Update parser to parse `block_title` - Fix parser for `mobile` `ads_results` not extracting `inline sitelinks` - Fix parser for `mobile` `ads_results` with `card carousel expanded sitelinks` parsed as `inline sitelinks` - Add unit tests - Add unit test examples
• Standardise selector for getting badges

2022-10-27

• Remove installment_payment
• fix(google-inline-image): not parsing images on top
• fix(google-knowledge-graph): parse new dynamic block
• Remove unnecessary condition
• Disable Rubocop Layout/HashAlignment

2022-10-26

• Extract CN unit numbers
• Fix parser
• Add main parser
• Add discover_more parser
• Add market_trends parser
• Add markets parser
• Add news_results parser
• Add support for index_market parameter
• Add parameters, validator, api, model... specs
• Parse avatar of the source

2022-10-25

• Support answer_box - finance_results on Baidu mobile
• Add bing images api documents
• Add serpapi link for Bing inline images
• Add tests for Baidu mobile answer_box - currency_converter
• Support answer_box - currency_converter on Baidu mobile
• feat(google-trends-api): add timestamp to interest over time

2022-10-24

• Support forum_results on Baidu mobile
• Support inline_videos on Baidu mobile
• Add pagination and first spec for bing images
• Scrape related searches and enhance shopping, videos results
• Rename read_count to reads

2022-10-22

• Add initial parser support

2022-10-21

• Fix expanded sitelinks multiple issues - Update parser to catch multiple expanded sitelinks - Update parser for parsing multiple expanded sitelink nodes for same organic result - Refactor deep links parser(expanded sitelinks) to be parsed as caption in order to parse more keys from it(`thumbnail`, `displayed_link`, `sitelinks-> inline`) - Update caption parser for displayed links, thumbnails, and inline sitelinks - Update expanded sitelinks parser not to return empty title - Remove optional parameter on xray markings in common parsers(`parse_caption`, `parse_deep_links`) - Update expanded sitelinks parser for ads results - Update caption parser to return empty hash if no result is parsed - Add unit tests for expanded sitelinks in organic results and ads - Add unit test examples
• Update note
• Hardcode BDORZ for not changing
• fix(google-search-api): news result got updated selector

2022-10-20

• Ensure Google Local Ads results work on desktop
• Refine code
• Simplify css selector
• Fix snippet
• Fix inline_images
• Fix xray
• Parse `verified`, `comments` and `likes`
• Rename content to snippet
• Change css targeting
• Ensure no NoMethodError will be thrown
• Change the selector to avoid prevention of parsing for other kinds
• Fix duplicate results - Update unit tests with duplicate results - Make code more elegant in parser - Add a unit test example for duplication
• Support related_topics on Baidu mobile
• Fix Google partial `knowledge_graph` parsed as `organic_results` - Update parser - Add a unit test - Add a unit test example
• Fix Google Organic Results `rich_snippet` not parsed - Update parser - Add a unit test - Add a unit test example
• Fix Google `organic_results` `sitelinks` `inline` not parsed - Update parser - Add a unit test - Add a unit test example
• Fix Google Organic Results mobile results `thumbnail` not parsed - Update parser - Add one unit test - Add one unit test example
• Use class selectors rather than jsname
• Add missing `mark_for_xray`
• Refactor to shorten logic

2022-10-19

• Support documents_results on Baidu mobile
• Support social_media_results on Baidu mobile
• Fix parsers - Refactor caption parser to parse multiple elements at the same time - Refactor ads parser to parse the entire ad body for caption elements - Fix a formatting error on Organic Results unit test
• Fix missing thumbnail
• fix(google-search-api): parse single google result
• fix(google-search-api): prevent empty hash in result
• Remove unused `index` block argument
• Ensure key moments are extracted

2022-10-18

• Support news_results on Baidu mobile
• Fix inline sitelinks for Bing Organic Results and Ads - Fix Organic Results Parser - Fix Ads Parser - Fix Caption Parser (Common Method) - Add a unit test for organic results - Add a unit test for ads - Add 4 unit test examples
• Update tests
• Update parser
• Display Yahoo domain in JSON response
• Fix extraction for mobile view
• Refactor to use `get_valid_link`
• Extract link for grid view
• Support people_also_search_for on Baidu mobile
• feat(google-search-api): add unit converter to answer box
• fix(google-search-api): parse organic result thumbnail for mobile
• Don't rescue JSON.parse error
• Refine code
• fix(google-buying-guide): parse table in expanded result

2022-10-17

• Minor safeguarding for results outside Amharic Language
• Update regex for hours
• Add a missing css target and a unit test example
• Add xrays for `displayed_link` and `thumbnail` - Add xrays for `thumbnail` and `displayed_link` - Narrow down thumbnail selector to avoid scraping icons - Fix a typo in unit test
• Remove unnecessary safeguard
• [Google Play Store API] Fix key name in output JSON
• Extract direct link of sitelinks
• Scrape suggested searches, refined searches, shopping results...

2022-10-16

• fix(google-search-api): add buying guide parser

2022-10-15

• Scrape Bing images results
• Improve empty search yahoo videos

2022-10-14

• Fix Yahoo `rich_snippet` not parsed for mobile searches - Update parser - Add a unit test - Add a unit test example
• Fix Bing Organic Results `displayed_link`, and `thumbnail` not parsed - Update parser to pick right captions - Update unit test with optional `thumbnail` check - Add 2 unit test examples
• Add parser for extracting category values
• Fix Yahoo Empty `total_results` - Safegaurd parser - Add unit tests - Add unit test examples
• Add minor safeguarding
• Fix Google Images Filetype Search Command infinite loop - Add a parameter validator with instructions on error - Update parameters file - Update Controller Spec with 2 unit tests
• Fix Google Local Results `address` is parsed in `hours` - Update parser - Add a unit test - Add a unit test example - Minor fix on an old test
• Update app/models/search/google/parser/popular_destinations.rb
• Fix spec test

2022-10-13

• Support mobile sources - Update parser - Add a unit test example
• Fix Google rich_snippet bottom mobile - Update parser - Add a unit test - Add a unit test example
• Update parser
• Fix parser
• Revert "Update parser"
• Update parser
• Fix Google `top_stories` `live` key in `date` - Update parser - Add a unit test - Add a unit test example
• Fix Google Organic Results `sitelinks` -> `inline` not parsed - Update parser - Add a unit test - Add a unit test example
• Fix Google Video Results `found_in_videos` not parsed - Update parser - Update unit test - Add a new unit test example
• Correct yahoo videos result status
• Refine yahoo videos search results
• Change parsing method
• Revert the PR for Safety Concerns

2022-10-12

• Fix twitter results `published_date` not parsed
• Fix residues in `source`
• Add serpapi link for yahoo video search in yahoo inline videos
• Add shopping results for Baidu mobile parser
• fix(google-search-api): local pack missing
• Scrape ads result for yahoo videos
• Scrape yahoo videos results

2022-10-11

• Fix parser
• Fix parser
• Fix Google Search Commands causing infinite loop - Fix parameters when filetype search - Add standard filetypes to constants - Add a unit test for filetype searches - Add unit test for checking displayed query for scenarios - Add a unit test for comparison of normal search vs filetype search - Add unit test examples
• Fix parser
• Try to use direct link if available
• Update link condition
• Fix parser
• Remove redundant code
• Refine xray
• Add validator for tbm parameter
• Fix Bing News empty `source`

2022-10-10

• Fix Google Mobile Results first organic result not parsed correctly
• Fix Google Local Results `address` containing `hours`
• Fix mixmatch of `answer_box` and `sports_results` - Fix parser - Add a unit test example

2022-10-08

• Fix baidu mobile rich_snippet xray

2022-10-07

• Bypass baidu mobile captcha
• Remove displayed link manipulation
• Fix typo
• Fix doubling of extensions
• Support `htichips` parameter - Support Google Events API `htichips` parameter - Update playground - Update unit tests - Update parameters - Update documentation - Add 2 unit test examples
• Fix typos
• Update app/models/search/google_about_this_result/parser.rb
• fix(google-shopping-results): missing properties in image search

2022-10-06

• Add tests
• Detect empty results
• Update parser to support empty results error
• Fix empty link
• Clear residues of `pz` parameter in Yahoo Search - Clear residues of `pz` parameter in Yahoo Search - Update documentation with a notification
• Fix Ebay Search API extensions not parsed completely - Update parser - Add new unit test examples - Update unit test to check if more than one extension is being parsed at least once
• Fix `address` being parsed in `years_in_business` for some `en` results
• Add missing validation for all engines
• fix: remove spaces
• feat(google-image-results): add original width and height
• Parse rich_snippet
• Change inline_images to related_images and fixes
• Classify jy_dict_video as answer box

2022-10-05

• Fix double coding
• Fix residue in `url` and `thumbnail` of `inline_images`
• Parse date
• Update organic results parser
• Add parser
• Add new engine
• fix(organic-results-mobile): remove empty organic result
• fix(google-news-api): people also search for news results not parsing

2022-10-04

• Support ads_results for Baidu mobile parser
• Fix parser and add test
• Update documentation, and add links for related engines - Update Apple App Store Documentation with new examples, and restructring of items - Add new images for documentation examples - Delete old documentation images - Update parser to return `apple_product` and `apple_reviews` links in results - Update responsible unit test for new keys
• fix(google-knowledge-graph): website not parsing for new layout
• fix(google-knowledge-graph): profiles not parsing
• Add `rn` validator
• fix(google-news-results-api): not parsing news result
• Use m.baidu.com for baidu mobile
• Add a note about bypassing 302 captcha page for Baidu mobile
• Fix potential empty results
• Safeguard code

2022-10-03

• Add validator for integer win min max values
• Fix parser
• Use better css selector to get quality full-hd
• Remove unused extract js div

2022-09-30

• Add parser
• Scape when date from event details
• Parse `images` for google mobile
• Add sitelinks, fix snippet and thumbnail
• Use get_thumbnail

2022-09-29

• Scrape source and resolution for yandex videos
• Update parser

2022-09-28

• Revert "Disable `Location` search by geo coordinates"
• Refactor code and reverse yandex inline videos items
• Add parser
• Add test
• Add support for search in category
• Safeguard code

2022-09-27

• Add translation to answer box

2022-09-26

• Google remove ivg-i class for knowledge graph logo result
• Refine documents and add spec for views
• refactor code

2022-09-25

• Link yandex videos search engine to yandex video inline results

2022-09-23

• Add support for non-chromium browsers as well - Add support for non-chromium browsers as well - Refactor a unit test
• Add Yandex videos document
• Update Regexp escape
• Scrape text ads
• feat: add new google_maps_photo_meta_api

2022-09-22

• Scrape yandex videos results

2022-09-21

• Update parser
• Update parser to escape the plus sign
• fix(google-search-api): add breadcrumbs to knowledge graph doc
• Update Baidu Search mobile parser
• Simplify parameters
• Stick `position` with index
• feat(google-search-api): add breadcrumbs to knowledge graph api
• Support Baidu mobile organic results
• Update documentation, and parsers - Fix parsers for off cases - Refactor old parsers to work with new methods - Add images - Add documentations - Update parser to parse menu items - Update unit tests to clean unnecessary parts (parts that'll be added in the future) - Move unit test examples to their new location - Add engine button text - Update text manipulation to show mobile htmls

2022-09-20

• Refactor Google Shopping into another Engine - Define a separate new engine Google Shopping - Delete old residue of tbm=shop method - Add backwards compatibility - Add new spec tests - Update Google parser - Add Google Shopping Parser - Add Unit tests for Google Shopping Parser
• Fix empty snippet
• Update parser
• fix(google-answer-box): currency converter not parsing from and to

2022-09-19

• Code style changes

2022-09-18

• add if

2022-09-16

• Support serpapi_link for Google Autocomplete - Update parser - Update Documentation - Update Documentation demo - Update unit tests
• Get original shopping result link
• Add yahoo image pagination

2022-09-15

• Fix Google Flights not being parsed - Update parser - Update unit tests
• Rename chips to suggested searches
• fix(google-search-api): parse top stories subtitle for other search

2022-09-14

• Fix parser
• Update parser
• Scrape shopping results for yahoo images results

2022-09-13

• Add xray for Bing Ads
• Fix Parser to work with mobile
• Add parser
• Fix xray not working on brand_ad
• fix(google-search-api): inline_images is present when no needed

2022-09-12

• Polish code for baidu news
• Fix Yahoo Organic Results External Sitelinks not parsed - Update parser - Add a unit test - Add unit test examples - Update documentation
• Add xray for individual store offer
• Remove unused user_id param
• Rename Yahoo!, refine document and validations
• Fix Bing Ads residue in description
• Show empty showing fixed spelling results message
• fix(google-search-api): related searches not scraping on mobile
• Add source and currency fields. Refine document
• Store offers html content will be replaced by js so it remove our xray class
• Change price hash response

2022-09-10

• Add support for Baidu News

2022-09-09

• fix(google-events-api): add image

2022-09-08

• Remove \ in the regex and cover new case $
• Prefer squish over strip
• Exclude more edge cases
• Handle empty suggestion in erb
• Fix parsing google inline image link on mobile

2022-09-06

• Scrape store offers for yandex images results

2022-09-03

• Add yahoo images utils and specs
• Rename suggested searches to related searches

2022-09-02

• Fix error for existing searches
• Escape cp param
• Cover single quote case
• Scrape chips for yahoo images results
• Extra fixes and spec updates
• Update the Graphql
• Fix parser

2022-09-01

• Fix extraction of date and image_resolution
• Manually build google inline image link
• fix(google-sport-results): simpify solution on misparsing sport results
• Scrape more yahoo image info
• Notify airbrake when can not parse menu items

2022-08-31

• Update google inline image parser
• fix(walmart-api): store id always give timeout
• Cover query with +, =, ?
• Fix no results and google return suggested new keyword

2022-08-30

• fix(google-search-api): mobile knowledge graph not parsing header images
• Fix scraping menus with special search keywords
• Specify the knowledge graph target further
• Add serp api for yahoo inline images results

2022-08-28

• Add more yahoo images specs

2022-08-26

• Update yandex inline videos parser
• Fix Google Knowledge Graph not parsed
• Fix Google Knowledge Graph dynamic carousel name is not parsed - Fix Google Knowledge Graph dynamic carousel name is not parsed - Fix Google Knowledge Graph dynamic carousel extensions returning empty strings - Update an old unit test giving false positive
• Refine params and support pagination

2022-08-25

• Update parser on Yahoo! ads result sitelinks
• fix(google-search-api): sport results is in nothing sports related

2022-08-24

• Update favicon xray
• Update parser
• Remove vc, vl param
• Scrape suggested searches

2022-08-23

• Scrape all yahoo images
• Fix Google Search Product Results Rating Ratios are mistakenly parsed - Update parser - Update mobile and desktop unit tests - Add unit test examples
• Update parser
• Add `crop_id` parameter support
• fix(google): time out when q parameters missing

2022-08-22

• Fix home depot issue
• Parse game spotlight for football
• Fix Yandex Images links
• Safeguard parser
• Make cosmetic changes to parser
• Support all yahoo images search queries and init the request
• fix: constant variable to be uppercase
• fix(baidu): validate ct value

2022-08-19

• Add serpapi_link
• Add support for reverse image parameters
• Add parser for Yandex reverse image search
• Fix regex in order agnostic parser

2022-08-18

• Add image for parameters - Update parser - Update unit test - Update documentation
• Fix sport result specs
• Reduce checking condition

2022-08-17

• Refactor inline images code, spec and document
• Scrape inline images for more image results
• Scrape inline images for yahoo search results

2022-08-16

• Update about_this_result parser
• Restructure some conditions
• Support Google Knowledge Graph Formulas - Update Google Knowledge Graph Parser - Add a unit test - Add unit test examples - Update documentation
• fix(youtube): search query is optional when sp is present
• Update parser
• Fix broken spotlight and remove unused code
• Remove location param when uule present

2022-08-15

• Fix extract jsdata id regex
• Exclude serpapi link for other google domain
• Fix Google Scholar Author shifting graph issue in cited_by - Fix Google Scholar Author shifting graph issue in cited_by - Add a unit test - Add a unit test example
• Change reverse_each statements to each
• Remove location param when uule is present
• Only add serpapi_link for google search link
• fix(google-sport-results-api): scrape total score
• Parse new rich list of related questions
• Fix related question missing title

2022-08-14

• Fix serpapi ulr for google product variation

2022-08-11

• Add serpapi link for product variations
• fix(google-search-api): visual stories not parsing

2022-08-10

• Support shown price at the ticker in typical results
• Reverse the order of ratings to make it one to one with visible data
• Check product variations array present
• remove accidentally code
• fix(google-sports-results): score not returning correctly

2022-08-09

• Fix Google Shopping Mobile and Tablet Results - Update Google Parser with necessary changes to parse mobile and tablet (presence of mobile and tablet elements, prevention of pagination parsing from the main parser ) - Update Shopping Results parser for results hidden in js - Update Shopping Results parser for exclusion of empty organic results key when empty results - Update Shopping Results parser with filters functions for mobile and tablet (with and without js) - Update Shopping Results parser with inline_shopping_results functions for mobile and tablet (with and without js, with correct positioning) - Separate old parser to get_shopping_results_desktop function - Update Shopping Results parser with mobile and tablet parser functions (with and without js) - Update Shopping Results parser with a pagination parser(with and without js, for all devices, corrects some results in accordance with organic results) - Update Google Validator for shopping results(all devices, prevents htmls without data in it) - Update Google Validator Spec for shopping results( all devices, different kinds of bodies) - Add Google Shopping Results > Filters unit tests (all devices) - Add Google Shopping Results > Pagination unit tests (all devices, checks in accordance with organic results) - Update Google Shopping Results > Inline Shopping Results unit tests with mobile and tablet - Update Google Shopping Results > Shopping Results unit tests with mobile and tablet - Add 40 unit test examples alongside old unit test examples to toughen the testing process
• Add product variations for google shopping results

2022-08-08

• Update parser with a simpler method
• Add support for Google Product typical prices across the web - Update parser - Add a unit test - Add a unit test example - Update documentation - Add a recent documentation image
• Fix Google Search API mobile snippets are parsed as base links for some results - Fix Google Search API mobile snippets are parsed as base links for some results - Update mobile unit test - Add a unit test example
• Fix Menu Items wrongfully parsed as spelling fix - Fix Menu Items wrongfully parsed as spelling fix - Update menu items parser for the presence of spelling fix - Update the order of parsing between spelling fix and menu items - Add a unit test example

2022-08-05

• Home depot missing taxonomy
• fix(sport-results-api): add stadium to game spotlight

2022-08-04

• Some search has no pagination info

2022-08-03

• Fix carousel missing author and source exception

2022-08-02

• Fix parser

2022-08-01

• fix(google-search-api): answer box scraping shopping results (#2970)
• [Google Search API] Video carousel should show in organic results (#2817)
• Fix xray markings - Fix xray markings - Add a recent example
• Fix Bing Organic Results links are giving tracking links - Fix Bing Organic Results links are giving tracking links - Add a unit test - Add 2 unit test examples
• Fix Google Search API Product Result Rating Results - Fix Google Search API Product Result Rating Results wrong distribution of rating ratios - Fix multiplication of ratings - Update tests
• Some searches wont contain volume data
• Support more operations

2022-07-28

• Update video_results.rb
• Update app/models/search/naver/parser/video_results.rb
• Clear the repeated condition
• fix(google-shopping-api): update link to show direct link to source website
• [Support New API] Google Immersive Product API (#2913)
• [Walmart] Update `store_id` validation message (#2950)
• Fix empty html result
• Fix menu items causing 500 - Safeguard parser - Separate one of the conditions - Add a unit test example from airbrake
• Retry twice
• [skip ci] fix typo

2022-07-27

• Add full support for all charts in all tabs of Baidu Line Chart Answer Box - Update the parser for more fields - Add a unit test - Add a comprehensive unit test example - Update documentation
• Fix serpapi_link extraction
• Add support for Baidu line chart answer box - Update parser
• fix(google-events-api): events results got wrong thumbnail for some result
• Retry when can not get cookies from google trends
• Add check for substituted substrings as regex and add +
• Fix malformed unescape
• Fix safesearch choice appearing in the menu items - Fix safesearch choice appearing in the menu items - Fix documentation accordingly
• Fix Google Images menu items not parsed - Fix menu items not parsed for images results - Fix menu item titles containing css - Add a unit test for with css residue - Add a unit test example for css residue

2022-07-26

• Support Google Knowledge Graph Formula Type - Add parser - Add unit tests - Add unit test examples for desktop and mobile - Add documentation
• Check missing html
• Initial commit on Google Knowledge Graph Formula Box

2022-07-25

• Fix parser & spec tests
• fix parser
• fix: syntax
• fix: simplify jsdata code
• fix: change method name
• fix: simplify code for retrieving jsdata
• Remove blank answer box

2022-07-24

• [skip ci] Use util to get thumbnail
• [skip ci] Squish all texts

2022-07-22

• Add a missing example of related questions with rich snippets - Fix some related questions not parsing rich list - Add a unit test - Add a unit test example
• fix(google-search-api): add sport results spec for stadium
• Fix Multiple Google Organic Results Tablet Issues - Fix Snippets are missing in organic results for some searches on tablet - Fix Duration is not parsed for some organic results in tablet searches - Fix Date is being parsed as snippet in organic results for some tablet searches - Fix Video thumbnail in organic results for some tablet searches is not parsed - Fix Pagination button is parsed as organic result for some tablet searches - Fix Displayed link not parsed for some organic results in tablet searches - Fix Some organic results links are incorrect for tablet searches - Add a unit test - Add a unit test example

2022-07-21

• feat(google-search-api): add tournament and stadium to sport results
• Fix parse URl included [] in fragment

2022-07-20

• Exclude sport results in organic results
• Add xray and fix node selector issue
• Support sport standing tablet new UI

2022-07-19

• Add support for current page item - Add support for current page item - Make the code backwards compatible - Safeguard the code
• Refactor Menu Items - Parse Menu Items deductively to control diversity - Add exceptions for unloaded order of items for some results in parser - Change `search_intents` to `menu_items` - Update documentation with a new image that is closer to old font and size - Update documentation - Update unit tests to check for order of titles - Update unit tests with more off cases

2022-07-18

• Scrape other leagues on mobile

2022-07-15

• Remove unused code
• Trim apostrophes at the start and end of examples
• Parse found in videos in javascript
• Exclude time from title
• Add support for mobile parsing
• Scrape found in videos on mobile videos search
• Refactor code and specs

2022-07-14

• Update parser to add `serpapi_link`

2022-07-13

• Local result has ID Odp5De
• Scrape inline images suggested searches
• Fix Google Answer Box Dictionary Pronunciation Audio not parsed - Update parser - Update unit tests - Update documentation
• Description has multiple ways to display
• Fix conflicts
• fix(google-search-api): remove & from link parsing in inline shopping results
• Fix Google Answer Box Dictionary results `examples` and `phonetic` not parsed - Fix Google Answer Box Dictionary results `examples` and `phonetic` not parsed - Update old unit test example - Fix a typo on documentation
• Parse image with special div container

2022-07-12

• Prevent inline_images to be mistaken with answer_box
• Change `google_images_link` to `more_images_link`
• Allow unconverted keys t obe returned as unknown
• Add new files
• Fix Google Search Intents not parsed - Update parser - Add unit tests - Add unit test examples - Update documentation
• Refactor getting facts code
• Fix Naver Web Organic Results `related_wiki_results` to return empty hashes - Fix Naver Web Organic Results `related_wiki_results` to return empty hashes - Update unit tests - Update documentation
• Show invisible text
• fix: unescape url
• fix(google-search-api): ads shopping should show direct link
• Fix Naver Search API Web Organic Results Related Results not parsed - Fix Naver Search API Web Organic Results Related Results not parsed - Update unit tests - Update documentation
• Add xray for all knowledge graph fields
• Fix yandex parse new selector
• remove empty space commit
• fix(google-search-api): organic results not parsed on single result
• Fix Google Rich lists not parsed - Fix Google Rich lists not parsed - Update unit tests - Update documentation

2022-07-11

• Update parser
• Fix Google Product API returning empty `tax` - Fix Google Product API returning empty `tax` - Use an existing example, restructure and update unit tests
• Add prettified HTML
• Add playground support
• Add initial engine support and the parser
• Add xray for sports results
• Fix Google Answer Box not parsing highligthed words in snippet - Fix Google Answer Box not parsing highligthed words in snippet - Use an existing unit test example and add a new unit test example - Update unit test - Postpone documentation (not all details are parsed)
• Fix Google Answer Box Organic Results Images not fixed - Fix Google Answer Box Organic Results Images not fixed - Update unit test with recently added example
• Revert unnecessary check for key - Revert unnecessary check for symbolized string - Add a mobile unit test example
• Fix Google Answer Box Organic Results dynamic info pairs not being parsed - Apply same structre in Knowledge Graph - Use a recently added existing unit test example to update unit tests - Postpone updating documentation (images not parsed)

2022-07-08

• Enhance specs and doc code
• Update documentation
• Revert `date` key in `wind_forecast`
• Revert `date` key in `precipitation_forecast`
• Fix number of installments not being parsed - Fix number of installments not being parsed - Add a unit test - Add test examples from US and BR - Update documentation without an example (`additional_price -> tax`) key is broken

2022-07-07

• Fix Baidu Calculator Answer Box not parsing - Fix Baidu Calculator Answer Box not parsing - Fix Baidu Calculator Answer Box mistaken with organic results - Update documentation - Add unit tests - Add a test example
• Parse videos results with new google response
• Parse inline images on mobile
• Safeguard and simplify the code - Simplify the code - Restructre tests by separating mobile and desktop - Remove newly added test example - Update old desktop tests - Fix unnecessary `||` - Safeguard the code in some places
• Fix typo mistake and restructre tests - Fix `precitipation` to `precipitation` in all places - Simplify time extraction logic - Restructre tests by separating mobile and desktop - Remove newly added test example - Update old desktop tests - Fix unnecessary `||` - Simplify `aria-label` extraction
• Fix thumbnails starting with `//` - Fix thumbnails starting with `//` - Update old unit test and add a unit test - Add a unit test example
• Raise error when lm parameter is used - Add lm parameter back to raise error - Add a validation for lm parameter - Update documentation with a temporary warning.

2022-07-06

• Add xray for individual image
• Google image has new id for images container

2022-07-05

• Fix empty place results field
• Parse google maps plus code
• Refactor `dialy_wind_forecast` to `wind_forecast` - Refactor `dialy_wind_forecast` to `wind_forecast` - Revert xray markings for the time of `wind_forecast` element - Refactor documentation - Refactor unit test
• Fix Google Answer Box Weather Precitipation Forecast Not Pared - Fix Google Answer Box Weather Precitipation Forecast not parsed - Add a unit test - Add a unit test example - Update Documentation

2022-07-04

• Check vertical when new type layout present
• Fix Google Answer Box not returning dialy wind forecast - Fix Google Answer Box not returning dialy wind forecast - Add a unit test - Add a unit test example - Update documentation
• Each teams has their own result
• Fix unnecessary condition

2022-07-01

• Parse first league on mobile
• Update parser
• Mobile layout return 2 separated data for team and stats
• Extract title with new standings layout
• Extract full standing results for mobile page
• Parse standing results from js

2022-06-30

• Revert getting th_label
• Fix Google Related Questions now showing thumbnails - Fix Google Related Questions now showing thumbnails - Add a unit test - Add a test example - Update documentation
• Reduce scope of local variable
• [skip ci] Remove redundant check
• Parse type with new layout for famous profile
• [New Bing API] Let's add support for Bing's Inline Images (#2855)
• Refactor `second_hand_condition` key to `condition` - Update parser - Update documentation - Fix documentation rendering malfunctions

2022-06-29

• Fix tests
• Fix parser
• refacor: remove space commit
• fix(google-search-api): remove subtitle from top stories mobile
• Rename Top Charts JSON key
• Add support for videos in app_highlight
• Fix Google Related Questions `snippet` is mistaken for `title` - Fix Google Related Questions `snippet` is mistaken for `title` - Update an existing unit test - Add a test example
• Skip labels by matched labels and html structure

2022-06-28

• refactor: remove see more link encapsulate in a method
• refactor(google-search-api): make cleaner code
• Fix duplicated list elements - Fix duplicated list elements - Update documentation
• Fix missing standing results for mobile
• Add Baidu Answer Box Organic Results List Type - Add Baidu Answer Box Organic Results List Type parser - Update unit tests - Update and replace test examples - Update documentation
• Remove labels comment
• Keep displayed labels match with api response
• Revert checking hash name
• Remove always true check
• Parse team rank and skip label
• Remove unnecessary test and update former tests
• fix(google-search-api): update refine by to parse filter on top page
• Fix Google Answer Box giving wrong `people_also_search_for` - Fix Google Answer Box giving wrong `people_also_search_for` - Add a unit test - Add test example
• Fix parsing of `info` key - Fix parsing of `info` key - Update documentation - Delete outdated image - Add new images to assets - Add a unit test - Add a unit test example

2022-06-27

• Prevent common kg elements to cause disorder in results
• Add new types of organic result answer box - Add new types of organic result answer box - Add another unit test for it - Add text example
• Correct selector for date, duration and views
• Add xray for see more link
• Add xray for each individual item
• Update app/models/search/yahoo/parser/inline_videos.rb
• Mark individual videos
• Add xray for all visible parts
• We want to start from 0 because of xray.
• Update parser and fix mistakes in spec - Update parser for newly found result that doesn't have `title` and `snippet` (`displayed_link` is not there) - Fix missing check for title in a spec - Add a new unit test for `title` and `snippet` not being extracted - Add unit test example
• Fix Google Knowledge Graph not extracting What to Watch Blocks - Fix Google Knowledge Graph not extracting What to Watch Blocks - Create conditions to parse `watch_options` dynamically named inner key - Add unit tests - Add unit test examples
• Update parser

2022-06-25

• Fix Google Places local_results

2022-06-24

• Check for 200 Google Internal Server Error
• Remove redundant code
• Fix typo
• Fix Baidu Direct Answer Box not parsing - Fix Baidu Direct Answer Box not parsing - Fix Baidu Direct Answer Box mistaken with organic results - Update documentation - Add unit tests - Add a test example
• Fix rendering issue and add xray marking for individual tweets
• Fix Google twitter results missing - Fix parsing of new class named twitter results (nfl_tweets.html) - Fix parsing of partial results in the carousel by double js extraction (nfl_tweets_original.html) - Refactor twitter results parser with safeguarding - Add unit tests - Add test examples

2022-06-23

• Refactor to find right element to extract `source` in full - Refactor to find right element to extract `source` in full - Refactor unit test to simpler version
• Refactor to prevent `title` to be taken as `question` in parser - Refactor to prevent `title` to be taken as `question` in parser - Remove newly added test examples - Refactor unit test to check for title - Add one new test example from old examples
• Fix for amp link
• Fix typo - Fix a typo in unit test - Add a safeguarding for rating parser
• Refactor Baidu Search API `lm` parameter to `gpc` parameter - Replace `lm` parameter with `gpc` in all places - Replace engine definitions - Add a Baidu constants file to store dynamically predefined time ranges - Add a unit test to check for time_filters - Add unit test examples
• Safeguard against empty arrays
• Safeguard code

2022-06-22

• Add `fr2` parameter in Yahoo! Search API - Add `fr2` parameter to Yahoo Search API (responsible for expanding some elements on HTML response) - Add a parser for `related_trending_searches` - Add unit tests for `related_trending_searches` both for expanded and minimized versions for tablet and desktop - Add documentation for `related_trending_searches` - Refactor playground to show `fr2` parameter for Yahoo! Search API - Update `trending_searches` documentation - Move images to assets in their corresponding place for `yahoo-trending-searches` and `yahoo-related-trending-searches`
• Refactor fr parameter not to be in a definite position in shuffling
• Force trending searches to apper in Yahoo API
• Fix Yahoo API 500 error caused from trending searches - Fix 500 error caused from trending searches time block is nil - Add xray markings for trending searches - Add position key into yahoo trending searches - Update unit tests with position parameter
• Fix confusing with news block
• Parse see more link mobile page
• Support mobile inline videos
• Fix Google Maps local_results not showing operating hours of the week - Fix Google Maps local_results not showing operating hours of the week - Add a unit test - Add a test example - Update documentation
• Add see more link
• Support another layout of inline videos
• Should be source instead of channel
• Add video position for yahoo inline videos
• Parse duration for yahoo inline videos
• Parse date for yahoo inline videos
• Parse title for yahoo inline videos
• Parse channel for yahoo inline videos
• Parse inline videos link and thumnail

2022-06-21

• Implementing julien's solution & hopefully cleaning whitespaces
• Update tests
• Fix Google Product API `conditions`, and `second_hand_conditions` to be parsed - Add `conditions` array in `product_results` to give conditions of buying options - Update current unit tests with `conditions` array - Add a separate unit test to check whether or not `conditions` array is not full of `New` condition for some results - Add `second_hand_condition` (same with the upcoming Google Shopping API update) for `sellers_results` - Add unit tests for `second_hand_condition` - Refactor `/online-sellers` documentation with a recent result containing `second_hand_condition` - Update `/product-page` with `conditions` array - Move all old images of Google Product API to assets with order
• Add Google Visual Stories API - Add parser for visual stories (only present in mobile) - Add unit test - Add test examples - Add documentation
• fix(google-search-api): change json key name for inline product link

2022-06-20

• Fix for title and date
• Replacing wildcard by img element
• Fix duplicated available on items
• Remove listen block func
• Remove listen block check
• Avoid defining new listen block
• Add serpapi_link to related searches
• Fix top charts
• Fix Google Inline Shopping Results `rating` missing - Fix Google Inline Shopping Results `rating` missing - Add unit test - Add test example
• Fix Google Inline Shopping `source` is redacted - Fix Google Inline Shopping `source` is redacted - Add a unit test - Add a unit test example
• Support mobile listen
• Select available on list
• Duplicated selector with .wDYxhc:not(:has(.ynrNJf))
• Hardcode the listen name
• Query listen items based on a and class
• Name the listen block to read easier
• Check listen block based on the data-attrid
• fix(google-search-api): local result not working
• Fix Google Related questions redirect link instead of actual link - Fix Google Related questions redirect link instead of actual link - Add a unit test - Add a test example
• Fix Google Search `related_questions` missing - Fix Google Search `related_questions` missing - Refactor `title` to be omitted when it is a duplication of `question` - Refactor unit tests accordingly - Refactor unit tests to check for the size of array to be at least 1 - Add unit test examples

2022-06-17

• Fix Google Direct Answer Box Results (Organic Results Type) `title` and `snippet` are the same - Fix Google Direct Answer Box Results (Organic Results Type) `title` and `snippet` are the same - Add unit test - Add test example
• Revert unintended changes
• Fix Google Answer Box Results (Organic Result Type) returning empty `people_also_search_for` array - Fix Google Answer Box Results (Organic Result Type) returning empty `people_also_search_for` array - Add a unit test - Add a test example
• Fix Google Direct Answer Box Results (Organic Results Type) `title` and `snippet` are the same - Fix Google Direct Answer Box Results (Organic Results Type) `title` and `snippet` are the same - Add unit test - Add test example
• Fix Apple Product `serpapi_link` containing `no_cache=true` - Fix `serpapi_link` containing `no_cache=true` parameter in the url - Update specs
• fix(google-search-api): add view more link to inline products

2022-06-16

• Parse links from js div
• More consistent wording
• Don't allow q parameter to start with %00 (NUL)
• fix(walmart-search-api): current page appear in other pages link
• feat(google-top-stories): add subtitle

2022-06-15

• fix(google-related-questions-api): handle empty result
• refactor(google-related-questions-api): update copy and naming
• Select search suggestion based on html structure
• Select search suggestion based on new class

2022-06-13

• Reject `nil` rows
• .ynrNJf and .TzHB6b.Hwkikb.WY0eLb returns duplicated watch_now items
• Adding new class name for img element
• No reviews div has the same selector with total_time

2022-06-12

• Add support for games parser when using pagination
• Update parser
• Update prettified html

2022-06-11

• Correct synthax error, add whitespaces.
• Adding again removed whitespace
• Adding Julien's improvement
• Update parser

2022-06-09

• Elem accepts both string and symbol
• Missing source with limited search result images
• Update app/models/search/google/parser/top_stories.rb

2022-06-08

• Update app/models/search/google_trends/validator.rb
• Update app/models/search/google_trends/parser.rb

2022-06-07

• Fix amp_link in mobile top stories with test
• refactor(google-related-questions): change to one request per validator

2022-06-03

• Update parsers

2022-06-02

• Update documentation
• Adding test if link is a string
• Update parser
• fix(google-search-api): local pack not parsing
• Fix inline top_stories carousel

2022-05-31

• Remove episode_id parameter

2022-05-30

• fix(local-pack): not parsing local results and map
• fix: spec failing
• feat(google-related_questions): add prettify html
• feat(google-related-questions): retrieve more than initial response related questions

2022-05-26

• Add tests
• Update parser to include serpapi_link

2022-05-25

• Update date validation
• feat: add google_related_questions engine

2022-05-24

• Update validator
• Update Google Trends constants
• Fix parser error when `q` is missing
• Fix merge conflict
• Update parser
• Update parameter validators
• Add prettified HTML

2022-05-23

• Add expanded_description or description
• Add support for region parameter
• Fix a tag appearing in title of related images - Fix a tag, and   appearing on title - Add check for type in specs

2022-05-18

• Fix parser
• Update validator wording
• fix(apple-product): 500 error and image extraction

2022-05-17

• Simplify code structure
• Fix place id extraction
• Update parameter validation
• fix(google-search-api): update short videos node selector
• Add initial Google Trends engine support
• Add parser
• fix(googele-search-api): short videos not parsing on mobile

2022-05-16

• Update documentation

2022-05-06

• Add tests
• Rename `gl` to `hl`

2022-05-05

• Rename object `languages` to `review_languages`
• Update proxies for yelp reviews
• Add business name to prettified HTML
• Update parser to remove HTML tags
• Add support for empty response
• Update validator
• Update parser
• Add languages constant
• Add prettified HTML
• Update nexearch_results.rb

2022-05-03

• Add parser

2022-05-02

• Add support for reviews_link to Yelp Search API
• Add initial support for Yelp Reviews API
• Extract `place_id` in Yelp Search API

2022-04-29

• Update parameters.rb

2022-04-27

• fix(google-search-api): top carousel not parsing for some search
• fix(google-news-api): only part of date is parse

2022-04-26

• Add extraction of old price - Add extraction of old price - Add tests for it - Update documents - Fix false parsing of non-shopping results (Results based on reviews) - Add tests for it

2022-04-25

• Refactor the parser checks - Refactor extraction of shopping results to main parser - Fix tax statements showing up on second hand results - Fix minor refactoring residues in filters, and inline shopping results - Change related searches tests with results with shopping items

2022-04-22

• Fix Google Product Reviews pagination
• Fix parser

2022-04-20

• Complete refactor for parsing js comment hashes

2022-04-16

• Disable `Location` search by geo coordinates
• Fix dynamic positioning of inline_shopping_results
• Add new indexes for parsing

2022-04-15

• Temporary fix for shopping (not complete)

2022-04-14

• Remove highlight from snippet
• Update app/models/search/yelp/parser/extract_results.rb
• Update app/models/search/yelp/parser/extract_results.rb

2022-04-13

• Add parser

2022-04-11

• Update parser
• fix(google-search-api): add snippet highlighted words

2022-04-08

• Fix parser
• Create ad results parser

2022-04-07

• Update validator
• Fix parser to extract the date in some results

2022-04-06

• parse source
• Fix displayed link parser
• Add pagination
• Add tests
• test(google-map-reviews): fix spec failing
• fix(google-map-reviews): squish review text

2022-04-05

• Fix Displayed_link parser
• Fix some snippet issues
• Fix parser for date
• Fix snippet parser
• Update validator
• Update parser to extract offer details
• fix(google-map-reviews): snippet combine next line without spacing

2022-04-04

• Add initial tests
• fix(walmart-product-reviews): remove unnecessary review
• Restructure parser

2022-04-01

• Fix Related Images not being parsed completely - Fix parser for recent kinds of results - Add support for sitelinks - Add documentation - Add tests
• Refactor `related_cctv` -> `other_area_names` -> `area_name` to `name`
• Change code style for organic results parser

2022-03-31

• Fix parser.

2022-03-30

• Fix place_id and lsig being truncated to 1 length
• Fix local_results not being parsed on mobile
• test(walmart-product-reviews): fix spec failing
• fix(walmart-product-reviews): add filter to validator
• fix(walmart-product-reviews): accept filter as parameter
• refactor(walmart-product-reviews): categorized parser
• fix(node-extractor): prevent link of javascript:void(0);

2022-03-29

• Update parser to support more filters

2022-03-28

• fix(google): add serpapi link to top carousel knowledge graph
• Make location only required parameter
• Fix playground wrongfully adding `q` parameter for Yelp
• Update filter extraction

2022-03-25

• Rename parameters to match Yelp website
• Handle empty results

2022-03-24

• allow_blank: false
• Enforce a minimum of 8 characters for product_id
• Add support for more parameters and filters
• fix(google): finance answer box convert 5 digits number to 2 digits decimal
• fix(google): related searches not parsed on loaded async
• fix(google): add si parameters

2022-03-23

• fix(google): inline people also search not displaying for some result
• fix(google): related searches not scraping all list
• Move xray markings into condition
• Fix merge conflict on parser

2022-03-22

• Add one more xray
• Add xray markings for sitelinks - Add xray markings - Replace test with other query - Change Spec Definition to Rich Snippet
• Fix Baidu related_cctv description containing CSS
• Change safeguarding condition for duration node - Change safeguarding condition for duration node - Change spec style
• Add backwards compatibility for thumbnail extraction - Replace test file and put the test in a definite place in spec
• Better selector
• Fix parser
• Update phone extraction classes in ads_results.rb
• Replace test example for Baidu related_searches - Delete old example - Add new example - Clear empty spaces for links - Update documentary accordingly
• refactor(google): make code cleaner
• fix(google): not extracting all short videos

2022-03-17

• fix(google-maps-api): parse multiple hours

2022-03-16

• Update parser
• Improve parser
• Remove HTML parser
• Add parser
• Add initial Yelp engine support
• Change safeguarding and spec style
• Fix unnecessary safeguardings
• Add Baidu `source` in `organic_results`, `related_videos` - Fix test location to its correct place - Add `source` extraction - Update spec and documentation

2022-03-15

• Change safeguarding style
• Change Safeguarding Style
• Make safeguarding into alternative style
• Add Safeguard logic wherever possible - Add further safeguard logic for strings that only consist of spaces - Add `comparisons_link` in `shopping_results` - Add xray for filters - Update specs for `comaprisons_link` - Remove unnecessary condition in parameters - Update documentation for `comparisons_link`
• feat(walmart-product-reviews): add sort and rating to docs
• feat(walmart-product-reviews): add sort and filter to the playground
• fix: xray syntax error
• feat(walmart-product-reivew): add sort and rating filters

2022-03-14

• Separate tests for search_information
• Fix Baidu rich snippet wrong parsing of `detected_extensions`
• Add safeguard for strings consisting of only spaces
• Add safeguard for strings only consisting of spaces
• Add safeguard for strings consisting of only spaces
• Safeguard for strings consisting of spaces
• Add safeguard for strings composing of spaces
• Add safeguard for Strings composing of spaces
• Add Baidu related_cctv xray markings
• Refactor Baidu `related_cctv` - Change the multiple hash to one hash in parser - Change safeguard and refactor parser - Refactor specs

2022-03-11

• Add xrays - Add xrays for inner elements of `related_news` - Fix a minor unnecessary safe navigation
• Fix method used for extracting image
• Fix yielding condition - Fix yielding to be in condition - Fix xray markings for individual items - Fix minor wordings on menu and documenatation
• Refactor `text` key in `related_searches` to `query` - Fix minor documentation and menu wordings - Refactor `text` key in `related_searches` to `query` in parser and spec

2022-03-10

• fix(google): nitpick
• fix(google): add serpapi link to broaden searches
• fix(google): address broaden searches copies

2022-03-09

• Improve validation condition
• refactor(google-reverse-image): change sizes to image sizes
• fix(yahoo-shopping): special character break graphql search
• fix(google-reverse-image): empty error is shown even result present

2022-03-08

• feat(google-reverse-image): add sizes

2022-03-07

• fix(google): remove unnecesary parse inline video in tags function
• fix(google): remove unneeded comment in inline video
• fix(google): use /text() to avoid getting style content

2022-03-05

• fix(google): tag wrong xray object
• feat(google): add broaden searches result
• fix(google): not parsing result with carousel
• fix(google): inline people search not parsing for some searches
• fix(google): related searches mobile not working for some search
• Fix Baidu related_images (not complete)
• Add baidu Related Searches
• Add Baidu People Also Search For - Add Baidu People Also Search For - Add Documentation - Add Tests
• Fix Baidu organic_results related_cctv nil error - Fix Baidu organic_results related_cctv nil error - Add Tests
• Fix Baidu Organic Results error - Fix Baidu Organic Results nil text error - Safeguard Baidu Organic Results short videos - Add tests

2022-03-04

• Add Baidu Top Searches - Add parser for Baidu top_searches - Add tests - Add documentation
• Add specs for rich snippet - Fix links not parsed as String - Add specs for rich snippet
• Fix Baidu rich_snippet not appearing - Fix, simplify and safeguard Baidu rich snippets - Add test file

2022-03-03

• fix(google): short video title xray broken
• fix(google): fix short video not parsing on mobile for some search

2022-03-02

• fix(google): parse inline video carousel for mobile
• Fix Baidu organic results not parsing thumnnails - Fix thumbnail parser - Add tests
• Fix Baidu related_news missing elements - Fix Baidu related_news missing title - Fix Baidu related_news missing icon - Fix Baidu related_news missing snippet - Add Baidu related_news thumbnail to parser - Safeguard parsing functions - Add tests

2022-03-01

• Fix Baidu empty organic results - Fix search information being parsed as an empty organic result - Add search_information > information_text within result JSON - Add test
• feat(google): scrape inline video tags

2022-02-28

• Fix source
• Fix parser
• test(inline-search): remove CTET inline poeple search mobile
• test(inline-search): fix spec failing

2022-02-27

• fix(google-shopping): first result include css in the source

2022-02-26

• fix(answer-box): not parsing people also search for in answer box

2022-02-25

• fix(knowledge_graph): parse cite link
• fix(knowledge-graph): fix exam dates not parsing for some exam

2022-02-23

• Fix parser

2022-02-21

• Fix top_stories parser
• fix: add safe guarding
• fix: add new css class to inline people search

2022-02-18

• Fix Apple Product API odd key in version history - Fix Apple Product API odd key in version history - Revert accidental change to search processor
• Update parser
• Add device outputs on specs

2022-02-17

• Update parser
• add types for sub nutrition rows
• add tablet to answer box wellness

2022-02-16

• Fix Parser & add specs
• add type to nutrition facts
• parse syllabus and dates for CTET on desktop

2022-02-15

• Reverse key and value for eBay domain constant
• feat(knowledge-graph): extract CTET result for knowledge graph in mobile

2022-02-14

• Fix failing specs
• Fix failing specs
• Fix commit mistakes
• Remove Shopping Results parser from Google Parsers - Remove Shopping Results parser from Google Parsers - Fix minor conditionals to prevent errors - Seperate pagination

2022-02-10

• more stats link and token key extract
• Update sports_results.rb
• Change the word `lables` to `labels`

2022-02-09

• Fix Google Scholar Author pagination
• Add currency parser
• Update parser
• Fix source extraction on mobile and tablet
• Rename `status` to `tag`

2022-02-07

• Update parser
• Update graphql url

2022-02-04

• Sanitize snippets to remove HTML from it
• Extract `posted_at` from images

2022-02-03

• Fix parser
• Don't show events results inside knowledge graph
• Update parser

2022-02-02

• Minor change on validator
• Fix picker and parameter specs
• Update parser
• Fix pagination and invalid byte sequences

2022-02-01

• Update organic results classes
• Update parser
• Update parser

2022-01-31

• Refactor Google Shopping API into a new engine

2022-01-29

• Remove test field from parser
• Add tests

2022-01-28

• Add Yandex images size parameters validator
• Add support for advanced parameters
• Add pagination to JSON
• Refactor Google Shopping into a new engine
• fix(google): not parsing inline people also search for

2022-01-27

• Handle empty results
• Add suggested searches parser
• Add images results parser
• Add initial support for Yandex Images API
• fix(google): X-Ray node not found for some searches in google flights answer box

2022-01-26

• Refactor Google tbm="shop" as another engine - Create parameters (not final) - Create parser (not final) - Create validator (not final) - Add Google Shopping API to Engine Database
• fix(google): change parser algorithm
• Update documentation for Google Shopping - Update Google Shopping `inline_shopping_results` documentation - Remove one unneccessary false positive class name for `second_hand_condition` in Google Shopping Results Parser
• Update `inline_shopping_results` `second_hand_condition` key - Refactor extraction of second_hand_condition into a new function - Fix false positive second_hand_condition results - Update inline_shopping_results extraction of second_hand_condition - Update inline_shopping_results specs (Desktop, Mobile, Tablet) - Add test files
• Fix Parser

2022-01-25

• Fix `related_shopping_results` duplicated in `shopping_results` - Refactor defining both nodesets outside condition to eliminate duplicates - Refactor `related_shopping_results` specs with test metadata to cover for recent kinds of results lacking some fields - Add new test file
• refactor(bing): make sure link is valid before adding

2022-01-23

• fix(google): answer box table error on some layout

2022-01-22

• fix(home_depot): spelling fix error on empty result for some searches
• styles(bing): rename recipe to recipes for all the naming
• fix(google): inline videos video URL is parsed incorrectly

2022-01-21

• Update parser

2022-01-20

• New class
• Revert "Update app/models/search/google/parser.rb"
• Revert "Update app/models/search/google/parser.rb"
• Update app/models/search/google/parser.rb
• Update app/models/search/google/parser.rb
• Update app/models/search/google/parser.rb
• Update app/models/search/google/parser.rb
• Update app/models/search/google/parser.rb
• Update app/models/search/google/parser.rb
• fix(bing-recipe): make sure image and link are present before adding
• fix(bing-recipe): prevent empty array
• refactor(bing-recipe): remove X-Ray from link
• refactor(bing-recipe): make sure link is present
• refactor(bing-recipe): make sure image is present
• style(bing-recipe): remove comment
• refactor(bing-recipe): rename module name to RecipesResults

2022-01-19

• Fix thumbnails not being properly selected
• Update parser
• Add x-ray
• Stop showing empty results error when there are results
• Fix minor typo errors

2022-01-18

• Remove first page from other pages and update spec
• Simplify the checking.
• fix(google): add source link to inline images
• fix(google): inline images link not showing full path
• feat(bing): parse recipe results for grid and mobile

2022-01-17

• Update parser and add new spec
• feat(bing): add recipe results to bing

2022-01-15

• Update Google Shopping Parser - Update Google Shopping parser for Nearby Stores - Revert `used` tag and refactor it to `second_hand_condition` which is a string - Fix mistakeful css parses in `source` field - Update one more validation for Google Shopping Desktop - Update validator_spec for the faulty result - Add `nearby_shopping_results` spec - Update shopping result specs with new and refactored fields (desktop, tablet, mobile) - Add HTMLS

2022-01-14

• Update video_results.rb
• Update places_results.rb
• Update parser
• feat(bing): add new answerbox for wellness

2022-01-13

• Update Google Shopping parser for problematic results - Update problematic HTMLs to be parsed completely - Add new field `used` TrueClass - Add linkless `Other Options` to extensions - Fix tags to be repeated in extensions - Fix minor change on proxy and useragent distribution to improve performance
• Combine mobile results with desktop, and add spec for it
• Fix desktop video blocks
• Fix parser

2022-01-12

• Refactor Google Shopping Desktop proxy/useragent selection - Add new proxy distribution for Google Shopping Desktop - Add new method to pick Google Shopping Desktop useragents - Update the method to pick different proxy/useragent options for different Google `tbm` values - Add new useragents that give <6s on localhost - Add new method to pick Google shopping proxy distribution - Add new HTML with problematic form for testing (Desktop) - Update Google Shopping parser, and Google parser for the problematic form (Desktop) - Update Google Shopping Desktop `shopping_results` spec to test for problematic form

2022-01-11

• Add specs for newly parsed elements in Google Shopping - Change .?* to .* to omit warning in specs - Update Google Shopping parser for `related_searches` - Update Google Shopping validator for server error to be used generally across different search languages - Add `filters` specs for Google Shopping (desktop, mobile, tablet) - Add `inline_shopping_results` specs for Google Shopping (mobile, tablet) - Add `related_searches` specs for Google Shopping (desktop, mobile, tablet) - Add `shopping_results` specs for Google Shopping (mobile, tablet) - Add `various_blocks` specs for Google Shopping (desktop, mobile, tablet) - Add example HTMLS
• Fix the term parameter to accept spaces, and added spec test.
• fix(bing): add X-Ray to entire related question block

2022-01-10

• Update specs for Google Shopping - Refactor proxy distributions for Google Shopping `mobile` and `tablet` under a different constant - Update picker for google_shopping_mobile_and_tablet accordingly - Fix filters returning empty results for empty results on Google Shopping parser - Fix Google Shopping returning empty results that doesn't have to be empty for some searches in validations for desktop - Update picker_spec for new set of distributions - Fix a typo error on related_shopping_results spec - Fix structral error on various blocks spec - Update specs for all 3 devices with successful results in validator_spec - Update specs for all 5 new conditions in validations in validator_spec - Add HTMLs for successful results - Add HTMLs for problematic results - Update empty results spec for Google Shopping for all 3 devices to make sure the new validations aren't causing any blockage on real empty results - Add HTMLs for empty results
• fix(bing): add X-Ray to related questions
• fix(google): proper result showing no result in search information
• refactor: use && instead of and

2022-01-07

• Fix pagination not appearing in mobile and tablet
• Optimize proxy & useragent performance in Google Shopping - Fix a typo error in Google Shopping string manipulation - Add a new distribution of proxies for Google Shopping mobile and tablet - Add two new methods for picking useragents for Google Shopping mobile and tablet - Add one new function to pick proxies for Google Shopping tablet and mobile - Add conditions to pick right proxies and useragents for Google Shopping tablet and mobile - Add new useragents that are fast for all proxies used in Google Shopping tablet and mobile - Update the structre of validations for Google Shopping mobile and tablet
• fix(bing-knowledge-graph): wrongly select another element

2022-01-06

• Update parser and spec
• refactor(walmart-product-review): return early on object nil
• refactor: simplify spec

2022-01-05

• Update Google validator for new sets of empty results and fix prioritization
• Update Google Shopping mobile parser with new sets of class targets
• Fix typo errors and change first regex match to regex match in some methods
• refactor(bing-knowledge-graph): improve code readability

2022-01-04

• Update parser and playground demo JS file
• Add Google Shopping Results validation for problematic forms
• Refactor Google Shopping Results - Add Mobile Parser - Refactor get_shopping_results as a general method - Refactor parsing of filters for reusability - Update filters parser - Refactor parsing of inline_shopping_results - Refactor parsing of variable blocks - Update parsing of variable blocks - Add Mobile pagination
• Update parser
• Update Google Shopping Parser - Update safe condition for thumbnail extraction - Update safe condition for link extraction - Update squish for title - Update squish for price
• fix(bing-knowledge-graph): remove empty result in country facts
• refactor(bing-knowledge-graph): rename variables in country facts

2022-01-03

• Update parser and documentation
• Fix and Update Google Shopping Results - Fix the block with header `Deals related to your search` parsed incorrectly - Add `related_shopping_results` key instead - Refactor parser to parse `shopping_results` and `related_shopping_results` with the same method - Update parser to parse various fields such as `filter_by_variety`, `you_might_like` - Add specs for `related_shopping_results` - Add specs for `various blocks` - Add HTML test files
• Add `serpapi_more_results_link` to Google Answer Box `organic_result` type `carousel`
• Remove unnecessary check for presence
• fix(bing): knowledge graph error on some searches

2021-12-31

• Fix `apple_app_store` `num` parameter not working - Remove `num` variable manipulation in parameters - Add `num` parameter manipulation in `search.rb`

2021-12-30

• Hide the pagination for empty results
• Parser update
• Fix failing specs
• Support Google asnwer box flights updated legacy version - Update css targets for updated legacy version - Extract deeper information from the updated legacy version on desktop - Fix `flight_info` with`-` and `.` to be shown as one value ( .....s, ------s ) - Fix organic results to avoid conflict with answer box flights on mobile - Update parser to extract links from deeper info - Update parser to extract search_information for updated legacy version - Update parser to extract icon of airline company - Add specs - Add test HTMLs
• First part of addressing changes
• Update app/models/search/naver/parser/image_results.rb
• Update app/models/search/naver/parser/image_results.rb
• Update app/models/search/naver/parser/image_results.rb
• Update app/models/search/naver/parser/custom_pagination.rb
• Update app/models/search/naver/parameters.rb
• Update app/models/search/naver/parameters.rb
• Update app/models/search/naver/parameters.rb
• Update app/models/search/xray.rb
• Update app/models/search/metadata_and_parameters.rb
• Update parser

2021-12-29

• Remove commented lines in parser
• Remove unnecessary conditions and safe guards in parser - Remove unncecessary consitions and safe guards in parser - Add spacing in reviews spec - Update variable names with more understandable ones in sort_spec and add spacing - Add spacing in validator_spec
• Bring description extraction out of conditional
• Update Google Answer Box Flights deeper info - Update extraction of deeper infor for Google Answer Box Flights - Add new key `ad` - Fix misparse of `ad` as flight - Move all the images in Google Answer Box Documentation for faster load time - Add new image to Google Answer Box Documentation Flights - Update Google Answer Box Flights Documentation with `First Class, Non-Stop, Round-Trip Flights from Austin to Orlando` to showcase the precision of the update - Add new specs with different kinds for destktop and mobile - Add new test files
• Simplify parser
• Update parser

2021-12-28

• Fix parser for duplicated result and add spec tests.
• Update Google answer_box `about_this_result` - Extract `about_this_result` and `about_page_link` for answer_box type:`organic_result`
• Update parser

2021-12-27

• Update parser
• refactor(local-ads): parse missing attribute
• Naver Image API spec test

2021-12-26

• Update parser
• Pretty HTML
• Add custom pagination

2021-12-25

• Fix conflicting css with older translate answer boxes
• Update Google related_questions parser - Add new types of related_questions - Add list key to be parsed
• Update safe navigation for Google inline_people_also_search_for
• Refactor the new elements to be answer_box with organic_result type
• Revert changes in Google organic_results

2021-12-23

• Add safe_query to the search
• Update local_ads with serpapi link
• Add tests
• fix(local-ads): nil in selecting ads node
• Update Google `inline_people_also_search_for` with new safe navigations
• Fix Google `related_questions` not present for some results
• Fix Google `organic_results` not parsing some elements
• fix(bing): parse nutrition facts error

2021-12-22

• Add validator
• Add prettified HTML support
• Add parameters support
• Add parser
• Fix typo mistake in parser
• fix(walmart-review): catches more error state from walmart endpoint
• test(inline-video): fix spec failing
• Add fundamentals of the pretty HTML Part 1

2021-12-21

• Add X-ray and validations
• fix(inline-video-carousel): conflicted with answer box elements

2021-12-20

• fundamentals for Naver Image Api.
• fix(inline-people-search-for): error thrown when clone link doesn't have query params
• fix(organic-results): rich snippet parse wrongly in mobile

2021-12-18

• Fix the css class, to remove the duplicated result

2021-12-17

• Fix Google Knowledge Graph `audience_reviews` not parsing all reviews - Fix parser - Add spec file - Add test file
• Fix where key moments include is
• Update parser
• fix(map-place): not checking longtitude
• Fix Google local pack description not appearing for some results
• Add safe navigation

2021-12-16

• Fix typor error on parser
• Fix minor mistake in parsing websites
• Fix minor mathematical statement for pagination
• Add safety to parsing websites in local_results
• Remove unnecessary safe navigations
• Fix Google Knowledge Graph not parsing `hours` for some results
• Change mistaken key `years_in_bussiness` to `years_in_business`
• Add safe guard
• nit(google): use && instead of and

2021-12-15

• Fix missing big top stories
• Fix local_results parsed as organic_results in some searches
• Fix new kinds of local_results to be parsed - Add recent local_results that are not parsed - Fix keys that are not parsed for desktop - Fix keys that are not parsed for mobile - Fix misparses by introducing a method - Fix maps not being picked up for some results in desktop - Fix maps not being picked up for some results in mobile - Fix dummy coordinate values to be avoided (0,0) for some searches
• Update parser
• Use get_valid_link to retrieve URL
• Fix date showing in snippet
• Add safe guards
• Handle potential error on missing lat long in place result

2021-12-14

• Remove redundant css lookup
• Fix Google not parsing `local_ads` for some searches - Update parser with new class names - Update parser with new fields for tablet - Update xray markings and setting - Update spec with new examples covering all new aspects - Add new test files
• Fix answer_box parsing the wrong box
• Add X-Ray to answer box directions train result
• Fix answer box destination not parsing to and from correctly for some locale
• Fix error when gps_coordinates not available
• Fix direct answer box table error on certain table format
• Fix organic result containing video not showing in mobile
• Trim down code
• Fix typo
• Fix Naver pagination for non-ASCII queries

2021-12-13

• Add support for key_moments in organic_results
• Fix parser and add spec test.
• Fix indentation
• Fix src images not loaded on mobile for key_moments
• Add extracting for key moments that are not on page
• Fix Google `local_ads` not appearing for some searches
• Update inline people also search for parser for mobile
• Update default `page` parameter - Update playground and documentation explanation for `page` - Update default of `page` from 0 to 1 - Update handling of parameters - Update catching the error for `page=0` - Update examples on documentation for `page` change - Update pagination spec values with new ones for `page` change - Update reviews spec with a clearer order
• Update tablet extraction
• Update parser
• Update parser and spec - Refactor `about_this_result` extraction to be used by both desktop and mobile - Update spec output order - Update spec to give types as output - Fix minor indentation style in parser
• Add squish text
• Use text method directly
• Add directions answer box parser

2021-12-11

• Update Apple Product API - Update handling parameters - Fix unnecessary safe navigation values in parser - Add parsing of `snippet` key - Update 500 and 204 errors that are recent in validator - Update test HTMLs with recent ones - Update product page spec file - Fix typo mistake on validator spec

2021-12-10

• Use index instead of length
• Add parsing for link
• Fix the CSS class
• Remove unneeded escaping
• Ensure thumbnail exists before adding xray
• Add xray to moment
• Update comma to last index of Hash
• Parse variant swatches in Product Result in Walmart Product engine (#2270)

2021-12-09

• Don't write gps_coordinate if nil
• Add parser for key moments in videos

2021-12-08

• Fix unaccepted page values to return error - Fix negative numbers giving error - Fix string containing values to give error
• Update app/models/search/google/parser/direct_answer_box.rb
• update node parser names
• Update parser

2021-12-07

• Fix long text not using alt text for some people also search for
• Update direct_answer_box.rb
• Update app/models/search/google/parser/direct_answer_box.rb
• Update app/models/search/google/parser/direct_answer_box.rb
• Update app/models/search/google/parser/direct_answer_box.rb
• Add parsing for url and title in answer hover box
• Update parser
• Fix error on direct answer box for some searches
• Fix Walmart `page` parameter to only support values between 1 and 100
• Fix rich snippet rating parse decimal to whole number
• Refactor code
• Refactor google local pack reviews parser
• Remove unnecessary spaces and declaration
• Fix spec error
• Refactor home depot check empty result

2021-12-06

• Update parser add the missing travel
• Fix Google Search `cached_page_link` key appearing when there isn't a cached web page - Fix Google Search `cached_page_link` key appearing when there isn't a cached web page - Fix `answer_box` and `organic_results` have same item. - Add `about_page_link`
• Update hash keys, percentage and chart
• Handle home depot empty results
• Fix bing finance when volume data not present
• Fix google local pack reviews more than 1000 not parsed correctly
• Remove unnecessary declaration

2021-12-03

• Make sure extracted_gps_parameters isn't nil
• Fix mobile product result getting extra producs
• Fix manufacturer link missing on mobile
• Fix infinite loop when page is set to less then 1
• Fix date extraction
• Fix parser
• Revert "Revert "Add Google Top Sights API""
• Revert "Add Google Top Sights API"
• Imrpove sport results spec and code
• Imrpove sport results thumbnail parser
• Fix Xray apply wrongly
• Fix google sport results sets on parsing
• Remove mobile from Walmart Product Reviews xrat template
• Remove device from Walmart Product Reviews
• Remove search information

2021-12-02

• Change key names, to not have any conflicts.
• Change time to UTC timezone
• Check if details is not empty
• Fix Apple App Store `organic_results' invalid `links`
• Update validator
• Fix race condition in Walmart's `validator_spec`
• Add X-Ray to google sport results
• Update parser to support more tennis olympics games

2021-12-01

• Fix ratings and reviews extraction
• Update price and currency css class
• Add link
• Add xray to product_result and fix pricing not being parsed
• Update parser
• Fix time in the chart
• Add X-Ray
• Fix google olymipics tennis sport results
• Refactor Google Search `top_stories` `link`
• Refactor encrypt json path
• Refactor extract_ludocid_and_lsig in google local_results

2021-11-30

• Fix answer box, and add the currency chart
• Add `store_id` parameter to Walmart Product
• Fix Google Search `top_stories` duplicate items issue - Fix `live` key appearing breaking down visuals in `mobile` - Fix duplication of items inside keys - Remove unnecessary lines of code for link parsing - Check for duplication within tests for `mobile` and `desktop`
• Refactor Apple Reviews `review_date` key from Hash to String
• Fix merge conflict for Apple Reviews API
• Parse top sights map
• Fix undefined method 'scan' in local pack

2021-11-29

• Fix thumbnails and added extension for video or the number of photos in the blog post
• Fix parser not extracting lyrics, and fix the play_link
• Update parser
• Add documentation
• Add top sights parser
• Fix Apple App Store giving 500 error
• Update parser
• Fix Google Search Top Stories `link` not present in some results
• Fix local result type return true

2021-11-28

• Fix Google Search vertical top stories duplicated keys(not final)
• Add Google Search Top Stories `live` key for desktop and mobile - Update parser for `live` badge - Make `live` badge appear in HTML - Update parser for new kind of top stories (horizontal array) - Update top stories desktop and mobile specs - Move images - Expand documentation for live key

2021-11-26

• Add presence check to more css selectors in baidu organic results
• Add popular destinations parser
• Fix google local pack mobile type is parsing incorrectly
• Add years_in_business for Google Search Local Pack mobile
• Revert "Add years_in_business to Google Search Local Pack mobile"
• Revert "Update yeras_in_business regex to suppor other languages"
• Revert "Squish rating text in Google local results"
• Fix Google Map search parsing error (#2301)

2021-11-25

• Fix #2346 nil error
• Check any instead of just if nil
• Add xray for hover info
• Compact hash
• Update nexearch_results.rb
• Update nexearch - add music_results
• nexearch update - apps_results spec tests
• Update nexearch - add apps_results
• Update nexearch - add academic_results
• Parse hover info and remove from snippet in answer box
• Revert `rating` parser changes

2021-11-24

• Use unescape_js in JSDivExtractor
• Update app/models/search/naver/parser/ads_results.rb
• Remove the dot from the total_time
• Update nexearch - add influencer_results
• revert not forcing the domain in xray resources, and convert when updating demos
• Update nexearch - Travel spec tests.
• Add unclaimed_listing check for Google Maps
• Add video author parser on mobile
• Update nexearch - add travel to knowledge_graph
• Check for presence in Google `local_ads`
• Add multiple options state and product page url id on Walmart engine

2021-11-23

• Don't force domain in xray file
• Force xray click-blocking to activate right away, wait for load for rest of code
• Update nexearch - add knowledge_ results spec tests.
• update nexearch - add knowledge_results
• Update mobile parser
• Update parser
• Revert "Update parser"
• Revert "Update parser logic"
• Update parser logic
• Update parser
• Update parser
• Don't show empty text field in content rating
• Add video duration support on desktop
• Update Walmart Product Reviews validator to handle 404
• Fix json path can't decrypt non ascii characters
• Update to new method name
• Fix the cause of fail for older spec files in Google 'discover_more_places'
• Fix Google Search `local_ads` mobile not showing results anymore - Fix mobile not showing up in recent results - Add new test files to spec - Expand `hours` element to be scraped in different results

2021-11-22

• Fix missing keys for mobile, desktop and improved the test
• extract float using utils
• Update empty video results detection
• Update mobile video results parser
• Update desktop video results parser
• Update validator logic
• Squish rating text in Google local results

2021-11-21

• Improve js object parsing

2021-11-19

• update the parser & add tests
• Apply suggestions from code review
• Extract ad results, add tests and fix conflicts with web organic results
• Refactor tieba parser for baidu engine
• Add X-Ray to Walmart Product Reviews for mobile and tablet
• Add X-Ray to Walmart Product Reviews for desktop

2021-11-18

• Add Walmart Stores List
• Add parser for hotels answer box
• Fix workflow spec failing
• Add walmart_product_reviews engine

2021-11-17

• Fix duration
• Update Google Search `top_stories` in mobile - Add `live` key - Refactor mobile specs - Add a test file
• Update yeras_in_business regex to suppor other languages
• Update regex for years_in_business to support general languages
• Update years_in_business regex to support general languages
• Fix yeras_in_business regex not working for other language

2021-11-16

• Pass location cookie to Walmart Search
• Add years_in_business to Google Search Local Pack mobile
• Add X-Ray to local_results in Google Search Local Pack mobile
• Change engine name to `apple_app_store`
• Add years_in_business to Google Search Local Pack desktop
• Parse number of years in business on Google local results correctly
• Add X-Ray to Google Knowledge Graph name and extensions

2021-11-15

• Fix Google Knowledge duplicated results
• Fix parser & add spec
• Fix extensions in Google Knowledge Graph not present for some searches
• Fix Google Knowledge duplicated results
• Update business open status to select single css with at_css

2021-11-12

• Fix parser
• Fix direct answer box error when when row is merged (#2284)
• Fix parser
• Fix direct answer box error when when row is merged (#2284)

2021-11-11

• Make come back fo fulfillment_badge in Walmart Search
• Update movie_showtimes.rb
• Refactor images extraction for mobile
• Minor change in parser
• Update movie_showtimes.rb
• Add new api business open status to the knowledge graph doc
• Parse business future date and recently open in Google Search knowledge graph
• Make come back fo fulfillment_badge in Walmart Search

2021-11-10

• Update app/models/search/google/parser/direct_answer_box.rb
• Fix images not extrated from google top carousel
• Fix & add spec test file
• Update app/models/search/google/parser/direct_answer_box.rb
• Fix images not extrated from google top carousel
• Fix some image links appearing as protocol relative
• Fix & add spec test file

2021-11-09

• Add dates results
• Add parsing for popular times and live data on google maps
• Parse location in Walmart Product
• Update parser
• Add parsing for popular times and live data on google maps
• Refactor `text` key to `snippet` - Resize images in documentation - Make images appear on playground - Simplify desktop image extraction - Refactor `text` key to `snippet` key - Add more examples in documentation - Add new spec files - Seperate mobile and desktop tests - Add a seperate test for optional snippet value
• Implement X-Ray to location in Walmart Product
• Parse location in Walmart Product
• Add mobile template to Walmart Product playground
• Update parser
• Add dates results
• Parse variant swatches for Walmart Product Result

2021-11-08

• Revert "Fix parser and add spec test."
• Fix parser and add spec test.
• Fix parser extract query
• Add user_nickname and customer_type to Walmart Product top reviews
• Update parser to fix extracting the reviews
• Fix Walmart search spec failing
• Revert "Add location with own defined constant key"
• Add location with own defined constant key
• Update comment
• Update parser
• Fix typo
• Update parser to fix extracting the reviews
• Revert "Fix parser and add spec test."
• Fix parser and add spec test.
• Fix parser extract query
• Update documnetation - Fix a mistake in documentation - Remove `see_more_link` - Update documentation
• Add user_nickname and customer_type to Walmart Product top reviews
• Update parser
• Fix Walmart search spec failing
• Revert "Add location with own defined constant key"
• Fix typo
• Update comment
• Add location with own defined constant key

2021-11-05

• Revert "Update Gems (#2086)"
• Update Gems (#2086)
• Add parser
• Revert "Update Gems (#2086)"
• Add parser
• Update Gems (#2086)
• Add X-Ray to Walmart Product desktop
• Fix minPrice is not parse in Walmart search
• Responsibe Walmart search view

2021-11-04

• Add swatch image to Walmart search variant swatches
• Parse Walmart search location
• Rename fulfillment_badge to fulfillment_badges as now it takes array
• Fix Walmart search fulfillment_badge key change in Walmart data
• Parse sponsored product in Walmart search
• Fix Walmart search organic results consist null item
• Revert "Fix Walmart search has empty organic result item"
• Fix Walmart search has empty organic result item
• merge ratings fix
• fix ratings
• fix thumbnails
• tests pass, updated parser for non english
• Parse Walmart search location
• Parse sponsored product in Walmart search
• Add swatch image to Walmart search variant swatches
• Rename fulfillment_badge to fulfillment_badges as now it takes array
• Fix Walmart search fulfillment_badge key change in Walmart data
• Fix Walmart search organic results consist null item
• Revert "Fix Walmart search has empty organic result item"
• Fix Walmart search has empty organic result item

2021-11-03

• add spec, update to catch more user reviews
• Safe navigation
• add spec for olympic tennis results
• Use hl for cookie language
• add spec, update to catch more user reviews
• forgot ?
• fix up hours logic
• fix double count
• add more specs
• Refactor key names - Refactor `service_availability` to `hours` - Refactor `phone_number` to `phone` - Fix wrong tests in dashboard menu - Fix typo error in menu name - Update specs in accordance with key names change - change unless statements to if - Update documentation accordingly
• Fix `images` not appearing in mobile
• Support Google Search `discover_more_places` - Fix issue with `related_search_boxes` appearing in irrelevant places. - Fix issue with `related_search_boxes` using `discover_more_places` as xray - Add parser - Add tests - Add documentation
• Add X-Ray to Walmart search
• Add a validator
• Add conditional variable for ltr pages

2021-11-02

• better hours handling
• better hours handling
• better hours handeling
• mark more fore xray
• fix hours, and double listing

2021-11-01

• fix undefined method size causing airbrake
• fix google news on mobile (#2186)
• add google as_* params (#2170)
• use both classes
• fix tennis players parsing
• fix undefined method size causing airbrake
• fix google news on mobile (#2186)
• fix phone parsing
• add google as_* params (#2170)
• Fix empty results error
• Support Google Search Local Ads - Add Parser - Add Specs - Add Documentation

2021-10-29

• Fix Parser
• add test
• add tieba result parsing
• Update App Store - Update `mobile` default value and push it on playground (only select parameter is not working for engine=app_store) - Update engine and documentation definitions - Update route /app_store to /app-store - Update new titles for Documentation - Update `mobile` default parameter in backend - Update prettified html structure - Add Mobile Layout - Add Tablet Layout - Refactor `release_date` and `size` key value structres - Refactor relevant tests
• Check proxy country_code not the ll country_code
• Fix Parser

2021-10-28

• Fix the parser (Bing knowledge graph)
• Remove redundant code
• Add spec
• Fix the parser (Bing knowledge graph)
• Fix google KG and add direct answer box on RTL language results (#2208)
• Adjustments
• Dynamic geo proxy url

2021-10-27

• Don't return nil values
• Correct the node name
• Don't return nil values
• Final changes and fixes.
• Update `id` of review to be string
• Update parameter fallback values - Remove `country`, `id`, `sort` fallback values - Update `page` value and remove fallback value of `page` ( Default is declared in search parameters )

2021-10-26

• extract without style
• Fix baidu page navigation, and fix issues with updated baidu tests
• fix google local results places on mobile
• fix airbrake #6074 (#2201)
• Fix baidu page navigation, and fix issues with updated baidu tests
• extract without style

2021-10-25

• Update parser to add support for additional serpapi links
• Update google play parser to support serpapi links to play product pages

2021-10-24

• Fix issues
• Improve knowledge_graph part 5 (Game results) with spec tests
• Knowledge_graph part 5 (Game results) with spec tests
• Knowledge_graph part 5 (TV Show results) with spec tests

2021-10-23

• Improve knowledge_graph (Movie results) with spec tests
• Retry Google Maps using a country-targeted proxy
• Knowledge_graph part 4 (Movie results) with spec tests

2021-10-22

• Remove binding.pry
• Resolve one of the missing cached links to be shown
• Add customer review badges - Add `customer_type` to customer reviews - Add `customer_type` to spec tests
• Revert nil? to blank?
• add new class & spec

2021-10-21

• Resolve walmart customer username and date not appearing
• add google lyrics (#2149)
• Knowledge_graph part 3 (Person full results) with spec tests
• fix again for ratings (#2182)
• google: fix local results rating (#2172)
• Adjust Google Reverse Image Search error handling

2021-10-20

• Update presence of parameters in search - Update presence of parameters - Update validation concern for product id - Remove unnecessary failsafe values for product id
• country knwoledge_graph and specs

2021-10-19

• Update Prettified HTML - Remove loading icon in prettified HTML - Remove extra space with iTunes Layout in prettified HTML - Remove unnecessary safe navigations and conditional hashes
• Update key `reviews_state` to `reviews_results_state` - Update `reviews_state` - Update simply parser - Update reorder pagination keys
• Update simplify parameters
• Update parameters - Update validation concern for product_id - Remove unnecessary safe_query for `product_id`
• Add Errors in returning final hash
• Update parameters - Update validator for `term` with a different validator - Remove unnecessary parameter value assignment for `page` - Remove unnecessart check for `term`
• Update Parser - Include Error type in returning final hash. - Simplify pagination hash.
• Fixing title X-ray
• add summary and some fixes

2021-10-18

• Update app/models/search/bing/parser/finance_answer.rb
• Update app/models/search/bing/parser/finance_answer.rb
• Update app/models/search/bing/parser/finance_answer.rb
• Update app/models/search/bing/parser/finance_answer.rb
• Knowledge_graph Part 1
• Update Apple Regions Documentation - Update Apple Regions in line with new design - Update default country from UAE to US - Fix typo error in Apple Constants
• Fix parsing developers with artist links

2021-10-17

• Naver changes.

2021-10-15

• move custom header method to validator.rb
• Slight code changes

2021-10-14

• Update validator
• Update pagination - Update parser with 'current' key in pagination - Update pagination spec with 'current' key
• Update prettified HTML - Refactor prettified HTML entirely - pass query term to prettified html - Change Documentation Images

2021-10-13

• use nil coalescing here (#2150)
• Stock infromation with specs
• Don't escape "<" and ">"

2021-10-12

• Use regex instead of nokogiri for faster about_this_result parsing
• Update tests
• Update parser

2021-10-11

• Remove reused parameters - Remove reused parameters - Update some statements with ruby style - Remove extra spaces
• Refactor prettified HTML - Delete recreated htmls - Add xray markings on parser -

2021-10-09

• Update validator.rb
• Fix typo
• Check if place is from a different country than ll

2021-10-08

• fix x-ray selector and use TextUtils.convert_to_snakecase_sym for keys for bing knowledge graph (#2125)
• Using Xpath and commenting the X-ray for title
• Update app/models/search/yahoo/parser/organic_results.rb
• Add documentation

2021-10-07

• Rename symbol to currency
• Fix google inline shopping result ratings on mobile (#2101)
• add correct "when" key for first google event (#2105)
• Change from shipping to delivery

2021-10-06

• remove redundant titles in keys
• Use "price" and "symbol" keys
• Update gemfile
• Fixes and specs
• Fix `about_this_result` parsing

2021-10-05

• Remove escaped control characters
• Fix Yahoo snippet
• Remove escaped control characters

2021-10-04

• fix getting image when js extraction fails (#2094)
• Add Prettfied HTML, specs, Documentation
• Bing knowledge graph updates (#2064)
• Don't include empty arrays in about_this_result
• Update dynamic block parser with info comments
• Add support for reviews responses
• Update reviews parser
• Use money-heuristics to determine if keys contain a currency symbol
• Fix for playground HTML not loading JS
• Apply various fixes and updates to the parser

2021-10-03

• Fixing and include tests

2021-10-01

• Fix icons in prettified html
• Fix home depot airbreak #2090
• update google local_results addresses (#2079)
• fix airbreak (#2085)
• Add safe navigation in walmart parser
• Update prettified html - Render Sections in partial files - Fix Rating parse error

2021-09-30

• Fix indentation
• Remove more control codes
• Fix html UI
• Fix indentation
• Remove more control codes

2021-09-29

• Squish text
• Don't show empty links
• Get links from knowledge graph items instead of list of strings
• Fix issues from review comments

2021-09-28

• Parse related_questions and related_results
• Update app/models/search/walmart_product/parser.rb
• Add support for Google Play Reviews
• Update app/models/search/google/parser/shopping_results.rb

2021-09-27

• Update app/models/search/walmart_product/parser.rb
• item_nil? is a better name
• fix various nil errors in the walmart parser
• add number of pictures to google mobile knowledge graph
• Change page_size and num to ps
• Update parser, refactor prettified html
• Update app/models/search/home_depot/parser/pagination.rb
• Update app/models/search/home_depot/parser/pagination.rb
• Update app/models/search/home_depot/parser/pagination.rb
• Update app/models/search/home_depot/parser/pagination.rb
• Update app/models/search/home_depot/parameters.rb
• Update app/models/search/home_depot/parameters.rb

2021-09-24

• parse related keywords and fix issue with changing indexes
• Move displayed query to search_information

2021-09-23

• Add initial about this result parsing
• Add support for tv shows and episodes

2021-09-22

• Update shopping_results.rb
• Parse search query for reverse image search
• Revert "Fix Google missing sport results"
• Revert "Fix Google missing sport results"
• Update pagination and pagination spec - 'current' key in pagination
• Update field names
• Update Parser and Constants -Update App Page Parser -Update Developer Platform Constants

2021-09-21

• use compact with blank check
• add optional chaining for idml and reviews
• use compact
• merge new and old walmart product parsing
• game fix and add new spec
• merge!
• Fix how we extract individual items on page
• Exclude posts from parsing as knowledge graph attributes
• Update parser
• Add parser
• Update parser

2021-09-20

• Parse mobile customer reviews
• Escape `.` and `$` in error names as Mongo disallows them in keys
• merge source if exists
• rethink source: nil for now
• fix
• cleanup some nil keys
• merge master
• Fix sport results, Desktop and mobile.
• fix knowledge_graph title for tv-shows
• Update parser and parameters
• update google related questions parser
• Update pagination method name
• Fix parser
• Use query_scheme_and_domain
• Show "0" for `extracted_price` when `price` is "free"
• Remove `InclusionValidator` class

2021-09-17

• Fix multiline JSON parsing
• YAML not working on "End-of-text" (\u0003) unicode
• Adjust regex
• Rescue when date parsing fails
• Fix tests in `google/shopping_results/inline_shopping_results_spec.rb`
• Fixing
• add tag to google shopping results
• Fix offer_type wording and convert time to ms
• Update related_questions.rb
• Add offer_type and arrival_date and fix tests
• Add Apple Product API Initial Commit
• Update Spec Files and Documentation - Add Apple Reviews API Documentation - Update Apple Search API Documentation - Add Documentation Images - Fix Menu Names - Fix pagination not showing for some results - Fix results_stats showing wrong value for some results - Spec tests for pagination and sort - Spec files for pagination and sort
• Fix multiline JSON parsing
• YAML not working on "End-of-text" (\u0003) unicode
• Adjust regex

2021-09-16

• Initial impl for new walmart product page
• Revert "Initial impl for new walmart product page"
• Initial impl for new walmart product page
• Fix parsing the correct link on google related questions
• Conflicts fixes
• Ad publisher to google info
• Update validator.rb
• Make sure, pagination parameter is a string
• Fix `disallow_explicit` parameter - Update Documentation - Update Menu - Update Organic Results (more compact solution) - Update specs to work with `should` - Fix `disallow_explicit` in initialization

2021-09-15

• Fix some lists appearing as strings
• Fix people also searched for picking up extra data
• Stock_information spec
• Remove rescue
• Update frontend and backend to minimal working conditions - Fix Playground Layout - Fix Proxy Preference for Apple Reviews - Fix Pagination - Fix Results Stats - Fix Parameters - Add Validator - Fix Parser not showing rating - Update individual review prettified item - Update xray markings - Add Empty Results spec test and files - Add Reviews spec files - Add Validator spec (to be updated) and files
• Update validator.rb
• Update validator.rb
• Update validator.rb
• Update parser and add x-ray
• Fix 'organic_results' parsed from a seperate file - Fix 'organic_results' parsed from a seperate file - Add 'POST' in validator - Fix 'disallow_explicit' in parameters

2021-09-14

• Support map "People Also Search For" links
• Update organic_results.rb
• Update organic_results.rb
• Add Apple Reviews API -Initial Commit
• Fix 'explicit' parameter in Playground Remove 'explicit' parameter Add 'disallow_explicit' boolean parameter Add checkbox for 'disallow_explicit' in Playground Add number boxes for 'num' and 'page' parameters in Playground Fix 'empty results' giving the error 'Malformed Request'
• Update parser
• Stock information and inline videos for nxearch

2021-09-13

• Don't parse CSS in Google Job chip types
• Add App Store API - Add Localization Headers - Add Parameters - Add Parser - Add Spec Tests - Define Proxy Strategy - Add Documentation - Add Prettified HTML File (to be renewed later) - Add Playground Page - Support Handling Empty Result and Bad Requests in Validator and Parser - Add Apple Regions and Apple Languages Lists
• Fix parser

2021-09-10

• Add initial support for Google Play Product

2021-09-09

• Apply code style changes
• Update validators

2021-09-08

• Display actual ZIP code on the prettified Home Depot page
• Correctly generate search ID for the Home Depot when ZIP code and Store ID change

2021-09-07

• Remove dupe code
• rubocop:disable Layout/HashAlignment
• Add offers (mobile)
• Add desktop Local Posts for Google Maps Place
• Add knowledge graph Local Posts and Products

2021-09-06

• Refactor to work with pretty HTML

2021-09-03

• Fix X-Ray for Google `shopping_results[].extensions`
• Filter out empty `lists` and `paragraphs` from Bing Knowledge Graph
• Fix error in Bing Knowledge Graph parser
• Add custom headers support
• Fix code typo

2021-09-02

• Update direct_answer_box.rb
• Extract price to float
• Fix extensions missing for some searches
• Fix 500 error on Google Shopping
• Fixing the issues and improving the answer box

2021-09-01

• Revert "Google Events not scraping multiple days event's time"

2021-08-31

• Inconsistent indentation fix
• Specs & mobile & doc

2021-08-30

• Update app/models/search/google/parser/direct_answer_box.rb
• Update app/models/search/google/parser/direct_answer_box.rb
• Update app/models/search/google/parser/direct_answer_box.rb
• Update app/models/search/google/parser/direct_answer_box.rb
• Update app/models/search/google/parser/direct_answer_box.rb
• Update app/models/search/google/parser/direct_answer_box.rb
• Update app/models/search/google/parser/direct_answer_box.rb
• Update app/models/search/google/parser/direct_answer_box.rb
• Changes & improvements
• Parser update
• Add prettified html support for items highlights
• Add items highlight parser

2021-08-29

• Update events_results.rb

2021-08-28

• Add translation box & specs

2021-08-27

• Fix some empty space issues

2021-08-26

• Add tests
• Fix link extraction

2021-08-25

• Add more parameter validations
• Add support for price parameter
• Show `see_more_link` for pagination results
• Extract `serpapi_pagination` to separate method
• Home depot finish

2021-08-24

• Allow raw HTML data to be parsed without status to fix tests
• Display `error_breakdown` in our JSON only in development environment for now
• Rename `proxy_errors` to `error_breakdown`
• Add "403 - Forbidden" validation
• Update pagination to be multilingual
• Don't show empty pagination hash
• Support pagination for different html structures, when using pagination

2021-08-23

• Return early hash with metadata if search is queued, processing, or errored
• Handle JSON errors raised by `SearchSplitter` at the same spot

2021-08-22

• Pagination, specs and improvements

2021-08-21

• Improvement
• Pagination & Improvement
• Spelling fix remove and pagination template add
• Improvements
• Refactor home depot parser part 2

2021-08-20

• Support Google Knowledge Graph Movies Add 'knowledge_graph.movies' Add 'knowledge_graph.movies' spec files Add 'knowledge_graph.movies' documentation
• Fix parser naming
• Update prettified HTML
• Remove topic_id parameter support
• Add parser for app screenshots
• Refactor parser

2021-08-19

• Don't treat ReCaptcha text in snippet as an actual ReCaptcha page
• Support Google Knowledge Graph Episodes Support Google Knowledge Graph Episodes Add related spec files Add related documentation Fix minor documentation problem in Google Knowledge Graph Notable Moments

2021-08-18

• Changes

2021-08-17

• Home depot changes
• test specs update & mobile selector
• Test specs update
• Google Shopping Product Fixes
• Fix Google Inline Videos

2021-08-14

• link correction
• Support new Google Knowledge Graph Layout Fix horizontal extra bar layout Fix vertical extra bar layout Fix mobile horizontal extra bar layout Fix and Update knowledge_graph.cast Fix and Update knowledge_graph.related_media Fix and Update knowledge_graph.people_also_search_for Fix and Update knowledge_graph.see_results_about Add x-ray markings for all fixed and added blocks and their elements Add spec file and html files for horizontal, vertical and mobile-horizontal extra bar layouts Fix spec file mislabeled as knowledge_graph.header_images to knowledge_graph.related_media Add old class names for new layout on mobile Add x-ray markings for 'header_images' Add documentation for new blocks and example for different layout Add knowledge_graph.editorial_reviews Add knowledge_graph.clip Add knowledge_graph.audience_reviews Add knowledge_graph.user_statistics Add knowledge_graph.watch_now
• Requested changes on Yahoo! Shopping
• Improvements

2021-08-13

• fix problems and improvement

2021-08-12

• New link selector
• Update organic_results_videos_mobile.rb
• Naver changes

2021-08-10

• Improvments
• Fix missing keys, Google_jobs and new specs
• Fix null value and spec
• Fix missing job_id

2021-08-09

• Filtering, Specs and Screenshots quality improve.
• Update selectors

2021-08-06

• Improve and fix some problem

2021-08-05

• Fix extraction of images results with products

2021-08-04

• User agent update
• Unstuck HomeDepot tests on GitHub Actions
• Headers by bart

2021-08-03

• Improvements
• Add prettify HTML support for Google Play API
• Update parser
• Fix some errors and add test
• Support Google Knowledge Graph Notable Moments Add Google Knowledge Graph 'notable_moments' Add spec files for Google Knowledge Graph 'notable_moments' Add documentation for Google Knowledge Graph 'notable_moments'

2021-08-02

• Support Google Search Results Short Videos Add Google 'short_videos' Add spec files for Google 'short_videos' Add documentation for Google 'short_videos'
• Support Google Knowledge Graph 'ordering_options' -Add Google Knowledge Graph 'ordering_options' -Add Google Knowledge Graph 'ordering_options' x-ray markings -Support Google Knowledge Graph 'service_options' text version -Add Google Knowledge Graph 'service_options' text version x-ray markings -Support Google Knowledge Graph 'health_and_safety' -Add Google Knowledge Graph 'health_and_safety' x-ray markings -Fix Google Knowledge Graph 'reviews_from_the_web' not parsing on some results -Add Google Knowledge Graph 'reviews_from_the_web' x-ray markings -Add spec file for 'ordering_options', 'service_options', 'health_and_safety', 'reviews_from_the_web' -Extend Google Knowledge Graph Documentation

2021-07-31

• Remove spaces

2021-07-30

• Update `graph_ql` method
• Refactor product API
• Fix Google Search Results 'inline_people_also_search_for'
• Add parser for additional html blocks
• Add x-ray for first pages

2021-07-29

• Update validator.rb
• Update parse_html.rb
• Update parse_html.rb
• shopping_yahoo_gql, safe_get, and parse_html mix multiple responsibilities.
• Remove num parameter
• Update parser
• Update Google Play pagination

2021-07-28

• Changes
• Update parse_html.rb
• Update parser.rb

2021-07-27

• Add pagination parser
• Fix POST request to Google Play Store

2021-07-26

• Update parser

2021-07-23

• Add initial parser
• Add initial Google Play support

2021-07-16

• Check Fix odd layout
• Fix odd layout (top-level filter)
• Improvements and changes
• Home Depot product parser

2021-07-15

• Fix thumbnail extraction for Google inline people also search for block

2021-07-14

• Pagination and product wip
• Fix Google related_questions on mobile
• Add merchant_description to Google knowledge_graph

2021-07-13

• Add `place_id` parser for Googles knowledge graph
• Simplifying and fixes
• Pagination & parser fixes (Home depot)

2021-07-12

• Fix merge conflict
• Update app/models/search/duckduckgo/parser/inline_videos.rb
• Update parser

2021-07-11

• Parser refactor (GraphQL)

2021-07-10

• Fix Knowledge Graph not parsing full menu url
• Fix snippet class for expanded sitelinks

2021-07-09

• menu_array to menu_links
• //WIP

2021-07-08

• Revert "All problems fixed under a new pr."
• All problems fixed under a new pr.
• Correct date extraction from organic results on mobile too
• Fix to latest failing spec
• Update parser
• Restore missing # AUTO_ADD_HTMLS
• Revert "Changes to last accidental commit and new additions"
• Fix x-ray html scrolling on some engines
• Final changes before the push Missing Auto add HTMLS Indentation Differences
• parser and util refactor

2021-07-07

• Simplify date extraction from Google organic results
• Changes to last accidental commit and new additions
• X-Ray for `rich_snippet_table` and `rich_snippet_list`
• Fix X-Ray for Text nodes
• Revert accidental commits

2021-07-06

• Support for older versions of knowledge graph
• Use single `at_css` call instead of multiple `search`es
• New spec file, changes on spec files, new class names, old snippet class names
• Fix extraction of Google Images when Ads present on page
• Fix finance results price movement extraction
• Update direct_answer_box.rb
• Fix parser
• Fix edge case of `original` image extraction on Google Images
• Update direct_answer_box.rb
• Fix extraction of `original` image on Google Images

2021-07-05

• Fix rating extraction in regular Google search
• Fix rating and reviews extraction in local_results
• Creating the parser
• Spec fail fix and name change
• Improvements and updates
• Update parser
• Fixed according to failing specs
• Mobile parser fix
• Update parser to scrape sources
• Add parser

2021-07-03

• Changes made according to reviews by Zil
• Duplicate definitions for date merged Reviews addressed Co-author : Ilya Zub <ilya@serpapi.com>

2021-07-02

• Update parser
• Update Google Maps
• Remove events localization parameters
• Update documentation icon
• Remove "gl" support
• Fix parser
• Update prettify html

2021-07-01

• Extract other options in Google Shopping
• Exclude links to other google_product variants
• Update validator
• Add prettify HTMLs
• More changes made from reviews
• Changes committed regarding rewiews by Ilya
• Add parser for open_state
• Miscellaneous fixes

2021-06-30

• Update facts extraction
• Revert four accidential commits to master

2021-06-29

• Resolves #1804
• Resolves #1804
• Changes to spec and better parsing
• Fixes Description not parsing
• Revert change in file permissions
• Add parser
• Changes to spec and better parsing
• Update documentation
• Update parser and add x-ray
• Fixes Description not parsing

2021-06-28

• Add playground support
• Add initial Google Maps Photos engine support
• Remove duplicate
• Add support for sitelinks expanded and inline

2021-06-26

• Fix title, adding sitelinks and xrays
• Profiles (old knowledge graph)

2021-06-25

• Possible fix to last spec fail
• Revert changes to file permissions
• Little changes
• Mark with X-Ray Baidu Related News
• Last Push
• new knowledge graph finish

2021-06-24

• Documentation for all changes and missing parts Simplified related_videos parsing
• Just before the documentation Spec file fixed, sparatic parsing failure fixed, other_area_names unique now
• Add finance answer box parser

2021-06-23

• Fix thumbnail links
• Fix duration in mobile videos
• Resolves #1799 along with requested changes

2021-06-22

• get valid link method
• get valid link method
• Update app/models/search/bing/parser/knowledge_graph_results.rb
• Removing position from knowledge graph
• get valid link method
• Removing position from knowledge graph
• Removing position from knowledge graph
• Removing position from knowledge graph
• Removing position from knowledge graph
• Improve field extraction
• Fix parser
• Parse weather answer box
• Resolves #1788 alongside related spec files and test.

2021-06-21

• Update pricing (offers) selectors
• Support for Israeli pages and spec files
• Add results state for `Not many results contain`

2021-06-20

• Hardened with safe navigation
• Related videos missing alerted by spec
• Fixes related news, false snippet taking Fixes related news, age and source mistakes

2021-06-18

• Update app/models/search/google/parser/product_result.rb
• Revert "Squashed commit of the following:"
• Squashed commit of the following:
• Update parser
• Add pagination
• Add duckduckgo main documentation
• Update knowledge_graph.rb
• Remove shopping results parser
• Update parser to escape HTML
• Rename region parameter to `kl`
• Update organic_results.rb
• Update app/models/search/baidu/parser/organic_results.rb
• Update app/models/search/baidu/parser/organic_results.rb
• Update organic_results.rb
• Resolves #1701
• Update app/models/search/baidu/parser/organic_results.rb
• Update app/models/search/baidu/parser/organic_results.rb
• Update app/models/search/baidu/parser/organic_results.rb

2021-06-17

• Fix thumbnail extraction
• Update some block extractions
• Update app/models/search/baidu/parser/organic_results.rb
• Rename Google Maps Reviews `place_id` parameter to `data_id`
• Resolves #1012 New test htmls for #1012 Resolves partial description issue for #1776

2021-06-16

• Fix request url
• Update parser
• Add DuckDuckGo prettify support
• Update parser
• Add support for place_id
• Fixes #1780, Partially fixes #1776, retracts #1609

2021-06-15

• Tests for (places,view,books) and more improvements.
• Add new lines at the end of the files
• Test upadtes and improvements
• Extract place_id

2021-06-14

• Fixes #1013 , #1609
• Update app/models/search/naver/parser/organic_results_nexearch.rb
• Update app/models/search/naver/parser/organic_results_nexearch.rb
• Update app/models/search/naver/parser/organic_results_nexearch.rb
• Update app/models/search/naver/parser/organic_results_nexearch.rb
• Update app/models/search/naver/parser/organic_results_videos.rb
• Update app/models/search/naver/parser/organic_results_videos.rb
• Update app/models/search/naver/parser/organic_results_videos.rb
• Update app/models/search/naver/parser/organic_results_videos.rb
• Update app/models/search/naver/parser/organic_results_videos.rb
• Update app/models/search/naver/parser/organic_results_videos.rb
• Update app/models/search/naver/parser/organic_results_videos.rb
• Update app/models/search/naver/parser/organic_results_videos_mobile.rb
• Update app/models/search/naver/parser/organic_results_videos.rb
• Update app/models/search/naver/parser/organic_results_videos_mobile.rb
• Update app/models/search/naver/parser/organic_results_videos_mobile.rb
• Update app/models/search/naver/parser/organic_results_videos_mobile.rb
• Update app/models/search/naver/parser/organic_results_videos_mobile.rb
• Update app/models/search/naver/parser/organic_results_videos_mobile.rb
• Fix video pagination and ducomentation highlights
• Update related_searches.rb
• Update app/models/search/naver/parser/related_searches.rb
• Update app/models/search/naver/parser/related_searches.rb
• Update app/models/search/naver/parser/organic_results_nexearch.rb
• Update app/models/search/naver/parser/organic_results_nexearch.rb
• Update app/models/search/naver/parser/organic_results_nexearch.rb
• Update app/models/search/naver/parser/organic_results_nexearch.rb

2021-06-13

• Update app/models/search/naver/parser/organic_results_news.rb
• Update app/models/search/naver/parser/organic_results_news.rb
• Update app/models/search/naver/parser/organic_results_news.rb
• Update app/models/search/naver/parser/organic_results_news.rb
• Adding squish to the organic result (link)
• Removing x-ray. One shadows another.
• Update app/models/search/naver/parser/organic_results.rb
• Update app/models/search/naver/parser/organic_results.rb
• Update app/models/search/naver/parameters.rb
• Update app/models/search/naver/parameters.rb
• Update app/models/search/naver/parameters.rb
• Update app/models/search/naver/parameters.rb
• Update knowledge_graph_results.rb

2021-06-12

• Update organic_results.rb

2021-06-11

• Update product_result mobile
• Update product_result desktop
• Retrieve search token in the background and use it to make the request
• Fix failing tests for Walmart parser
• Add spelling_fix parser
• Add knowledge_graph parser
• Add shopping_results parser
• Add related_searches parser
• Add organic_results parser
• Add news_results parser
• Add inline_videos parser
• Add inline_images parser
• Add ads parser
• Add playground support
• Playrgound, documentation, fixes, tests (Naver)
• Fix snippet and link
• Fix snippet (Yandex)

2021-06-10

• Extract table `parse_organic_results` method
• Update parser
• New bing knowledge graph
• Fix Incorrect Yandex search results

2021-06-09

• Don't fail Walmart tests when optional data is missing
• Parse pagination on Walmart beta website

2021-06-08

• Parse filters on Walmart beta website
• Extract rating and reviews from Walmart beta website
• Parse search results on the Walmart beta website
• Naver Tests finished

2021-06-07

• Rename answer box hash with multiple boxes

2021-06-06

• Remove comments
• Page parameter & general fixes

2021-06-04

• Add initial parser commit
• Finishing mobile & tablet and updates on the playground

2021-06-03

• Fix Google Product seller results parsing with the Arabic language
• Fix extraction of `duration` of `inline_video_carousels` on tablet
• Rename `inline_videos_groups` to `inline_video_carousels`
• Add TODO comment for unused conditional branch in Google Product reviews content
• Add `link`, `position` to `inline_videos_groups`

2021-06-02

• Correct spelling on the Inline Videos documentation page
• Fix parser

2021-06-01

• Rename selling_points to highlights
• Remove unneeded comment
• Extract selling_points and features
• Adding mark for xray and fix the hash issue.
• Finish the Validator.
• Fix parser
• Fix parser
• add the shopping to nexearch
• Update parameters.rb
• Update validator.rb
• Playground updates

2021-05-31

• Extract full description of Google Product review content
• Fix features extraction
• Replace old features with new features block
• Extract rating from stars if reviews not present
• Remove old HTML files of `inline_videos`

2021-05-28

• Add comment about google product review content length
• Fix reviews pagination
• Add x-ray to reviews
• Fix rating extraction in Google Product
• WIP: Add tests for inline_videos_groups
• Add x-ray
• Add thumbnail parser

2021-05-27

• Fix Google Product description extraction

2021-05-26

• Add Naver API
• Fix parser
• Fix Bing x-ray node

2021-05-25

• Add x-ray markup to Google Product API
• Fixing missing `next` & `previous`

2021-05-24

• Fix extraction of `top_stories` on mobile

2021-05-21

• Update app/models/search/yandex/parser/pagination.rb

2021-05-20

• Mark Google Flights Answer Box with X-Ray
• Fix Google Flights parsing in `answer_box`

2021-05-18

• Fix related questions extraction
• Update xray.rb

2021-05-17

• Capture more elements in related searches block
• Edit file name from Pagination to pagination

2021-05-14

• Fix the missing eBay pagination from JSON

2021-05-12

• Fix Google Related Searches parsing on mobile
• Add X-Ray for Google `inline_videos`

2021-05-11

• Extract `platform` for horizontal Inline Videos on mobile

2021-05-10

• Adding yandex pagination
• Adding Yandex pagination to xray
• Create Pagination.rb
• Update parser.rb

2021-05-06

• Refactor Twitter Results parser

2021-05-05

• Implement Twitter results parser on desktop
• Fix parser
• Update parser
• Update parser
• Validate product_id
• Validate product_id
• Validate hd_filter_tokens
• Validation for product_id

2021-05-04

• Update parser
• Clear `start` and `num` parameters when generating `google_scholar` links
• Remove x-ray for citation table

2021-04-30

• Fix failing test
• Rename some parameters
• Update validator
• Update parser

2021-04-29

• Fix validator
• Add test for 302
• Exclude related searches from profiles

2021-04-28

• Add new element - review
• Add x-ray markup for discover_more_places
• Mark rating with x-ray

2021-04-27

• Support `position` field on `inline_videos`
• Fix Ads recognition and add x-ray markup

2021-04-26

• Improve variable naming
• Parse next div if price is present
• Fix Google local results address and hours extraction
• Fix parser

2021-04-23

• Update parser

2021-04-22

• Rename `user_id` to `author_id`
• Update parser
• Exclude Related Questions from Google Organic Results

2021-04-21

• Add pagination
• Extract SerpApi author link
• Update `get_valid_link` method
• Add prettify html for co author section
• Add engine parser
• Add parameter support

2021-04-19

• Add profiles parser for main Google Scholar organic results section

2021-04-16

• Fix parsers empty field
• Update parser
• Extract user ID
• Add Google Scholar Profiles parser

2021-04-14

• Update parser to fix position issue

2021-04-13

• Update documentation
• Fix block_position parser
• Fix parser

2021-04-09

• offers param is the same type as specs and reviews
• Make "offers" consistent with "specs", "reviews"
• Set offers=true for sellers=online parameters

2021-04-08

• Add xray markup for business operational status
• Remove unused code
• Add parser for inline shopping results
• Differentiate between temporarily and permanently closed

2021-04-07

• Fix Google Knowledge Graph business operational status

2021-04-06

• Update parser
• Improve Google Ads description selector in mobile and tablet
• Update parser
• Improve Google Ads description selector in desktop
• Add displayed_link and description x-ray markup to Google desktop ads
• Add x-ray markup for Google mobile Ads
• Fix description extraction in Google Ads desktop and mobile
• Capture new Yandex captcha page

2021-04-02

• Improve Google Autocomplete engine Youtube client regex
• Fix service_options extraction and JSON result key name
• Fix service options extraction in Google Local results
• Fix service options extraction in Google Maps Places
• Fix service options extraction in Google Search

2021-04-01

• Updated google products tests.

2021-03-30

• Updated doc pages removing local_results. Removed num parameter from google product.
• Replaced sellers with boolean offers.
• Removed nodes traversing. Added xrath text lookup.
• Updated google shopping pagination params. Changed local sellers doc page pointing deprecation.

2021-03-26

• Disable LinkedIn APIs
• Handle local_pack case with wrong address/missing phone.
• Fix exceptions in direct_answer_box_table

2021-03-25

• Fix NoMethodError in Google Local Results mobile
• Fix typo. Added guard clause.
• Fixed local pack: - title beign polluted with 'Why this ad?' - address being polluted with JS. - missing phone extraction.
• Update all engines to use the same playground wrapper

2021-03-24

• Fix Google Inline People Also Search For extraction
• Updated selectors and tests for related searches.

2021-03-23

• Fix knowledge graph hours extraction. Extract hours break.
• Fixed profiles data slipping into people_also_search_for in Google knowledge graph.
• "0" specs, reviews parameters handling

2021-03-20

• Add BooleanConflictWithOtherParametersValidator

2021-03-16

• Fix Google Organic results missing snippet, date, resolution and displayed_results

2021-03-15

• Fix 'streetview' extraction for Google Local Pack

2021-03-12

• Update app/models/search/google_product/parser/sellers_results.rb
• Update app/models/search/google_product/parser/sellers_results.rb
• Update app/models/search/google_product/parser/sellers_results.rb
• Fix online_sellers

2021-03-10

• Fix Google Maps parser

2021-03-08

• Fix Google Images without auto-corrected results
• Update parser

2021-03-04

• Hardcode baidu_domain
• Fix SSRF in yandex_domain

2021-03-03

• Update parser
• Don't extract original images as thumbnails
• Don't symbolize_keys in JSON.parse
• Fix Google Images Results extraction

2021-03-02

• Test Google Maps validation for incorrect symbols
• Require comma between coordinates in `ll` parameter
• Changed svg path selectors to classes of parent elements.
• Do not parse Google's related_questions as answer_box
• Fix extraction of an answer in the knowledge graph
• Fix parser

2021-03-01

• Validate `ll` parameter for `google_maps` engine
• Update code
• Update parser
• Add documentation

2021-02-27

• Fix parser

2021-02-26

• Remove comments
• Extract related search boxes from Javascript

2021-02-25

• Extract dictionary result of Answer Box from JS
• Add `reviews_link` to Google Maps API
• Remove false duplicates of answer_box in organic results
• Fix parser
• Fix mobile parser
• Update parser

2021-02-24

• Fix Google Answer Box tests
• Update parser
• Rename data_id to place_id
• Add more parameters support
• Update parser
• Fix answer_box extraction
• Fix table answer box extraction
• Fix parser
• Extract full title without domain name.
• Revert "Revert "Scrape answer box content""

2021-02-23

• Revert "Fix wrong query_ordered when `q` is missing"

2021-02-22

• Add parser
• Add Google Maps Reviews engine
• Fix wrong query_ordered when `q` is missing
• Require `q` parameter unless Google Reverse Image
• Update google_maps_search_pagination? method

2021-02-19

• Require `q` parameter for Google engine
• Move Google Maps search pagination condition to separate method

2021-02-18

• Add more explicit message for missing `ll` parameter when using pagination

2021-02-17

• Extract local_results extraction to parse_search_results_by_node
• Remove `alt` parameter from SerpApi link
• Fix empty results
• Add pagination to Google Maps

2021-02-16

• Improve readability of query_generic method

2021-02-12

• Fixed google knowledge graph fields extraction.

2021-02-10

• Fix parser

2021-02-09

• Reuse latitude and longitude from `ll` parameter
• Remove Google Maps Search engine
• Small fixes
• Use google_maps engine for pagination too

2021-02-08

• Changed htilrad to lrad for radius refinement.
• Include JSDivExtractor module in reverse image API
• Prettify Google Scholar Cite HTML response

2021-02-05

• Fix parser
• Revert "Fix parser"
• Fix parser
• Update parser

2021-02-04

• Codestyle changes.
• Fix Google Jobs htidocid extraction

2021-02-03

• Extract structured extensions as detected_extensions.
• Use calculate_md5 for hashing instead of query_ordered
• Fix ratings extraction in Google Jobs Listing engine
• Fix Google Jobs Listing API caching issue
• Make constants use moduls

2021-02-02

• Update parser
• Fix parser

2021-02-01

• Update parser

2021-01-29

• Use clone_with_new_url_parameters to build serpapi link
• Restored compatibility with older html examples.
• Fix parser
• Update parser
• Update parser
• Update parser to include title in Google's kg mobile

2021-01-28

• Use unpack1 instead of unpack for performance
• Fixed selector to obtain full-length title.
• Rename constants

2021-01-27

• Added htilrad for Google Jobs.
• Update searches_controller
• Update parsers
• Add pretty html for Google Maps Search

2021-01-26

• Parse see more link in Google inline people also search for block
• Add serpapi_link inline people_also_search_for items
• Refactor Google parameters
• Scrape Google inline people also search for section
• Updated LinkedIn Profile: changed naming in json response. updated tests and documentation. fixed typos in xray marking. added xray markings for thumbnails.
• Add support for Google Maps Search engine
• Extract whole title in Google Jobs API.

2021-01-21

• Remove empty string info elements in graph_results
• Add X-Ray markings for popular times
• Rename typical_time_to_spend to typipal_time_spent
• Prefer && instead of and operator
• Scrape undisplayed images. Undisplayed images are marked as "The rest of the results might not be what you're looking for".

2021-01-19

• Update documentation
• Handle one more edge case
• Move main get_knowledge_graph method to the top
• Rewrite popular times knowledge graph parser and update test htmls

2021-01-18

• Fix desktop ad phone selector

2021-01-15

• Do not use encrypted_start_of_api_key
• Fix Google images thumbnail extraction

2021-01-12

• Added validation of profile_id.
• Added xray for LinkedIn Profile.
• Update parser

2021-01-11

• Style code to be more consistent

2021-01-08

• Add initial encryption / decryption of image JSON path

2021-01-06

• Fix job listing job ID validation error message
• Stop polluting base class with validations

2021-01-05

• Added newlines.

2021-01-03

• Fixed typo in documentation. Fixed empty result parsing. Added test.

2020-12-30

• Added uule param to be passed when used directly, not as a location.
• Removed code duplication.
• Changed YAML #load -> #safe_load. Removed unnecessary navigation operator.

2020-12-29

• Added params validation for linkedin.

2020-12-28

• Added xray marking for LinkedIn public search.
• Updated documentation.
• Fix condition ordering to fix query_displayed value
• Refactor inclusion_with_value_modifier to inclusion_case_insensitive validator
• Replace if presence checks with allow_blank option in InclusionWithValueModifierValidator
• Use allow_blank: true instead of explicit presence check
• Fix Google Scholar link extraction edge case

2020-12-25

• Updated title extraction for yandex tablet search.
• Fix google jobs params extraction. Prevent google job listing fail if param is missing.

2020-12-23

• Revert "Scrape answer box content"
• Do not display query_displayed for Google Scholar if it is empty

2020-12-21

• Add explicit cites_id and cluster_id to final json response
• Usage of cites + q triggers a search within cited articles

2020-12-18

• Support cites parameter and fix serpapi URL building

2020-12-17

• Add uncommited change.
• Fixed yandex mobile search.

2020-12-16

• Fix date, displayed_results and image resolution extraction
• Fix Rubocop linting
• Make q optional if tbs is present for google engine
• Add rich snippet `reviews` support on mobile

2020-12-15

• Added documentation for LinkedIn profile. Added tests for LinkedIn profile. Covered missing sections from profile.
• Fix X-Ray markings for Google Mobile Search
• Fix Google reverse image search returning 500
• Fix displayed results, snippet and date
• Fix parser
• Update parameters.rb

2020-12-14

• Add `lr` parameter and update code
• Update parser

2020-12-11

• Add back markings for Google top stories
• Add X-Ray markings for Yahoo! engine
• Update parser
• Test unsuccessful searche statuses for google_reverse_image validator
• Update link handling
• Update validator

2020-12-10

• Make sure we are talking about the same `self.doc`
• Refactor regex for extracting body tag
• Remove duplicate line (typo)
• Remove typo
• Support X-Ray markings for Google Autocomplete pages

2020-12-09

• Fix issue with Google Maps being purely JSON and JS
• Fix parser
• Revert "Fix parser"
• Fix parser
• Add another condition to distinguish between kg and product result blocks

2020-12-08

• Extract selector for inline products to a variable

2020-12-07

• Fix rating selector for Google Shopping
• Update parser
• Fix Google gps coord extraction
• Improve Google lcl parser code

2020-12-04

• Changed json response scheme. Renamed 'description' -> 'occupation' Added 'about' Transformed 'meta' into 'last_job' and 'last_education'
• Revert "Revert "Fix address extraction from local results""

2020-12-02

• Removed outdated test examples.

2020-11-30

• Update parser
• Stop showing empty results
• Update parser
• Update parser
• Fix Google top stories empty results
• Removed stripping of a RLM and hypen minus in organic results date extraction. Code style changes. Changed test descriptions.
• Remove left-to-right marks.

2020-11-27

• Fix Bing JS images extraction
• Remove unneeded kg_block_header_name assignment

2020-11-26

• Handle empty thumbnails.
• Added bot user agents that work for LinkedIn scraping. Modified LinkedIn playground to display first and last name inputs in the top. Updated documentation, used Jimmi Neutron as a name for search. Fixed error when search gives a single result redirecting to profile page. Updated tests.
• Code style fix.
• Fix regex to include optional white space
• Catch first occurrence
• Update Youtube parsing Regex
• Reuse selectors in kg for items and headers

2020-11-25

• Rename LocationValidator to LocationPresenceValidator
• Add q parameter validation to google_scholar_cite engine
• Update parser to not mute airbrake notification
• Fix parser

2020-11-24

• Improve X-Ray markings for inline products
• Group regexes for JS image extraction again
• Update parser
• Update parser
• Update parser

2020-11-23

• Linkedin profile API initial commit.
• Fix Bing location cookie value for desktop
• Add Location validator for Bing search engine
• Add AtLeastOnePresentValidator and improve google scholar and walmart validation
• Double check and fix all validations based on parameters used
• Add custom conflicting parameters validator
• Move google_listing query encoding validation to engine module
• Move Bing, Yahoo, eBay validation to search engine modules
• Move hl and gl validation to all google search engine modules
• Add google domain validation to all google search engines
• Move query params validations to engine modules

2020-11-19

• Update app/models/search/xray.rb
• Add X-Ray markings for Google inline products
• Refactor parameter validations into `#{engine}/parameters.rb` module
• Don't group JS_IMAGE_REGEXES due to no performance improvement
• Speed up Google Images extraction

2020-11-18

• Handle Walmart Product 520 responses
• Added missing selector for rich_snippet on desktop.
• Added questions extraction for questions in mobile rich snippet.
• Update parser and tests

2020-11-17

• Update parser
• Get extracted JS thumbnails in O(1) time

2020-11-16

• Fix thumbnail extraction for Google Shopping
• Update and fix google product result parser
• Fix Yandex query_randomized
• Fix yandex_domain not being displayed in search parameters
• Mobile organic results missing rich snippet.

2020-11-13

• Fix Google product review results on mobile
• Revert "Fix address extraction from local results"
• Enable usage of lr parameter in Yandex search engine

2020-11-12

• Fix thumbnail extraction for cases when several thumbnail IDs point to the same image
• Fix another parser bug with treating local_results phone as address

2020-11-11

• Inject <head> if not present and switch to Nokogiri injection for X-Ray assets
• Expand X-Ray marking to more fields
• Defer responsability of setting xrayHtml classes to Ruby instead of JS
• Fix Google shopping missing title, link and remove empty extensions
• Remove shopping results in ads

2020-11-09

• Add linkedin public search API.
• Normalize all X-Ray markings to one style
• Clean up original prototyping code
• Expand X-Ray markings to more parsers
• Handle cases with unavailable item
• Capture Walmart product engine ReCaptcha

2020-11-06

• Fix Home Depot Product Search engine

2020-11-04

• Expand X-Ray markers to more data
• Get the parent Nokogiri node when getting an array of them
• Display better errors when not getting a Nokogiri node for X-Ray
• Move prototyped Nokogiri fixes from Google HTML beautifuler to X-Ray
• Mark snippet with `inner_wrap` to make it more readable
• Inject X-Ray markings via Nokogiri instead of JS
• Improve pagination
• Refactor code to get next page token

2020-11-03

• Correctly parse Youtube next page token

2020-11-02

• Changed online sellers retrieving by appending /offers to url. Updated documentation for Google Product API.

2020-11-01

• Fix parser

2020-10-31

• Fix empty results
• Update parser
• Update documentation
• Add results state
• Add spelling fix
• Fix and update ads parser

2020-10-30

• Added displayed_results/date for mobile results.
• Fix parser when horizontal ads appear
• Add support for Horizontal Ads
• Update youtube parser

2020-10-29

• Adderessed comments.
• WIP fixing edge cases
• Prototype injection of X-Ray classes using Nokogiri on the Ruby side instead of JS
• Changed name extraction to exclude redundant text.

2020-10-28

• Fix google jobs chips encoding
• Expand X-Ray markings to more JSON fields
• Scan HTML for thumbnails only once Related #948.
• Separate 'address' and 'phone' on Serbian local_results again

2020-10-27

• Use Addressable::URI for encoding and unencoding
• Fixed missing google product online sellers.
• Extend X-Ray markings to more engines and more JSON stuctures
• Fix Google Images having a <base> tag breaking relative paths
• Remove pagination parameter

2020-10-26

• Fix tests by preserving conditional mobile parsing order
• Fix mobile video results
• Fix Google inline videos desktop parsing
• Add video results parser
• Add support for new YouTube layout

2020-10-23

• Changed selector for google mobile related results.
• Fixed sitelinks extraction in google desktop. Skipped spec for local results direction link.
• Fixed google mobile results missing some organic results.

2020-10-22

• Added displayed_results extraction. Added tests for different localisations.
• Mark X-Ray for Google related searches
• Corrected selector for spelling fix.
• Encode filters parameter in Bing search
• Revert "Revert "Added spell check 'including results for' case.""

2020-10-21

• Mark for X-Ray a sample of JSON paths and Nokogiri nodes
• Add X-Ray JSON to CSS paths generations and injection to X-Ray view
• Re-use get_local_map for desktop and mobile Google local results
• Make sure 'address' and 'type' differ
• Fix phone extraction on French local results
• Fix phone extraction on non-English locale

2020-10-20

• Update validator.rb
• Update validator.rb
• Update Youtube validator

2020-10-19

• Revert "Added spell check 'including results for' case."

2020-10-16

• Extract resolution_image. Unified date extraction from organic results.
• Renamed thumbnail_link to thumbnail_destination_url.

2020-10-14

• Added spell check 'including results for' case.
• Fix address extraction on Serbian local results
• Fix description extraction on mobile Google local_results
• Added missing case for extract_js_image. Removed redundant condition.

2020-10-13

• Fixed incorrect extraction of a thumbnail. Added tests for image extractor.

2020-10-12

• Bind X-Ray files to homepage, playground, and inspect pages
• Update parser

2020-10-09

• Do not include rich snippets as date in mobile results
• Fix snippet extraction in Google mobile results
• Extract date and fix snippet in mobile organic results
• Fix snippet extraction from Google organic results

2020-10-08

• Add a date representation to Google organic results
• Make non-failing tests fail for Google local_parser

2020-10-07

• Remove empty lines and fix code style
• Fix phone extraction for some Google local results on mobile
• Handle empty sitelink element.

2020-10-06

• Fix 'address' extraction for desktop and mobile Google local results

2020-10-05

• Fix Bing links
• Fix Google organic results carousel extraction
• Get full snippet contents.

2020-10-02

• Added extra selector for answer box list extraction.
• Added new case for answer box table extraction.
• Force JSON output to use stricly 2 decimals for certain Floats
• Fix error 500 on blocked and recaptcha responses from Scraper API
• Add ads documentation
• Upgrade parser and tests

2020-10-01

• Added tests for different localisations. Fixed rating extraction for arabic language.
• Added new selectors for desktop snippet and inline sitelinks.
• Fix ads position
• Update and fix parsers
• Add support for mobile ads
• Update ads parser
• Update inline_shopping_results

2020-09-30

• Added serpapi_link for products in Home Depot. Updated documentation.
• Fix Yandex failures
• Update parser
• Update parser

2020-09-29

• Add support for Bing filters parameter
• Fix links

2020-09-25

• Added raw table array to result.
• Update parser
• Fix inline yandex images

2020-09-24

• Added fix to handle mobile expanded sitelinks cases.

2020-09-23

• Revert "Revert "Added extraction for mobile organic results.""
• Revert "Added extraction for mobile organic results."

2020-09-22

• Add conditional engine check for regex
• Fix getting thumbnails on Bing mobile
• Added new selector for rich snippet.

2020-09-21

• Moved collection assignment to conditional.
• Remove old Bing knowledge graph test
• Parse header images in Bing knowledge graph
• Fix bug of returning blank subtitle
• Add missing link and title to Bing mobile organic search results
• Extract description for retail store

2020-09-18

• Use get_valid_link method.
• Do not include empty thumbnails in results.

2020-09-17

• Added thumbnail_link for organic results.
• Added guard clauses for mobile sitelinks extraction.

2020-09-10

• Fix Google Knowledge Graph missing right image

2020-09-04

• Improve Bing parsing performance and fix ads parsinng
• Improve link selector in Google organic results

2020-09-02

• Remove duplication in extensions.
• Use get_valid_link to grab header image link in knowledge graph

2020-09-01

• Added fix for rating/votes in rich_snippet.
• Remove modular transformer and remove google redirection inline

2020-08-31

• Added more examples for answer box extraction.

2020-08-28

• Added extraction for mobile organic results. Added thumbnails. Added favicons. Added rich_snippet. Added sitelinks.
• Fix spec/search/youtube/video_results/star_wars_spec.rb
• Added price to knowledge_graph.

2020-08-26

• Improve knowledge graph answer selector
• Fix Google knowledge graph answer selector
• Fix spec/search/google/inline_videos/inline_videos_mobile_spec.rb
• Fix spec/search/ebay/spell_check/spell_check_spec.rb
• Fix and test knowledge graph header images extraction on mobile
• Add new parameter to Youtube engine to force old layout
• Add validator test

2020-08-25

• Add header images to knowledge graph

2020-08-24

• Fix spec/search/yahoo/validator_spec.rb
• Fix spec/search/walmart/featured_item/featured_item_spec.rb

2020-08-21

• Fix spec/search/google/sports_results/new_york_mets_spec.rb

2020-08-20

• Remove a level of abstraction from transformers
• Fix spec/search/google/sports_results/boston_red_sox_spec.rb
• Added fix for apply_options should be an array.
• Fix spec/search/google_reverse_image/reverse_image_devito_spec.rb
• Added first apply option from jobs api html to jobs listing api through job_id.
• Update app/models/search/google/parser/js_image_extractor.rb

2020-08-19

• Updated documentation.
• Style changes to direct box table parser. Modified tests to ease html refreshing.

2020-08-18

• Aligned image result for Home Depot product. Updated documentation.
• Refreshed tests and updated parser for HomeDepot.
• Remove location parameter

2020-08-17

• Updated Home Depot Product parser.
• Implement Google final hash transformer to modify parsed content
• Update validator and add empty results test

2020-08-14

• Add support for similar results on mobile
• Update parser

2020-08-13

• Add date parameter and remove location parameters

2020-08-12

• Fix spec/search/google/product_result/product_result_for_gpu_mobile_spec.rb
• Fix spec/search/google/local_results/local_results_for_products_spec.rb
• Add trending results parser
• Add organic results parser
• Update `get_thumbnail` method to accept custom id attributes
• Update js_image_extractor.rb

2020-08-11

• Fix spec/search/google/local_pack/local_pack_for_stores_spec.rb
• Set initial Bing News engine

2020-08-10

• Addressed comments.

2020-08-06

• Renamed variable.

2020-08-05

• Extract answer box table contents as a meaningful structure.
• Removed '/v' option from link construction.
• Fix inline product extraction.
• Partially fix Google ads

2020-08-03

• Fix spec/search/google/local_pack/local_pack_for_products_spec.rb
• Fix spec/search/google/knowledge_graph/top_carousel_spec.rb
• Revert "Revert "Fix parser""
• Update knowledge_graph.rb

2020-08-01

• Update HTML's, update and fix specs
• Fix massage_link() nil if link starts with http
• "//" to "https://"
• Fix Phone parsing for updated HTML's

2020-07-31

• Add parsing of Photos
• Fix spec/search/google/knowledge_graph/popular_times_spec.rb
• Fix spec/search/google/knowledge_graph/critic_reviews_user_reviews_spec.rb
• Remove unused code.
• Utilize text utils for salary extraction. Fix node extractor spec.

2020-07-30

• Trim whitespace for total_results
• Add impressions_per_month
• Changed specs collection matching to a style used across the project.
• Revert "Fix Google dictionary results parsing"
• Fix Google dictionary results parsing
• Fix eBay organic results spec
• Fix Google images results related searches selector

2020-07-29

• Fix google places title in address
• Fix mobile organic results extraction from Google results

2020-07-27

• Fix some deprecation warnings
• Fix 'inline_videos' extraction in Google results
• Fix 'related_searches' extraction on mobile Google searches
• Prepend URL origin to the links of Google mobile results

2020-07-24

• Fix typo in 'botttom' word

2020-07-23

• Add sourceid=chrome-mobile parameter
• Fix ie=UTF-8 parameter (was e=UTF-8)
• Fix old Google News design parsing for French language
• Treat Google Scholar 302 as 302 instead of success

2020-07-22

• Removed extraction of answer box as knowledge graph.

2020-07-21

• Added guard clause for local results address extraction.
• Fixed spelling for quotes type.

2020-07-20

• Extract serpapi_link to inline_images for reverse image search.

2020-07-17

• Added extraction of links for amp cached pages.

2020-07-16

• Fixed selector for answer box dynamic result.

2020-07-13

• Update validator.rb
• Fix captcha detection when following redirect

2020-07-10

• Extract block_position for shopping_results.

2020-07-08

• Fix for knowledge graph top snippet extraction.
• Revert "Revert "Updated knowledge graph""
• Revert "Updated knowledge graph"
• Adjusted parsing to consider i18n in responses.

2020-07-07

• Added amp to organic_results. Added amp to carousel in organic_result.
• Added top_snippet extraction for knowledge_graph. Added revenue extraction as a detail_node from top block.

2020-07-03

• Updated knowledge_graph see_more_about extraction. Added test for see_more_about section.

2020-06-29

• Updated documentation for google jobs.

2020-06-26

• Added image extraction. Fixed documentation image path.

2020-06-25

• Added google jobs listing api.
• Revert "Fix parser"
• Add service_options parser to Google kg
• Fix parser

2020-06-23

• Remove `disable_polymer` from URL

2020-06-22

• Fix parser

2020-06-19

• Update app/models/search/google/parser/knowledge_graph.rb

2020-06-17

• Fix parser
• Update parsers and rename to `amenities` `service_options`

2020-06-16

• Update parsers

2020-06-11

• Update parser

2020-06-09

• Handle other empty product case.

2020-06-08

• Removed commented unnecessary code.
• Use direct proxy for home depot. Fixed price extraction for home depot product.
• Fix and normalize authentification via API key

2020-06-04

• Update app/models/search/google/parser/js_image_extractor.rb
• Update app/models/search/google/parser/js_image_extractor.rb

2020-06-03

• Update js_image_extractor.rb
• Update sports_results.rb
• Update js_image_extractor.rb
• Fix parser

2020-06-02

• Split Walmart tests into required and opional

2020-05-29

• Added Home Depot product scraping.

2020-05-28

• Added autocomplete changes. Added browser mimicking for playground.

2020-05-22

• Exclude Walmart filters and include optionally to the response

2020-05-15

• Update product_result parser

2020-05-14

• Update selectors
• Update events_results.rb
• Update product_result.rb
• Update refine_by.rb
• Update news_results.rb
• Add amenities to tbm=lcl results
• Update local_results.rb
• Update knowledge_graph.rb

2020-05-13

• Update inline_shopping_results.rb
• Update inline_shopping_results.rb
• Update extract_js_image_by method
• Update images_results.rb
• Update direct_answer_box.rb
• Update Bing local results parser
• Update events_results.rb
• Update Google Events results parser

2020-05-12

• Added toolbar and youtube clients.
• Fix YouTube spec
• Document Walmart Product
• Document Walmart Search API

2020-05-11

• Add soft_sort and grid parameters
• Update parser
• Update parser

2020-05-08

• Add parser

2020-05-06

• Added prettifier for autocomplete result. Changed autocomplete to use netnut. Updated doc. Added test for multiple request ends up as 403 forbidden.
• Handle the case when params is nil
• Fix regular searches being misclassify as Google Image searches as `params["ijn"]` is always an array`
• compact to compact!
• Fix getting dupe :text and :carousel top_stories

2020-05-01

• Add Walmart search parameters to playground
• Add 'grid' parameter to Walmart search
• Test Walmart Product reviews
• Test Walmart pagination

2020-04-30

• Fixed emapty results message for HomeDepot.
• Changed rating_count to reviews. Changed HomeDepot to The Home Depot in documentation.
• Test Walmart product_result
• Fix desktop top_stories selectors
• Fix Google Product parsing error on empty non-English results
• Update parser

2020-04-29

• Fixed typo. Changed tests for node_extractor.
• Removed redundant price and rating fields from response for HomeDepot. Updated documentation.
• Added missing icon for autocomplete. Changed client attribute default to chrome for Search model.
• Fix phone parser
• Update app/models/search/google/parser/local_results.rb
• Update parser

2020-04-28

• Changed raw price to have dot delimiter. Changed rating to display number of stars. Added rating_raw to display rating percent. Updated documentation.
• Updated documentation.
• Added psy-ab as a client option for autocomplete.
• Test Walmart Product product results
• Revert "Fix parser"
• Fix lsig extraction
• Update gps extraction
• Fix parser
• Revert "Update parser"
• Update parser

2020-04-27

• Fixed thumnail extraction for HomeDepot.
• Move old price and price save into price group in response.
• Changed price and rating to numeric values. Changed price response to have raw, extracted and currency values. Changed tests.
• Fixed price extraction. Added link extraction. Added product_id extraction. Added badge extraction. Fixed documentation link to sorting/ordering.
• Add hl support to Google Maps

2020-04-24

• Add google autocomplete. Add autocomplete documentation.
• Extract Walmart spelling fix
• Update organic_results.rb
• Revert "Update organic_results.rb"
• Update organic_results.rb
• Update parser
• Update parser
• Pass all Walmart parameters to url
• Add available_on parser
• Add Walmart search pagination

2020-04-21

• Test Walmart filters and related queries

2020-04-16

• Integration test for Walmart organic results and Walmart Product result

2020-04-15

• Add walmart_product engine
• Escape chips for google_jobs.

2020-04-14

• Update thumbnail node
• Update inline_videos.rb
• Update top_stories source

2020-04-13

• Fix thumbnail extraction. Removed device parameter.
• Added pagination extraction for HomeDepot.
• Update inline_videos.rb
• Update inline_videos.rb
• Update news results
• Update parser
• Update parser

2020-04-10

• Extract related queries and filters from Walmart search
• Extract featured item on Walmart
• Stop showing event results when top_carousel is present
• Update parser

2020-04-09

• Added highlighting for HomeDepot in playground dropdown menu. Used paragraphs for documentation_examples. Used get_valid_link and get_thumbnail.
• Update parser

2020-04-08

• Update top carousel

2020-04-07

• Updated documentation. Changed images to jpg in order to reduce size.
• Added spelling check for HomeDepot.
• Parse Walmart organic results

2020-04-03

• Added filtering for HomeDepot. Added documentation for filtering, price bounds, sorting.
• Safe navigation for array and and if it's a string
• Update Google Jobs validator
• Fix phone extraction for Google local pack
• Update yahoo organic results tests
• Remove unused code from Google parser
• Update top_stories parser and tests
• Fix boston red sox sports results
• Update Google product result parser and tests
• Update popular products parser
• Fix specific address local pack parser
• Fix Google local pack gps coordinates

2020-04-02

• Implement changes
• Update app/models/search/google/parser.rb
• Update node_extractor.rb
• Update Bing local_results parser

2020-04-01

• Added HomeDepot scraping.
• Update app/models/search/google/parser.rb
• Update app/models/search/google/parser.rb
• Update app/models/search/google/parser.rb
• Update app/models/search/google/parser/local_results.rb
• Update parser

2020-03-31

• Fix places results gps for some searches
• Exclude related searches widget from images_results

2020-03-30

• Add parser

2020-03-28

• Fix lsig extraction

2020-03-27

• Fix parser

2020-03-26

• Update rich_snippet parser
• Fix and update popular_times parser
• Add question mark at the end of method name to indicate boolean

2020-03-25

• Added new selector for top stories title.
• Update parsers
• Restructure parser logic

2020-03-24

• Fix displayed_brand and displayed_link
• Fix people_also_search_for parser
• Fix phone parser across apis
• Add constraints on thumbnail extraction as an option.
• Fix parameter encoding
• Rename hashes

2020-03-23

• Update spec
• Fix thumbnail extraction from inline top stories

2020-03-20

• Clean up the checks for empty organic results
• Look for preceding g-tray-header to define group name.
• Add validator
• Remove json parser
• Sepparate main `organic` results parser to its own file
• Update documentation
• Changed tests for mobile top stories.
• Remove unnecessary whitespace

2020-03-19

• Fix inline top stories for mobile.
• Update parser
• Revert "Fix Google product pagination"
• Fix Google Product star rating extraction with no reviews

2020-03-18

• Implement changes
• Update parser to support live videos
• Add hl parameter
• Finish parser

2020-03-17

• Add request retrying to Baidu for empty results

2020-03-14

• Update selectors

2020-03-13

• Fix selector
• Unit-test get_thumbnail
• Fix google_reverse_image_spec
• Fix design_alt_spec
• Fix local_results_for_products_spec

2020-03-12

• Use get_thumbnail everywhere
• Normalize Google Scholar Cite
• Normalize Google Product

2020-03-11

• Update Google Product "Product not found" selector
• Normalize Google Jobs
• Normalize Google Events
• Normalize Google
• Fix Yandex infinite loop on empty results response
• Normalize Yandex
• Normalize Yahoo!
• Normalize Bing
• Normalize Baidu

2020-03-10

• Fixed missing inline videos for mobile. Added carousel parsing for organic results for mobile. Fixed related searches for mobile.
• Fix spec/search/google/images_results/images_results_spec.rb
• Fix spec/search/yandex/yandex_organic_results_coffee_spec.rb
• Add parser for old youtube layout

2020-03-09

• Extract JS image for Google Local results thumbnails
• Fix events_results_mobile_spec tests: added selector for table layout of events
• #611: Get additional inline_top_stories from js.

2020-03-06

• Extract query_displayed from script.
• Added comment on query_displayed for spelling fix result.
• Added spelling fix for yandex search.
• Convert location_requested into location for and api request.

2020-03-05

• Fix token authentication for Guests
• Add new saved HTMLs

2020-03-04

• Fixed misspelling for multiword query.
• Fix Baidu captcha not getting detected
• Check for num parameter value and set to 21 if value is 20
• Fix 302 loop
• Disable youtube polymer

2020-03-03

• Added search_information for Yandex engine.
• Add device and location to the cypher key to make it more unique (search md5 uses user_id which is not know in token auth)
• Update parser
• Implement the changes
• Remove device from query
• Fix Ebay encoding

2020-03-02

• Fix api pagination bug with escaping location.
• Remove debugging statements
• Convert to binary encoding to store more hex chars in the same space
• Fix Yandex encoding

2020-02-28

• Fixed yandex loop issues.
• Add more parsers
• Add support for more parameters

2020-02-27

• Add video_results parser
• Add initial parser support

2020-02-26

• Fixed selectors for total_results and time_taken_displayed.
• Update sports results parser
• Update Google results_state parser

2020-02-25

• Fix spelling_fix selector.
• Fix Bing local pack gps extraction
• Fix Bing organic results thumbnail extraction
• Add and update js_image_extractor for Bing
• Revert "Revert "Fix spelling result for image searches.""
• Update Google product API results parser

2020-02-24

• Update Google product result parser
• Update google flights parser
• Update images_results.rb
• Fix Google direct answer box parsing against new design

2020-02-21

• Revert "Fix spelling result for image searches."

2020-02-20

• Fix and update Google lcl parser
• Fix and update Google local pack mobile parser
• Fixe spelling result for image searches.
• Make parameters escaping explicit.

2020-02-19

• Update sports results parser

2020-02-18

• #554: Fixed uule for reverse image search.
• #554: Fixed uule escaping for google jobs and events. Added location to events playground. Added location and uule to jobs playground. Added tests to ensure uule is encoded correctly.
• Transformed inline images tests. Fixed case for js image extraction for non-base64 images.

2020-02-14

• Added new selector for google total results.
• #198: Removed present_with_type helper. Moved yandex languages into a separate page Changed validator to compare against html elements.

2020-02-13

• Fixed yahoo local pack reviews for mobile.
• Fixed reverse images devito spec. Transformed reverse images witcher spec and made non-strict.
• Requested fixes and changes
• Changes and fixes
• Transformed yahoo organic tests.

2020-02-12

• Update parsers
• Return key as "unknown" if value empty

2020-02-11

• #198: Addressed PR comments.
• Make sure key is never blank to ensure JSON.load work in every languages
• Update parsers
• Add TextUtils.snakecase method
• Update ebay tests
• Update Google events

2020-02-10

• Remove comments.

2020-02-07

• #198: Addressed pr comments.
• Fix empty hashes by replacing parameterize method
• Update parser
• Update parser

2020-02-06

• Implement changes
• Transform test for samsung tv `tbm=shop` into # AUTO_ADD_SAVED_HTMLS
• Revert "Fix typo"
• Add documentation
• Update validator
• Restore old selectors for google jobs backwards compability.
• Update validator
• Add support for events_results_state
• Fixed google jobs parsing after updated jobs page design.

2020-02-05

• #198: removed unused code.
• #198: Added yandex search api. Added tests for yandex search parser organic search and knowledge graph. Added parser for organic search and knowledge graph. Added language and page parameters. Added playground. Added domains and languages. Added parser for inline images and inline videos. Added documentation. Added tests for inline images and inline videos. Fixed bug in organic results parsing. Fixed bug in inline images parsing. Made correction to documentation.
• Implement changes
• Extract anchor tags instead of list items and remove if link = item.at_css('a') check
• Use serpapi.com for all urls except development
• Add parser
• Remove location support from Google events
• Move extract_trending_searches_by_node
• Rename rank to position
• Fix typo
• Update extract_js_image method
• Add initial support for Google events engine API

2020-02-04

• Update app/models/search/ebay/parser/organic_results.rb
• Remove safe navigation as nil key for pagination should break anyway
• Implement the TrendingSearches parser
• Fix link and title parsing when tbs=shop
• Remove unnecessary check for existence of anchor tag in refine_by_item
• Add parser
• Fix 'serpapi_link' when in 'suggested_searches' of Google Images results when chips query is separated by spaces or '+' signs

2020-02-03

• Only use `serpapi.com` for production URLs
• Revert the location_requested backup
• Add parser

2020-01-31

• Move SearchSplitter Bing cookie logic to Validator
• Update links extraction to use get_valid_link method
• Update Ebay search API
• Add Google shopping results filters parser

2020-01-30

• Update Google product result mobile parser

2020-01-29

• Update parameters.rb
• Add price parameters
• Fix links in Google product results

2020-01-28

• Support new layout format that uses `span` instead of `cite` for inline top stories
• Update Ebay engine with proper parameter names
• Update parser.rb
• Update parser

2020-01-27

• Check if the result is sponsored
• Add raw and extracted price
• Fix related_searches parser when empty results
• Add pagination
• #160: Renamed organic_results to image_results for reverse image search.

2020-01-24

• #160: Prevent knowledge_graph image to appear null.
• #160: Removed unused headers from utils return value. Changed error message.
• Fix Google flight parser issue
• Fix typo
• Revert the check for "$$$"
• Update app/models/search/google_maps/parser/maps_search_results.rb
• Update parsers
• Add spelling fix

2020-01-23

• Addressed PR comments.
• Add comments
• Fix price parsing on restaurants and hotels
• Update organic_results to support results from foreign domains
• Update validator
• Update validator
• Add new parameters (pgn, ipg, ebay_domain)
• Add related_searches parser
• Add related_items parser
• Update get_valid_link method

2020-01-22

• #160: Fixed knowledge graph image.
• #160: Fixed knowledge graph image.
• Add results_state parser
• Add organic_results parser
• Add initial support for ebay engine

2020-01-21

• #160: Added playground. #160: Fixed storing reverse image result url.
• Prioritize `data-iurl` actual image urls over `data:image` ones
• Fix pagination
• Update parameters.rb

2020-01-20

• Update parameters.rb
• Return empty error if no results for a query
• Update validator

2020-01-17

• Fix rating extraction for Yahoo searches
• Add playground support for google scholar cite engine
• Add parser
• Add Google scholar cite engine support
• Generate google scholar cite link

2020-01-16

• #160: Added parameters, parser and validator. #160: Added extra request to get link to images. #160: Added specs. #160: Reuse google parser modules.
• Update hash names

2020-01-15

• Update menu_pages.rb
• Update knowledge_graph.rb
• Update top_stories parser
• Fix news parser
• Extrack `stick` parameter
• Update parser
• Remove top_carousel endpoint
• Add aditional info to Google's KG popular_times
• Update validator

2020-01-14

• Fix google_product validator
• Update regex

2020-01-13

• Enforce indentation cop on all .rb files

2020-01-10

• Fix product_results test by making rating a float number
• Fix rating extraction in google product results
• Add comment about music block in Baidu organic results

2020-01-09

• Exclude music block from organic search
• Fix Baidu parser
• #431: Replaced URI with Addressable. Removed unused code.
• #237: Changed default query form coffee to barista for jobs engine in playgroud. Fixed typo in query_ordered.
• #237: Added defensive guard clauses.
• #237: Addressed comments for documentation and tests.
• Update local_pack parser
• Rename PATTERNS to IMAGE_URL_PREFIXES

2020-01-08

• Disable all Rubocop cops except indentation-related ones
• Fix video_results extraction
• Fix inline_images parsing from Google SERPs
• Fix google_flight parsing
• Check source attribute prioritization of get_valid_link
• Revert changes in Bing local parser

2019-12-30

• Undo unescaped link error changes to validator.
• Fix unescaped urls raising InvalidURIError.
• Add missing saved HTMLs for sports results
• Improve get_valid_link
• Check if place is verified
• Extract gps on desktop
• Don't show `top_carousel` if `stick` parameter is used
• #237: Added parser, validator, parameters modules for jobs engine. #237: Scrape basic jobs data. #237: Added tests for jobs scraping. #237: Added chips scraping. #237: Added playground page for jobs engine. #237: Added documentation page for jobs. #237: Changed tests to use automatic test file generation.

2019-12-27

• Fix ruby style
• Don't use to_sym in Bing local results
• Update parser
• Add yahoo local pack parser
• Add device support to Yahoo

2019-12-26

• Update ads_results.rb
• Update ads_results.rb
• Update organic results for mobile support
• Update related_searches.rb
• Add device support to Yahoo
• Add tests
• Add rich_snippet parser to Yahoo organic results
• Update parser.rb

2019-12-25

• Fix inline_shopping_results_spec
• Update parsers
• Fix possible top stories exceptions
• Add yahoo related_searches parser
• Fix Google's top_stories Airbrake error
• Add Yahoo related_questions parser
• Add device support to Yahoo
• Add get_valid_link method to yahoo
• Escape url before parsing.

2019-12-24

• get_valid_link in all engines
• Update knowledge_graph.rb
• Update top_carousel parser
• Fix people_also_search_for carousel
• Add top_carousel parser
• Add profiles parser

2019-12-23

• Fix boston_red_sox_spec
• Fix shipping_results spec again
• Fix lionel messi stats spec
• Fix local_pack_for_specific_address_desktop_spec
• Fix tests in refiny_by spec
• Fix tests in refine_by

2019-12-20

• Fix Bing local pack
• #250: Fixed doc example.
• #250: Handled ads links.
• Fix thumbnail and source extraction in inline top stories
• Update sports_results.rb

2019-12-18

• #250: Added extensions. Fixed incorrect price extraction.

2019-12-17

• Re-create .rubocop_todo.yml

2019-12-16

• #250: extract_numbers -> extract_number.
• Move top_stories_inline to mobile only method

2019-12-13

• #250: fixed mobile reviews parsing.
• Add more classes for mobile
• Update top_stories parser
• Added guard clause.
• Added refine_by key fallback. Added test for Japanese search.

2019-12-12

• Add hotel pricing parser
• #260: Fixed mobile parsing.
• Fix top stories links
• Fix popular_times parser
• Revert "Revert "260 add refine by""

2019-12-11

• Revert "260 add refine by"
• Revert "Google kg popular_times results"

2019-12-10

• #250: Added mobile inline shopping results.
• #260: Convert refine by text to symbol.
• Update Google shopping_results

2019-12-09

• Add popular_times parser
• Update sports_results.rb
• Fix and improve Baidu answer box results
• Fix Google sports parser

2019-12-06

• Add "missing" and "must_include" annotations parser
• Rename review_summary to user_reviews
• Add critic_reviews and reviews_summary to Google's kg
• Update Google's get_valid_link method
• Show original description and saved HTML file path in RSpec output
• Add another selector for related_questions

2019-12-04

• #260: Addressed review comments.
• Update local_results.rb
• Fix parser, add more options to extract gps

2019-12-03

• Update bing local pack
• revert changes made to clone_with_new_parameters method
• Update Bing local pack parser

2019-12-02

• #260: Added 'Refine by' extraction.
• Add Bing local pack parser
• Update error message
• Update bing validator
• Update some ad results to include title

2019-11-30

• Fix #367

2019-11-29

• #101: Added inline images for google organic search.
• Revert "Revert "Bing related searches and inline ads""

2019-11-27

• Revert "Bing related searches and inline ads"
• Lint Ruby code on pre-commit hook
• Remove debug.
• Use extract_integer.
• Added total_results for bing search.
• Improve top and bottom ads results
• Improve Bing parsers
• Add support for inline ads
• Add related searches parser to Bing
• Made fixes after code review.
• Be more defensive when extracting video duration
• Add "unsupported device" error message
• Fix typo

2019-11-26

• Add support for mobile sports parser
• Parse favicon in Yahoo! ads results
• Extract thumbnail urls for organic search from js.
• Fix maps API parser
• Move Yahoo! parsers to the separate modules

2019-11-25

• Parse Yahoo! search results on different locales
• Test Yahoo! on different languages
• Small fix
• yahoo_domain for Yahoo!
• Knowledge graph fields added: - See photos - See outside
• Update and improve sports parser

2019-11-22

• Update parser
• Remove 'n' parameter from Yahoo! search API
• Yahoo! search parameters on playground
• Add documentation
• Update parser and add support for baseball

2019-11-21

• Add parser for formula and moto gp
• Add parser for tennis athletes
• Handle Yahoo! web search parameters
• Add parser for basketball and american football athletes
• Improve `get_valid_link` method

2019-11-20

• Update team sport and add individual athlete (soccer) parser
• Yahoo! pagination
• Update results_state_and_spelling_fix.rb
• Improve parser error handling
• Re-use `dig`
• Update maps validator and tests
• Yahoo! query_displayed and total_results
• Button in ads_results for Yahoo! Search API

2019-11-19

• Yahoo! ads results with sitelinks
• WIP sports_results parser
• Yahoo! ads results with list sitelinks
• Replace blank? with present?
• Re-using `dig` in maps parser
• Add `google_domain` support to Maps
• Test Yahoo! agains real queries

2019-11-18

• Parse Yahoo! organic results
• Improve Maps parser
• Make sure place and search results are not loaded at the same time

2019-11-17

• Fix for when html shows wrong type of results
• Separate results state and spelling fix to different file

2019-11-15

• Update and improve `Maps` parser
• Add validator test
• Add `position` to some place results and rename hash

2019-11-12

• Extract total_results on Baidu
• Improve and update parser
• Revert "Improve and update parser"
• Add ReCaptcha validator
• Improve and update parser
• Fix Google Maps URL constructor
• Update inline products screenshots in documentation
• Extract thumbnails in compact inline products

2019-11-11

• Extract compact inline products
• Extract regular inline products
• Move inline_products extraction to own module
• Extract extended inline_products
• Add support for countries cities and provinces
• Add support for hotel ads

2019-11-08

• Improve `Maps API` parser
• Rename state from `place` to `local`
• Update public JSON sample

2019-11-07

• Add pagination.previous and fix pagination.next on Baidu

2019-11-06

• Update `Maps API` parser
• Fix Google video results parsing on mobile devices
• Fix tests
• Clarify code comment for extract_js_image_by
• Unit test performance-optimized JSUtils.unescape
• Rename `place_results` hash to `local_results (tbm=lcl)

2019-11-05

• WIP update Google Scholar validator
• Decode HTML entities with regular expressions instead of loops
• Add support to Google Scholar API for `empty results state`

2019-11-04

• WIP Add Google Maps API parser
• Add support for `type` parameter
• Test saved HTML with different designs of Google Images results
• Extract original image URL from JS
• Remove unused code

2019-11-03

• Fix and improve places_results parser

2019-11-01

• WIP Initial files for `Google Maps API`
• Parse Google Images from both HTML and JS

2019-10-31

• Start images extraction
• Change spelling in the documentation

2019-10-30

• Add top_stories_link

2019-10-29

• Fix tests
• Move top_stories extraction to own file
• Parse top_stories defensively
• Support list format of top stories
• Support carousel format of top stories
• Fix mobile ads parsing without description

2019-10-28

• Add support for results state to all Google API endpoints
• Improve places reuslts parser

2019-10-25

• Exclude 'rich_snippet.bottom' for organic results with list format for sitelinks
• Parse list format for sitelinks
• Update bing related questions parser
• Add related questions parser
• Document Direct Answer Box snippet selector usage in organic_results
• Code improvements for places results parser

2019-10-24

• Fix parsing of hotel bookings
• Exclude incomplete answer_box duplicates in organic_results
• Use new gps_coordinates method

2019-10-23

• Improve reliability of geting a place type
• Enable `organic_results_state` for `tbm` results
• Add places results parser
• Rename file to `places_results` and move under proper conditional
• Use both of new and old answer box CSS selectors

2019-10-22

• Test Bing pagination with high number of results
• Handle pagination without 'other_pages'
• Fix GPS coordinates extraction in local results
• Fix 'knowledge_graph.people_also_search_for' parsing
• Update selectors to parse dictionary answer box of Google results
• WIP tbm local results parser

2019-10-21

• Move pagination-specific methods to correct modules

2019-10-19

• Fix for product results
• Fix for local pack results

2019-10-18

• Add support for editorial_reviews to knowledge_graph and update description
• Test pagination in Google Scholar results
• Small fix

2019-10-17

• Baidu integration test
• Update `local_pack` parser
• Bing pagination
• Edit `get_valid_link` method to support `data:image` links
• Update product_results parser

2019-10-16

• Add serpapi_pagination to Baidu
• Unify pagination in Google Scholar
• Unify pagination and serpapi_pagination in google's organic results
• Improve gps and valid link methods

2019-10-14

• WIP local_pack parser
• Update `get_valid_link` method
• Extract pagination to separate method
• Don't set organic_results_state when no results are found
• Rework `extract_gps_coordinates` to enable `js` extraction
• Additional case for 'Empty showing fixed spelling results' state
• Code review fixes
• Add the `get_valid_link` method
• Be sure title is returned for `product_result`

2019-10-11

• Remove TODO
• Remove method for setting organic results state
• Extract spelling fix type
• Guard agains nonexisting nodes in inline videos

2019-10-10

• Fix Airbrake errors for `product_result`
• Fix bed links and comment out `lsig` parameter
• Fix NoMethodError for inline video title parsing

2019-10-09

• Move set_organic_results_state to parser
• Test empty results in Arabic and German locale
• Fix for when labels have space in them
• Fix syntax issue
• WIP - local map for specific address
• Add support for the new `lsig` parameter
• Fix some syntax issue
• Handle inline videos without 'channel' in HTML
• Tests for quotes spell fix

2019-10-08

• Improve method arguments naming
• Empty search result states implementation
• WIP - Add support for `Mobile v2` results
• Rename to `product_result` and other small improvements

2019-10-07

• Google product results parser

2019-10-04

• Test inline videos in three different languages
• Support results_for_exact_spelling and fully_empty states

2019-10-03

• Describe challenges of parsing :video_thumbnail

2019-10-02

• Document inline videos in organic results
• Base64-encode thumbnails
• Convert indentation to spaces
• Fix Google and Google Scholar parser

2019-10-01

• Improve organic results parser
• Support inline video results

2019-09-30

• Remove Google Search specific validators from Google Scholar
• Improve organic results
• Fix syntax
• Show error if Google hasn't returned any results

2019-09-25

• Remove parameters that don't work with Google Scholar
• Add support for Google Scholar `cites` and `cluster` parameters

2019-09-24

• WIP Google Scholar organic results
• WIP Google Scholar parser

2019-09-21

• Add support for stats to Google Scholar results
• Reduce some whitespace
• Add support for related searches to Google Scholar results

2019-09-20

• Add support for Google Scholar specific parameters

2019-09-17

• WIP Add basic skeleton to support Google Scholar API

2019-09-10

• WIP Support Bing organic search in mobile

2019-09-04

• Fix #234
• Fix #228

2019-09-03

• Fix issue with new Google design

2019-08-26

• Fix Baidu issues

2019-08-22

• Fix issue with rich snippet questions not being supported and messing with direct answer

2019-08-20

• Fix issue with related searches including link with blank hrefs
• Fix data extraction for Google Shopping API
• Fix product headers extraction
• Fix product reviews, ratings, and number of reviews extraction
• Don't transform a Float product rating into an Integer
• Fix extensions not being correctly extracted
• Snakecase keys to avoid JSON parsing issue for Google Product API
• Fix issue with extracting related searches on mobile
• Fix issue on mobile with organic results and ad results getting mixed up
• Extract more rich data form organic results extensions
• Stop removing direct answer from document

2019-08-19

• Add Google News API support for people also search for blocks
• Fix issue with knowledge graph when key already exists
• Add `raw_hours` key for people relying on a string instead of an object
• Fix rating and review extraction for knowledge graph API
• Fix position issue with Google Images Results API
• Add support for extra layout variation

2019-08-15

• Fix Bing search ads
• Fix issues with stopping recreating hash dynamically

2019-08-13

• Fix Google Images pagination end infinite loop

2019-08-11

• Fix issue with knowledge graph
• Fix issues with direct answer box and regular hash
• Use regular hash instead of recursive created hash

2019-08-09

• Fix price, extension, and snippet for shopping results #210
• Actually avoid parsing parent knowledge graph section
• Ensure `reviews_from_the_web` has results
• Don't parse results from parent knowledge graph section for `see_more_about1`

2019-08-08

• Only create hash for `see_more_about` if it exist
• Avoid setting empty reviews
• Add support for `See more about` section

2019-08-07

• Fix issue with valid searches about Captcha being blocked

2019-08-06

• Don't include missing hours in local results #93

2019-08-05

• Update rating to float
• Extract number from `review_count`
• Add support for businesses' weekly schedule and reviews from the web

2019-08-02

• Add support for custom location by latitude and longitude input
• Fix naming of hours, address and description in local pack results
• Fix missing ratings in local results box #93
• Fix links and title links for Binb organic results
• WIP Rewrite final_hash and spec logic

2019-07-30

• Display JSON and HTML endpoints for Baidu search
• Allow guest to reach to their JSON and HTML endpoints via tokens

2019-07-26

• Add Google Spell Check API
• Fix commits and PR to streamline tokens

2019-07-25

• Fix typo
• Avoid parsing answer divs when they are not there
• WIP Use shorter tokens that look more user friendly

2019-07-23

• Add `location_used` to search json when using Bing #183
• Fix thumbnail and images

2019-07-19

• Fix missing parameters error with validators
• Add support for setting of precise location #181
• Early preparation, add cookie setting

2019-07-18

• Fix mobile title for desktop news
• Fix mobile search

2019-07-16

• Fix Google News API

2019-07-14

• Fix extraction of ratings from reviews

2019-07-09

• Remove conditional pagination (trouble on mobile)

2019-07-05

• Add product id to shopping results

2019-07-02

• Update knowledge graph to support parsing of business' operational status #116

2019-06-27

• Fix issue when knowledge graph answer type is missing
• Remove raw_submodules from final hash
• chore(app/*) whitespace changes and newlines at EOF

2019-06-23

• merge organic_results and video_results for desktop fix video_results parser for mobile

2019-06-21

• Fix temporary live total results empty error
• Handle edge cases of Google pagination

2019-06-19

• Fix Base64 Errors

2019-06-17

• update pagination parameters with Search#clone_with_new_parameters
• remove `then`'s from Bing knowledge graph results

2019-06-16

• convert bing knowledge graph into a hash vs. list of hashes
• add `formatted` field to parseable bing submodules
• Resolve conflicts
• Actually fix bugs

2019-06-15

• Local map bug fixes

2019-06-13

• remove unique top-level type parsers & fallback to generics
• remove unused include for bing ads results

2019-06-11

• Beta operational Baidu parsing

2019-06-10

• Fix emtpy hash being evuated as true issue #167
• Add new serpapi search links for `place_id` #167
• Rename location_id to place_id to match Google own documentation #167
• Use `api_key` instead of `user_id` to encrypt in order to allow re-generation
• Use only "Success" status instead of both "Cached" and "Success" for successful searches
• Use tokens to authentifcate users tied to one specific request
• stub out unimplemented Bing KG submodule type
• filter out JavaScript links in bing KG link parser
• add bing 'people_also_search' submodule formatter
• add written_works, timeline, and interesting_stories bing formatters
• add generic bing header submodule format to food & famous people
• use Ruby's splat operator to get head & tail of Bing submodules
• add generic format logic for Bing's knowledge graph header submodule
• add 'Organization SocialActive' to Bing Organization types

2019-06-07

• Extract `ludocid` (location_id) from local pack

2019-06-06

• Add local map support and GPS coordinates for knowledge graph results
• Support for dictionary answer box
• Add parsing support for "Related news" and "rich snippet" results
• Update parser with dynamic module loading based on required element
• [squash] start on organization-specific Bing KG fields
• add Bing 'flatmodule' knowledge graph parsing
• add Bing 'slide' submodule parser

2019-06-05

• parse multiple Bing knowledge graph lists when they exist
• replace Bing 'answers' with 'knowledge_graph'

2019-06-03

• add framework that captures typical Bing answer types

2019-06-02

• refactor bing ads pull helper fns from organic results module
• move bing organic results into their own parser module
• move bing ads results into their own parser module

2019-05-31

• Fail-safes for missing elements in local pack mobile
• Add pagination, source icon for Baidu

2019-05-29

• Add parsing of address on specific style for local pack mobile
• Extract phone number from local pack mobile
• Add support for local packs with mobile device #109
• Initial commit

2019-05-26

• Fix media parsing in `product_results`

2019-05-25

• Add fail safes for when elements don't exist or output is nil

2019-05-24

• add title field to Bing ad results
• fixup tracking_link (vs. link) field names for Bing ad sitelinks

2019-05-23

• Ensure values aren't null, actually add parsing of related_search_boxes
• Remove `log` in validator
• add basic serpapi_pagination logic
• Modify validation of successul request in product page
• Finalize renaming `product` paramter to `product_id`
• Strip quotes at start and end of review summary
• Avoid parsing unrelated sections for `related_search_boxes`

2019-05-22

• Rename`related_search_box` to `related_search_boxes`
• Add documentation page, navbar menu item, and icon for related searches
• Routine cleanup

2019-05-21

• Update `specs_results` for alternate product page
• Update `related_products_results` for alternate product page
• Fix removal of text for expand description

2019-05-20

• Update `sellers_results` to work alongside alternate product page
• Update `reviews_results` to work alongside alternate product page
• Update `product_results` to work alongside alternate product page
• Use parameters `specs` and `reviews` as booleans
• Refactor get_related_search_box

2019-05-19

• Add additional selectors for parsing results on alternate product page #105
• Check if `title` tag contains `Product not found` text
• Checking if the url contains `/shopping/product` isn't really necessary
• Use `if` `end` block if two or more conditions are used

2019-05-18

• Initial write-up for related search box #129
• Bug fixes and adjustments

2019-05-17

• add Bing-specific query parameters

2019-05-15

• Support knowledge graph type extraction for business listing

2019-05-14

• replace the link with a tracking_link url
• Use a more precise selector for knowledge graph type extractor
• Fix knowledge graph type support
• Changes per code review

2019-05-13

• add bing ad parsing logic

2019-05-12

• add in `form=QBRE` to trigger Bing ad results
• move bing organic parsing into it's own fn
• only return valid Bing organic results

2019-05-10

• Support float and integer number extraction for all locales
• Massive overhaul
• add basic Bing organic result parsing logic

2019-05-03

• Rename store/seller to name in sellers results, more adjustments to rating/reviews parsing in reviews_results

2019-05-02

• Fix stars and total reviews regex for international languages (hopefully), begin documentation for subparts of product API

2019-05-01

• Fix google_domain setting for engines "google" and "google_product", pagination fix for sellers and revies

2019-04-30

• Fix parsing of Google calculator direct answer
• Add local stores parsing, rating and reviews to integer, fix param name on Google Prod API documentation,

2019-04-28

• Add some documentation for global api, reviews results. Adjust sellers results for online and local, refine filtering/sorting inline with documentation,
• Fix reviews, adjust price extraction for multilingual
• Add filtering, sorting, pagination, full results for reviews and specs/details

2019-04-26

• Use `unless` in replace of `if` blocks
• Add feature to iterate through shopping results and fetch product page results (Not completed)
• Restructuring
• Naming, ensure elements exist before creating hash and iteration
• There are ads but needs seperate parsing
• Initial Google Product parsing integration

2019-04-24

• Support ludocid parameter (Google My Business Listing CID)
• Structuring adjustments
• Migrating to respective branch for development

2019-04-23

• Add parsing for specification details section
• Add parsing for additional review data
• Fix skeleton for other engine support
• Initial product page parsing

2019-04-18

• Fix conflicting html attribute
• Seperate search domain and search engine parameters

2019-04-05

• Support basic Bing search WIP