Best Bugs Yet

Leave a reply

This week ended up representing the cumulation of a years worth of experience at my current employer and it happened through some very special bugs that had been haunting us for months.

1. Ajax Timing

First of all, our UI automated tests have been failing due to ajax and javascript timing issues.  Intermittently and actually quite rarely, say 1-5%.  That’s a problem though because, including branches, we frequently do several hundred CI runs a day, so that means sometimes dozens of runs fail.  This leads to the dreaded lack of trust in test runs which has very negative consequences as ultimately you no longer can trust them to reflect if the product is actually working.

The mystery was why.  The page loads up in 2-3 seconds on our macs and we had a 15 second max wait time which seemed ample.

The clue finally came when I ran the UI tests locally at home on my Ubuntu machine, which ironically is the same OS as the CI machines and also with lower power than my mac.  I ran the specs and they brought up the browser locally, but instead of 2-3 seconds for the page and all the js to load it took 30-40 seconds!

Finally I had the cause of our issues and was able to adjust the 15 in this piece of code for implicit waits to 60 to allow the page to fully load.  Boom.  5 months of a mysterious ajax loading issue finally nailed!

                                         -->  <-- 15 changed to 60
def wait_for(condition_name, max_wait_time: 15, polling_interval: 0.01)
  wait_until = Time.now + max_wait_time.seconds
  while true
    return if yield
    if Time.now > wait_until
      raise "Condition not met: #{condition_name}"
    else
      sleep(polling_interval)
    end 
  end 
end

2. Slow Automated Mobile Tests

Finally tests are passing for mobile devices through the browserstack automated service.  But they are soooo slow.
One of the most simple’fixes’ for this was to simply reduce the tests being run to reflect those workflows actually used on mobile devices.  Two of our workflows are only on desktop.  This reduced the mobile test specs from 24 to 9 and the run time from 42 mins to 18 mins

3.  Passing, Failing, Passing and stopping randomly

The mobile tests have been behaving very erratically in other ways.  At various different points it seems like the device suddenly stops working or responding.  Then we would get 3 test suites in a row passing.  With that good sign we ran a bunch more.  But most failed.  Except the last one.  After a long day of runs, pattern finally spotted.  The different test runs on circleCI (we can have 3 running simultaneously) have 4 slots (machines) with each run).  They create **and destroy** a tunnel to virtual devices at Browserstack, so different runs are killing each others connections.  Not obvious until running multiple (or the same) branch at the same time as other CI runs.  Proved by running one run at a time.  Fix is to start/stop the tunnel once.

command line playtime, grep, xargs, cat and head

Leave a reply

Search for this in any of files with filename *.rb

  grep 'this' *.rb

Search for this in files and cat them

  grep -l 'this' b*.rb | xargs cat

Search for this in files and then show their contents with filename on the first line:

  grep -l 'this' b*.rb | xargs head -999

Search for this AND that:

  grep -l 'this' b*.rb | xargs grep --color 'that'

Search for (this OR that) AND other:

  grep -E '(this | that).*other' b*.rb

Rspec Style

Leave a reply

My current style for writing feature specs

require 'rails_helper'
RSpec.feature 'Consumer completes 3 step minimal flow' do
  include ThreeStepMinimalFoundationSetupHelper
  include LandingPageHelper, AutoPoliciesHelper, ExitPageHelper
  let!(:zip) { generate_zipcode }
  let!(:vehicle) { FactoryGirl.create(:polk_vehicle, :with_year_make_and_model) }
  let!(:insurer) { FactoryGirl.create(:insurer) }
  let(:auto) { FactoryGirl.build(:auto) }
  let(:s2_path) { '/auto_policies/three_step_minimal_foundation/s2_primarydriver' }
  let(:s3_path) { '/auto_policies/three_step_minimal_foundation/s3_coverage' }
  let(:p) { PageObject.new }
  before :each do
    setup_three_step_minimal_foundation
    setup_landing
    setup_landing_page('myquote_non_sem', config_set, mobile: false)
    setup_auto_page('three_step_minimal_foundation')
    setup_exit_page('myquote_non_sem', config_set, layout: 'empty_page_with_theme', mobile: false)
  end

  context '3_step_minimal_foundation UI test', :js do
    before :each do
      visit_ready landings_path
    end
    scenario 'landing page with a required field NOT entered', :sad do
      fill_in :landing_zip_code, with: ''
      click_button p.eqv_continue_button
      expect(current_path).to eq landings_path
    end

    context 'landing page with all required fields entered' do
      before :each do
        fill_out_form zip.zip_code
        with_readiness_wait do
          click_button p.eqv_continue_button
        end
      end
      scenario 'now we are on stage#1, the auto_policies path', :happy do
        expect(current_path).to eq auto_policies_path
      end
      scenario 'happy thru all the stages', :happy do
        fill_out_auto auto
        select vehicle.submodel, from: p.css_vehicle1_auto_submodel
        select p.eqv_label_ownership, from: p.css_vehicle1_ownership
        select p.eqv_label_primary_use, from: p.css_vehicle1_primary_use
        select p.eqv_label_miles_per_year, from: p.css_vehicle1_miles_per_year
        select p.eqv_label_parking, from: p.css_vehicle1_parking
        with_readiness_wait do
          find(p.css_submit_form).click
        end
        find(p.css_has_license).click
        fill_out_driver p
        fill_in p.css_driver1_age_licensed, with: p.eqv_age_licensed
       select p.eqv_credit_rating, from: p.css_credit_rating
        select p.eqv_education, from: p.css_driver1_education
        with_readiness_wait do
          find(p.css_submit_form).click
        end
        select p.eqv_coverage_type, from: p.css_vehicle1_coverage_type
        fill_in p.css_street_address, with: p.eqv_street_address
        select p.eqv_residence, from: p.css_residence
        fill_in p.css_phone, with: p.eqv_phone
        fill_in p.css_email, with: p.eqv_email
        with_readiness_wait do
        find(p.css_submit_form).click
        end
        expect(page).to have_css p.css_quotes
      end
    end
  end

  context 'sad stages', :js do
    before :each do
      visit_ready landings_path
      fill_out_form zip.zip_code
      with_readiness_wait do
        click_button p.eqv_continue_button
      end
    end

    scenario 'submit of stage1 with vehicle year not selected redisplays with errors and retains changed field values', :sad do
      select p.eqv_option_auto_year_blank, from: p.css_vehicle1_auto_year
      select ownership_option, from: p.css_vehicle1_ownership
      select primary_use_option, from: p.css_vehicle1_primary_use
      select miles_per_year_option, from: p.css_vehicle1_miles_per_year
      select parking_option, from: p.css_vehicle1_parking
      with_readiness_wait do
        find(p.css_submit_form).click
      end
      expect(page).to have_css p.css_select_vehicle1_auto_year
      expect(page).to have_select p.css_vehicle1_ownership, selected: ownership_option
      expect(page).to have_select p.css_vehicle1_primary_use, selected: primary_use_option
      expect(page).to have_select p.css_vehicle1_miles_per_year, selected: miles_per_year_option
      expect(page).to have_select p.css_vehicle1_parking, selected: parking_option
      expect(page).to have_error_fields
    end

    scenario 'submit of stage2 with driver last name blank redisplays form with error and retains changed field values', :sad do
      visit_ready auto_policies_path(current_step: s2_path)
      fill_in p.css_driver1_last_name, with: ''
      select gender_option, from: p.css_driver1_gender
      select marital_status_option, from: p.css_driver1_marital_status
      with_readiness_wait do
        find(p.css_submit_form).click
      end
      expect(page).to have_css p.css_input_driver_last_name
      expect(page).to have_select p.css_driver1_gender, selected: gender_option
      expect(page).to have_select p.css_driver1_marital_status, selected: marital_status_option
      expect(page).to have_error_fields
    end

    scenario 'submit of stage3 with street address blank redisplays form with error and retains changed field values', :sad do
      def choose locator
        page.choose locator
      end
      visit_ready auto_policies_path(current_step: s3_path)
      fill_in p.css_street_address, with: ''
      select years_at_residence_option, from: p.css_years_at_residence
      select residence_option, from: p.css_residence
      choose p.css_vehicle1_parked_at_mailing_address_false
      choose p.css_homeowner_insurance_true
      choose p.css_add_violation_true
      choose p.css_add_insurance_claim_true
      choose p.css_add_driver_true
      choose p.css_add_auto_true
      with_readiness_wait do
        find(p.css_submit_form).click
      end
      expect(page).to have_css p.css_input_street_address
      expect(page).to have_select p.css_driver1_years_lived_there, selected: years_at_residence_option
      expect(page).to have_select p.css_residence, selected: residence_option
      expect(page).to have_field p.css_vehicle1_parked_at_mailing_address_false, checked: true
      expect(page).to have_field p.css_homeowner_insurance_true, checked: true
      expect(page).to have_field p.css_add_violation_true, checked: true
      expect(page).to have_field p.css_add_insurance_claim_true, checked: true
      expect(page).to have_field p.css_add_driver_true, checked: true
      expect(page).to have_field p.css_add_auto_true, checked: true
      expect(page).to have_error_fields
    end
  end
end