Sunday, January 29, 2017

AUTOMATION 29 - The need for speed

This is part 29 of my automation series - for our story to date, check out the list of articles here.

Technical level: **

Just a couple posts ago we were looking at the goal of automaton.  In general terms, what I get from every group I've talked to is something along the line of "speedy feedback".  We're trying to shorten and supercharge the feedback loop from someone committing a change to being told "the system doesn't work".

So what's an ideal time for it to take?  This is important.

Obviously we want it to take longer than doing it manually, otherwise what's the point?

Different groups had different ideas about this.  Most just wanted some kind of "thumbs up" or "thumbs down" indicator on build quality.  And they wanted it quite quickly.  For others there was a rigorous series of regression tests they needed to run on the system, and they were trying to speed this up from "currently takes a couple of weeks".

I'm going to call these two drivers "build quality" and "regression".  They will dramatically change the timeframe we'll be looking at.  For both these drivers though, the first port of call for making things as fast as possible is to embrace the automation pyramid and to be sure to be doing at the unit or API level those checks which make sense to do there.

Driven by "build quality"

A developer finishes a code or configuration change, and wants to commit to the test environment.  But first they have to run a series of checks to make sure what's being deployed isn't obviously broken.



Ideally you want these to run as fast as possible.  The ideal time I've often heard of is about 20 minutes (the office joke is we've got a cafe on our ground floor, and this is the time it takes to have a pizza made and delivered).  Certainly any more than 40 minutes is painful.  It leads to "I guess I just kick this off before I leave and hope I'm first in tomorrow morning" *

There are ways to go faster.  Selenium GRID allows you to perform automation tests in something like a multi-threaded environment (running multiple things concurrently).  Katrina Clokie wrote an article about adoption of this at BNZ here.

Using technology to help you go faster is a definite must, it's the driving force behind trying to use the pyramid to make sure your checks are at the place where they can run most efficiently.  If you're testing hundreds of business logic combinations, do it at the unit level.

This picture though highlights something to be aware of ...


If you've got an elephant on a motorbike, and need to go faster, the first port of call is to buy a more powerful motorcycle (faster machines, Selenium GRID).

If that's still not working, it might be time to put the elephant on a diet.  That means looking through your build tests, and taking a hard critical look as "do we need them all?".  Potentially scaling it back.

Driven by "regression"

Is the driver for your automation is regression test coverage, you're probably aiming for a much bigger suite, with much longer runtimes.

It's as important even though you're looking at maybe running over days vs running in well under an hour, to still embrace the pyramid and try to have your checks running as efficiently as possible.

A few areas I've spoken with to have embraced both "build driven" and "regression".  Typically the post-build checks that they run are a cut down version of the regression test suit (hey, that means reuse), but of the top critical behaviours.

There's still a need to try and avoid the feeling of the elephant on the bike.  More automation means more to maintain, and you want to try and avoid trying to service and carry going forward tests which are trivial in nature.

At the beginning of this series I told about the test manager who'd make us run scripts which checked for every bug that we'd ever found (no matter how trivial), and how painful that got in time.  When I was at Agile Test Days in Germany, I happened to meet a fellow tester from the project included in that experience report, and we talked about how painful that approach had become.  So much so that the prohibitive cost of testing like that eventually led to us losing future contract work with that customer (the unit manager happened to be a school friend, and had confirmed that).  And for all that rerunning of those tests, we never once found a recurrence of one of those bugs to warrant the time.

[To be clear, I learned from that test manager the important of going deeper in testing than I'd normally do.  But I learned in trying to engage with that test manager and the business involved, the importance of revising testing to quite that level if we're not getting defects about the product illuminated.]







*  When I was a developer, waiting for the build to finish drove me nuts.  If it was a short build, I might update some documentation, then write a humorous meme to my friends.  If it was longer, I'd try and remote log in on spare machine, and trawl through the defect list to find a relatively simple bug I might be able to knock out of the park.  This might be a bigger mistake than the comedy meme as I'd end up switching context so would forget some of what I'd done from the compile and test that was going on!

21 comments:

  1. Great site for these post and i am seeing the most of contents have useful for my Carrier.Thanks to such a useful information.Any information are commands like to share him.
    Pest Control in Chennai

    ReplyDelete


  2. Thanks for sharing article like this. The way you have stated everything above is quite awesome. Keep blogging like this.



    Best Dental Clinic in Tambaram

    ReplyDelete
  3. nice blog too informative. looking and reading your points its so impressive. doing more blog like this. i really appreciated doing like this.
    GMAT coaching chennai

    ReplyDelete
  4. Thank you for taking the time to provide us with your valuable information. We strive to provide our candidates with excellent care and we take your comments to heart.As always, we appreciate your confidence and trust in us
    SEO Company in India
    Digital Marketing Company in India

    ReplyDelete
  5. Excellent and very cool idea and the subject at the top of magnificence and I am happy to this post..Interesting post! Thanks for writing it.What's wrong with this kind of post exactly? It follows your previous guideline for post length as well as clarity.
    Web Development Company in India

    ReplyDelete
  6. nice steps you are covered in this topic. its much useful to me. keep update more things about more test preparation and questions and how to rectify absolute answer.
    Germany Education Consultants in Chennai

    ReplyDelete
  7. Truly a very good article on how to handle the future technology. After reading your post,thanks for taking the time to discuss this, I feel happy about and I love learning more about this topic
    Tooth Braces In Chennai

    ReplyDelete
  8. Manual testing is performed by a human sitting in front of a computer carefully executing the test steps.

    Automation Testing means using an automation tool to execute your test case suite.

    The automation software can also enter test data into the System Under Test , compare expected and actual results and generate detailed test reports.

    Test Automation demands considerable investments of money and resources.

    Successive development cycles will require execution of same test suite repeatedly.

    Using a test automation tool it's possible to record this test suite and re-play it as required.

    Once the test suite is automated, no human intervention is required .

    This improved ROI of Test Automation.
    Selenium Training|Selenium Training in Chennai

    ReplyDelete
  9. I just see the post i am so happy to the communication science post of information's.So I have really enjoyed and reading your blogs for these posts.Any way I’ll be replay for your great thinks and I hope you post again soon...
    Software Testing Training in Chennai

    ReplyDelete
  10. Now it is known to me that articles is nothing but inspiring is everything to do something great. This is a great article for the people who want to come in freelancing.
    GMAT Training in Chennai | Best GMAT Coaching Classes in Chennai

    ReplyDelete
  11. I am very happy when read this blog post because blog post written in good manner and write on good topic. Thanks for sharing valuable information



    Bangalore Web Designing Company, Web Development Company in Bangalore

    ReplyDelete

  12. تختلف انواع الحشرات وعليه تختلف طرق مكافحتها لذلك وفرنا كل السبل لمكافحة الحشرات والقضاء عليها فوفرنا بمدينة الرياض
    شركة ابادة الفئران بالرياض
    وفي مجال القضاء علي الحشرات بمختلف انواعها بالرياض وفرنا
    اما ما يخص مدينتي الخرج وجدة مكافحة حشرات بالخرج
    لاكن كن علي يقين بانك سوف يرتاح بال نهائيا من تلك الحشرات بعد تعاملك معنا
    شركة رش مبيدات بالخرج
    وايضا في مجال رش الدفان

    ReplyDelete
  13. It was a great information and Its really worth reading it. The author did an mind blowing work by describing each and every concept in detail. Thanks for such an informative post. Please keep up your good work.
    Engineering Colleges in Chennai

    ReplyDelete
  14. I liked your blog, it shares many information regarding new technologies, and keeps us up to date.

    ReplyDelete
  15. Great post as usual Mike, I've loved the automation series.
    I've only just checked back here and caught up with this latest post today, though I see none since January, looking forward to the next one :-).

    ReplyDelete
  16. Ass Wr Wb Saya ingin berbagi cerita kepada anda bahwa sy seorang TKW dari malaysia dan secara tidak sengaja saya buka internet dan saya melihat komentar IBU LUSI yg dari singapura tentang AKI yg telah membantu dia menjadi sukses dan akhirnya saya juga mencoba menghubungi beliau dan alhamdulillah beliau mau membantu saya untuk memberikan nomor Togel/lottrey 4D dr hasil ritual/ghaib melalui jarak jauh dan alhamdulillah itu betul-betul terbukti tembus dan menang RM.270.000 Ringgit ,kini saya kembali indon membeli rumah dan kereta walaupun sy cuma pembantu rumah tanggah di selangor malaysia, sy sangat berterimakasih kepada AKI MAULANA dan tidak lupa mengucap syukur kepada ALLAH karna melalui AKI saya juga sudah bisa se sukses ini, pesan AKI yg slalu sy ingat setiap manusia bisa menjadi kaya, hanya saja terkadang mereka tidak tahu atau salah jalan, Banyak orang menganggap bahwa miskin dan kaya merupakan bagian dari takdir, Takdir macam apa? Tuhan tidak akan memberikan takdir yang buruk terhadap kita semua, cobaan yang Tuhan berikan merupakan pembuktian seberapa kuat Anda bertahan di dalamnya. Tuhan tidak akan merubah nasib Anda jika Anda tidak berusaha untuk merubahnya sendiri, Jadi teman2 yg dalam ke susahan jgn pernah putus asah, kalau sudah waktunya tuhan pasti kasi jalan asal anda mau berusaha. AKI MAULANA adalah guru spiritual terkenal di indonesia yg bisa melakukan ritual ghaib seperti:
    1.Pesugihan bank ghaib
    2.Ritual tembus togel/lottrey
    3.Transfer janin
    4.Pelaris usaha, jodoh DLL
    jika anda ingin mengubah nasib seperti saya silahkan ☎KLIK DISINI PESUGIHAN DUNIA GHAIB



    THE STORY OF SUCCESS MOTHER RATNAH PUSPITA

    I want to share a story to you that sy a migrant worker from malaysia and i accidentally open internet and i see the comment LUSI mom from singapore about AKI who has helped him become successful and finally i also try to contact him and alhamdulillah he would help me to Give the number Togel / lottrey 4D dr ritual results / ghaib through long distance and alhamdulillah it really proved translucent and won RM.270.000 Ringgit, now I am back indon buying house and train even though sy just housemaid in selangor malaysia, sy very thankful To AKI MAULANA and do not forget to give thanks to ALLAH because through AKI I also can be this successful, AKI message which slalu sy remember every human being can become rich, only sometimes they do not know or wrong way, Many people assume that poor and rich Is a part of destiny, what kind of fate? God will not give a bad destiny to us all, God's temptation is proving how strongly you endure it.God will not change your fate if you do not try to change it yourself, So teman2 deep into tusah jgn never despair, when it's time god must be the road to where you want to try.AKI MAULANA is a famous spiritual teacher in Indonesia who can perform the occult rituals such as:
    1.Presugihan bank ghaib
    2.Ritual translucent togel / lottrey
    3.Transfer fetus
    4.Paris business, matching DLL
     If you want to change my destiny please ☎CLICK HERE WORLD WIDE GHAIB




    ReplyDelete
  17. Nicely explained. Here you described the well written article from your in-depth knowledge. Truly impressive and nice information

    Java Training in Chennai Core Java Training in Chennai Core Java Training in Chennai

    Java Online Training Java Online Training JavaEE Training in Chennai Java EE Training in Chennai

    ReplyDelete
  18. A nice article here with some useful tips. Thanks for this helpful information I agree with all points you have given to us. I will follow all of them.
    Hadoop Training in Chennai
    Software Testing Training in Chennai
    Selenium Training in Chennai
    Dot Net Training in Chennai

    ReplyDelete
  19. Terimakasih banyak AKI karna melalui jalan togel ini saya sekarang sudah bisa melunasi semua hutang2 orang tua saya bahkan saya juga sudah punya warung makan sendiri hi itu semua berkat bantuan AKI JAYA yang telah membarikan angka 4D nya menang 275 jt kepada saya dan ALHAMDULILLAH berhasil,kini saya sangat bangga pada diri saya sendiri karna melalui jalan togel ini saya sudah bisa membahagiakan orang tua saya..jika anda ingin sukses seperti saya hubungi no hp O85-244-015-689 AKI JAYA,angka ritual AKI JAYA meman selalu tepat dan terbukti..silahkan anda buktikan sendiri. 2D 3D 4D 5D 6D






    Terimakasih banyak AKI karna melalui jalan togel ini saya sekarang sudah bisa melunasi semua hutang2 orang tua saya bahkan saya juga sudah punya warung makan sendiri hi itu semua berkat bantuan AKI JAYA yang telah membarikan angka 4D nya menang 275 jt kepada saya dan ALHAMDULILLAH berhasil,kini saya sangat bangga pada diri saya sendiri karna melalui jalan togel ini saya sudah bisa membahagiakan orang tua saya..jika anda ingin sukses seperti saya hubungi no hp O85-244-015-689 AKI JAYA,angka ritual AKI JAYA meman selalu tepat dan terbukti..silahkan anda buktikan sendiri. 2D 3D 4D 5D 6D







    Terimakasih banyak AKI karna melalui jalan togel ini saya sekarang sudah bisa melunasi semua hutang2 orang tua saya bahkan saya juga sudah punya warung makan sendiri hi itu semua berkat bantuan AKI JAYA yang telah membarikan angka 4D nya menang 275 jt kepada saya dan ALHAMDULILLAH berhasil,kini saya sangat bangga pada diri saya sendiri karna melalui jalan togel ini saya sudah bisa membahagiakan orang tua saya..jika anda ingin sukses seperti saya hubungi no hp O85-244-015-689 AKI JAYA,angka ritual AKI JAYA meman selalu tepat dan terbukti..silahkan anda buktikan sendiri. 2D 3D 4D 5D 6D


    ReplyDelete