Robotic Process Automation (RPA)

 View Only
  • 1.  Defining Region in Wait for Image commands

    IBM Champion
    Posted Fri April 09, 2021 03:20 PM
    Hi,

    I've been trying to understand how to limit the area the robot should look for an image without success. No matter how I tried it, if I don't select to search in the entire screen, it does not work. Has anyone made this work? Here is a sample script comparing the two "methods", which is searching the entire screen and limiting a region (using the rectangle from the window object itself).

    import --name image1 --type "Image" --content iVBORw0KGgoAAAANSUhEUgAAAFcAAAAMCAYAAAAXgSFYAAAAAXNSR0IArs4c6QAAAARnQU1BAACxjwv8YQUAAAAJcEhZcwAADsMAAA7DAcdvqGQAAAN4SURBVFhH3Vc9bqtAEP6SE/gMxJIlSgpugF24onBPiZvIDQUnoHBjucElfQoqCpsbUFAiITk+AzfIm1mGP//ESfSS95RPQvbOetmdb2a+HT+8OHhL5hU2FgQpVqM1JvkBS60xrTBK5qiej5gaMez+3Idw5Z3fjFM4hVF6qFrHTginBvxMhmaA/LCEps62QCRmhvPS8HHr3JdrzCCHVxpYT3Ic5MeP1txBlKRqoHA6okCGeH8SA5sKmJMnQFviUP0cQV/GKYQb28gbYmk8HRkovQpVJc8O2LdumwhysVO2RYsV0XcPvTX0MKHWJocduwiFukc8TWAWR4prjdM+hh4EQPnaWEAm2LP/ndEO7AMdmLJSjRC6PvQ2GwWUKMv+uIE1h0Ppdexy6xPQsPR0+Ns6NI/QZrARo07UmsjJbAw9SiR6rygzHWN1Ui6HqURGvockGaOReqZNyBgqWxr7UYyC3hw/K0mTdNV9H+5F4DVqksv7cm2Hs2Q47RFnDubXiLyGNEFk2vhyLvWS9ZHZHuuZJCoRSVTPNAtzR6LHmzlzXD9bBr8kLZZyyvytBISIMeps4bkdBa/Tp+FclQcoFjWJA4mifQsTrTypbGRpSrfw9Ze2HAfZqNBPBoE5Aa18B+SHIQHju0Vp8T301vSDrFFiZiWdQpHbc4qJ1MfqxRQA5Virt1dBuvMs3vXLiXWbLoxmSlt6NCdQmt7LJCpPz5Hg9qKeJgVsz+ZD0LiXjfybaDGsknsQZ29D9JMCbbYVew9Dze2C/ISJWfOgyG2cSolIR7zWZuRYuf9ZvW0lKkVSUAVZPCZiuLRVRfFv+FLlanBvyMIZ+J1mhP6dfRP07l1QYP2ZwF2gq5yaXOWUj4VfV15tY931qXU5K7GPQJWGD9F1kst1Jws8R6PWWdLSddRksgaOaeyuUagLiccFkm3Zu6BqaMsD8sBEcXHzdJlTgy8Z7gDOAkH7hlcI5yrT/e7G/zRU1dYyVJOrnCCBGwg56y593NTb92Bho1qaWo9cCl0rCzyndFb0SvXNm3YPVTHUizbVwuMiIoloDsY9t+ic4evwLvpCCVCvlaQeSbRd9uTHBVWGzA9g4ZmaJd8NlTyda+tdOXotkYm0PpBevNXWXwTuLIi83Ycupr8J7mTUxuq/wO8kl3D5D+37wa3kAtTJyJ6/ltx/D+APaWreeC7YT5gAAAAASUVORK5CYII=
    defVar --name window --type Window
    defVar --name successWaitImage --type Boolean
    defVar --name successWaitImageRegion --type Boolean
    launchWindow --executablepath "notepad.exe" window=value
    waitWindow --classname Notepad --processname notepad
    focusWindow --window ${window}
    waitImage --searchonscreen  --image ${asset.image1} --similarity 100 --timeout 00:00:10 --interval 00:00:01 --selector "Vision" --comment "With Search on Screen" successWaitImage=success
    waitImage --image ${asset.image1} --similarity 100 --timeout 00:00:10 --interval 00:00:01 --region "${window.Bounds}" --selector "Vision" --comment "With Region" successWaitImageRegion=success
    closeWindow --window ${window}
    logMessage --message "Success without region: ${successWaitImage}\r\nSuccess with region: ${successWaitImageRegion}" --type "Info"​

    Any help is much appreciated.

    By the way, if anyone has information/guides/tutorials on how to use computer vision commands with IBM RPA (useful if working with VDI, Citrix, and others), it would be great if you shared. Everything I could find was a short video on YouTube, not even the Learning Journey has something about it.

    ------------------------------
    Vinicius Marques
    ------------------------------


  • 2.  RE: Defining Region in Wait for Image commands

    Posted Mon April 12, 2021 09:21 AM
    Hi Vinicius

    When using the command 'waitImage' with 'Search on Screen' enabled, the command searches the image on the entire screen, when disabled it searches only the window that is in context.

    What I can see in your code is that you are looking for an image with the text 'Windows (CRLF)' with a gray background. I don't know if there was this text on the test screen, if so, it can decrease the similarity a little, to see if it gets more accurate.

    Another way to search for an image limiting the search area is the command 'FindImage', it returns to the position of the screen where the image is and with the command 'Mouse Command' you can click. But the findImage command has no time limit, so it will not search for whether it exists or not, it will only validate once.

    Some notes on working with image:
    • The resolution of the development machine with the production machine, both have to be the same
    • The smaller the image, the faster its location
    • If I managed to map the region of the image, with a color that has little on the screen, it is faster too.


    ------------------------------
    Angelo Alves
    ------------------------------



  • 3.  RE: Defining Region in Wait for Image commands

    IBM Champion
    Posted Tue April 13, 2021 09:03 AM
    Hi Angelo,

    Thank you very much for the information. I was looking exactly for this behavior you described "...when disabled it searches only the window that is in context.". This wasn't clear to me, so I was trying to use the Region as a delimiter for the search and with no success.

    Could you please select some optimal images that could be used for detection and post it here as an example? For this I'm sending the notepad screen, which is very simple but probably enough for you to extract some good images for detection. What I aim to do is just understand how big should detection images be.


    Thank you in advance!

     


    ------------------------------
    Vinicius Marques
    ------------------------------



  • 4.  RE: Defining Region in Wait for Image commands

    Posted Tue April 13, 2021 03:53 PM
    Edited by Angelo Alves Tue April 13, 2021 03:54 PM
    Using the context window is when you want the image search to be in a specific window, for that you need to locate the window through WaitWindow or FindWindow. For example, you can find the "Remote Desktop" window to automate a system in a virtual environment. Because the images you are looking for can contain on your local machine.

    Below are some screenshots that I took during the mapping of the script.



    import --name image1 --type "Image" --content iVBORw0KGgoAAAANSUhEUgAAAAsAAAAKCAIAAADtkjPUAAAAAXNSR0IArs4c6QAAAARnQU1BAACxjwv8YQUAAAAJcEhZcwAADsMAAA7DAcdvqGQAAAELSURBVChTHc7LTsMwEAXQZvLwY5yEIIpoQbBDqD/A/+/4CjYIFjwaj51XGydhUvnKu3vmRm/vH+OyjNM8hPk0TUOYhnnux6kdw9fP7+vTHrTJNRrFv0FpDEdoFGgybSAVL/d3UDtXk6stHcmRdeS8Je+c59d6X0YLKESOXGMEYqa1RJ1pjIUor8rH7TUw8EfuSGSJ2GOAyFtPTU1JGMl7kHweUWiTXgBuJ1onSqdS7rc3VVHA0XLP8QDr1hHkeIN31DREJppZBXFpJAoTiSAVKAVSg5SJEIeH3Wp0TTO0zalrz107dl3o+2XoNuee5xSwrMZniL7n2G7iNk5DJpg0eV7lhamq591tVRT/I86gus2xbF8AAAAASUVORK5CYII=
    import --name image2 --type "Image" --content iVBORw0KGgoAAAANSUhEUgAAABsAAAAICAIAAACH9BM3AAAAAXNSR0IArs4c6QAAAARnQU1BAACxjwv8YQUAAAAJcEhZcwAADsMAAA7DAcdvqGQAAADISURBVChTrZGxDYMwEEWPzAJKxQCwAVSpUlKaFg+QigGghZKSGtkjeIAUKMK7EOw7uwApUOQ1tv6X79+dYTUIBkmz2CtxVDw/LMMN/s2FirpNA6SUJCGyDNJWkpu2GlVfUfHIOkTeky7LaHzijAJq98yheA3DZi0N8AJNX/G4R4v+vH1Y3qt5QdmRNEMVbmdYvRiZV/bIBMWsa5eRtsckI2cVw3sM/WFYjxona+lpVOxh4057zDqzI7fj3d9AEs+F0SMeC9s/wBf2xaBff+zCqAAAAABJRU5ErkJggg==
    import --name image3 --type "Image" --content iVBORw0KGgoAAAANSUhEUgAAAB8AAAAGCAIAAAC0FdI9AAAAAXNSR0IArs4c6QAAAARnQU1BAACxjwv8YQUAAAAJcEhZcwAADsMAAA7DAcdvqGQAAADvSURBVChTnZC9DcIwEIUfHiBLGImCKk0mQFDQIRZIFYkmHfIIUTooIqXyAoguBRETuElFgYSX8AScHdsICiT4Cv+8073z88QYo5tFgbZFQet1x2Hpy6SeDeH2M76dAfpyxmbF+WqD80X7soPGJpZFE+SgEGU/SmQU6+OZ1q2EEmnCgjnwaa9EjZaiDRVE4fr7MhXzE0lWvG1fQ99ZHswpR1YNhkXz0V4c/ZNA5dZ9DN/tc3UnI/24IV8vXTGKX2FHoWwGRyoUZBftA9b1P5i0CSJDlUV75b+J0in3ZD6dx+G6qeWYg8+yEKLvpNs9wBOcLIRuWLSenAAAAABJRU5ErkJggg==
    defVar --name notepad --type Window
    defVar --name success --type Boolean
    launchOrAttach --executablepath "C:\\windows\\system32\\notepad.exe" --useregex  --regexPattern "Untitled - Notepad" --regexOptions "0" --processname notepad notepad=value success=success
    focusWindow --window ${notepad}
    waitImage --image ${asset.image1} --similarity 100 --timeout 00:00:05 --interval 00:00:00.5000000 --selector "Vision" success=success
    if --left "${success}" --operator "Is_True"
    	click --selector "Vision" --visionimage ${asset.image2} --visionsimilarity 100 --timeout "00:00:15" --comment Help
    	click --clickOnScreen  --selector "Vision" --visionimage ${asset.image3} --visionsimilarity 100 --timeout "00:00:15"
    endIf​

    ------------------------------
    Angelo Alves
    ------------------------------



  • 5.  RE: Defining Region in Wait for Image commands

    IBM Champion
    Posted Tue April 13, 2021 04:07 PM
    Awesome!

    Thank you for all the information.

    ------------------------------
    Vinicius Marques
    ------------------------------