An example of UIAutomation
Not logged in
Home
Forum
Login
Chronological
This forum is sponsored by Mistachkin Systems.
Eagle: Secure Software Automation

An example of UIAutomation

(1.1) By Yusuke Yamasaki (yyamasak) on 2022-10-06 12:48:49 edited from 1.0 [link] [source]

Hello,

I wrote an example code at the bottom of this post referring to your example eagle code to automate Windows calc.exe. (examples\ex2.eagle)

However, it failed to run due to this error.

object invoke ClientSettings RegisterClientSideProviderAssembly [$assembly GetName]
#=> {object or type "ClientSettings" not found} {expected type value but got "ClientSettings"}

How can I invoke the class, ClientSettings?

Versions are as follows.

Eagle 1.0.7945.33333 trusted genuine official stable beta {14th Anniversary Extra Special Edition, Beta 49} NetFx20 Release {Tcl 8.4.21} {2021.10.02T02:03:20.000 +0900} daa68e5dfb3a38507491509ea8bca0e9be3c937b {2021-10-30 21:49:53 UTC} {Microsoft.NET 4.0.30319.42000 [4.8.4515.0 built by: NET48REL1LAST_C] 528372} {Windows NT} intel

Garuda 1.0 daa68e5dfb3a38507491509ea8bca0e9be3c937b {2021-10-30 21:49:53 UTC}

package require Garuda

puts [eagle version]
puts [garuda packageid]

eagle {
	object load UIAutomationClient
	object load UIAutomationTypes
	object load UIAutomationProvider

	set assembly [object load -alias UIAutomationClientsideProviders]
	object import System.Windows.Automation
	object invoke ClientSettings RegisterClientSideProviderAssembly [$assembly GetName]

	set prop(aid) [object invoke AutomationElement AutomationIdProperty]
	set prop(name) [object invoke AutomationElement NameProperty]
	set patt(invoke) [object invoke InvokePattern Pattern]

	set elem(root) [object invoke -alias AutomationElement RootElement]

	set cond(sample) [object create PropertyCondition $prop(name) "Open the Sample Window"]
	set elem(sample) [$elem(root) -alias FindFirst Children $cond]

	[$elem(sample) -alias GetCurrentPattern $patt(invoke)] Invoke
}

(2.4) By mistachkin on 2022-10-05 22:06:56 edited from 2.3 in reply to 1.0 [link] [source] 

My first guess is that you’ll want to use the .NET 4.0 build of Eagle to do this (i.e. for compatibility with all the necessary assemblies).

EDIT #1: Slightly revised example that is tested and working on Windows 7 (32-bit) using the .NET Framework 4.0.

EDIT #2: Also, you may want to upgrade to the latest Eagle release (beta 50).

EDIT #3: Yeah, it’s a bit atypical to use the .NET Framework 2.0 (CLR v2) build of Eagle running on the .NET Framework 4.x (CLR v4).

object load UIAutomationClient
object load UIAutomationTypes
object load UIAutomationProvider

set assembly [object load -alias UIAutomationClientsideProviders]

object import System.Windows.Automation

object invoke ClientSettings \
    RegisterClientSideProviderAssembly [$assembly GetName]

set prop(pid) [object invoke AutomationElement ProcessIdProperty]
set prop(class) [object invoke AutomationElement ClassNameProperty]
set prop(name) [object invoke AutomationElement NameProperty]
set patt(invoke) [object invoke InvokePattern Pattern]

# optional, close calc.exe
# kill -all calc.exe

set pid [exec calc.exe &]
set pidObject [object invoke -create Int32 Parse $pid]
after 2000; # wait for calc.exe input idle?

set elem(root) [object invoke -alias AutomationElement RootElement]

set cond(1) [object create PropertyCondition $prop(pid) $pidObject]
set elem(pid) [$elem(root) -alias FindFirst Children $cond(1)]

set cond(2) [object create PropertyCondition $prop(class) Button]

if {0} then {
  #
  # NOTE: Show all the button names within the target window.
  #
  object foreach -alias elem(this) \
      [$elem(pid) -alias FindAll Descendants $cond(2)] {
    puts stdout [$elem(this) GetCurrentPropertyValue $prop(name)]
  }
}

set cond(3) [object create PropertyCondition $prop(name) 1]
set cond(4) [object create AndCondition $cond(2) $cond(3)]
set elem(btn1) [$elem(pid) -alias FindFirst Descendants $cond(4)]

set cond(5) [object create PropertyCondition $prop(name) Add]
set cond(6) [object create AndCondition $cond(2) $cond(5)]
set elem(btn+) [$elem(pid) -alias FindFirst Descendants $cond(6)]

#
# HACK: Apparently, name for plus button can be different?
#
if {![isNonNullObjectHandle $elem(btn+)]} then {
  unset cond(5) cond(6) elem(btn+); # dispose?
  set cond(5) [object create PropertyCondition $prop(name) +]
  set cond(6) [object create AndCondition $cond(2) $cond(5)]
  set elem(btn+) [$elem(pid) -alias FindFirst Descendants $cond(6)]
}

set cond(7) [object create PropertyCondition $prop(name) Equals]
set cond(8) [object create AndCondition $cond(2) $cond(7)]
set elem(btn=) [$elem(pid) -alias FindFirst Descendants $cond(8)]

#
# HACK: Apparently, name for equals button can be different?
#
if {![isNonNullObjectHandle $elem(btn=)]} then {
  unset cond(7) cond(8) elem(btn=); # dispose?
  set cond(7) [object create PropertyCondition $prop(name) =]
  set cond(8) [object create AndCondition $cond(2) $cond(7)]
  set elem(btn=) [$elem(pid) -alias FindFirst Descendants $cond(8)]
}

[$elem(btn1) -alias GetCurrentPattern $patt(invoke)] Invoke
after 1000

[$elem(btn+) -alias GetCurrentPattern $patt(invoke)] Invoke
after 1000

[$elem(btn1) -alias GetCurrentPattern $patt(invoke)] Invoke
after 1000

[$elem(btn=) -alias GetCurrentPattern $patt(invoke)] Invoke
after 1000

# optional, close calc.exe
# kill $pid

(3) By Yusuke Yamasaki (yyamasak) on 2022-10-06 13:34:02 in reply to 2.4 [link] [source]

I forgot to write my environment. Yes, I wanted to use .NET 4.0 build for compatibility among Windows XP, 7 and 10.

(PC1) OS: Windows 10 Pro 64bit 21H2
Tcl/Tk: BAWT Tcl 8.6.4 Batteries Included (32 bit)
Eagle/Garuda: Beta 49

(PC2) OS: Windows 7 Professional SP1 32bit
Tcl/Tk: ActiveTcl 8.6.4.1 32bit
Eagle/Garuda: Beta 50

This is the shortest script. In both PCs, before doing automation work, the script raises an exception.
object or type "ClientSettings" not found

set exehome [pwd]
set EAGLE_DLL [file join $::exehome Eagle.dll]

namespace eval ::Garuda {
    variable methodFlags 0x40; # METHOD_PROTOCOL_V1R2
    variable assemblyPath $::EAGLE_DLL
}

package require Garuda

puts [eagle version]
puts [garuda packageid]

eagle {
    object load UIAutomationClient
    object load UIAutomationTypes
    object load UIAutomationProvider

    set assembly [object load -alias UIAutomationClientsideProviders]

    object import System.Windows.Automation

    object invoke ClientSettings \
		RegisterClientSideProviderAssembly [$assembly GetName]
}

The output is as follows.

Eagle 1.0.8192.54321 trusted genuine official stable beta {Fire Dragon Series, Beta 50} NetFx20 Release {Tcl 8.4.21} {2022.05.06T21:42:00.000 +0900} 48b99035ca39c1100f84abd801d0309f6ce25106 {2022-05-26 21:44:39 UTC} {Microsoft.NET 4.0.30319.42000 [4.8.4110.0 built by: NET48REL1LAST_B] 528049} {Windows NT} intel
Garuda 1.0 48b99035ca39c1100f84abd801d0309f6ce25106 {2022-05-26 21:44:39 UTC}
{object or type "ClientSettings" not found} {expected type value but got "ClientSettings"}

In my weak memory, your calc.exe automation script (ex2.eagle) worked when my main PC was Windows 7 and was using Garuda fetched from teapot repository hosted by ActiveState.

I tried again after making wish.exe.config beside wish.exe hoping that it works under Windows 10 + .NET 4.8 but it didn't help.

<?xml version="1.0"?>
<configuration>
<startup><supportedRuntime version="v4.0" sku=".NETFramework,Version=v4.0"/></startup></configuration>

(4.1) By mistachkin on 2022-10-07 18:07:09 edited from 4.0 in reply to 3 [source]

Please try using the .NET Framework 4.0 specific builds of Eagle and Garuda for the latest release (Beta 50). The downloaded file will be marked with “NetFx40_”, e.g.: https://download.eagle.to/releases/1.0.8192.54321/EagleBinaryNetFx40_1.0.8192.54321.exe

EDIT #1: Also, is there a “UIAutomationClient.dll” file somewhere on the system in question?

(5) By Yusuke Yamasaki (yyamasak) on 2022-10-12 02:33:14 in reply to 4.1 [link] [source]

Thank you very much. I didn't notice that I was using .NET Framework 2.0 version of Eagle and Garuda. When I replaced both with Beta 50 targeting .NET Framework 4.0, class loading got successful.

Eagle 1.0.8192.54321 trusted genuine official stable beta {Fire Dragon Series, Beta 50} NetFx40 Release {Tcl 8.4.21} {2022.05.06T21:42:00.000 +0900} 48b99035ca39c1100f84abd801d0309f6ce25106 {2022-05-26 21:44:39 UTC} {Microsoft.NET 4.0.30319.42000 [4.8.4515.0 built by: NET48REL1LAST_C] 528372} {Windows NT} intel  

Garuda 1.0 48b99035ca39c1100f84abd801d0309f6ce25106 {2022-05-26 21:44:39 UTC}

One issue left might be specific to the GUI which I was going to automate. So, I'm not sure if I can ask of your help. According to Inspect.exe, this button supports InvokePattern. But when I try GetCurrentPattern, it throws System.InvalidOperationException: Unsupported Pattern. Do you think it has something to do with the usage of eagle script?

How found:	Selected from tree...
Name:	"Open the Sample Window"
ControlType:	UIA_ButtonControlTypeId (0xC350)
LocalizedControlType:	"button"
:
IsInvokePatternAvailable:	true

Test script.

package require Garuda

puts [eagle version]
puts [garuda packageid]

eagle {
	object load UIAutomationClient
	object load UIAutomationTypes
	object load UIAutomationProvider
}

eagle {
	set assembly [object load -alias UIAutomationClientsideProviders]
	object import System.Windows.Automation
	object invoke ClientSettings RegisterClientSideProviderAssembly [$assembly GetName]
}

eagle {
	set prop(aid) [object invoke AutomationElement AutomationIdProperty]
	set prop(name) [object invoke AutomationElement NameProperty]
	set patt(invoke) [object invoke InvokePattern Pattern]
}

eagle {
	set elem(root) [object invoke -alias AutomationElement RootElement]
}

eagle {
	set cond(sample) [object create PropertyCondition $prop(name) "Open the Sample Window"]
	set elem(sample) [$elem(root) -alias FindFirst Descendants $cond(sample)]
	[$elem(sample) -alias GetCurrentPattern $patt(invoke)] Invoke
}

The line of error and the stack trace.

eagle {$elem(sample) -alias GetCurrentPattern $patt(invoke)}
System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.InvalidOperationException: Unsupported Pattern.

   at System.Windows.Automation.AutomationElement.GetCurrentPattern(AutomationPattern pattern)

   --- End of inner exception stack trace ---

   at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)

   at System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments)

   at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)

   at Eagle._Commands.Object.Execute(Interpreter interpreter, IClientData clientData, ArgumentList arguments, Result& result)

(6) By mistachkin on 2022-10-12 05:59:00 in reply to 5 [link] [source] 

That sounds like a problem with the target application itself.  When working
with and troubleshooting the UI Automation stuff, it can be quite helpful to
make use of some of the tools provided with the Microsoft Platform SDK, e.g.:

    %ProgramFiles%\Windows Kits\10\bin\x86\AccChecker\acccheckui.exe
    %ProgramFiles%\Windows Kits\10\bin\x86\accevent.exe
    %ProgramFiles%\Windows Kits\10\bin\x86\UIAVerify\VisualUIAVerifyNative.exe

(7) By Yusuke Yamasaki (yyamasak) on 2022-10-13 05:23:12 in reply to 6 [link] [source]

Yes, I'm using such tools. Those can find the button I wanted to click. But there seems to be a case where a button cannot be found via UIAutomation.

By the way, I tried to enumerate elements which supports InvokePattern. But this line failed to create a condition. I tried 1, true, True but all failed.

set cond(invoke) [object create PropertyCondition $prop(invoke) 1]

# =>System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.ArgumentException: PropertyCondition value for property 'AutomationElementIdentifiers.IsInvokePatternAvailableProperty' must be 'Boolean'.

   at System.Windows.Automation.PropertyCondition.Init(AutomationProperty property, Object val, PropertyConditionFlags flags)

   at System.Windows.Automation.PropertyCondition..ctor(AutomationProperty property, Object value)

   --- End of inner exception stack trace ---

   at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)

   at System.Reflection.RuntimeConstructorInfo.Invoke(BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)

   at Eagle._Commands.Object.Execute(Interpreter interpreter, IClientData clientData, ArgumentList arguments, Result& result)

Full script

package require Garuda

puts [eagle version]
puts [garuda packageid]

eagle {
	object load UIAutomationClient
	object load UIAutomationTypes
	object load UIAutomationProvider

	set assembly [object load -alias UIAutomationClientsideProviders]

	object import System.Windows.Automation
	object invoke ClientSettings RegisterClientSideProviderAssembly [$assembly GetName]

	set prop(name) [object invoke AutomationElement NameProperty]
	set prop(invoke) [object invoke AutomationElement IsInvokePatternAvailableProperty]

	set patt(invoke) [object invoke InvokePattern Pattern]

	set elem(root) [object invoke -alias AutomationElement RootElement]

	set cond(elan) [object create PropertyCondition $prop(name) "ELAN Instrument Control Session"]
	set elem(elan) [$elem(root) -alias FindFirst Children $cond(elan)]

	set cond(tbar) [object create PropertyCondition $prop(name) "Elan Toolbar"]
	set elem(tbar) [$elem(elan) -alias FindFirst Children $cond(tbar)]

	set cond(invoke) [object create PropertyCondition $prop(invoke) 1]
	set elems(invoke) [$elem(tbar) -alias FindAll Children $cond(invoke)]
	$elems(invoke) Count
}

(8) By mistachkin on 2022-10-13 05:50:24 in reply to 7 [link] [source] 

Please try using the following:

    set boolTrue [object invoke -create System.Boolean Parse true]
    set cond(invoke) [object create PropertyCondition $prop(invoke) $boolTrue]

(9.1) Originally by Yusuke Yamasaki (yyamasak) with edits by mistachkin on 2022-10-14 05:53:13 from 9.0 in reply to 8 [link] [source]

The error disappeared. But finally, I concluded that this GUI doesn't expose the button to UIAutomation.

I don't know why it appears on "%ProgramFiles(x86)%\Windows Kits\10bin\10.0.18362.0\x86\inspect.exe". There might be any other ways to find it in C#.
For this specific GUI, I will take legacy, coordinate based approach using twapi.

eagle {
	set bool(true) [object invoke -create System.Boolean Parse true]
	set prop(invoke) [object invoke AutomationElement IsInvokePatternAvailableProperty]
	set cond(invoke) [object create PropertyCondition $prop(invoke) $bool(true)]
	set elems(invoke) [$elem(elan) -alias FindAll Descendants $cond(invoke)]
	$elems(invoke) Count
}
# => 0

Thank you very much for your advice.
It will help me automate modern GUIs in future.

(10.1) By mistachkin on 2022-10-14 06:11:31 edited from 10.0 in reply to 9.1 [link] [source]

FYI, you can also declare and call native Win32 API functions directly from Eagle, e.g.:

set module [library load kernel32.dll]

set function [library declare -functionname GetStdHandle \
    -returntype IntPtr -parametertypes [list int32] -module $module]

set STD_OUTPUT_HANDLE -11; # constant from "WinBase.h"
set handle [library call $function $STD_OUTPUT_HANDLE]
Fossil 2.15 [eb1a722f4e] 2021-02-07 14:59:40