Automating GUI interaction

[fractionMan]

Put simply I have a program that wants to interact with another via clicking buttons in a (multi) windowed application. It also needs to scrape data from that application to know what to click.

I've heard of autoit but not looked super deeply at it. Anyone have any experience of this?

 

[mauvais]

If you own the code then you're basically looking at automated UI unit testing, for which there are various frameworks.

If you don't, and it's a prebuilt app, then things probably get a bit trickier as it has to try and kludge integration.

So, which? If it's your code, what language? If it's not, any idea what it was built in? What does it run on - Windows?

 

[fractionMan]

If you own the code then you're basically looking at automated UI unit testing, for which there are various frameworks.

If you don't, and it's a prebuilt app, then things probably get a bit trickier as it has to try and kludge integration.

So, which? If it's your code, what language? If it's not, any idea what it was built in? What does it run on - Windows?

It's not my code. It's a 'random' windows application, which could be flash, java or native.

My code is written in java, which doesn't exactly help. One thing I can make it do though is expose all the logic through a web server, which makes using a GUI interaction framework of some sort easier. I can script that, then make calls to the webserver to decide which button to click etc.

 

[Crispy]

This doesn't answer your question in any way, but it's cool so I'm putting it here.

The software I use at work, Revit, logs every single UI interaction in a .journal file. Every single click, drag, button press etc. The journal file's main use is for submitting alongside bug reports or tech support for a corrupted project file. But you can also use it to replay your actions. It means if the app crashes without saving, you can just drop the .journal file onto the app's icon and it replays all your work at lightning spped. It's hypnotic to watch - Zooming in and out, launching new views, creating objects, deleting things. All your trial and error, but at 100x speed.

 

[fractionMan]

One of the real problems I've got is scanning a graphical app and working out where the bloody elements I need to click are. That plus scanning it for the data I need to make a descision.

It's a bloody nightmare which is why I've not done it yet. Anything that makes it easier would be gold.

 

[magneze]

If it's a web application then Selenium (http://seleniumhq.org) is very good. You can make decisions based upon data and the elements clicked on are identified by id rather than screen position.

 

[fractionMan]

If it's a web application then Selenium (http://seleniumhq.org) is very good. You can make decisions based upon data and the elements clicked on are identified by id rather than screen position.

Unfortunately it's not a web application

 

[mauvais]

That is a difficult problem then - it's basically OCR and pattern recognition but in a pretty dynamic environment. Autoit does look like the established tool, but I'd never heard of it before.

Java integration is the least of your worries - for example you can compile Autoit into an executable and run that from Java.

 

[fractionMan]

Java integration is the least of your worries - for example you can compile Autoit into an executable and run that from Java.

Ah, that sounds promising. I was thinking I had to do it the other way round (connect autoit to java somehow)