GitHub - tebelorg/TagUI: General purpose tool for automating web interactions
source link: https://github.com/tebelorg/TagUI
Go to the source link to view the article. You can view the picture content, updated content and better typesetting reading experience. If the link is broken, please click the button below to view the snapshot at that time.
TagUI
Why This | Set Up | To Use | Cheat Sheet | Developers | Credits | Video
TagUI is a command-line tool for digital process automation (RPA)
- automate Chrome in visible / invisible mode
- visual automation of websites and desktop
- write in 20+ human languages & JavaScript
- Chrome extension for recording web actions
- Python & R integrations for machine learning
Why This
The goal of UI (user interface) automation is to reproduce cognitive interactions that you have with websites or your desktop, so that your computer can do it for you, base on your schedule or conditions. TagUI helps you rapidly automate repetitive or time-critical tasks - use cases include digital process automation, data acquisition and testing apps.
Read on for more info or jump right to the flow samples section to see examples of TagUI automation in natural-language-like syntax, which makes automation easy to maintain. This is a full-feature and free open-source tool, so there's nothing to upgrade to or any paid subscription. To feedback suggestions or bugs, raise an issue on GitHub.
Click to show differences between TagUI open-source RPA and commercial RPA software
HOW IT WORKS
TagUI converts your intentions from over 20 human languages to lines of working JavaScript code that perform UI automation. Under the hood, it uses Chrome DevTools Protocol, SikuliX, CasperJS, PhantomJS & SlimerJS.
// sample TagUI flow to log into Typeform and download a survey report
https://www.typeform.com
click login
type username as [email protected]
type password as 12345678
click btnlogin
download https://admin.typeform.com/xxx to report.csv
// besides web element identifiers, images of the elements can be used
click login_button.png
type username_box.png as [email protected]
// in addition (x,y) coordinates of user-interface elements can be used
click (1200,200)
type (800,400) as [email protected]
Click to show powerful and user-friendly features of TagUI that come right out of the box
Set Up
TagUI is in v5.11 - it unzips and runs directly on Windows, macOS, Linux (link to release notes)
PACKAGED INSTALLATION
- TagUI is easy to use right away, in most environments all dependencies are already packaged in
- To use visual automation on desktop or browser, be sure to have OpenJDK v8 (64-bit) or later
| Platform | macOS | Linux | Windows | Node.js (macOS/Linux) |
|---|---|---|---|---|
| Package | unzip and run | unzip and run | unzip and run | npm install tagui |
Recommended locations to unzip to
- Windows - c:\ or desktop
- macOS - your desktop
- Linux - /home/your_userid
Avoid spaces in the folder path as some components of TagUI don't work well with spaces in folder and file names. Optionally, configure web browser settings in tagui_config.txt, such as browser resolution, step timeout of 10s etc.
Tip - to try cutting edge version with the latest features, download master.zip to overwrite your existing packaged installation. Be sure to manually select and move the folders & files inside master.zip's TagUI-master/src folder to replace your existing tagui/src folder, some OSes will delete existing target folders that are missing from source folder.
MANUAL INSTALLATION
Click to show step-by-step setup if you prefer to download dependencies manually from their websites
To Use
COMMAND LINE
tagui flow_filename option(s) for Windows and ./tagui flow_filename option(s) for macOS/Linux
- Flow filename can be a local file or the URL of an online file
- Filename can have no extension, .txt or .js or .tagui extension
- Type tagui without parameters to see its version and options
By default, log files (.log .raw .js) are created after running. To switch off, put an empty file tagui_no_logging in tagui/src folder. If your command prompt or terminal font size is too small, you can set it to much larger font sizes for easier reading. Below example runs a script to search on Yahoo website and capture screenshots of the results.
Windows - unzip the tagui folder to c:\. Open command prompt with Start Menu -> type run -> type cmd and enter
c:
cd c:\tagui\src
tagui samples\1_yahoo chrome
macOS - unzip the tagui folder to your desktop. Open terminal from Apps -> Utilities -> Terminal and enter commands
cd /Users/your_userid/Desktop/tagui/src
./tagui samples/1_yahoo chrome
Linux - unzip the tagui folder to a convenient folder on your laptop for eg /home/your_userid and enter commands
cd /home/your_userid/tagui/src
./tagui samples/1_yahoo chrome
If the script works successfully, you will notice 5 .png files - congratulations, you have run your first TagUI script!
Troubleshoot OS-specific exceptions
- For Windows computers, if you see 'MSVCR110.dll is missing' error, install this from Microsoft website (choose vcredist_x86.exe) - this file is required to run the Windows PHP engine packaged with TagUI. Some IT policies restrict TagUI from writing to c:\tagui and working properly, in that case please unzip to user desktop folder.
- For newer macOS versions, if you get a 'dyld: Library not loaded' error, install OpenSSL in this way. macOS Catalina update has introduced tighter security controls, see solutions for the PhantomJS and Java popups.
- For some flavours of Linux (Ubuntu for example) which do not have PHP pre-installed, google how to install PHP accordingly (eg Ubuntu, apt-get install php). Most Linux distributions would already come with PHP.
- If you Ctrl+C to break a TagUI automation, you can use tagui/src/end_processes command (for macOS/Linux) or end_processes.cmd (for Windows) to kill any dead processes of TagUI integrations (Chrome, SikuliX, Python etc)
Troubleshoot browser-specific exceptions
- Google Chrome browser has to be installed by user before trying to automate Chrome browser. TagUI creates a separate Chrome browser user profile for use in automation, it is stored in tagui\src\chrome\tagui_user_profile.
- Make sure TagUI's Chrome browser is set to 100% zoom. As TagUI mimics the user mouse-clicks at the (x,y) coordinates of web elements, using a different zoom level will cause clicks to be triggered at wrong locations.
- For Microsoft Edge or Internet Explorer, TagUI can only automate them using visual automation, keyboard and mouse automation. There is no backend integration to automate through web element identifiers like Chrome or Firefox.
- For Firefox automation, download an older version here (SlimerJS doesn't work with Firefox v60 onwards).
More details on TagUI command-line usage
Click to show the command line options for TagUI tool and their purposes (eg chrome, headless, report) Click to show info on automation logs (.log, .js, .raw), and how to run tagui from any directory Click to show how to run TagUI scripts by double-clicking as desktop icons
BY SCHEDULING
To schedule an automation flow with crontab (for macOS/Linux), for example at 8am daily
0 8 * * * /full_path/tagui/src/tagui flow_filename option(s)
For Windows, use Task Scheduler (search schedule from Start Menu) or something like Z-Cron
CHROME EXTENSION
Download from Chrome Web Store to use TagUI Chrome web browser extension for recording automation flows. TagUI Chrome extension records steps such as page navigation, clicking web elements and entering information.
Click to show details on TagUI Chrome extension - based on https://github.com/ebrehault/resurrectio
NATIVE LANGUAGES
To run TagUI flows in human languages or output flow execution in other languages (see demo run)
- set your default flow language with tagui_language variable in tagui_config.txt
- write automation flow in human language base on language definition .csv files
- optionally set tagui_language in flow to any other languages as output language
Click to show the 20+ human languages supported by TagUI and how to self-build language definitions
VISUAL AUTOMATION
TagUI has built-in integration with SikuliX (base on OpenCV) to allow visually identifying the web elements and desktop UI (user interface) elements for interaction by providing their images or (x,y) coordinates. Applicable steps are click, hover, type, select, read, show, save, snap, keyboard, mouse. To use visual automation, OpenJDK v8 (64-bit) or later is needed.
Click to show how to download and check for Java, how to use visual automation & OCR, and a demo video
PYTHON INTEGRATION
TagUI has built-in integration with Python (works out of the box for both v2 & v3) - a programming language with many popular frameworks for big data and machine learning. The py step can be used to run commands in Python and retrieve the output of those commands. To use Python integration in TagUI, first download Python for your OS. macOS and Linux normally come pre-installed with Python. Make sure that python command is accessible from command prompt.
Click to show how to use py step in your automation flow to send and receive data from Python frameworks
R INTEGRATION
TagUI has built-in integration with R - an open-source software environment for statistical computing and graphics. R can be used for big data and machine learning. The r step can be used to run commands in R and retrieve the output of those commands. To use R integration in TagUI, first download R software for your OS. Make sure that Rscript command is accessible from your command prompt (added to path or symbolically linked).
Click to show how to use r step in your automation flow to send and receive data from R frameworks
CLI ASSISTANT
TagUI scripts are already in natural-language-like syntax to convert to JavaScript code. What's even better is having natural-language-like syntax on the command line. Instead of typing tagui download_bank_report june creditcard to run the automation flow download_bank_report with parameters june creditcard, you can type erina download my june creditcard bank report. For a demo of the CLI (command-line interface) assistant in action, see this video.
Click to show details on how you can rename your CLI assistant and the syntax used to invoke automations
TAGUI WRITER, SCREENSHOTER & EDITOR
TagUI Writer is a Windows app created by Arnaud Degardin / @adegard which makes it easy to write TagUI scripts. By pressing Ctrl + Left-click, a popup menu will appear with the list of TagUI steps for you to paste into your text editor. Arnaud also created a ScreenShoter app which makes it easy to capture snapshots for TagUI visual automation. Lastly, TagUI Editor allows you to edit and run TagUI scripts via AutoHotKey. To download, click here.
Click to show a preview of how TagUI Editor can be used to edit and run TagUI scripts
TAGUI FOR PYTHON
TagUI for Python is a Python package created by TagUI's creator Ken Soh / @kensoh. It brings the digital process automation capabilities of TagUI directly to the Python environment through a simple, expressive and powerful API. To install, pip install tagui. For its Python API and architecture, refer to the project homepage. At merely ~1k lines of code, the package is built on TagUI and its architecture is based on integration with TagUI's live mode.
FLOW SAMPLES
TagUI includes the following automation flow samples (tagui/src/samples folder)
| Flow Sample | Purpose |
|---|---|
| 1_yahoo | searches github on Yahoo and captures screenshot of results |
| 2_twitter | goes to a Twitter page and saves some profile information |
| 3_github | goes to a GitHub page and downloads the repository file |
| 4_conditions | goes through examples of using conditions in natural language |
| 5_repositories | shows using repositories on Russian social media site VK.com |
| 6_datatables | set of flows uses datatables to retrieve and act on GitHub info |
| 7_testing | shows how to use check step assertions for CI/CD integration |
| 8_hastebin | used by upload option to upload flow result to hastebin.com |
| 9_misc | shows how to use steps popup, frame, dom, js, { and } block |
| a_facedetect | uses face recognition to detect profile images on webpages |
| b_visualoutlook | uses visual recognition for desktop MS Outlook email sending |
| c_chineseflow | run flow in other languages (first, change src/tagui_config.txt) |
Cheat Sheet
AUTOMATION WORKFLOW
- What happens behind the scenes when you run an automation flow (TagUI architecture diagram)
FIND XPATH OF WEB ELEMENT
- In Chrome browser, right-click on the element, click Inspect, right-click on HTML code block, then
STEPS DESCRIPTION
- TagUI will auto-wait for an UI element to appear and interacts with it as soon as it appears
- Element identifier can be auto-recorded using TagUI Chrome extension, or found from web browser
- TagUI auto-selects provided identifier in this order - xpath, css, id, name, class, title, aria-label, text(), href
| Basic Step | Parameters (separator in bold) | Purpose |
|---|---|---|
| http(s):// | just enter full url of webpage (`variable` for variable) | go to specified webpage |
| click | element to click | click on an element |
| rclick | element to right-click | right-click on an element |
| dclick | element to double-click | double-click on an element |
| hover | element to hover | move cursor to element |
| type | element as text ([enter] = enter, [clear] = clear field) | enter element as text |
| select | element to select as option value ([clear] = clear selection) | choose dropdown option |
| read | element to read (page = webpage) to variable name | fetch element text to variable |
| show | element to read (page = webpage, ie raw html) | print element text to output |
| save | element (page = webpage) to optional filename | save element text to file |
| snap | element (page = webpage) to optional filename | save screenshot to file |
| snap (pdf) | page to filename.pdf (headless Chrome / PhantomJS) | save webpage to basic pdf |
| load | filename to variable name | load file content to variable |
| echo | text (in quotation marks) and variables | print text/variables to output |
| dump | text (in quotation marks) and variables to optional filename | save text/variables to file |
| write | text (in quotation marks) and variables to optional filename | append text/variables to file |
| variable_name | = value (for text, put in quotes, use + to concat) | define variable variable_name |
| // (on new line) | user comments (ignored during execution) | add user comments |
| ask | question or instruction for user (reply stored in ask_result) | ask user for input |
| live | try steps or code interactively for Chrome / visual automation | enter live mode (Firefox not yet) |
Tip - to use variables where text is expected, `variable` can be used. XPath is an expressive way to identify web elements. If you know xpath and use xpath for element identifier, use double quotes for text //*[@title="Login"]
Click to show pro steps - tagui, keyboard, mouse, table, wait, check, upload, api, run, dom, js, r, py, vision, timeout
CONDITIONS EXAMPLES
Click to show how conditions and loops can be expressed in natural language or JavaScript
HELPER FUNCTIONS
Click to show csv_row(), present(), visible(), count(), clipboard(), url(), title(), text(), timer(), mouse_xy() functions
MODULES
Click to show how you can use tagui step to call other TagUI automation flows within an automation flow file
REPOSITORIES
Click to show how you can use repositories csv to make objects reusable and improve readability
DATATABLES
Click to show how datatable csv files can be used to perform batch automation at scale
AUDIT & REPORTING
Click to show how the report option can be used to track and store automation results
Developers Reference
TagUI excels in automating user-interface interactions. It's designed to make prototyping, deployment and maintenance of UI automation easier by minimizing iteration time for each phase. Originally developed by a frustrated test automation engineer, to maximize automation code output with minimal coding efforts when automating web interactions.
Click to show how to use API webservice that comes with TagUI and how to make complex outgoing API calls
: : :::CHROME
Click to show how TagUI has native integration with visible and headless Chrome using DevTools Protocol
TESTING
Click to show details of how TagUI can be used for test automation with integration to CI tools
FILES
Click to show details of the different files used by TagUI and their purposes
SECURITY
Click to show details of security concerns and features of TagUI
Credits
- TagUI - Ken Soh from Singapore / @kensoh
- SikuliX - Raimund Hocke from Germany / @RaiMan
- CasperJS - Nicolas Perriault from France / @n1k0
- PhantomJS - Ariya Hidayat from Indonesia / @ariya
- SlimerJS - Laurent Jouanneau from France / @laurentj
License
TagUI is open-source software released under Apache 2.0 license
Recommend
About Joyk
Aggregate valuable and interesting links.
Joyk means Joy of geeK