selenium

fijiaaron

What is Selenium? Does it support multiple users?

Selenium is a programming tool used to automate the browser — simulating a real user by opening windows, going to URLs, clicking buttons, filling in forms, etc. Selenium is primarily used for test automation, but can also be used for other things.

Selenium is available as a library for most popular programming languages, including Java, JavaScript, TypeScript, C#, Python, Ruby, PHP, etc.

It is also referred to as Selenium WebDriver, because there were two different projects (Selenium & WebDriver) which did similar things and eventually merged. Selenium uses the WebDriver protocol (a W3C standard now) to communicate over the network via a REST API. This allows for remote automation,

There are other tools associated with Selenium, including Selenium Grid — which enables remote execution of automation and Selenium IDE — which allows you to record and play back automated steps without writing code, and has (limited) ability to to export from Selenium IDE to code that can run independent.

Selenium IDE does not support multiple users, but scripts can be exported and shared from one user to another.

Selenium Grid allows for parallel remote execution of Selenium scripts, which allows multiple people (or tests) to execute tests at the same time.

The concept of “supporting multiple users” does not really make sense in terms of Selenium as an open source development tool or coding library.

It would be like saying:
“Does Microsoft Word support multiple users?” or
“Does the Java programming language support multiple users?”

In the case of Microsoft Word, every user that has the program can use it, but collaboration is (primarily) done outside of the tool. With proprietary software like Microsoft Word, each user may need a license to run their own copy of the application, but Selenium is open source, so does not require the purchase of any license to use.

And as a programming library, any number of users can reference Selenium in their own code and execute it. Users can run multiple automated tests (or other scripts) at once — if they write their program to run in parallel.

But in order to maximize parallel execution (for single or multiple users) you need to have a remote Selenium grid. There is an open source grid that anyone can deploy, but there are also commercial services the host Selenium Grid with additional tools at a cost. These companies include Sauce Labs, BrowserStack, LambdaTest, and Applitools. Each company has their own policy about multiple users, and of the ones I mentioned, they all support multiple users in their own way.

This post is based on a question asked on Quora at: https://www.quora.com/What-is-Selenium-Does-it-support-multiple-users

fijiaaron

How to get cryptocurrency prices using Python (and various tools)

Here’s another post based on a question from Quora:

Can I export the contents of an HTML table to Excel or MySQL via Selenium/Python?

No, you can’t export a table from HTML to Excel or MySQL using Selenium with Python.

But you’re in luck! You just asked the wrong question.

It’s like if you had asked: “Can I build a bookshelf using my DeWalt Table Saw?”

The answer is, of course, still no, but a reasonable person would say that your table saw is one of the tools you could use, but you’re also going to need a hammer, and some nails — or a screwdriver and screws — and maybe some paint or varnish. And a couple clamps to hold it all to together while you’re building it.

Unfortunately, I am not a reasonable person. But I’ll show you how to do it anyway.

You can use Selenium with Python to extract content from an HTML table. You can then use other tools (with Python) to import that content into an Excel spreadsheet or MySQL database.

For example, I’ll fetch cryptocurrency prices from CoinMarketCap using Selenium:

# get the HTML table data using Selenium
from selenium import webdriver

url = "https://coinmarketcap.com/"
table_locator = "xpath", "//table"

driver = webdriver.Chrome()
driver.get(url)

table_webelement = driver.find_element(*table_locator)
table_html = table_webelement.get_attribute("outerHTML")

In addition to Selenium, I’d recommend using Pandas DataFrames to export to Excel — both because it’s easier than working with openpyxl directory (or xlwt, or xlsxwriter, or one of several other lower level libraries) and because pandas has all sorts of other great data manipulation features that might come in handy. (And it looks great on a resume!)

Here’s how you can use Python to read an HTML table directly into a Pandas Dataframe and then export it to a Microsoft Excel Spreadsheet using DataFrame.to_excel()

# load the HTML table to Pandas DataFrame
import pandas

dataframes = pandas.read_html(table_html)

# get the first and only table on the page
table_dataframe = dataframes[0] 

# export data to Excel
table_dataframe.to_excel("data.xlsx")

Here is the resulting spreadsheet:

Excel Spreadsheet with Most Recent Cryptocurrency Pricing

Or you can export the Dataframe to a database.

Here, we use MySQL Connector with SQL Alchemy to append our results from the HTML table to a table named “prices” into the MariaDB “test” database. If the table does not exist, it creates it.

Using Pandas Datataframe.to_sql()

# export data to MySQL (or MariaDB)
import sqlalchemy
from mysql import connector

conn = sqlalchemy.create_engine("mysql+mysqlconnector://username:password@localhost/test")

table_dataframe.to_sql(con=conn, name="prices", if_exists='append', index=False)

And our resulting table will look like this:

MYSQL Table with Most Recent Cryptocurrency Pricing

There are also other tools you may be able to use:

requests HTTP library instead of Selenium if you don’t need to manipulate the browser to fetch data

# get prices using requests
import requests

response = requests.get("https://coinmarketcap.com/")

BeautifulSoup4 to parse the HTML into a List of Lists

# parse data using Beauitful Soup
from bs4 import BeautifulSoup

soup = BeautifulSoup(response.content, "html.parser")
table_soup = soup.find("table")

headings = [th.text.strip() for th in table_soup.find_all("th")]

rows = []
for tr in table_soup.find_all('tr', limit=11):
    row = []
    for td in tr.find_all('td'):
        row.append(td.text.strip())
    rows.append(row)

csv to parse into a CSV instead of Excel format (a CSV file can also be loaded directly into MySQL without Pandas).

#export data to csv file
import csv 
 
with open("data.csv", mode="w") as csvfile: 
    writer = csv.writer(csvfile) 
    writer.writerow(headings) 
    writer.writerows(rows)

Of course , you can also load your data List into pandas as well:

table_dataframe = pandas.DataFrame(rows)

Here is a gist showing all the code together:

pip install selenium
pip install pandas
pip install openpyxl
pip install sqlalchemy
pip install mysql-connector-python
pip install beautifulsoup4
pip install requests
view raw dependencies hosted with ❤ by GitHub
# get prices using requests
import requests
response = requests.get("https://coinmarketcap.com/")
# parse data using Beauitful Soup
from bs4 import BeautifulSoup
soup = BeautifulSoup(response.content, "html.parser")
table_soup = soup.find("table")
headings = [th.text.strip() for th in table_soup.find_all("th")]
rows = []
for tr in table_soup.find_all('tr', limit=11):
row = []
for td in tr.find_all('td'):
row.append(td.text.strip())
rows.append(row)
# save data using CSV
import csv
with open("data.csv", mode="w") as csvfile:
writer = csv.writer(csvfile)
writer.writerow(headings)
writer.writerows(rows)
view raw get_prices_using_requests_beautiful_soup_and_csv.py hosted with ❤ by GitHub
from selenium import webdriver
url = "https://coinmarketcap.com/"
table_locator = "xpath", "//table"
driver = webdriver.Chrome()
driver.get(url)
table_webelement = driver.find_element(*table_locator)
table_html = table_webelement.get_attribute("outerHTML")
# load the HTML table to Pandas DataFrame
import pandas
dataframes = pandas.read_html(table_html)
table_dataframe = dataframes[0] # get the first and only table on the page
# export data to Excel
table_dataframe.to_excel("data.xlsx")
# export data to MySQL (or MariaDB)
import sqlalchemy
from mysql import connector
conn = sqlalchemy.create_engine("mysql+mysqlconnector://username:password@localhost/test")
table_dataframe.to_sql(con=conn, name="prices", if_exists='append', index=False)
view raw get_prices_using_selenium_and_pandas.py hosted with ❤ by GitHub
fijiaaron

When should you use JavaScriptExecutor in Selenium?

When you want to execute JavaScript on the browser :)

This was my answer to a question on Quora

https://www.quora.com/When-should-I-use-JavaScriptExecutor-in-Selenium-WebDriver/answer/Aaron-Evans

JavaScriptExecutor is an interface that defines 2 methods:

in Java (and similarly in C#):

Object executeScript(String script, Object... args)

and

Object executeAsyncScript(String script, Object... args)

which take as an argument a string representing the JavaScript code you want to execute on the browser and (optionally) one or more arguments. If the second argument is a WebElement it will apply the script to the corresponding HTML element. Arguments are added to the JS magic arguments variable which represents the values passed to a function. If the code executed returns a value, that is returned to your Selenium code

Each driver is responsible for implementing it for the browser.

RemoteWebDriver implements it as well.

But when *you* as a Selenium user want to use JavaScriptExecutor is when you assign a driver to the base type WebDriver, which does not implement it.

in this case, you cast your driver instance (which really does implement executeScript() and executeScriptAsync().

For example

WebDriver driver = new ChromeDriver();  

// base type ‘WebDriver’ does not define executeScript()  although our instance that extends RemoteWebDriver actually does implement it.  

// So we need to cast it to ‘JavaScriptExecutor’ to let the Java compiler know.  

JavaScriptExecutor js = (JavaScriptExecutor) driver;  

js.executeScript(“alert(‘hi from Selenium’);”

if you keep your instance typing, you do not need to cast to JavaScriptExecutor.

RemoteWebDriver driver = new RemoteWebDriver(url, capabilities);  

// information about our type is not lost so the Java compiler knows our object implements executeScript()  

WebElement element = driver.findElement(By.id(“mybutton”));  

driver.executeScript(“arguments[0].click();", element);  

// in the above case it adds the element to arguments and performs a click() event (in JavaScript in the browser) on our element

String htmlsnippet = driver.executeScript(“return document.querySelector(‘#myid’).outerHTML” , element);  

// this time we use native JavaScript on the browser to find an element and return its HTML, bypassing Selenium’s ability to do so.

The above two examples illustrate ways you can accomplish in JavaScript what you would normally use Selenium for.

Why would you do this?

Well, sometimes the driver has a bug, or it can be more efficient (or reliable) to do in JavaScript, or you might want to combine multiple actions in 1 WebDriver call.

fijiaaron

Scheduling tests to monitor websites

If you have access to your crontab you can set a Selenium script to run periodically. If you don’t have cron, you can use a VM (with Vagrant) or Container (with Docker) to get it.

Cron is available on Linux & Unix systems. On Windows, you can use Task Scheduler. On Mac, there is launchd, but it also includes cron (which wraps launchd).

You could also set up a job to run on a schedule using a continuous integration server such as Jenkins. Or write a simple, long running script that runs in the background and sleeps between executions.

I have a service that runs Selenium tests and monitoring for my clients, and use both cron and Jenkins for executing test runs regularly. I also have event-triggered tasks that can be triggered by a checkin or user request.

Each line represents a task with schedule in the following format:

#minute   #hour     #day      #month    #weekday  #command

# perform a task every weekday morning at 7am
*         7         *         *         1-5       wakeup.sh

# perform a task every hour
@hourly python selenium-monitor.py

You can edit crontab to create a task by typing crontab -e

You can view your crontab by typing crontab -l

If you just want to repeat your task within your script while it’s running, you can add a sleep statement and loop (either over an interval or until you kill the script).

#!/usr/bin/env python

from time import sleep
from selenium import webdriver

sites = ['https://google.com', 'https://bing.com', 'https://duck.com']

interval = 60 #seconds
iterations = 10 #times

def poll_site(url):
	driver = webdriver.Chrome()
	driver.get(url)
	title = driver.title
	driver.quit()
	return title

while (iterations > 0):
	for url in sites:
		print(poll_site(url))
	sleep(interval)
	iterations -= 1

See the example code on github:

#!/usr/bin/env python
from time import sleep
from selenium import webdriver
sites = ['https://google.com', 'https://bing.com', 'https://duck.com']
interval = 60 #seconds
iterations = 10 #times
def poll_site(url):
driver = webdriver.Chrome()
driver.get(url)
title = driver.title
driver.quit()
return title
while (iterations > 0):
for url in sites:
print(poll_site(url))
sleep(interval)
iterations -= 1
view raw selenium-monitor.py hosted with ❤ by GitHub

Originally posted on Quora:

https://www.quora.com/How-can-I-schedule-simple-website-test-scripts-Selenium-to-run-regularly-like-Cron-jobs-and-notify-me-if-it-fails-for-free/answer/Aaron-Evans-56

fijiaaron

Weekly Wednesday Webinar on Selenium & Sauce Labs

I’ve been working at Sauce Labs for a while now, helping enterprise users build test automation frameworks and implement continuous integration using Selenium & Sauce Labs.

In order to reach a larger audience — and to learn more about people’s challenges developing test automation — I’m going to be hosting a weekly webinar on using Selenium with Sauce Labs for test automation.

So, starting this week, each Wednesday during lunch (12:30pm Mountain Time) I’ll host a webinar / office hours.  I’ll begin with a brief presentation introducing the topic, followed by a demo (live coding — what could go wrong?), and then open it up for questions & comments.

The first webinar will be tomorrow at 12:30pm MST.  The topic is DesiredCapabilities.

I’ll talk about what desired capabilities are, how to use desired capabilities with Sauce Labs, and show how you can use the  Sauce Labs platform configurator to generate desired capabilities.  I’ll also talk about Sauce Labs specific capabilities used to report on tests and builds.

Register on EventBrite here: Selenium & Sauce Labs Webinar: Desired Capabilities

Join the WebEx directly: Selenium & Sauce Labs Webinar: Desired Capabilities

Contact me if you’d like a calendar invite or more info.