python

Aaron

How long does it take to learn Selenium and Python

This is another post that grew out of a question asked by someone else online.

Here is the original question (and my answer) on Quora:

https://www.quora.com/How-long-does-it-take-to-learn-selenium-Webdriver-with-Python

There are several different things here, and they affect how long it will take you to learn Selenium with Python.

Let’s break it down:

  1. Learning Selenium
  2. Learning Python
  3. Learning programming
  4. Learning test automation

Your existing knowledge in each of these topics will affect how easy it is.

For example, if you’ve already used Selenium with another programming language, that means that (to some degree) you also know programming and test automation principles. So all you need to learn is Python.

But if you have some programming experience in JavaScript but have never done test automation (with or without Selenium) or used Python, you come with general programming knowledge (but using a different paradigm) there are more obstacles.

Alternately, you may have experience with test automation so you understand the goals, but have used a commercial low-code record and playback automation tool. This may actually be harder than starting from a programming background because it requires a paradigm shift in your strategy to test automation.

However, probably most people asking this question have some experience manually testing software, a basic knowledge of programming (either in Python, or some other language — but would not consider themselves expert), and want to know how long it would take them to become competent enough with Selenium and Python to either:

A. Get a job doing test automation with Selenium and Python or
B. Apply test automation with Selenium and Python in their current job
(which may be manual testing, or some other role).

So, I’ll try to answer this question.

Give yourself a few weeks to learn the basics of Python, general programming principles, and the “Pythonic” idioms. A good course on Udemy or book about Python is about what you need. Subtract if you already understand some of this, if you’re a fast learner, or have guidance (such as a mentor.)

But it’s not really about how quickly you can absorb knowledge, it’s practice to retain it, and having the time to make mistakes and experiment.

And then it will only take a week or two to pick up Selenium. It has a fairly straightforward API, and the concepts of automating finding buttons and clicking on them is fairly simple to understand. There are some obstacles that can trip you up though — things like provisioning and configuring a WebDriver, managing sessions, “getting” locators like XPath and CSS, data driven, using Pytest and fixtures, etc can trip you up and lead to hours (or days) of frustration — or you can establish bad habits that will bite you in the future.

But applying Selenium in a meaningful way to write useful test automation may take additional weeks, or months (or years) of practice. Again, this depends on your personal curiosity, cleverness, mentoring opportunities, and above all, practice and ability to apply it in the real world.

These challenges fall under writing good maintainable test automation, knowing what to test and how to approach it, and even little things like picking good locators, or naming functions and variables well.

If you were smart, and scrolled to the bottom, short answer is:

It should only take you a few weeks to pick up Python and Selenium, depending on where you are coming from experience-wise and how fast you learn. Having mentors and or guided learning can help you focus on what’s important and get past silly obstacles. But like anything, it will take a much longer time to master, and depend on real world experience applying the principles, making mistakes, and learning from them.

A reasonable time to go from basic programming knowledge and no test automation experience to competent enough to make an impact at a job (e.g. get hired as a junior test automation engineer) is a few months of steady learning and practice.

Aaron

How to get cryptocurrency prices using Python (and various tools)

Here’s another post based on a question from Quora:

Can I export the contents of an HTML table to Excel or MySQL via Selenium/Python?

No, you can’t export a table from HTML to Excel or MySQL using Selenium with Python.

But you’re in luck! You just asked the wrong question.

It’s like if you had asked: “Can I build a bookshelf using my DeWalt Table Saw?”

The answer is, of course, still no, but a reasonable person would say that your table saw is one of the tools you could use, but you’re also going to need a hammer, and some nails — or a screwdriver and screws — and maybe some paint or varnish. And a couple clamps to hold it all to together while you’re building it.

Unfortunately, I am not a reasonable person. But I’ll show you how to do it anyway.

You can use Selenium with Python to extract content from an HTML table. You can then use other tools (with Python) to import that content into an Excel spreadsheet or MySQL database.

For example, I’ll fetch cryptocurrency prices from CoinMarketCap using Selenium:

# get the HTML table data using Selenium
from selenium import webdriver

url = "https://coinmarketcap.com/"
table_locator = "xpath", "//table"

driver = webdriver.Chrome()
driver.get(url)

table_webelement = driver.find_element(*table_locator)
table_html = table_webelement.get_attribute("outerHTML")

In addition to Selenium, I’d recommend using Pandas DataFrames to export to Excel — both because it’s easier than working with openpyxl directory (or xlwt, or xlsxwriter, or one of several other lower level libraries) and because pandas has all sorts of other great data manipulation features that might come in handy. (And it looks great on a resume!)

Here’s how you can use Python to read an HTML table directly into a Pandas Dataframe and then export it to a Microsoft Excel Spreadsheet using DataFrame.to_excel()

# load the HTML table to Pandas DataFrame
import pandas

dataframes = pandas.read_html(table_html)

# get the first and only table on the page
table_dataframe = dataframes[0] 

# export data to Excel
table_dataframe.to_excel("data.xlsx")

Here is the resulting spreadsheet:

Excel Spreadsheet with Most Recent Cryptocurrency Pricing

Or you can export the Dataframe to a database.

Here, we use MySQL Connector with SQL Alchemy to append our results from the HTML table to a table named “prices” into the MariaDB “test” database. If the table does not exist, it creates it.

Using Pandas Datataframe.to_sql()

# export data to MySQL (or MariaDB)
import sqlalchemy
from mysql import connector

conn = sqlalchemy.create_engine("mysql+mysqlconnector://username:password@localhost/test")

table_dataframe.to_sql(con=conn, name="prices", if_exists='append', index=False)

And our resulting table will look like this:

MYSQL Table with Most Recent Cryptocurrency Pricing

There are also other tools you may be able to use:

requests HTTP library instead of Selenium if you don’t need to manipulate the browser to fetch data

# get prices using requests
import requests

response = requests.get("https://coinmarketcap.com/")

BeautifulSoup4 to parse the HTML into a List of Lists

# parse data using Beauitful Soup
from bs4 import BeautifulSoup

soup = BeautifulSoup(response.content, "html.parser")
table_soup = soup.find("table")

headings = [th.text.strip() for th in table_soup.find_all("th")]

rows = []
for tr in table_soup.find_all('tr', limit=11):
    row = []
    for td in tr.find_all('td'):
        row.append(td.text.strip())
    rows.append(row)

csv to parse into a CSV instead of Excel format (a CSV file can also be loaded directly into MySQL without Pandas).

#export data to csv file
import csv 
 
with open("data.csv", mode="w") as csvfile: 
    writer = csv.writer(csvfile) 
    writer.writerow(headings) 
    writer.writerows(rows)

Of course , you can also load your data List into pandas as well:

table_dataframe = pandas.DataFrame(rows)

Here is a gist showing all the code together:

pip install selenium
pip install pandas
pip install openpyxl
pip install sqlalchemy
pip install mysql-connector-python
pip install beautifulsoup4
pip install requests
view raw dependencies hosted with ❤ by GitHub
# get prices using requests
import requests
response = requests.get("https://coinmarketcap.com/")
# parse data using Beauitful Soup
from bs4 import BeautifulSoup
soup = BeautifulSoup(response.content, "html.parser")
table_soup = soup.find("table")
headings = [th.text.strip() for th in table_soup.find_all("th")]
rows = []
for tr in table_soup.find_all('tr', limit=11):
row = []
for td in tr.find_all('td'):
row.append(td.text.strip())
rows.append(row)
# save data using CSV
import csv
with open("data.csv", mode="w") as csvfile:
writer = csv.writer(csvfile)
writer.writerow(headings)
writer.writerows(rows)
view raw get_prices_using_requests_beautiful_soup_and_csv.py hosted with ❤ by GitHub
from selenium import webdriver
url = "https://coinmarketcap.com/"
table_locator = "xpath", "//table"
driver = webdriver.Chrome()
driver.get(url)
table_webelement = driver.find_element(*table_locator)
table_html = table_webelement.get_attribute("outerHTML")
# load the HTML table to Pandas DataFrame
import pandas
dataframes = pandas.read_html(table_html)
table_dataframe = dataframes[0] # get the first and only table on the page
# export data to Excel
table_dataframe.to_excel("data.xlsx")
# export data to MySQL (or MariaDB)
import sqlalchemy
from mysql import connector
conn = sqlalchemy.create_engine("mysql+mysqlconnector://username:password@localhost/test")
table_dataframe.to_sql(con=conn, name="prices", if_exists='append', index=False)
view raw get_prices_using_selenium_and_pandas.py hosted with ❤ by GitHub