Tag: python

Website Automation for Complete Noobs Part 2

In the first article in the series, we installed Python & Selenium and used them to write a script that opens a browser. In today’s article, we’ll look at how to do some basic inspection of web pages and modify our script to interact with them.

Selenium can do most of the same things you can do with web pages–namely, clicking & typing. But, just like you read and interpret what you see to know what to do, you need to tell your script how to find and do the things you want it to.

The easiest way to do this is to use the “inspect” functionality that exists in most modern browsers to help find identifiers you can use in your code. Let’s try it out with everybody’s ol’ buddy Google. Browse to google.com, right-click the search box, and pick Inspect.

This will take you to the selected element in the web page’s code. This is the part where you start to get some choices. Selenium has a lot of different ways to find elements, so you need to know about what you can use and compare it to what’s available. Usually the most reliable way to get an element is by using its id.

Usually, using an element’s id property along with Selenium’s find_element_by_id method is the most reliable way to find the element, but we don’t see an id when we look at Google’s search box.

We can see a name property with the value q, though, and there’s a find_element_by_name in Selenium, too; so let’s use that! With the element selected, you can use Selenium’s send_keys method to simulate typing. Add the following lines to your script:

from selenium import webdriver

driver = webdriver.Chrome('chromedriver.exe')
driver.get('https://www.google.com')
search = driver.find_element_by_name('q')
search.send_keys('test search')

When you run this script, the browser will open and send the specified text to the search box. Now we need to submit the search. Google has a search button, so we can repeat the steps to find the button and click it:

button = driver.find_element_by_name('btnK')
button.click()

But, that’s not really how you search with Google, right? No–you type your search and press enter. To simulate this, you can use the submit function on the search element:

from selenium import webdriver

driver = webdriver.Chrome('chromedriver.exe')
driver.get('https://www.google.com')
search = driver.find_element_by_name('q')
search.send_keys('test search')
search.submit()

Run the script again, and you’ll see your search submitted and search results displayed. Cool stuff! Let’s do a quick review of all the tools we have now:

  • Browse to sites using webdriver.get
  • Find elements using webdriver.find_element methods
  • Enter text with webdriver.send_keys
  • Click things with webdriver.click
  • Submit forms–like you would do by pressing enter on your keyboard after entering data–with webdriver.submit

Website Automation For Complete Noobs Part 1

Image by mohamed Hassan from Pixabay

Have you ever been doing something on the internet and been like, “Ugh. I wish there was a way to do this automatically.” Well, most of the time there is: you can build some website automation using Python and Selenium. In part 1 of this series, I’ll walk you through installing Python and Selenium, and we’ll write a short script that opens a browser.

Install Python

For folks that don’t know, “Python is a scripting language that lets you work quickly and integrate systems more effectively.” (python.org) That means you write code in files and run the files without having to compile and deploy. In order to run the files, though, you need to install Python on your computer.

One way to do this is to download the installer from the official Python website, here. However, I prefer to use an application called PyCharm. PyCharm is an IDE (integrated development environment) available for Windows, Linux, and Mac that gives you lots of nice helpers when writing scripts, and there’s a very nice free community edition that’s just perfect for people like you who are just getting started.

So that’s step 1 for us today: download and install PyCharm Community Edition. Installation is pretty straightforward; you can just Next through everything and accept the defaults.

Once the install is complete, you can launch PyCharm. The first time you run it, PyCharm will ask you about which theme you prefer. Pick what you like and click Skip Remaining and Set Defaults.

Create Project

PyCharm is ready to go, but we need to do a little more setup before we’re ready to write our code. Begin by creating a new project from the PyCharm launch screen.

Give your project a name to specify where it will live on your computer, and click Create. If you don’t already have Python installed (the actual programming language) at this point, PyCharm will install it for you.

Install Selenium + Chromedriver

Selenium WebDriver is what we’ll use to launch and control the browser in our script. In order to use it, we need to install the libraries. We can install it directly from PyCharm by running a command in the terminal.

Open the terminal by clicking the button.

Now run the following command:

pip install selenium

Selenium also needs an extra application to help control the browser. In this example, I want to use Chrome as my browser, so I need to download ChromeDriver from here. When you visit the ChromeDriver page, it offers a few different versions. You need to pick the version based on the version of Chrome you have installed. This can be found in Chrome’s About page.

Once you determine which version you need, download the Zip file for your OS and extract chromedriver.exe to your project directory. You can verify that it’s in the right spot because it’ll be visible in PyCharm.

Write Script

Flip up those hoodies because it’s time to code. Add a new Python file to your project by right-clicking the project folder.

Add the following code to your script:

from selenium import webdriver

driver = webdriver.Chrome('chromedriver.exe')
driver.get('https://www.google.com')

The first line tells our script that webdriver comes from the Selenium library that we installed. Then, it creates a variable named driver (but we could name it anything) that’s a Selenium WebDriver object using the ChromeDriver executable we downloaded. The third line uses driver to browse to www.google.com. Nothing too crazy, right?

Run Script

Go back to your terminal and run your script like this:

python <your-script>.py

When you press enter, a Chrome browser should open and navigate to the URL specified in your script. Chrome also gives an indicator that the browser is being controlled by automation software.

Pretty cool, right? There’s not much to experiment with this script. You can change the URL or add more driver.get lines to visit multiple sites. In the next article in the series, we’ll look at how to use WebDriver to interact with web sites and do things.

python – Selenium: FirefoxProfile exception Can’t load the profile – Stack Overflow

I ran into a problem last week where Selenium python scripts were just opening a blank Firefox window and doing nothing. Running this from the command line fixed it:

pip install -U selenium

via python – Selenium: FirefoxProfile exception Can’t load the profile – Stack Overflow.

Getting Started with Selenium in Python

I just spent a day doing some browser automation with Selenium in Python, and man-oh-man is it fun! I was surprised at how quickly I was able to get setup and productive, and I thought I’d put together a quick getting started guide to celebrate.

Installation

If you’re starting from scratch (no Python), then the first thing you’ll want to do is install Python. Head on over to python.org and grab a copy. If you install Python 2, you’ll also want to install pip. (Python 3 ships with pip, so it is not necessary to install it yourself.)

With Python and pip, installing Selenium is done with a single command:

pip install selenium

That’s it, now you’re ready to get to automatin’!

Hello World

Open your favorite text editor and create a new file. The first thing you’ll do is import the selenium.webdriver module. Then, you’ll instantiate a webdriver. And then you’ll automate whatever you want!

Here is a script that goes to google and types “hello world” into the box.

from selenium import webdriver
from selenium.webdriver.common.keys import Keys

driver = webdriver.Firefox()
driver.get("http://www.google.com")
driver.find_element_by_id("gbqfq").send_keys("hello world")
driver.close()

Development

I only have one day’s worth of knowledge as I’m writing this, but what I’ve found to work well is to use Chrome’s right-click > Inspect element function for each element that I need to find or interact with and type it into my Python script.

For example, if I wanted to automate entering data into a form, I’d inspect the element to find its id or other identifiable trait (name, css), then type it into my script.

Here’s how you might automate logging into a website:

<form>
    <input id="username" type="text"></input>
    <button id="login-submit" type="submit">Submit</button>
</form>

driver.find_element_by_id("username").clear()
driver.find_element_by_id("username").send_keys("admin")
driver.find_element_by_id("login-submit").click()

And here’s how you might validate that the login was successful:

<div class="banner">
    <span>Welcome, admin</span>
</div>

assert "Welcome, admin" in driver.find_element_by_css_selector("div.banner span").text

Super easy, super fun! Want to learn more? Start here!