from requests_html import HTMLSession session = HTMLSession() r = session.get('http://www.yourjspage.com') r.html.render() # this call executes the js in the page
As a bonus this wraps
BeautifulSoup, I think, so you can do things like
which returns the content of the HTML element as you’d expect.
However, if you want to use Selenium, I recommend Chrome over PhantomJS. Many users have problems with PhantomJS where a website simply does not work in Phantom. Chrome can be headless (non-graphical) too!
First, make sure you have installed ChromeDriver, which Selenium depends on for using Google Chrome.
Then, make sure you have Google Chrome of version 60 or higher by checking it in the URL chrome://settings/help
Now, all you need to do is the following code:
from selenium.webdriver.chrome.options import Options from selenium import webdriver chrome_options = Options() chrome_options.add_argument("--headless") driver = webdriver.Chrome(chrome_options=chrome_options)
If you do not know how to use Selenium, here is a quick overview:
driver.get("https://www.google.com") #Browser goes to google.com
Use either the ELEMENTS or ELEMENT method. Examples:
driver.find_element_by_css_selector("div.logo-subtext") #Find your country in Google. (singular)
- driver.find_element(s)_by_css_selector(css_selector) # Every element that matches this CSS selector
- driver.find_element(s)_by_class_name(class_name) # Every element with the following class
- driver.find_element(s)_by_id(id) # Every element with the following ID
- driver.find_element(s)_by_link_text(link_text) # Every with the full link text
- driver.find_element(s)_by_partial_link_text(partial_link_text) # Every with partial link text.
- driver.find_element(s)_by_name(name) # Every element where name=argument
- driver.find_element(s)_by_tag_name(tag_name) # Every element with the tag name argument
Ok! I found an element (or elements list). But what do I do now?
Here are the methods you can do on an element elem:
- elem.tag_name # Could return button in a .
- elem.get_attribute(“id”) # Returns the ID of an element.
- elem.text # The inner text of an element.
- elem.clear() # Clears a text input.
- elem.is_displayed() # True for visible elements, False for invisible elements.
- elem.is_enabled() # True for an enabled input, False otherwise.
- elem.is_selected() # Is this radio button or checkbox element selected?
- elem.location # A dictionary representing the X and Y location of an element on the screen.
- elem.click() # Click elem.
- elem.send_keys(“thelegend27”) # Type thelegend27 into elem (useful for text inputs)
- elem.submit() # Submit the form in which elem takes part.
- driver.back() # Click the Back button.
- driver.forward() # Click the Forward button.
- driver.refresh() # Refresh the page.
- driver.quit() # Close the browser including all the tabs.
its a wrapper around pyppeteer or smth? 🙁 i thought its something different
async def browser(self): if not hasattr(self, "_browser"): self._browser = await pyppeteer.launch(ignoreHTTPSErrors=not(self.verify), headless=True, args=self.__browser_args) return self._browser