Parse Dynamic Power BI table with selenium

15 Views Asked by At

This is an extension of another stack overflow question. The problem I have is that I can't properly parse the data. The column headers are missing the first 10 columns. Only columns you have to scroll to see appear in my parsed columns headers output. Also, the rows I parse are only a fraction of total rows. I tried a few ways to extract the entire table but fell very short each time.

I've included how I tried to parse the dynamic table as the other code is copied from the above answer.


# find the desired 2nd table
table = driver.find_elements(By.CSS_SELECTOR, 'div.tableExContainer')[1]


# now you can parse this desirable as you want.
column_header = table.find_elements(By.CSS_SELECTOR, 'div[role="columnheader"]')

# Initialize a list to hold the text of each column header
header_text = []

for header in column_header:
    # Extract and append the text from each header
    header_text.append(header.text)

# Now 'header_texts' contains the text of all column headers
print(header_text)

#rows
rows = table.find_elements(By.CSS_SELECTOR, 'div[role="row"]')
row_texts = []

for row in rows:
    cells = row.find_elements(By.CSS_SELECTOR, 'div[role="gridcell"]')
    row_texts.append([cell.text for cell in cells])

# Ensure you press Enter twice here if in an interactive shell

import csv

# Define the path to your CSV file
csv_file_path = 'extracted_rows.csv'

# Open a new CSV file to write into
with open(csv_file_path, 'w', newline='', encoding='utf-8') as csvfile:
    # Create a CSV writer object
    csvwriter = csv.writer(csvfile)

This is the outputted csv:
My Outputted CSV Table

My help would be much appreciated.

0

There are 0 best solutions below