storing data retrieved from REST endpoint into variables in python

Question

I'm trying to retrieve data from SEC EDGAR database, I get a valid endpoint url, for example: https://data.sec.gov/submissions/CIK0000320193.json (this is the endpoint for Apple Inc) but when I try to store the results in a variable I get an empty list: results = [] and when I save results in a txt file... well... the file is obviously empty, here's the chunk of code that handles it:

def get_sec_data(cik):
    try:
        # Define the SEC data.gov API endpoint for fetching filings in JSON format
        endpoint = f"https://data.sec.gov/submissions/CIK{cik}.json"
# Print the API endpoint
        print("API Endpoint:", endpoint)

        # Define the headers with the required User-Agent
        headers = {
            'User-Agent': "dom dom [email protected]",
            'Accept-Encoding': 'gzip, deflate'
        }

     
        # Make a GET request to the SEC data.gov API with the specified headers
        response = requests.get(endpoint, headers=headers)

        # Check if the request was successful (status code 200)
        if response.status_code != 200:
            raise Exception(f"Failed to fetch SEC data. Status: {response.status_code}")

        # Parse the JSON response
        json_data = response.json()

        # Extract the XBRL data for the latest 10-K and 10-Q filings
        filings_data = json_data.get('hits', [])

        return filings_data

    except Exception as e:
        raise Exception(f"Error: {str(e)}")

And then I get the result and try to save it

# Retrieve the last 10-K and 10-Q filings for the specified company in JSON format
try:
    results = get_sec_data(cik)
    # Display the results or handle errors
    for result in results:
        if "Error" in result:
            print(result)
        else:
            print("Successfully retrieved data from SEC data.gov API.")
            # You can now use the variable 'results' for further processing.

    # Specify the file name without a path
    file_name = "test.txt"

    # Construct the full file path by joining it with the current working directory
    file_path = "/users/domdom/desktop/Project/" + file_name

    # Open the file in write mode
    with open(file_path, 'w') as file:
        # Iterate over the results and write each entry to the file
        for entry in results:
            file.write(str(entry) + '\n')

    print(f"Data has been written to {file_path}")

except Exception as e:
    print(e)

What do I get wrong? I tried different checks to see if endpoint link I obtain is correct and if I retrieve data from SEC website and it all works when I open the link in browser, I also tried saving a csv instead of a txt, same thing: empty. Seems like the problem is in the storing and then saving content from the endpoint.

Answer 1

What are you trying to do on this line: filings_data = json_data.get('hits', []) ?

It appears that you are trying to get the value of the property hits from the JSON data returned from EDGAR, but as far as I can tell there is no such property. Try getting a different property (like tickers, for example) and you should get better results.