Missing XML files in 990 bucket

[elijahgoldberg]

Hi there — I am seeing a large number of missing XML files, and wondering whether it is something on my fault or something in the S3 instance. Of 300 objects tested, 80% (241) threw a 404 error:

In download.file(url, file_path, quiet = TRUE)
cannot open URL 'https://s3.amazonaws.com/irs-form-990/201723259349300000_public.xml': HTTP status was '404 Not Found'

When I try to access one of these files through the browser, I see this message:

NoSuchKey The specified key does not exist. 201700379349300224_public.xml CFAMJNK6P9V903JZ TkCHSmCi+R7gHMceu2ZZp3jIrEGWT3IQlmAXn28iLU7S0pfltJ/Bvz+TZeGhhhBkEOT95EI3BQA=

3 sample keys that are in the index and return XML

• https://s3.amazonaws.com/irs-form-990/201743189349310624_public.xml
• https://s3.amazonaws.com/irs-form-990/201743189349310944_public.xml
• https://s3.amazonaws.com/irs-form-990/201703149349302400_public.xml

5 sample keys that are in the index but return 404

• https://s3.amazonaws.com/irs-form-990/201703189349302176_public.xml
• https://s3.amazonaws.com/irs-form-990/201703149349302624_public.xml
• https://s3.amazonaws.com/irs-form-990/201743199349311168_public.xml

For context, I tried a similar exercise about 2 years ago and anecdotally remember the occasional missing XML file, but not close to 80%.

Any ideas about what is going on? Thanks!

[jflasher]

Hi @elijahgoldberg, thanks for reporting this. The data ingestion pipeline is owned by the IRS and we (AWS) do not have any elevated access to the account or pipelines. From looking at your description, it does look like the index is out of sync with the actual files though. You can reach out to the IRS via the Contact Us methods at the bottom of https://www.irs.gov/e-file-providers/e-file-for-charities-and-non-profits.

We will also try to highlight this issue with them. Thanks again!

cc/ @pschmied