How to remediate – Common Files Detection

1. Introduction

2. Technical Explanation

The vulnerability occurs when unreferenced files remain in the web root directory. Attackers use techniques like forced browsing, making requests for common filenames to identify these files. These files can reveal framework versions or internal application logic. There is no specific CVE associated with this general detection method. An attacker could simply request a file named ‘robots.txt’ and if present, gain insight into the web server structure.

• Exploit mechanism: Attackers use wordlists to enumerate common filenames and directories via HTTP requests.
• Scope: All web servers hosting applications built with common frameworks (Apache, Nginx).

3. Detection and Assessment

Confirming the vulnerability involves checking for the presence of known sensitive files. A quick check is to browse common file locations directly. Thorough assessment requires automated scanning.

• Quick checks: Use a web browser to access URLs like /robots.txt, /.git/HEAD, and /server-status.
• Scanning: Nessus plugin ID 10385 can detect common files. Burp Suite’s Intruder can be used with a wordlist of common filenames.
• Logs and evidence: Web server access logs may show requests for these files, indicating reconnaissance activity.

curl -I http://example.com/robots.txt

4. Solution / Remediation Steps

The solution is to remove unreferenced files from the web root and application directory. Access control can provide an additional layer of security.

4.1 Preparation

• Ensure you have a list of expected files and directories for the application. A rollback plan is to restore from backup.
• Change windows may be needed depending on system criticality; approval from IT security may be required.

4.2 Implementation

1. Step 1: Identify and remove unreferenced files from the web root directory.
2. Step 2: Review application code to ensure no files are unexpectedly referenced.
3. Step 3: Verify that removing these files does not break any functionality.

4.3 Config or Code Example

Before

/var/www/html/robots.txt

After

(File removed)

4.4 Security Practices Relevant to This Vulnerability

Practices that directly address this vulnerability include least privilege and regular file system audits.

• Practice 1: Least privilege – restricting access to the web root directory limits potential damage if files are exposed.
• Practice 2: Regular File System Audits – Regularly review the contents of the web root directory for unexpected or unreferenced files.

4.5 Automation (Optional)

A script can be used to identify and remove common sensitive files. Use caution when automating file deletion.

#!/bin/bash
# Caution: This script deletes files. Test thoroughly before use!
find /var/www/html -name "robots.txt" -delete
find /var/www/html -name ".git" -type d -delete

5. Verification / Validation

Verify the fix by confirming that sensitive files are no longer accessible via HTTP requests. Perform a smoke test to ensure application functionality remains intact.

• Post-fix check: curl -I http://example.com/robots.txt should return a 404 Not Found error.
• Re-test: Re-run the quick checks from Section 3 to confirm files are no longer present.
• Smoke test: Verify that core application features (login, search, etc.) still function as expected.
• Monitoring: Monitor web server access logs for requests to common filenames; unexpected requests may indicate reconnaissance activity.

curl -I http://example.com/robots.txt

6. Preventive Measures and Monitoring

Update security baselines to include checks for unreferenced files. Implement CI/CD pipeline scans to detect these files during deployment.

• Baselines: Update a security baseline or policy to require regular removal of unreferenced files.
• Asset and patch process: Review web server configurations regularly to ensure appropriate access controls are in place.

7. Risks, Side Effects, and Roll Back

• Risk or side effect 1: Removing a required file may break application functionality; restore from backup if this occurs.
• Roll back: Restore the web root directory from the pre-change backup.

8. References and Resources

• Vendor advisory or bulletin: http://projects.webappsec.org/w/page/13246953/Predictable%20Resource%20Location
• NVD or CVE entry: Not applicable for this general detection method.
• Product or platform documentation relevant to the fix: http://httpd.apache.org/docs/2.0/mod/mod_access.html, http://nginx.org/en/docs/http/ngx_http_access_module.html