SSIS Proxy

A browser configured for the HTTP protocol sends client requests not directly, but through a proxy server, which in turn sends them on its own behalf to the destination host. The proxy server here acts as a link between the computer and the requested resource, and the response it immediately sends to the client.

Jsoup is a Java library for working with HTML documents. To scrape links using Jsoup, you can use its selector syntax to target the anchor elements and then extract the href attributes. Here's a simple example:

import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
import org.jsoup.select.Elements;
import java.io.IOException;

public class LinkScraper {
    public static void main(String[] args) {
        String url = "https://example.com";

        try {
            // Connect to the website and get the HTML document
            Document document = Jsoup.connect(url).get();

            // Select all anchor elements
            Elements links = document.select("a");

            // Iterate over each anchor element and print the href attribute
            for (Element link : links) {
                String href = link.attr("href");
                System.out.println("Link: " + href);
            }
        } catch (IOException e) {
            e.printStackTrace();
        }
    }
}

Make sure to replace the url variable with the URL of the website you want to scrape.

This example connects to the specified URL, retrieves the HTML document, selects all anchor elements using the "a" selector, and then iterates over them to print the href attributes.

You need to include the Jsoup library in your project. If you are using Maven, you can add the following dependency to your pom.xml:

    org.jsoup
    jsoup
    1.14.3 

To add a site to proxy exceptions, you need to configure your proxy settings to bypass the proxy for specific domains or websites. The process may vary depending on the browser or operating system you are using. Here, I will provide instructions for popular web browsers:

Google Chrome:
- Open Google Chrome.
- Click on the three dots (⠇) in the top right corner of the Chrome window.
- Select "Settings" from the dropdown menu.
- Scroll down and click on "Advanced" at the bottom of the page.
- Under the "System" section, click on "Open proxy settings."
- In the Windows Settings window, go to the "Exceptions" tab.
- Click on the "Add" button.
- Enter the domain or IP address of the site you want to add to the exceptions list in the "Address" field.
- Click "OK" to save the exception.

Mozilla Firefox:
- Open Mozilla Firefox.
- Click on the three lines (⠇) in the top right corner of the Firefox window.
- Select "Options" or "Preferences" from the dropdown menu.
- Go to the "General" tab, and click on "Settings..." in the "Network Proxy" section.
- In the Connection Settings window, click on "Settings..." under the "Dial-up networking" section.
- In the Internet Properties window, go to the "Security" tab.
- Click on "Restricted Sites" and then "Sites."
- Click on "Add" and enter the domain or IP address of the site you want to add to the exceptions list.
- Click "Close" and then "OK" to save the exception.

In a local network, you will need two computers to do this. One will be used as a proxy server, the other as a client. Then you need to activate the proxy on the server. And on the client PC - choose to access the Internet via a local network connection (i.e. from the server). Another option is to use a web server like Nginx.

In the browser menu (top right corner), find "Settings", and then, under "Network settings", go to "Settings" to select "Manual network configuration". Enter, depending on your network protocol, the IP address, the port and click on "OK". Open any website and in the window that appears, enter the proxy password and login, then click "Ok" again. A successful connection to the site means that the setup is successfully completed.