Dusk Proxies

When performing web scraping with authorization in Python, you typically need to simulate the login process of a user by sending the necessary authentication data (such as username and password) to the website. The exact steps depend on the authentication method used by the website, and there are several common approaches

Basic Authentication (using requests library)

If the website uses HTTP Basic Authentication, you can include the authentication credentials in the request headers using the requests library.

import requests

url = 'https://example.com/data'
username = 'your_username'
password = 'your_password'

response = requests.get(url, auth=(username, password))

if response.status_code == 200:
    # Successfully authenticated, you can now parse the content
    print(response.text)
else:
    print(f"Failed to authenticate. Status code: {response.status_code}")

Form-Based Authentication

For websites that use form-based authentication (login form), you need to send a POST request with the appropriate form data.

import requests

login_url = 'https://example.com/login'
data = {
    'username': 'your_username',
    'password': 'your_password',
}

# Use a session to persist the authentication across requests
with requests.Session() as session:
    response = session.post(login_url, data=data)

    if response.status_code == 200:
        # Authentication successful, continue with subsequent requests
        data_url = 'https://example.com/data'
        data_response = session.get(data_url)
        print(data_response.text)
    else:
        print(f"Failed to authenticate. Status code: {response.status_code}")

OAuth Authentication

For websites using OAuth, you might need to use an OAuth library like requests_oauthlib or oauthlib to handle the OAuth flow.

Handling Cookies

Sometimes, authentication is maintained using cookies. In such cases, you need to handle cookies in your requests.

import requests

login_url = 'https://example.com/login'
data = {
    'username': 'your_username',
    'password': 'your_password',
}

# Use a session to persist the authentication across requests
with requests.Session() as session:
    login_response = session.post(login_url, data=data)

    if login_response.status_code == 200:
        # Authentication successful, continue with subsequent requests
        data_url = 'https://example.com/data'
        data_response = session.get(data_url)
        print(data_response.text)
    else:
        print(f"Failed to authenticate. Status code: {login_response.status_code}")

Simply, in the connection properties of your PC or mobile device, you need to enter the data of the proxy server through which you will be connecting. In Windows, for example, this is done through "Settings", then "Network and Internet", and in the next window you should open the tab "Proxy server".

On smartphones, when a proxy is turned on, the corresponding indicator (the "VPN" icon) appears in the status bar. In Windows you have to go to "Settings", open "Network and Internet". Under "Proxy Server", if the item "Manual" is activated, it means that the proxy is engaged right now.

In the Windows Settings menu, go to "Network and Internet". At the very bottom, on the left side, find the item "Proxy server" and uncheck it so that it is no longer used. It is also desirable to uncheck the item "Automatic detection of parameters" in the section "Automatic configuration". If this is not done, there is a chance that the proxy will continue to be used. Reboot your laptop.

A proxy is responsible for forwarding traffic. Technically, it just copies the traffic and sends it to the Internet, but it also replaces various metadata (the type of equipment from which the request is sent, the port number, the IP address, and so on). Or it can be simply called a "mediator" in the computer network.