Proxy for Crawly

To organize multi-threaded scraping through a proxy in C#, you can use the HttpClient class along with tasks and threads. Additionally, you may use proxy rotation to avoid rate limiting and bans. Here's a basic example to get you started:

using System;
using System.Collections.Generic;
using System.Net.Http;
using System.Threading.Tasks;

class Program
{
    static async Task Main()
    {
        // List of proxy URLs
        List proxyList = new List
        {
            "http://proxy1.com:8080",
            "http://proxy2.com:8080",
            // Add more proxies as needed
        };

        // Create HttpClient instances with a different proxy for each thread
        List httpClients = CreateHttpClients(proxyList);

        // List of URLs to scrape
        List urlsToScrape = new List
        {
            "https://example.com/page1",
            "https://example.com/page2",
            // Add more URLs as needed
        };

        // Create tasks for each URL
        List tasks = new List();
        foreach (string url in urlsToScrape)
        {
            tasks.Add(Task.Run(() => ScrapeUrl(url, httpClients)));
        }

        // Wait for all tasks to complete
        await Task.WhenAll(tasks);

        // Dispose of HttpClient instances
        foreach (HttpClient client in httpClients)
        {
            client.Dispose();
        }
    }

    static List CreateHttpClients(List proxies)
    {
        List clients = new List();
        foreach (string proxy in proxies)
        {
            var httpClientHandler = new HttpClientHandler
            {
                Proxy = new WebProxy(proxy),
                UseProxy = true,
            };

            clients.Add(new HttpClient(httpClientHandler));
        }

        return clients;
    }

    static async Task ScrapeUrl(string url, List httpClients)
    {
        // Select a random proxy for this request
        var random = new Random();
        var httpClient = httpClients[random.Next(httpClients.Count)];

        try
        {
            // Make the request using the selected proxy
            HttpResponseMessage response = await httpClient.GetAsync(url);

            // Check if the request was successful
            if (response.IsSuccessStatusCode)
            {
                string content = await response.Content.ReadAsStringAsync();
                // Process the content as needed
                Console.WriteLine($"Scraped {url}: {content.Length} characters");
            }
            else
            {
                Console.WriteLine($"Failed to scrape {url}. Status code: {response.StatusCode}");
            }
        }
        catch (Exception ex)
        {
            Console.WriteLine($"Error scraping {url}: {ex.Message}");
        }
    }
}

In this example:

The CreateHttpClients function creates a list of HttpClient instances, each configured with a different proxy from the provided list.

The ScrapeUrl function performs the actual scraping for a given URL using a randomly selected proxy.

The Main method creates tasks for each URL to be scraped and waits for all tasks to complete.

It depends on the purpose for which you plan to work with proxies at all. Personally, one is enough for myself. But if you plan to do massive parsing, it may not be enough to have 100 pieces.

Such proxy redirects requests from clients to different servers (globally or within a single local network). It can be used for load balancing in different Internet services, for testing web applications, for secured access to local network servers (all "non-client" traffic is ignored).

For Telegram, it is recommended to use paid proxy servers of the Socks5 protocol. These proxies provide the user with data protection and high and stable connection speed. Telegram developers recommend using servers from European countries.

The main task is to monitor traffic on the local network, as all requests will be handled by an organized proxy. Most often it is used to block access to certain resources in offices.