Как получить ответ через POST с помощью CURL?

491

18 января 2017, 03:11

Здравствуйте! Нужно получить данные праздников, которые приходят с сервера через Ajax. Сайт https://www.investing.com/holiday-calendar/ Задача: спарсить календарь. Я спарсил только ту часть, которая видна при загрузке, а вот получить на весь год не получается.

Нужно нажать на календарь, выбрать дату. После этого, нужно проскроллить страницу вниз, чтобы отправился post запрос на сервер.

Я сохраняю куки, задаю те же параметры (см скриншоты), но ответ не могу получить.

ini_set('error_reporting', E_ALL);
ini_set('display_errors', 1);
ini_set('display_startup_errors', 1);
define('DIR', __DIR__);
$user_agent = isset($_SERVER['HTTP_USER_AGENT']) ? $_SERVER['HTTP_USER_AGENT'] : 'Traider';
$cookie = dirname(__FILE__).DIRECTORY_SEPARATOR.'init_cookie.txt';
$f = fopen('init_deb.txt', 'w');
$ch = curl_init();
$getUrl = 'https://www.investing.com/holiday-calendar/';
$postUrl = 'https://www.investing.com/holiday-calendar/Service/getCalendarFilteredData';
$dateFrom='2017-01-14';
$dateTo='2017-12-31';
$limit_from = 0;
$params = [
    'dateFrom' => $dateFrom,
    'dateTo' => $dateTo,
    'county' => '',
    'limit_from' => $limit_from
];
curl_setopt($ch, CURLOPT_URL, $postUrl);
curl_setopt($ch, CURLOPT_POSTFIELDS, $params);
curl_setopt($ch, CURLOPT_HEADER, 1);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_COOKIEFILE, $cookie);
curl_setopt($ch, CURLOPT_COOKIEJAR, $cookie);
curl_setopt($ch, CURLOPT_USERAGENT, $user_agent);
curl_setopt($ch, CURLOPT_VERBOSE, true);
curl_setopt($ch, CURLOPT_STDERR, $f);
$response = curl_exec($ch);
fclose($f);
curl_close($ch);
echo $response;

После отправки post запроса получаю главную страницу.

Сначала делаю get чтобы получить куки, заголовки, затем post, меняя только $getUrl на $postUrl. Куки сохраняются в файл, а вот заголовки - не знаю. Пробовал сохранить заголовки в файл, а потом добавлять через CURLOPT_HTTPHEADER , но безуспешно. Результат пост запроса

* Trying 185.94.84.100... 
* Connected to www.investing.com (185.94.84.100) port 443 (#0) 
* ALPN, offering http/1.1 
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH 
* successfully set certificate verify locations: 
* CAfile: c:/openserver/modules/php/PHP-7-x64/cacert.pem 
CApath: none 
* NPN, negotiated HTTP1.1 
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384 
* ALPN, server did not agree to a protocol 
* Server certificate: 
* subject: OU=Domain Control Validated; CN=*.investing.com 
* start date: Jan 18 09:48:38 2016 GMT 
* expire date: Jan 18 09:48:38 2019 GMT 
* subjectAltName: host "www.investing.com" matched cert's "*.investing.com" 
* issuer: C=US; ST=Arizona; L=Scottsdale; O=GoDaddy.com, Inc.; OU=certs.godaddy.com/repository/; CN=Go Daddy Secure Certificate Authority - G2 
* SSL certificate verify ok. 
> POST /holiday-calendar/Service/getCalendarFilteredData HTTP/1.1 
Host: www.investing.com 
User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36 OPR/42.0.2393.94 
Accept: */* 
Cookie: PHPSESSID=ec8j7t662mqidip4chg5fl9vq6; StickySession=id.18267366337.742www.investing.com; geoC=RU; gtmFired=OK; nyxDorf=YGcyYDVkYSNkNGFrM34wMzFlPmMxKDc3YmNuag%3D%3D; adBlockerNewUserDomains=1484480915 
Content-Length: 453 
Expect: 100-continue 
Content-Type: multipart/form-data; boundary=------------------------5773b84c8c410fd5 
 
< HTTP/1.1 100 Continue 
< HTTP/1.1 301 Moved Permanently 
< Server: nginx/1.10.2 
< Date: Sun, 15 Jan 2017 11:55:25 GMT 
< Content-Type: text/html; charset=utf-8 
< Content-Length: 0 
< Connection: keep-alive 
< X-Powered-By: PHP/5.4.43 
< Expires: Thu, 19 Nov 1981 08:52:00 GMT 
< Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0 
< Pragma: no-cache 
* Added cookie ses_id="deleted" for domain www.investing.com, path /, expire 1 
< Set-Cookie: ses_id=deleted; expires=Thu, 01-Jan-1970 00:00:01 GMT; path=/; domain=www.investing.com 
< Location: https://www.investing.com/ 
< Vary: User-Agent 
< Accept-Ranges: bytes 
< X-Varnish: 1244608783 
< Age: 0 
< Via: 1.1 varnish 
< Content-Security-Policy: upgrade-insecure-requests 
* HTTP error before end of send, stop sending 
<  
* Closing connection 0 
* Issue another request to this URL: 'https://www.investing.com/' 
* Switch from POST to GET 
* Hostname www.investing.com was found in DNS cache 
* Trying 185.94.84.100... 
* Connected to www.investing.com (185.94.84.100) port 443 (#1) 
* ALPN, offering http/1.1 
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH 
* successfully set certificate verify locations: 
* CAfile: c:/openserver/modules/php/PHP-7-x64/cacert.pem 
CApath: none 
* SSL re-using session ID 
* NPN, negotiated HTTP1.1 
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384 
* ALPN, server did not agree to a protocol 
* old SSL session ID is stale, removing 
* Server certificate: 
* subject: OU=Domain Control Validated; CN=*.investing.com 
* start date: Jan 18 09:48:38 2016 GMT 
* expire date: Jan 18 09:48:38 2019 GMT 
* subjectAltName: host "www.investing.com" matched cert's "*.investing.com" 
* issuer: C=US; ST=Arizona; L=Scottsdale; O=GoDaddy.com, Inc.; OU=certs.godaddy.com/repository/; CN=Go Daddy Secure Certificate Authority - G2 
* SSL certificate verify ok. 
> GET / HTTP/1.1 
Host: www.investing.com 
User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36 OPR/42.0.2393.94 
Accept: */* 
Cookie: PHPSESSID=ec8j7t662mqidip4chg5fl9vq6; StickySession=id.18267366337.742www.investing.com; geoC=RU; gtmFired=OK; nyxDorf=YGcyYDVkYSNkNGFrM34wMzFlPmMxKDc3YmNuag%3D%3D; adBlockerNewUserDomains=1484480915 
 
< HTTP/1.1 200 OK 
< Server: nginx/1.10.2 
< Date: Sun, 15 Jan 2017 11:55:26 GMT 
< Content-Type: text/html; charset=utf-8 
< Content-Length: 305967 
< Connection: keep-alive 
< X-Powered-By: PHP/5.4.43 
< Expires: Thu, 19 Nov 1981 08:52:00 GMT 
< Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0 
< Pragma: no-cache 
* Added cookie ses_id="deleted" for domain www.investing.com, path /, expire 1 
< Set-Cookie: ses_id=deleted; expires=Thu, 01-Jan-1970 00:00:01 GMT; path=/; domain=www.investing.com 
* Replaced cookie nyxDorf="OT4xY2Y3NnQ0ZDw2NXg1Nj9rYj8wKTU1PD01MQ%3D%3D" for domain www.investing.com, path /, expire 1484484926 
< Set-Cookie: nyxDorf=OT4xY2Y3NnQ0ZDw2NXg1Nj9rYj8wKTU1PD01MQ%3D%3D; expires=Sun, 15-Jan-2017 12:55:26 GMT; path=/ 
< Vary: Accept-Encoding,User-Agent 
< Accept-Ranges: bytes 
< X-Varnish: 1244608919 
< Age: 0 
< Via: 1.1 varnish 
< Content-Security-Policy: upgrade-insecure-requests 
<  
* Connection #1 to host www.investing.com left intact

Помогите разобраться как отправить post и получить правильный ответ.

Answer 1

Нужно добавить:

curl_setopt($ch, CURLOPT_HTTPHEADER, array('X-Requested-With:XMLHttpRequest'));

Тем самым оповестим сервер, что это Ajax.

$ch = curl_init();
$cookieFile = __DIR__ . '/cookie';
curl_setopt($ch, CURLOPT_URL, 'https://www.investing.com/holiday-calendar/Service/getCalendarFilteredData');
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_USERAGENT, $_SERVER['HTTP_USER_AGENT']);    
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_COOKIEFILE, $cookieFile);
curl_setopt($ch, CURLOPT_COOKIEJAR, $cookieFile);
curl_setopt($cu, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_POSTFIELDS, array('dateFrom' => '2017-01-15',
                       'dateTo' => '2017-04-16',
                       'country' => '',
                       'limit_from'=>'0'));
curl_setopt($ch, CURLOPT_HTTPHEADER, array(
'X-Requested-With:XMLHttpRequest'
));
$r = curl_exec($ch);
echo '<pre>';
print_r(json_decode($r, true));