[Solved] Cron job that pointing to other domain not working

[ipokane]

Hello,

I created some cron jobs using curl.
The cron job that goes to a website on the same server can run, but the cron job that goes to a website on another server does not run.

I see in /var/log/syslog there is a line that goes to another website, but the page on the target website is not accessed (the log is empty)

What is the cause? Thanks in advance

[maurice]

Check the output of the curl command.

[ipokane]

Thanks for the answer.

I didn’t think before to run curl through ssh. Previously I tested by accessing the url directly via browser and it worked normally (hits/logs recorded).

But when I run curl through ssh, it shows 301 Moved Permanently.

I am using htaccess for that page.

The page that cron is pointing to is
https://example.com/cron/cron.php
The actual url is
https://example.com/apps/public/cron/cron.php

Is it because of htaccess? And how to fix it?

[sahsanu]

@ipokane

But when I run curl through ssh, it shows 301 Moved Permanently.

Use option -L in curl to follow redirections.

curl -L "https..."

[ipokane]

After I investigated, the 301 Moved Permanently seemed to be due to the security module feature on my website which considered it a bad bot. After I disabled the module, I tried curl via ssh and it ran normally. But the cronjob didn’t seem to run even though there was a record in the syslog.

I know the cron job isn’t running, because the log of the targeted page doesn’t record a visit from the server (hit). However, when I tried curl from ssh, the log recorded a visit (hit)

Here it shows that there is a cron history, but it doesn’t seem to actually access the target page (there is no log recorded on the target page as when I tried curl via ssh)

Oddly enough, the cron accessing the website on the same server runs normally. (the 3rd line)

[ipokane]

Hi,

My problem has been solved.

What happened was, the cron job actually ran, but there was no hit on the target web. This happened because there was a certain line in my .htaccess file that prevented curl from accessing the web page.

After I edited that line, the web page can be accessed by curl from the cron job normally.

Here is the line I edited:
originally:

RewriteCond %{HTTP_USER_AGENT} (havij|libwww-perl|wget|python|nikto|curl|scan|java|winhttp|clshttp|loader) [NC,OR]

became:

RewriteCond %{HTTP_USER_AGENT} (havij|libwww-perl|wget|python|nikto|scan|java|winhttp|clshttp|loader) [NC,OR]

I removed curl from the blocked user agent.

And this line helps me in debugging.

curl -L "https..."

Thank you @sahsanu & @maurice