Beanstalk healthd not finding nginx log due to time zone

Question

I have my application environment set up to use my local timezone (*America/New_York*), but the healthd seems to look for the nginx log using the GMT hour. I'm using the stock nginx healthd.conf:

```
if ($time_iso8601 ~ "^(\d{4})-(\d{2})-(\d{2})T(\d{2})") {
    set $year $1;
    set $month $2;
    set $day $3;
    set $hour $4;
}

access_log /var/log/nginx/healthd/application.log.$year-$month-$day-$hour healthd;
```

And these are my current contents of the `/var/log/nginx/healthd` directory:

```
$ ls -l
total 12
-rw-r--r--. 1 nginx nginx  310 May 10 08:42 application.log.---
-rw-r--r--. 1 nginx nginx 1343 May 10 08:56 application.log.2024-05-10-08
-rw-r--r--. 1 nginx nginx   44 May 10 09:01 application.log.2024-05-10-09
```

And these are some relevant lines from the `/var/log/healthd/daemon.log` file:

```
W, [2024-05-10T09:12:42.027170 #4050447]  WARN -- : log file "/var/log/nginx/healthd/application.log.2024-05-10-13" does not exist
W, [2024-05-10T09:12:47.027424 #4050447]  WARN -- : log file "/var/log/nginx/healthd/application.log.2024-05-10-13" does not exist
W, [2024-05-10T09:12:52.027692 #4050447]  WARN -- : log file "/var/log/nginx/healthd/application.log.2024-05-10-13" does not exist
```

Is this a bug w/ healthd or is there a configuration I need to apply somewhere?

It looks like [this was also asked](https://repost.aws/questions/QUME-eX9y8RgaiGFEi0ca8bg/aws-elastic-beanstalk-healthd-uses-wrong-timezone-for-nginx-logs) (but unanswered) a few months ago.

Answer

It's a good idea to keep all logs in UTC to avoid any ambiguity around daylight savings time shifts, leap minutes, etc. That's particularly important when the filenames themselves are timestamped - i.e. you would have a "hole" in your logs when the local time leaps forward, and traffic would appear to double during one hour once it leaps back. Arguably, healthd is following the best practice and is in the clear here.

As per [nginx documentation](https://nginx.org/en/docs/ngx_core_module.html#env),

> By default, nginx removes all environment variables inherited from its parent process except the TZ variable.

but it can be overriden in nginx.conf:

```diff
  user  nginx;
  worker_processes  auto;
+ env TZ=UTC;
```

That would do the trick.

Beanstalk healthd not finding nginx log due to time zone

Relevant content