Convert the USE_LDAP=yes to USES=ldap and adds the following features:
- Adds the argument USES=ldap:server to add openldap2{4|5|6}-server as
RUN_DEPENDS
- Adds the argument USES=ldap<version> and replaces WANT_OPENLDAP_VER
- Adds OPENLDAP versions in bsd.default-versions.mk
- Adds USE_OPENLDAP/WANT_OPENLDAP_VER in Mk/bsd.sanity.mk
- Changes consumers to use the features
Reviewed by: delphij
Approved by: portmgr
Differential Revision: https://reviews.freebsd.org/D38233
During an exp-run for llvm 15 (see bug 265425), it turned out that
mail/dovecot failed to build with clang 15:
ioloop-notify-kqueue.c:70:2: error: call to undeclared function 'i_gettimeofday'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
i_gettimeofday(&ioloop_timeval);
^
ioloop-notify-kqueue.c:70:2: note: did you mean 'gettimeofday'?
/usr/include/sys/time.h:617:5: note: 'gettimeofday' declared here
int gettimeofday(struct timeval *, struct timezone *);
^
...
--- test-mail-index-transaction-update.o ---
test-mail-index-transaction-update.c:633:14: warning: comparison of function 'timezone' equal to a null pointer is always false [-Wtautological-pointer-compare]
test_assert(timezone == 0);
^~~~~~~~ ~
../../src/lib-test/test-common.h:20:8: note: expanded from macro 'test_assert'
if (!(code)) test_assert_failed(#code, __FILE__, __LINE__); \
^~~~
test-mail-index-transaction-update.c:633:14: note: prefix with the address-of operator to silence this warning
test_assert(timezone == 0);
^
&
../../src/lib-test/test-common.h:20:8: note: expanded from macro 'test_assert'
if (!(code)) test_assert_failed(#code, __FILE__, __LINE__); \
^
test-mail-index-transaction-update.c:648:42: warning: arithmetic on a pointer to the function type 'char *(int, int)' is a GNU extension [-Wgnu-pointer-arith]
hdr.day_stamp = tests[i].old_day_stamp + timezone;
^ ~~~~~~~~
test-mail-index-transaction-update.c:648:17: error: incompatible pointer to integer conversion assigning to 'uint32_t' (aka 'unsigned int') from 'char *(*)(int, int)' [-Wint-conversion]
hdr.day_stamp = tests[i].old_day_stamp + timezone;
^ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
test-mail-index-transaction-update.c:650:49: warning: arithmetic on a pointer to the function type 'char *(int, int)' is a GNU extension [-Wgnu-pointer-arith]
mail_index_update_day_headers(t, tests[i].now + timezone);
^ ~~~~~~~~
test-mail-index-transaction-update.c:650:36: error: incompatible pointer to integer conversion passing 'char *(*)(int, int)' to parameter of type 'time_t' (aka 'long') [-Wint-conversion]
mail_index_update_day_headers(t, tests[i].now + timezone);
^~~~~~~~~~~~~~~~~~~~~~~
./mail-index-transaction-private.h:127:77: note: passing argument to parameter 'day_stamp' here
void mail_index_update_day_headers(struct mail_index_transaction *t, time_t day_stamp);
^
test-mail-index-transaction-update.c:654:63: warning: arithmetic on a pointer to the function type 'char *(int, int)' is a GNU extension [-Wgnu-pointer-arith]
test_assert_idx(new_hdr.day_stamp == tests[i].new_day_stamp + timezone, i);
^ ~~~~~~~~
../../src/lib-test/test-common.h:26:9: note: expanded from macro 'test_assert_idx'
if (!(code)) test_assert_failed_idx(#code, __FILE__, __LINE__, i); \
^~~~
test-mail-index-transaction-update.c:654:37: warning: comparison between pointer and integer ('uint32_t' (aka 'unsigned int') and 'char *(*)(int, int)') [-Wpointer-integer-compare]
test_assert_idx(new_hdr.day_stamp == tests[i].new_day_stamp + timezone, i);
~~~~~~~~~~~~~~~~~ ^ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
../../src/lib-test/test-common.h:26:9: note: expanded from macro 'test_assert_idx'
if (!(code)) test_assert_failed_idx(#code, __FILE__, __LINE__, i); \
^~~~
Fix these warnings and errors by importing:
e983ead7751a7b1f66fe867a37fa7b
PR: 268204
Approved by: maintainer timeout (2 weeks)
MFH: 2022Q4
mail/dovecot-pigeonhole: bump to 0.5.20
mail/dovecot-fts-*, mail/dovecot-coi: bump portrevision
mail/dovecot: ChanggLog:
+ Add dsync_features=no-header-hashes. When this setting is enabled and
one dsync side doesn't support mail GUIDs (i.e. imapc), there is no
fallback to using header hashes. Instead, dsync assumes that all mails
with identical IMAP UIDs contains the same mail contents. This can
significantly improve dsync performance with some IMAP servers that
don't support caching Date/Message-ID headers.
+ lua: HTTP client has more settings now, see
https://doc.dovecot.org/admin_manual/lua/#dovecot.http.client
+ replicator: "doveadm replicator status" command now outputs when the
next sync is expected for the user.
- LAYOUT=index: duplicate GUIDs were not cleaned out. Also the list
recovery was not optimal.
- auth: Assert crash would occur when iterating multiple userdb
backends.
- director: Logging into director using master user with
auth_master_user_separator character redirected user to a wrong
backend, unless master_user_separator setting was also set to the same
value. Merged these into auth_master_user_separator.
- dsync: Couldn't always fix folder GUID conflicts automatically with
Maildir format. This resulted in replication repeatedly failing
with "Remote lost mailbox GUID".
- dsync: Failed to migrate INBOX when using namespace prefix=INBOX/,
resulting in "Remote lost mailbox GUID" errors.
- dsync: INBOX was created too early with namespace prefix=INBOX/,
resulting a GUID conflict. This may have been resolved automatically,
but not always.
- dsync: v2.3.18 regression: Wrong imapc password with dsync caused
Panic: file lib-event.c: line 506 (event_pop_global):
assertion failed: (event == current_global_event)
- imapc: Requesting STATUS for a mailbox with imapc and INDEXPVT
configured did not return correct (private) unseen counts.
- lib-dict: Process would crash when committing data to redis without
dict proxy.
- lib-mail: Corrupted cached BODYSTRUCTURE caused panic during FETCH.
Fixes: Panic: file message-part-data.c: line 579 (message_part_is_attachment):
assertion failed: (data != NULL). v2.3.13 regression.
- lib-storage: mail_attribute_dict with dict-sql failed when it tried to
lookup empty dict keys.
- lib: ioloop-kqueue was missing include breaking some BSD builds.
- lua-http: Dovecot Lua HTTP client could not resolve DNS names in mail
processes, because it expected "dns-client" socket to exist in the
current directory.
- oauth2: Using %{oauth2:name} variables could cause useless
introspections.
- pop3: Sending POP3 command with ':' character caused an assert-crash.
v2.3.18 regression.
- replicator: Replication queue had various issues, potentially causing
replication requests to become stuck.
- stats: Invalid Prometheus label names were created with specific
mail/dovecot-pigeonhole: ChangeLog:
* No changes - release done to keep version numbers synced.
Commit b7f05445c0 has added WWW entries to port Makefiles based on
WWW: lines in pkg-descr files.
This commit removes the WWW: lines of moved-over URLs from these
pkg-descr files.
Approved by: portmgr (tcberner)
It has been common practice to have one or more URLs at the end of the
ports' pkg-descr files, one per line and prefixed with "WWW:". These
URLs should point at a project website or other relevant resources.
Access to these URLs required processing of the pkg-descr files, and
they have often become stale over time. If more than one such URL was
present in a pkg-descr file, only the first one was tarnsfered into
the port INDEX, but for many ports only the last line did contain the
port specific URL to further information.
There have been several proposals to make a project URL available as
a macro in the ports' Makefiles, over time.
This commit implements such a proposal and moves one of the WWW: entries
of each pkg-descr file into the respective port's Makefile. A heuristic
attempts to identify the most relevant URL in case there is more than
one WWW: entry in some pkg-descr file. URLs that are not moved into the
Makefile are prefixed with "See also:" instead of "WWW:" in the pkg-descr
files in order to preserve them.
There are 1256 ports that had no WWW: entries in pkg-descr files. These
ports will not be touched in this commit.
The portlint port has been adjusted to expect a WWW entry in each port
Makefile, and to flag any remaining "WWW:" lines in pkg-descr files as
deprecated.
Approved by: portmgr (tcberner)
Dovecot Changelog:
+ Added mail_user_session_finished event, which is emitted when the mail
user session is finished (e.g. imap, pop3, lmtp). It also includes
fields with some process statistics information.
See https://doc.dovecot.org/admin_manual/list_of_events/ for more
information.
+ Added process_shutdown_filter setting. When an event matches the filter,
the process will be shutdown after the current connection(s) have
finished. This is intended to reduce memory usage of long-running imap
processes that keep a lot of memory allocated instead of freeing it to
the OS.
+ auth: Add cache hit indicator to auth passdb/userdb finished events.
See https://doc.dovecot.org/admin_manual/list_of_events/ for more
information.
+ doveadm deduplicate: Performance is improved significantly.
+ imapc: COPY commands were sent one mail at a time to the remote IMAP
server. Now the copying is buffered, so multiple mails can be copied
with a single COPY command.
+ lib-lua: Add a Lua interface to Dovecot's HTTP client library. See
https://doc.dovecot.org/admin_manual/lua/ for more information.
- auth: Cache lookup would use incorrect cache key after username change.
- auth: Improve handling unexpected LDAP connection errors/hangs.
Try to fix up these cases by reconnecting to the LDAP server and
aborting LDAP requests earlier.
- auth: Process crashed if userdb iteration was attempted while auth-workers
were already full handling auth requests.
- auth: db-oauth2: Using %{oauth2:name} variables caused unnecessary
introspection requests.
- dict: Timeouts may have been leaked at deinit.
- director: Ring may have become unstable if a backend's tag was changed.
It could also have caused director process to crash.
- doveadm kick: Numeric parameter was treated as IP address.
- doveadm: Proxying can panic when flushing print output. Fixes
Panic: file ioloop.c: line 865 (io_loop_destroy): assertion failed:
(ioloop == current_ioloop).
- doveadm sync: BROKENCHAR was wrongly changed to '_' character when
migrating mailboxes. This was set by default to %, so any mailbox
names containing % characters were modified to "_25".
- imapc: Copying or moving mails with doveadm to an imapc mailbox could
have produced "Error: Syncing mailbox '[...]' failed" Errors. The
operation itself succeeded but attempting to sync the destination
mailbox failed.
- imapc: Prevent index log synchronization errors when two or more imapc
sessions are adding messages to the same mailbox index files, i.e.
INDEX=MEMORY is not used.
- indexer: Process was slowly leaking memory for each indexing request.
- lib-fts: fts header filters caused binary content to be sent to the
indexer with non-default configuration.
- doveadm-server: Process could hang in some situations when printing
output to TCP client, e.g. when printing doveadm sync state.
- lib-index: dovecot.index.log files were often read and parsed entirely,
rather than only the parts that were actually necessary. This mainly
increased CPU usage.
- lmtp-proxy: Session ID forwarding would cause same session IDs being
used when delivering same mail to multiple backends.
- log: Log prefix update may have been lost if log process was busy.
This could have caused log prefixes to be empty or in some cases
reused between sessions, i.e. log lines could have been logged for the
wrong user/session.
- mail_crypt: Plugin crashes if it's loaded only for some users. Fixes
Panic: Module context mail_crypt_user_module missing.
- mail_crypt: When LMTP was delivering mails to both recipients with mail
encryption enabled and not enabled, the non-encrypted recipients may
have gotten mails encrypted anyway. This happened when the first
recipient was encrypted (mail_crypt_save_version=2) and the 2nd
recipient was not encrypted (mail_crypt_save_version=0).
- pop3: Session would crash if empty line was sent.
- stats: HTTP server leaked memory.
- submission-login: Long credentials, such as OAUTH2 tokens, were refused
during SASL interactive due to submission server applying line length
limits.
- submission-login: When proxying to remote host, authentication was not
using interactive SASL when logging in using long credentials such as
OAUTH2 tokens. This caused authentication to fail due to line length
constraints in SMTP protocol.
- submission: Terminating the client connection with QUIT command after
mail transaction is started with MAIL command and before it is
finished with DATA/BDAT can cause a segfault crash.
- virtual: doveadm search queries with mailbox-guid as the only parameter
crashes: Panic: file virtual-search.c: line 77 (virtual_search_get_records):
assertion failed: (result != 0)
Pigeonhole Changelog:
This release is done to maintain parity with dovecot 2.3.19 release, so
it does not contain any news-worthy changes.
Flavors currently breaks mail/dovecot-pigeonhole,
mail/dovecot-fts-xapian, mail/dovecot-fts-flatcurve.
nc & I (ler) will work to see if we can come to a better way to do this
This reverts commit 0dd69d0adf.
Dovecot ChangeLog:
* Removed mail_cache_lookup_finished event. This event wasn't especially
useful, but it increased CPU usage significantly.
* fts: Don't index inline base64 encoded content in FTS indexes using
the generic tokenizer. This reduces the FTS index sizes by removing
input that is very unlikely to be searched for. See
https://doc.dovecot.org/configuration_manual/fts/tokenization for
details on how base64 is detected. Only applies when using libfts.
* lmtp: Session IDs are now preserved through proxied connections, so
LMTP sessions can be tracked. This slightly changes the LMTP session
ID format by appending ":Tn" (transaction), ":Pn" (proxy connection)
and ":Rn" (recipient) counters after the session ID prefix.
+ Events now have "reason_code" field, which can provide a list of
reasons why the event is happening. See
https://doc.dovecot.org/admin_manual/event_reasons/
+ New events are added. See https://doc.dovecot.org/admin_manual/list_of_events/
+ fts: Added fts_header_excludes and fts_header_includes settings to
specify which headers to index. See
https://doc.dovecot.org/settings/plugin/fts-plugin#plugin-fts-setting-fts-header-excludes
for configuration details.
+ fts: Initialize the textcat language detection library only once per
process. This can reduce CPU usage if fts_languages setting has multiple
languages listed and service indexer-worker { service_count } isn't 1.
Only applies when using libfts.
+ lib-storage: Reduced CPU usage significantly for some operations that
accessed lots of emails (e.g. fetching all flags in a folder, SORT, ...)
+ lib: DOVECOT_PREREQ() - Add micro version which enables compiling
external plugins against different versions of Dovecot.
+ lmtp: Added new lmtp_verbose_replies setting that makes errors sent to
the LMTP client much more verbose with details about why exactly
backend proxy connections or commands are failing.
+ submission: Support implicit SASL EXTERNAL with
submission_client_workarounds=implicit-auth-external. This allows
automatically logging in when SSL client certificate is present.
- *-login: Statistics were disabled if stats process connection was lost.
- auth: Authentication master user login fails with SCRAM-* SASL mechanisms.
- auth: With auth_cache_verify_password_with_worker=yes, passdb extra
fields in the auth cache got lost.
- doveadm: Fixed crash if zlib_save_level setting was specified,
but zlib_save was unset. v2.3.15 regression.
- doveadm: Proxying can panic when flushing print output. v2.3.17
regression. Fixes:
Panic: file ioloop.c: line 865 (io_loop_destroy): assertion failed:
(ioloop == current_ioloop)
- doveadm: stats add --group-by parameter didn't work.
- fts: Using email-address fts tokenizer could result in excessive memory
usage with garbage email input. This could cause the indexer-worker
processes to fail due to reaching the VSZ memory size limit.
Only applies when using libfts.
- imap: A SEARCH command timing out while fts returns indexes may timeout
returning "NO [SERVERBUG]", while it should return "NO [INUSE]" instead.
- imap: LIST-EXTENDED doesn't return STATUS for all folders. Sending
LIST .. RETURN (SUBSCRIBED STATUS (...)) did not return STATUS for
folders that are not subscribed when they have a child folder that is
subscribed as mandated by IMAP RFCs.
- imapc: Mailbox vsize calculation crashed with
Panic: file index-mailbox-size.c: line 344 (index_mailbox_vsize_hdr_add_missing):
assertion failed: (mails_left > 0)
- indexer: If indexer-worker crashes, the request it was processing gets
stuck in the indexer process. This stops indexing for the folder until
indexer process is restarted. v2.3.14 regression.
- indexer: Process was slowly leaking memory for each indexing request.
- lib-event: Unnamed events were wrongly filtered out for event/metric
filters like "event=abc OR something_independent_of_event_name".
- lib-index: 64-bit big endian CPUs handle last_used field in
dovecot.index.cache wrong.
- lib-ssl-iostream: Fix buggy OpenSSL error handling without assert-crashing.
If there is no error available, log it as an error instead of crashing.
The previous fix for this in v2.3.11 was incomplete. Fixes
Panic: file istream-openssl.c: line 51 (i_stream_ssl_read_real):
assertion failed: (errno != 0)
- lmtp: Out-of-memory issues can happen when proxying large messages to
LMTP backend servers that accept the message data too slow.
- master: HAProxy header parsing has read buffer overflow if provided
header size is invalid. This happens only if inet_listener
{ haproxy=yes } is configured and only if the remote IP address is in
haproxy_trusted_networks.
- old_stats: Plugin kept increasing memory usage, which became
noticeable with long-running imap sessions.
- stats: Dynamically adding same metric multiple times causes multiple stats.
- submission-login: Authentication does not accept OAUTH2 token (or
other very long credentials) because it considers the line to be too long.
- submission-login: Process can crash if HELO is pipelined with an
invalid domain.
- submission-proxy: Don't use SASL-IR if it would make the AUTH command
line longer than 512 bytes.
- submission: Service would crash if relay server authentication failed.
- virtual: FTS search in a virtual folder could crash if there are
duplicate mailbox GUIDs. This mainly happened when user had both INBOX
and INBOX/INBOX folders and the namespace prefix was INBOX/. Fixes
Panic: file hash.c: line 252 (hash_table_insert_node):
assertion failed: (opcode == HASH_TABLE_OP_UPDATE)
- virtual: If mailbox opening fails, the backend mailbox is leaked and
process crashes when client disconnects. Fixes
Panic: file mail-user.c: line 232 (mail_user_deinit):
assertion failed: ((*user)->refcount == 1)
- virtual: Searching headers in virtual folders didn't always use
full-text search indexes, if fts_enforced=no or body.
Pigeonhole ChangeLog:
- duplicate: Users without a home directory can crash with Sieve when
using duplicate database. v2.3.17 regression.
- imapsieve: When mail was expunged when processing imapsieve events, a
crash could occur. Fixes Panic: file mail-index-map.c:
line 558 (mail_index_map_lookup_seq_range): assertion failed: (first_uid > 0)
- managesieve-login: Proxy didn't support forwarding the forward_* passdb fields.
- redirect: Sieve would crash if redirect after keep-equivalent action failed.
- sieve: Interpreter crashes when the Sieve index extension is used with
index zero.
- vnd.dovecot.filter: Envelope sender string may become corrupted when
Sieve scripts are using vnd.dovecot.filter. This could end up
corrupting mbox's From line and return wrong envelope sender string in
Bump mail/dovecot-fts-xapian portrevision.
ChangeLogs:
dovecot:
* Dovecot now logs a warning if time seems to jump forward at least
100 milliseconds.
* dict: Lines logged by the dict process now contain the dict name as
the prefix.
* lib-index: mail_cache_fields, mail_always_cache_fields and
mail_never_cache_fields now verifies that the listed header names are
valid. Especially the UTF8 "–" character has sometimes been wrongly
used instead of the ASCII "-".
+ *-login: Added login_proxy_rawlog_dir setting to capture
rawlogs between proxy and backend.
+ dict: The server process now keeps the last 10 idle dict backends
cached for maximum of 30 seconds. Practically this acts as a
connection pool for dict-redis and dict-ldap. Note that this doesn't
affect dict-sql, because it already had its own internal cache.
+ doveadm: New stats add/remove commands added to support changing the
metrics configuration on runtime.
+ lazy_expunge: Added lazy_expunge_exclude settings to disable
lazy_expunge for specific folders. \Special-use flags can be used as
folder names.
+ lib-lua: Added a new helper function dovecot.restrict_global_variables()
to disable or enable defining new global variables.
- LAYOUT=index List index rebuild was missing.
- LAYOUT=index: Duplicate GUIDs were not detected.
- acl: When using acl_ignore_namespace Dovecot attempted to access or
create dovecot-acl-list even when the namespace should have been
ignored. For virtual namespaces this could have yielded errors about
"Read-only file system" or "Permission denied".
- auth: Setting the "master" passdb field to empty value would
cause proxying to fail with an authentication error.
Now an empty "master" field is ignored.
- doveadm-server: Duplicate error lines were sent for failed commands.
This didn't normally cause visible problems, except when using
wildcards in usernames or -A parameter to go through multiple users.
- doveadm-server: Logs written by doveadm-server were often missing log
prefixes, especially mail_log_prefix for mail commands. Logs sent to
doveadm TCP client were also missing log prefixes.
- doveadm: v2.3 regression: batch command always crashes.
- doveadm: v2.3.11 regression: Commands failed if ssl_cert or
ssl_key files weren't readable by the user running doveadm, even
though doveadm didn't actually use these settings
- imap-hibernate: Process may crash at deinit:
Panic: file ioloop.c: line 928 (io_loop_destroy): assertion failed:
(ioloop->cur_ctx == NULL).
- imap: Using imap_fetch_failure=no-after can cause assert-crash
with some IMAP commands if reading the mail fails (e.g. wrong cached
mail size). Fixes:
Panic: file index-mail-headers.c: line 198 (index_mail_parse_header_init):
assertion failed: (!mail->data.header_parser_initialized)
- imap: v2.3.10 regression: When using INDEXPVT to enable private
\Seen flags (for shared or public namespaces) the STORE command did
not send untagged replies for the \Seen flag changes.
- imap: v2.3.15 regression: If PREVIEW/SNIPPET is not the final FETCH
option in the command, the IMAP FETCH response is broken.
- imap: v2.3.15 regression: MOVE command leaks mailbox if it can't be
opened and crashes at deinit:
Panic: file mail-user.c: line 229 (mail_user_deinit): assertion failed:
((*user)->refcount == 1).
- imapc: Copying nonexistent mail via imapc could have crashed. Fixes:
Panic: file mail-storage.c: line 2385 (mailbox_transaction_commit_get_changes):
assertion failed: (ret < 0 || seq_range_count(&changes_r->saved_uids) == save_count ||
array_count(&changes_r->saved_uids) == 0).
- indexer: v2.3.15 regression: Process crashes if indexer-client
disconnects while it's waiting for command reply. This happened for
example if IMAP SEARCH triggered long fts indexing and the IMAP
client disconnected while waiting for the reply.
- indexer: v2.3.15 regression: Process may have crashed in some situations.
- indexer: v2.3.15 regression: indexer-worker processes may not have
reached the process_limit in some situations, possibly even using just
one indexer-worker process even though there were many indexing
requests queued.
- lib-compression: Reading lz4 compressed mdbox mails may crash. Fixes:
Panic: file istream.c: line 345 (i_stream_read_memarea):
assertion failed: (!stream->blocking).
- lib-compression: bench-compress crashes due to xz being read-only.
- lib-lua: Fix linking libdict_lua for non-GNU linkers when Lua support
is disabled.
- lib-mail: There was no limit on how large an email header name could be.
Processable header names are now limited to 1000 bytes.
- lib-oauth2: Dovecot disallowed JWT tokens if their validity time was
older than token creation time (nbf < iat).
- lib-storage: Reduce memory footprint of certain storage operations.
- lib-storage: When listing mailboxes with storage name escape
characters (^ or .) as part of the mailbox name, the listing could
show corrupted mailbox names. Due to an issue in handling escaped
parent folders, the listing of other mailbox names would become
corrupted by prepending parts of the previously listed mailboxes
parent folder as prefix to the actual mailbox names. The corruption
can occur when using LAYOUT=INDEX and maildir or obox, or when using
the listescape plugin.
- mail-crypt: Fix "-O" argument for "doveadm mailbox cryptokey password"
command to be a boolean, and not expect a string.
- submission-login: Add support for not authenticating to next hop in
submission proxying.
- submission-login: EHLO was not sent again after XCLIENT when doing
submission proxying.
- virtual: Mailboxes do not correctly detect underlying mailboxes
getting re-created even though they have a different UIDVALIDITY or
GUID.
Pigeonhole:
- duplicate: The Sieve duplicate test is prone to false negatives when
the user receives many e-mails concurrently, meaning that duplicate
deliveries can still occur.
- fileinto: v2.3.16 regression: Sieve delivery crashes if mail is
delivered to non-existing and existing folder.
- imap-filter-sieve: v2.3.15 regression: The CPU limits on Sieve
execution are too easily exceeded in IMAP context (the IMAPSieve and
FILTER=SIEVE capabilities). Changed the default to unlimited CPU time
for IMAP context, since similar excessive resource usage can be caused
by other means as well. The CPU limits on Sieve scripts executed at
LDA/LMTP delivery are still enforced by default.
- redirect: The Sieve redirect action has protections against users
triggering mail loops. Unfortunately, the detection of a redirect mail
loop sometimes causes the message to get lost if no other Sieve action
is applied that delivers the message somewhere else.
- redirect: v2.3.16 regression: With certain Sieve scripts if redirect
fails due to temporary failure, the lmtp process may crash after the
delivery. Fixes:
Panic: file mail-user.c: line 229 (mail_user_deinit):
assertion failed: ((*user)->refcount == 1).
- CVE-2020-10957: lmtp/submission: A client can crash the server by
sending a NOOP command with an invalid string parameter. This occurs
particularly for a parameter that doesn't start with a double quote.
This applies to all SMTP services, including submission-login, which
makes it possible to crash the submission service without
authentication.
- CVE-2020-10958: lmtp/submission: Sending many invalid or unknown
commands can cause the server to access freed memory, which can lead
to a server crash. This happens when the server closes the connection
with a "421 Too many invalid commands" error. The bad command limit
depends on the service (lmtp or submission) and varies between 10 to
20 bad commands.
- CVE-2020-10967: lmtp/submission: Issuing the RCPT command with an
address that has the empty quoted string as local-part causes the
lmtp service to crash.
Clean up some REINPLACE warnings whilst we're here.
MFH: 2020Q2
Security: 37d106a8-15a4-483e-8247-fcb68b16eaf8
Security: CVE-2020-10957
Security: CVE-2020-10958
Security: CVE-2020-10967
ChangeLog:
* Disable retpoline migitations by default. These can cause severe
performance regressions, so they should be only enabled when
applicable.
* IMAP MOVE now commits transactions in batches of 1000 mails. This
helps especially with lazy_expunge when moving a lot of mails. It
mainly avoids situations where multiple IMAP sessions are running the
same MOVE command and duplicating the mails in the lazy_expunge folder.
With this change there can still be some duplication, but the MOVE
always progresses forward. Also if the MOVE fails at some point, the
changes up to the last 1000 mails are still committed instead of
rolled back. Note that the COPY command behavior hasn't changed,
because it is required by IMAP standard to be an atomic operation.
* IMAP EXPUNGE and CLOSE now expunges mails in batches of 1000 mails.
This helps especially with lazy_expunge when expunging a lot of mails
(e.g. millions) to make sure that the progress always moves forward
even if the process is killed.
* Autoexpunging now expunges mails in batches of 1000 mails. This helps
especially with lazy_expunge when expunging a lot of mails
(e.g. millions) to make sure that the progress always moves forward
even if the process is killed.
+ Add tool for generating sysreport called dovecot-sysreport.
This generates a bundle of information usually needed for support
requests.
+ Add support for the new IMAP \Important SPECIAL-USE flag (RFC 8457).
+ Add metric { group_by } setting. This allows automatically creating
new metrics based on the fields you want to group statistics by.
NOTE: This feature is considered experimental and syntax is subject
to change in future release.
+ auth: Support SCRAM-SHA-256 authentication mechanism.
+ imap: Support the new IMAP STATUS=SIZE extension.
+ Use TCP_QUICKACK to reduce latency for some TCP connections.
+ quota-status: Made the service more robust against erroneous use with
Postfix ACL policies other than smtpd_recipient_restrictions.
+ Add "revision" field support to imap_id_send setting. Using
"revision *" will send in IMAP ID command response the short commit
hash of the Dovecot git source tree HEAD (same as in dovecot --version).
+ IMAP ENVELOPE includes now all addresses when there are multiple
headers (From, To, Cc, etc.) The standard way of having multiple
addresses is to just list them all in a single header. It's
non-standard to have multiple headers. However, since MTAs allow these
mails to pass through and different software may handle them in
different ways, it's better from security point of view to show all
the addresses.
+ Event filters now support using "field_name=" to match a field that
doesn't exist or has an empty value. For example use "error=" to match
only events that didn't fail.
- acl: INBOX ACLs shouldn't apply for IMAP GETMETADATA/SETMETADATA
commands.
- cassandra: CASS_ERROR_SERVER_WRITE_FAILURE error should also be
treated as "uncertain write failure".
- dict-redis: Using quota_clone configured with dict-redis could have
crashed when Redis responded slowly.
- imap-hibernate: Communication trouble with imap-master leads to
segfault.
- imap-hibernate: Unhibernation retrying wasn't working.
- imap: Fixed auth lookup privilege problem when imap process was reused
and user was being un-hibernated.
- Fix potential crash when copying/moving mails within the same folder.
This happened only when there were a lot of fields in dovecot.index.cache.
- lib-index: Recreating dovecot.index.cache file could have crashed when
merging bitmask fields.
- lib-index: Using public/shared folders with INDEXPVT configured to use
private \Seen flags, trying to search seen/unseen in an empty folder
crashes with segfault.
- lib-mail: Large base64-encoded mails weren't decoded properly.
This could have affected searching/indexing mails and message snippet
generation.
- lib-mail: Message with only quoted text could have caused message
snippet to ignore its 200 character limit and return the entire
message. This was added also to dovecot.index.cache file, which
increased disk space and memory usage unnecessarily.
v2.3.9.2 regression (previous versions cached the quoted snippet as
empty). In a large mail quoted text could have become wrongly added
to the snippet, possibly mixed together with non-quoted text.
- lib-smtp: client could have assert-crashed if STARTTLS handshake
finished earlier than usually.
- lib-ssl-iostream: remove -static flag for lib-ssl-iostream linking to
prevent a compile issue.
- lib-storage: Mailbox synchronization may have assert-crashed in some
rare situations.
- lib-storage: mdbox didn't preserve date.saved with dsync.
- lib: Don't require EAI_{ADDRFAMILY,NODATA}, breaks FreeBSD
- master: Some services could respawn unthrottled if they crash during
startup.
- push-notification: Do not send push_notification_finished event if
nothing was done. This happens when mail transaction is started and
ended with no changes.
- quota-status: Addresses with special characters in the local part caused
problems in the interaction between Postfix and Dovecot. Postfix sent
its own internal representation in the recipient field, while Dovecot
expected a valid RFC5321 mailbox address.
- submission-login: SESSION was not correctly encoded field for the
XCLIENT command. Particularly, a '+' character introduced by the
session ID's Base64 encoding causes problems.
- submission: Fix submission_max_mail_size to work correctly on 32-bit
systems.
- submission: Trusted connections crashed in second connection's EHLO
if submission-login { service_count } is something else than 1 (which
is the default).
- submission: XCLIENT command was never used in the protocol exchange
with the relay MTA when submission_backend_capabilities is configured,
even when the relay MTA was properly configured to accept the XCLIENT
command.
- fts-solr: The XML response parser fails to parse large/chunked responses
correctly. This leads to spurious parse errors, most notably:
"Error: fts_solr: received invalid uid '0'".
Bumping PORTREVISION on dovecot-pigeonhole (explicitly ignoring 0.5.10, as
there are compile/configure/hang issues), and dovecot-fts-xapian.
Changelog:
* CVE-2020-7046: Truncated UTF-8 can be used to DoS
submission-login and lmtp processes.
* CVE-2020-7957: Specially crafted mail can crash snippet generation.
MFH: 2020Q1
Security: CVE-2020-7046
Security: CVE-2020-7957
Security: 74db0d02-b140-4c32-aac6-1f1e81e1ad30
* CVE-2019-19722: Mails with group addresses in From or To fields caused
crash in push notification drivers.
- additional fix for blank headers
PORTREVISION bump for mail/dovecot-pigeonhole, mail/dovecot-fts-xapian
*NOT* requesting MFH as 2.3.9 is not in 2019Q4.
Security: b7dc4dde-2e48-43f9-967a-c68461537cf2
Security: CVS-2019-19722
Bump PORTREVISION of mail/dovecot-fts-xapian for version change of dovecot.
Changelog:
Dovecot:
* Changed several event field names for consistency and to avoid
conflicts in parent-child event relationships:
* SMTP server command events: Renamed "name" to "cmd_name"
* Events inheriting from a mailbox: Renamed "name" to "mailbox"
* Server connection events have only "remote_ip", "remote_port",
"local_ip" and "local_port".
* Removed duplicate "client_ip", "ip" and "port".
* Mail storage events: Removed "service" field.
Use "service:<name>" category instead.
* HTTP client connection events: Renamed "host" to "dest_host" and
"port" to "dest_port"
* auth: Drop Postfix socketmap support. It hasn't been working
with recent Postfix versions for a while now.
* push-notification-lua: The "subject" field is now decoded to UTF8
instead of kept as MIME-encoded.
+ push-notification-lua: Added new "from_address", "from_display_name",
"to_address" and "to_display_name" fields. The display names are
decoded to UTF8.
+ Added various new fields to existing events.
See http://doc.dovecot.net/admin_manual/list_of_events.html
+ Add lmtp_add_received_header setting. It can be used to prevent LMTP
from adding "Received:" headers.
+ doveadm: Support SSL/STARTTLS for proxied doveadm connections based on
doveadm_ssl setting and proxy ssl/tls settings.
+ Log filters support now "service:<name>", which matches all events for
the given service. It can also be used as a category.
+ lib: Use libunwind to get abort backtraces with function names
where available.
+ lmtp: When the LMTP proxy changes the username (from passdb lookup)
add an appropriate ORCPT parameter.
- lmtp: Add lmtp_client_workarounds setting to implement workarounds for
clients that send MAIL and RCPT commands with additional spaces before
the path and for clients that omit <> brackets around the path.
See example-config/conf.d/20-lmtp.conf.
- lda/lmtp: Invalid MAIL FROM addresses were rejcted too aggressively.
Now mails from addresses with unicode characters are delivered, but
their Return-Path header will be <> instead of the given MAIL FROM
address.
- lmtp: The lmtp_hdr_delivery_address setting is ignored.
- imap: imap_command_finished event's "args" and "human_args" parameters
were always empty.
- mbox: Seeking in zlib and bzip2 compressed input streams didn't work
correctly.
- imap-hibernate: Process crashed when client got destroyed while it was
attempted to be unhibernated, and the unhibernation fails.
- *-login: Proxying may have crashed if SSL handshake to the backend
failed immediately. This was unlikely to happen in normal operation.
- *-login: If TLS handshake to upstream server failed during proxying,
login process could crash due to invalid memory access.
- *-login: v2.3 regression: Using SASL authentication without initial
response may have caused SSL connections to hang. This happened often
at least with PHP's IMAP library.
- *-login: When login processes are flooded with authentication attempts
it starts logging errors about "Authentication server sent unknown id".
This is still expected. However, it also caused the login process to
disconnect from auth server and potentially log some user's password
in the error message.
- dict-sql: SQL prepared statements were not shared between sessions.
This resulted in creating a lot of prepared statements, which was
especially inefficient when using Cassandra backend with a lot of
Cassandra nodes.
- auth: auth_request_finished event didn't have success=yes parameter
set for successful authentications.
- auth: userdb dict - Trying to list users crashed.
- submission: Service could be configured to allow anonymous
authentication mechanism and anonymous user access.
- LAYOUT=index: Corrupted dovecot.list.index caused folder creation to
panic.
- doveadm: HTTP server crashes if request target starts with double "/".
- dsync: Remote dsync started hanging if the initial doveadm
"dsync-server" command was sent in the same TCP packet as the
following dsync handshake. v2.3.8 regression.
- lib: Several "input streams" had a bug that in some rare situations
might cause it to access freed memory. This could lead to crashes or
corruption.
The only currently known effect of this is that using zlib plugin with
external mail attachments (mail_attachment_dir) could cause fetching
the mail to return a few bytes of garbage data at the beginning of the
header. Note that the mail wasn't saved corrupted, but fetching it
caused corrupted mail to be sent to the client.
- lib-storage: If a mail only has quoted content, use the quoted text
for generating message snippet (IMAP PREVIEW) instead of returning
empty snippet.
- lib-storage: When vsize header was rebuilt, newly calculated message
sizes were added to dovecot.index.cache instead of being directly
saved into vsize records in dovecot.index.
- lib: JSON generator was escaping UTF-8 characters unnecessarily.
Pigeonhole:
+ Added events for Sieve and ManageSieve, see
https://doc.dovecot.org/admin_manual/list_of_events/#pigeonhole
+ Pigeonhole: Implement the Sieve "special-use" extension described in
RFC 8579.
- duplicate: Test only compared the handles which would cause
different values to be cached as the same duplicate test. Fix to also
compare the actual hashes.
- imap_sieve_filter: IMAP FILTER Command had various bugs in error
handling. Errors may have been duplicated for each email, errors
may have been missing entirely, command tag and ERRORS/WARNINGS
parameters were swapped.
Add a note to the dovecot port about the requirement to bump the portrevision
each time dovecot is updated
PR: 241147
Reported by: Matthias Pfaller <matthias.pfaller@familie-pfaller.de>
